On Thu, 2006-06-22 at 08:12 +0100, Paul Cunningham wrote:
> > Thanks, that's good to know, although a day or two get one pkg updated
> > is still not acceptable
>
> A major part of that day or two was doing the full gate build (4 or 5
> hrs, but dam it failed so yet another 4 or 5 hrs - but it was 8hrs+ at
> one time on slow build machines and before kde was moved to its own gate).
Right. So that's another advantage of building and packaging the
modules one by one. You don't need to rebuild the entire stack to
find out if there are any packaging problems.
> But I have to admit though, that having played with 'spec' files and
> pkgbuild lately, its a lot easier and quicker to do it using 'spec'
> files :-) The only down side of 'spec' files, other than those pointed
> out by people in other emails on this thread (ie. about consistent pkg
> content),
Yes, we are fixing that and thanks again to Alan for raising it.
> is that if you are doing a complete build of n packages and
> package A depends on package B to build then package B has to be
> installed on the build box first before doing a complete build of the n
> packages. But I guess there may be ways around that that I don't know about.
Sure. There are 2 scripts: pkgbuild is the low level build script,
equivalent to rpmbuild on Red Hat / SuSE; pkgtool is the high level
script that figures out the build order from the dependencies and
builds and installs each package one by one.[1] It's also a good
way of exercising the dependencies.
So a full build looks something like:
pkgtool uninstall-pkgs *.spec
pkgtool build *.spec
Reading your comment again, if you mean whether it's possible to
build package A without installing package B at all, then the answer
is no. What you can do, though, and this is how we build JDS, is
using a chroot environment. It also makes the build environment
more controlled and easy to reinstall.
Laca
[1] It cannot currently build multiple packages in parallel, but it's
a planned feature. We currently only parallelize the build on a
single package level, using GNU make's -j option.
> > for JDS. A new set of development tarballs is
> > released every two weeks. During the 2.14 development cycle the number
> > of changed tarballs was usually between 10 and 20 each time, but
> > sometimes a lot more.
> >
> > I realise that the requirements for SFW/CCD are different and updates
> > happen much much less frequently.
> >
> > Laca
> >
> > [1] some additional statistics:
> >
> > Changed component versions from:
> > 2.10 to 2.12: 117
> > 2.12 to 2.14: 123
> >
> > 2.10 to 2.14: 143
> >
> > In comparison, the total number of components in these builds:
> > 2.10: 143
> > 2.12: 162
> > 2.14: 169
> >
> > On Wed, 2006-06-21 at 07:46 +0100, Paul Cunningham wrote:
> >
> >>FYI, in the Companion CD (CCD) gate there is (was) a utility call PUFA
> >>that allows you to update most files in the gate for a package update,
> >>or new package add, automatically. The auto generated files then just
> >>need a bit a of manually tweaking afterwards. I and others in the old
> >>CCD team in Watford used to find that using PUFA speeded up the process
> >>for the CCD considerably. There used to be a load of example scripts for
> >>using PUFA in the gate's bin dir in a sub-directory (I forget what I
> >>called it). The utility's source code is in the gates tool/javaTools
> >>dir, there is also documentation for it in there. On average it took a
> >>day or two to update a package and get the ws building cleanly again, it
> >>was all the other crap that goes with that took the time.
> >>
> >>When working on the SFW gate stuff, I used to use PUFA also to generated
> >>most of the files and then move them across the the SFW gate, eg. that's
> >>how I did all the packages used by solaris printing in that gate. I
> >>never did get around to changing PUFA so it worked in the SFW gate
> >>directly though (not enough time in the working date).
> >>
> >>But in the old CCD team it was a personal preference if the developer
> >>used it or not, some of the team preferred to do it all manually :-( But
> >>my preference was to use PUFA (biased as I wrote it, so all problems
> >>with it can be blamed on me), it made life so much easier and got rid of
> >>most of the boring bits.
> >>
> >>Paul
> >>
> >>Laszlo (Laca) Peter wrote:
> >>
> >>>Recently, I spent a month working on updating libxml2 and libxslt in SFW
> >>>to newer versions. The time I actually spent with this was about 2 or 3
> >>>weeks, another 2 weeks were spent getting the new packages tested.
> >>>
> >>>I'd like to share my experiences, as I may be one of the first "external"
> >>>contributors to SFW. I also promised I would do so. I'm hoping to
> >>>start a constructive discussion that will lead to improving the SFW/CCD
> >>>build environment and making it easier for other people to contribute.
> >>>
> >>>First of all, I have to say that I've been building 100s of Open Source
> >>>packages[1] for Sun for the past 5.5 years, yet this was a new experience
> >>>for me on several fronts:
> >>> - I never worked with SCCS/Codemgr/wx before
> >>> - I was (still am) unfamiliar with most tools and processes used by ON
> >>> and SFW
> >>> - I never used the ON/SFW build system before
> >>>
> >>>That said, if I was starting this work again today, I could only save
> >>>about 1 week. The reason being that this update required changing 30
> >>>files and creating 32, lots and lots of manual updating of files
> >>>containing lists of files...
> >>>
> >>>To be fair, it wasn't just a simple update, at the same time I split the
> >>>packages into runtime and development bits and added the python bindings,
> >>>also in new packages[2]. Still, changing 30 pre-existing files would be
> >>>pain enough. To achieve the above, a total of 9869 lines changed
> >>>(5235 inserted/1338 deleted/3296 modified/32457 unchanged).
> >>>So, for those not familiar with SFW/CCD, what are these files and why
> >>>did they change so much:
> >>>
> >>>DISTFILES.sfw, EXFILES.sfw, DISTDIRS.sfw: these files are basically the
> >>>contents of the source tarball. They are used for building the SFW
> >>>source packages (SUNWfooS). I re-generated them using find commands in
> >>>the untarred sources. It seems like this could be completely automated.
> >>>
> >>>README.sfw, mapfiles: these obviously needed to be updated, no problem
> >>>with that.
> >>>
> >>>Makefile.sfw unpacks the tarball, applies the patches, runs configure
> >>>and then usually makes one target, not the entire build. In this case,
> >>>I needed to add one more make command for building the python bindings.
> >>>
> >>>install-libxml2, install-libxml2-64: now these are really weird things,
> >>>in my opinion. Instead of running "make install DESTDIR=foo", SFW uses
> >>>custom scripts that copy the files from the source tree to the
> >>>"proto area"[3], listing each file, setting the permissions, copying
> >>>libs from libtool's .libs (temporary) directory. Not very nice, to say
> >>>the least.
> >>>
> >>>When I was updating this script, I needed to do 4 things in parallel:
> >>> - check the list of files that make install would install
> >>> (I actually used a pkgbuild-generated pkgmap for this)
> >>> - find the location of these files in the source tree
> >>> - update the install-foo script
> >>> - update the package prototype files to make sure they correspond.
> >>>Guess what, I made mistakes: 2 new headers were missing from the
> >>>prototype, 2 files appeared in multiple pkgmaps and one file had
> >>>inconsistent attributes with the proto area[3].
> >>>
> >>>Which leads us to the packaging bits: prototype_com, prototype_sparc,
> >>>prototype_i386: these could really be generated. It's really easy to
> >>>make mistakes when writing these manually. Most if not all of these are
> >>>caught by the nightly build scripts, but then it's really just an
> >>>exercise to see if a human can be as thorough as a shell script (;
> >>>
> >>>FWIW, I do believe that this system works great for ON and other
> >>>in-house developed projects where developers can update the Makefiles
> >>>and the pkgmaps when they introduce a new header or library or source
> >>>file. However, for Open Source packages we have to do this for the
> >>>community developers: lots of possible mistakes and in fairness, they
> >>>know it better than us, so it's better to use their Makefiles.
> >>>
> >>>In my day job, I work on JDS and I maintain the JDS build system,
> >>>so allow me to make a comparison. (I'm obviously biased, but I'll
> >>>try my best to be fair.) Our builds are also based on community
> >>>tarballs and source patches. In JDS, we use the Makefile system
> >>>supplied by the originating community, which usually means
> >>>configure; make; make install. The "make install" part
> >>>is redirected (using the DESTDIR Makefile variable) to a per-package
> >>>proto area[3]. This means that by default, any files that the community
> >>>maintainers intended to install are installed, and to the correct
> >>>(relative) locations. Then we remove any files that we decided not to
> >>>deliver, for example lib*.a, lib*.la. This is the opposite of SFW's
> >>>philosophy, where we decide which files we wish to deliver and deal with
> >>>them one by one. We are now ready to create packages. Pkgmaps are
> >>>created dynamically from glob lists.
> >>>The above steps are basically the same for all packages and all versions,
> >>>so updating to a newer version is often as simple as changing the
> >>>version number in the spec file[4]. Once the packages are built, it's
> >>>the same in both systems: we need to test the new packages.
> >>>
> >>>I realise we have long-standing traditions at Sun for doing things
> >>>a certain way and it's difficult to change. I'm just offering my
> >>>experiences and recommend that the SFW and CCD builds be made more
> >>>automatic and more convenient. In it's present form, SFW/CCD isn't
> >>>encouraging or inviting contributions, because of the tedious work
> >>>involved.[5]
> >>>
> >>>Comments, flames welcome.
> >>>
> >>>Laca
> >>>
> >>>
> >>>[1] http://opensolaris.org/os/community/desktop/communities/jds/building
> >>>[2] in case you wonder, they need to be in separate pkgs so that SUNWlxml
> >>> need not depend on SUNWPython
> >>>[3] a directory structure where the result of the build is staged for
> >>> packaging in the same structure as they will appear in the packages
> >>>[4] a spec file is a recipe for doing all that I described here, see
> >>> http://cvs.opensolaris.org/source/xref/jds/Solaris/SUNWlibpopt.spec
> >>> for an example
> >>>[5] When I was halfway or even 2/3 done with my changes both libxml2
> >>> and libxslt had new micro releases, with lots of nice bug fixes.
> >>> Even though I'm all for using the latest and greatest community code,
> >>> I decided not to waste a week and start again with the new releases.
> >>>
> >>>
> >>>_______________________________________________
> >>>companion-discuss mailing list
> >>>companion-discuss at opensolaris.org
> >>>http://opensolaris.org/mailman/listinfo/companion-discuss
> >>>
> >>
> >>_______________________________________________
> >>sfwnv-discuss mailing list
> >>sfwnv-discuss at opensolaris.org
> >>http://mail.opensolaris.org/mailman/listinfo/sfwnv-discuss
> >
> >
> >
