Recently, I spent a month working on updating libxml2 and libxslt in SFW to newer versions. The time I actually spent with this was about 2 or 3 weeks, another 2 weeks were spent getting the new packages tested.
I'd like to share my experiences, as I may be one of the first "external" contributors to SFW. I also promised I would do so. I'm hoping to start a constructive discussion that will lead to improving the SFW/CCD build environment and making it easier for other people to contribute. First of all, I have to say that I've been building 100s of Open Source packages[1] for Sun for the past 5.5 years, yet this was a new experience for me on several fronts: - I never worked with SCCS/Codemgr/wx before - I was (still am) unfamiliar with most tools and processes used by ON and SFW - I never used the ON/SFW build system before That said, if I was starting this work again today, I could only save about 1 week. The reason being that this update required changing 30 files and creating 32, lots and lots of manual updating of files containing lists of files... To be fair, it wasn't just a simple update, at the same time I split the packages into runtime and development bits and added the python bindings, also in new packages[2]. Still, changing 30 pre-existing files would be pain enough. To achieve the above, a total of 9869 lines changed (5235 inserted/1338 deleted/3296 modified/32457 unchanged). So, for those not familiar with SFW/CCD, what are these files and why did they change so much: DISTFILES.sfw, EXFILES.sfw, DISTDIRS.sfw: these files are basically the contents of the source tarball. They are used for building the SFW source packages (SUNWfooS). I re-generated them using find commands in the untarred sources. It seems like this could be completely automated. README.sfw, mapfiles: these obviously needed to be updated, no problem with that. Makefile.sfw unpacks the tarball, applies the patches, runs configure and then usually makes one target, not the entire build. In this case, I needed to add one more make command for building the python bindings. install-libxml2, install-libxml2-64: now these are really weird things, in my opinion. Instead of running "make install DESTDIR=foo", SFW uses custom scripts that copy the files from the source tree to the "proto area"[3], listing each file, setting the permissions, copying libs from libtool's .libs (temporary) directory. Not very nice, to say the least. When I was updating this script, I needed to do 4 things in parallel: - check the list of files that make install would install (I actually used a pkgbuild-generated pkgmap for this) - find the location of these files in the source tree - update the install-foo script - update the package prototype files to make sure they correspond. Guess what, I made mistakes: 2 new headers were missing from the prototype, 2 files appeared in multiple pkgmaps and one file had inconsistent attributes with the proto area[3]. Which leads us to the packaging bits: prototype_com, prototype_sparc, prototype_i386: these could really be generated. It's really easy to make mistakes when writing these manually. Most if not all of these are caught by the nightly build scripts, but then it's really just an exercise to see if a human can be as thorough as a shell script (; FWIW, I do believe that this system works great for ON and other in-house developed projects where developers can update the Makefiles and the pkgmaps when they introduce a new header or library or source file. However, for Open Source packages we have to do this for the community developers: lots of possible mistakes and in fairness, they know it better than us, so it's better to use their Makefiles. In my day job, I work on JDS and I maintain the JDS build system, so allow me to make a comparison. (I'm obviously biased, but I'll try my best to be fair.) Our builds are also based on community tarballs and source patches. In JDS, we use the Makefile system supplied by the originating community, which usually means configure; make; make install. The "make install" part is redirected (using the DESTDIR Makefile variable) to a per-package proto area[3]. This means that by default, any files that the community maintainers intended to install are installed, and to the correct (relative) locations. Then we remove any files that we decided not to deliver, for example lib*.a, lib*.la. This is the opposite of SFW's philosophy, where we decide which files we wish to deliver and deal with them one by one. We are now ready to create packages. Pkgmaps are created dynamically from glob lists. The above steps are basically the same for all packages and all versions, so updating to a newer version is often as simple as changing the version number in the spec file[4]. Once the packages are built, it's the same in both systems: we need to test the new packages. I realise we have long-standing traditions at Sun for doing things a certain way and it's difficult to change. I'm just offering my experiences and recommend that the SFW and CCD builds be made more automatic and more convenient. In it's present form, SFW/CCD isn't encouraging or inviting contributions, because of the tedious work involved.[5] Comments, flames welcome. Laca [1] http://opensolaris.org/os/community/desktop/communities/jds/building [2] in case you wonder, they need to be in separate pkgs so that SUNWlxml need not depend on SUNWPython [3] a directory structure where the result of the build is staged for packaging in the same structure as they will appear in the packages [4] a spec file is a recipe for doing all that I described here, see http://cvs.opensolaris.org/source/xref/jds/Solaris/SUNWlibpopt.spec for an example [5] When I was halfway or even 2/3 done with my changes both libxml2 and libxslt had new micro releases, with lots of nice bug fixes. Even though I'm all for using the latest and greatest community code, I decided not to waste a week and start again with the new releases.
