Thanks, that's good to know, although a day or two get one pkg updated
is still not acceptable for JDS. A new set of development tarballs is
released every two weeks. During the 2.14 development cycle the number
of changed tarballs was usually between 10 and 20 each time, but
sometimes a lot more.
I realise that the requirements for SFW/CCD are different and updates
happen much much less frequently.
Laca
[1] some additional statistics:
Changed component versions from:
2.10 to 2.12: 117
2.12 to 2.14: 123
2.10 to 2.14: 143
In comparison, the total number of components in these builds:
2.10: 143
2.12: 162
2.14: 169
On Wed, 2006-06-21 at 07:46 +0100, Paul Cunningham wrote:
> FYI, in the Companion CD (CCD) gate there is (was) a utility call PUFA
> that allows you to update most files in the gate for a package update,
> or new package add, automatically. The auto generated files then just
> need a bit a of manually tweaking afterwards. I and others in the old
> CCD team in Watford used to find that using PUFA speeded up the process
> for the CCD considerably. There used to be a load of example scripts for
> using PUFA in the gate's bin dir in a sub-directory (I forget what I
> called it). The utility's source code is in the gates tool/javaTools
> dir, there is also documentation for it in there. On average it took a
> day or two to update a package and get the ws building cleanly again, it
> was all the other crap that goes with that took the time.
>
> When working on the SFW gate stuff, I used to use PUFA also to generated
> most of the files and then move them across the the SFW gate, eg. that's
> how I did all the packages used by solaris printing in that gate. I
> never did get around to changing PUFA so it worked in the SFW gate
> directly though (not enough time in the working date).
>
> But in the old CCD team it was a personal preference if the developer
> used it or not, some of the team preferred to do it all manually :-( But
> my preference was to use PUFA (biased as I wrote it, so all problems
> with it can be blamed on me), it made life so much easier and got rid of
> most of the boring bits.
>
> Paul
>
> Laszlo (Laca) Peter wrote:
> > Recently, I spent a month working on updating libxml2 and libxslt in SFW
> > to newer versions. The time I actually spent with this was about 2 or 3
> > weeks, another 2 weeks were spent getting the new packages tested.
> >
> > I'd like to share my experiences, as I may be one of the first "external"
> > contributors to SFW. I also promised I would do so. I'm hoping to
> > start a constructive discussion that will lead to improving the SFW/CCD
> > build environment and making it easier for other people to contribute.
> >
> > First of all, I have to say that I've been building 100s of Open Source
> > packages[1] for Sun for the past 5.5 years, yet this was a new experience
> > for me on several fronts:
> > - I never worked with SCCS/Codemgr/wx before
> > - I was (still am) unfamiliar with most tools and processes used by ON
> > and SFW
> > - I never used the ON/SFW build system before
> >
> > That said, if I was starting this work again today, I could only save
> > about 1 week. The reason being that this update required changing 30
> > files and creating 32, lots and lots of manual updating of files
> > containing lists of files...
> >
> > To be fair, it wasn't just a simple update, at the same time I split the
> > packages into runtime and development bits and added the python bindings,
> > also in new packages[2]. Still, changing 30 pre-existing files would be
> > pain enough. To achieve the above, a total of 9869 lines changed
> > (5235 inserted/1338 deleted/3296 modified/32457 unchanged).
> > So, for those not familiar with SFW/CCD, what are these files and why
> > did they change so much:
> >
> > DISTFILES.sfw, EXFILES.sfw, DISTDIRS.sfw: these files are basically the
> > contents of the source tarball. They are used for building the SFW
> > source packages (SUNWfooS). I re-generated them using find commands in
> > the untarred sources. It seems like this could be completely automated.
> >
> > README.sfw, mapfiles: these obviously needed to be updated, no problem
> > with that.
> >
> > Makefile.sfw unpacks the tarball, applies the patches, runs configure
> > and then usually makes one target, not the entire build. In this case,
> > I needed to add one more make command for building the python bindings.
> >
> > install-libxml2, install-libxml2-64: now these are really weird things,
> > in my opinion. Instead of running "make install DESTDIR=foo", SFW uses
> > custom scripts that copy the files from the source tree to the
> > "proto area"[3], listing each file, setting the permissions, copying
> > libs from libtool's .libs (temporary) directory. Not very nice, to say
> > the least.
> >
> > When I was updating this script, I needed to do 4 things in parallel:
> > - check the list of files that make install would install
> > (I actually used a pkgbuild-generated pkgmap for this)
> > - find the location of these files in the source tree
> > - update the install-foo script
> > - update the package prototype files to make sure they correspond.
> > Guess what, I made mistakes: 2 new headers were missing from the
> > prototype, 2 files appeared in multiple pkgmaps and one file had
> > inconsistent attributes with the proto area[3].
> >
> > Which leads us to the packaging bits: prototype_com, prototype_sparc,
> > prototype_i386: these could really be generated. It's really easy to
> > make mistakes when writing these manually. Most if not all of these are
> > caught by the nightly build scripts, but then it's really just an
> > exercise to see if a human can be as thorough as a shell script (;
> >
> > FWIW, I do believe that this system works great for ON and other
> > in-house developed projects where developers can update the Makefiles
> > and the pkgmaps when they introduce a new header or library or source
> > file. However, for Open Source packages we have to do this for the
> > community developers: lots of possible mistakes and in fairness, they
> > know it better than us, so it's better to use their Makefiles.
> >
> > In my day job, I work on JDS and I maintain the JDS build system,
> > so allow me to make a comparison. (I'm obviously biased, but I'll
> > try my best to be fair.) Our builds are also based on community
> > tarballs and source patches. In JDS, we use the Makefile system
> > supplied by the originating community, which usually means
> > configure; make; make install. The "make install" part
> > is redirected (using the DESTDIR Makefile variable) to a per-package
> > proto area[3]. This means that by default, any files that the community
> > maintainers intended to install are installed, and to the correct
> > (relative) locations. Then we remove any files that we decided not to
> > deliver, for example lib*.a, lib*.la. This is the opposite of SFW's
> > philosophy, where we decide which files we wish to deliver and deal with
> > them one by one. We are now ready to create packages. Pkgmaps are
> > created dynamically from glob lists.
> > The above steps are basically the same for all packages and all versions,
> > so updating to a newer version is often as simple as changing the
> > version number in the spec file[4]. Once the packages are built, it's
> > the same in both systems: we need to test the new packages.
> >
> > I realise we have long-standing traditions at Sun for doing things
> > a certain way and it's difficult to change. I'm just offering my
> > experiences and recommend that the SFW and CCD builds be made more
> > automatic and more convenient. In it's present form, SFW/CCD isn't
> > encouraging or inviting contributions, because of the tedious work
> > involved.[5]
> >
> > Comments, flames welcome.
> >
> > Laca
> >
> >
> > [1] http://opensolaris.org/os/community/desktop/communities/jds/building
> > [2] in case you wonder, they need to be in separate pkgs so that SUNWlxml
> > need not depend on SUNWPython
> > [3] a directory structure where the result of the build is staged for
> > packaging in the same structure as they will appear in the packages
> > [4] a spec file is a recipe for doing all that I described here, see
> > http://cvs.opensolaris.org/source/xref/jds/Solaris/SUNWlibpopt.spec
> > for an example
> > [5] When I was halfway or even 2/3 done with my changes both libxml2
> > and libxslt had new micro releases, with lots of nice bug fixes.
> > Even though I'm all for using the latest and greatest community code,
> > I decided not to waste a week and start again with the new releases.
> >
> >
> > _______________________________________________
> > companion-discuss mailing list
> > companion-discuss at opensolaris.org
> > http://opensolaris.org/mailman/listinfo/companion-discuss
> >
> _______________________________________________
> sfwnv-discuss mailing list
> sfwnv-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/sfwnv-discuss
