[gentoo-dev] Re: [gentoo-osx] Package testing -- Automated initiative
Adding -dev to CC: in case someone has any meaningful input or has
already tackled this problem...
On Aug 14, 2005, at 8:21 AM, Grobian wrote:
Introduction
Recently, once again we were confronted with a package marked as
ppc-macos stable, while it didn't compile at all, let alone run.
It is
believed more of these packages are in portage, and need to be
found and
fixed. Keeping the cause of why they are marked stable up to another
discussion, and out of the scope of this discussion, I will focus
on how
to track these packages down and report them to us.
In the secondary line, all 'unstable' packages, marked ~ppc-macos
should
be tested as well, since they can be faulty as well. Since for OSX
much
is in ~ppc-macos, many users consider it a normal procedure to
switch to
the unstable side of portage, hence some extra need for careful
testing
of ~ppc-macos also.
Proposed Global Structure
=
Testing should be done on a regular basis, both push and pull based.
This means that the testing machine would start testing packages
itself
if it is out of work, and on the other hand starts testing packages as
soon as they are being added/changed in CVS. It may need no great
imagination to see that the latter 'push-based' activity has priority
over the 'pull-based' work.
I'm not sure direct interaction with CVS would be needed, usually
only takes a short time for cvs commits to hit the rsync mirrors
(hence the volatile nature of the tree)
Starting over, will for the test machine mean that it starts cleaning
out its world file. Cleaning this file out to a bare minimum is an
important aspect of getting a test environment that reflects the
situation on new user's machines. If an ebuild uses a package without
having it in it's DEPEND, this may get noticed only when starting on a
clean machine. This, however, will add a big delay in testing as many
packages will need to be built prior the right package can be
installed.
The testing machine will have a queue file, which it reads packages to
emerge from. If the queue file is empty, i.e. when there is no push
based work, the machine will generate work by starting to compile
uncompiled packages, or emptying the tree.
Because ~ppc-macos and ppc-macos packages interfere with each other
-- a ~ppc-macos package overwrites a ppc-macos package -- both
stable and unstable have to be dealt with separately, i.e. they
should both have their own environment either via two separate
machines, or through the use of a chroot jail.
I think seperate chroots are definitely the way to go. We can just
store a 'pristine' chroot in iso or dmg or whatever on the build
server and copy when needed.
Queues
--
In order not to drag in a full DBMS (in the end Portage already is
one) queues are just simple flat files consisting of absolute
package names, one per line. Table wise locking granularity is
handled by the OS as one process opens the file in write mode.
Consumers -- the testing box in this case -- read the first line
and delete it, while producers simply add one line (or more) to the
end of the file.
The queue itself, is more a set than a list. This means that
packages that are in the queue, should be unique. If a package is
added that is already in the queue, it is dropped such that the
original queue position of the package is maintained.
Maybe a 'proper' dbms wouldn't be such a bad idea, could also store
build logs, timestamps, etc. there and make it easier for multiple
build hosts to push/pull from a centralized server.
CVS Producer
To catch up automatically with changes made to the tree, it is
necessary to act upon any commit to the tree for an ebuild file. A
possibility to do this would be via processing of CVS commit
messages, sent out as email by the CVS server. It is a task of the
producer to find out whether the ebuild found applies to the
testing machine (ppc-macos) and add the package/ebuild to the queue.
Consumer (testing process)
--
The test machine reads a line from the queue, and basically
executes 'emerge ${PACKAGE}'. However, before doing this, first it
figures out which use flags can be used (emerge -pv) and which
dependencies will be pulled (emerge -pt). If portage returns the
message all ebuilds that could satisfy X have been masked, the
emerge is cancelled, the line is removed from the queue and an
email message will be sent out.
All dependencies are put in the right order and emerged as normal
packages, that is: all dependencies are pushed at the front of the
queue, thereby keeping uniqueness of the queue and removing
duplicates that appear later on in the queue. After this, the
consumer is restarted and reads again from the queue. This should
result in usually merging only one package at a time, and as such
quite isolated cases, which should improve
Re: [gentoo-dev] revdep-rebuild
On Sat, 2005-08-13 at 23:26 -0400, Mike Frysinger wrote: i've already contacted fuzzray about utilizing two packages solar put together (and can be found in portage already): pax-utils: scanelf portage-utils: qfile Thanks for the ideas. I had a quick look at the programs; qfile: This would be useful in cleaning up the the last part of finding which package the file belongs to. But that part is already fairly quick compared to the rest. scanelf: From what I can see scanelf can print what libraries a file needs but it cannot say if the libraries are present. For example: /usr/bin/scanelf /bin/ls -n TYPE NEEDED FILE ET_EXEC librt.so.1,libncurses.so.5,libc.so.6 /bin/ls So you would need to keep a list of all libraries to check against. Thus I prefer using: LD_TRACE_LOADED_OBJECTS=1 /lib/ld-linux.so.2 /bin/ls But you could use ldconfig -p to gain a list of all the libraries, put them in a hash table and then use scanelf. All seems good, Stefan -- Stefan Jones [EMAIL PROTECTED] -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] revdep-rebuild
On Sunday 14 August 2005 16:20, Stefan Jones wrote: But you could use ldconfig -p to gain a list of all the libraries, put them in a hash table and then use scanelf. Please don't. FreeBSD's ldconfig is *not* the same and this would mean breaking (again) revdep-rebuild on Gentoo/FreeBSD. -- Diego Flameeyes Pettenò Gentoo Developer - http://dev.gentoo.org/~flameeyes/ (Gentoo/FreeBSD, Video, Gentoo/AMD64, Sound, PAM) pgpwd9UQD69gb.pgp Description: PGP signature
Re: [gentoo-dev] revdep-rebuild
On Sun, 2005-08-14 at 16:32 +0200, Diego 'Flameeyes' Pettenò wrote: On Sunday 14 August 2005 16:20, Stefan Jones wrote: But you could use ldconfig -p to gain a list of all the libraries, put them in a hash table and then use scanelf. Please don't. FreeBSD's ldconfig is *not* the same and this would mean breaking (again) revdep-rebuild on Gentoo/FreeBSD. Actually, for the next major revision, I am planning on some major collaboration between myself, the Gentoo/FreeBSD team, the OSX team, solar/vapier, and anyone else that is interested. The biggest complaint right now is lack of speed and that means investigating and exploring the different ways of determining library breakage. Some of those solutions are definitely not portable. Regards, Paul -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] revdep-rebuild
On Sun, 2005-08-14 at 09:52 -0500, Paul Varner wrote: Please don't. FreeBSD's ldconfig is *not* the same and this would mean breaking (again) revdep-rebuild on Gentoo/FreeBSD. Some of those solutions are definitely not portable. Well all we really need is the same utility to work across platforms, different platforms could have different implementations for certain functions. This has to be so if we are every going to have reasonable performance. But FreeBSD must have an equivalent function for it's dynamic linker to ldconfig -p, if not some code could be written up to do it. fuzzyray: Want me to do anything useful in particular? Had a quick look at Bug 63643. From what I can see that is a different problem then what revdep-rebuild solves (missing shared libraries), as you said. I think it can be best described as changing API between minor revisions of libraries. Will think a bit more. Stefan -- Stefan Jones [EMAIL PROTECTED] -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Re: [gentoo-osx] Package testing -- Automated initiative
On Sun, Aug 14, 2005 at 09:00:19AM -0500, Kito wrote: Adding -dev to CC: in case someone has any meaningful input or has already tackled this problem... We do already have an application capable of doing tinderboxing for packages. It's one of the abilities of Catalyst. So all you would need to do is hook up your queue/control system to call Catalyst for actually doing the build env. setup and actual build. I've been playing with Catalyst's tinderbox myself recently, running multiple instances on a 128-node x86 cluster (1/node). There are one or two things I'd like to change in Catalyst, as they currently make it slightly dangerous for your system. If for some reason an ebuild goes wrong, the bind mounts allow it access into your core system, namely /dev and $DISTDIR and it really hurts to have a bunch of /dev erased on a box that you don't have quick physical access to (I toasted a few cluster nodes due to a bug in some scripts this way, but luckily I could force a remote reinstall). Queues -- [snip] I'm fully up with this idea. CVS Producer How about using the gentoo-cvs commits mailing list? Consumer (testing process) -- Catalyst's tinderbox goes here. Combining it with a package cache, you can easily test just what you need. -- Robin Hugh Johnson E-Mail : [EMAIL PROTECTED] Home Page : http://www.orbis-terrarum.net/?l=people.robbat2 ICQ# : 30269588 or 41961639 GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgpi9B60pIOXE.pgp Description: PGP signature
[gentoo-dev] Adding arch specific use flags
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi All, Just a friendly (unless you don't fix them soon) reminder to folks who've added arch specific use flags to the tree recently (i.e. ibm) to make doubly sure that the use flag has been masked on all other architectures than the one you've added it for. Thanks! :) - -- Jason Wever Gentoo/Sparc Co-Team Lead -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFC/+WcdKvgdVioq28RAh9JAJ9MEkIARHmkFq6nZ/hUa+46ergbcgCgsab4 i157VQ04ba6/iPcyaIorblc= =lZN5 -END PGP SIGNATURE- -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Devconference archives
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Corey Shields wrote: | Archive links for the complete morning and afternoon sessions are at | http://devconference.gentoo.org Thanks Corey! I was unable to sit around all day Friday watching them, so this will be really helpful. Donnie -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFDAB4FXVaO67S1rtsRAnPwAJ9+znBFZySWNR+uRz9hRUQJJ/9tuQCgj3jK lTaqKGyx0+T0+fySfHLth1o= =IoUy -END PGP SIGNATURE- -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] flat profiles punted (and a note about x86/OpenBSD)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mike Frysinger wrote: as previously mentioned, i've punted all the flat profiles since the 2005.1 release since it seems like no one is using x86-obsd, can their team (if there is one) please create the proper cascading profiles for those of you looking to upgrade older portage versions, profiles/obsolete/ARCH/ exists for you -mike If you haven't done so already, might want to fire off a mail to -user as well. - -Alec Warner -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQwAqL2zglR5RwbyYAQK53g//cVEcHtPUsvQ/opt2dDcMPcNJlApDk8Tx LDRH5QmX0C1fRyL+OhFvd+ZPXhfiLFPJDzJS6pNn68wf9m2w+tRch7U1l1QAnmpS osGIhL0xYjAZt1l8ETXDwc8mJ98mEePB3R4/tsxvYs7l0jV/9xAmtGoKl3pLvRft f0fLx8BzIBImLabENI/U+enPU1pzvNFxsJZjeJeosG+rXsktkHLWVjYOtT4794+Y vtEUZGPbdZAQBlJG5VTld1OoSlTuPoLEfFJHaU0kbcZjs4WE2WQ6UpALbb8ghwaL 0MbpWY6a8Ga/Dqxj2diyloogJAA5yNoKdFd402U9bfWekUVjt3tXbHu/BducKmhy zpK0GOstuTemFjYLqlpVtlYqDf7XnCsTpUEW1nc6/76uAeiNusNW9Ef1ZPqXh04+ 5/mliXzvOXk5vg01hV+BJmqb10vFwoWxakwx74lAzdKcskRwqn7Yfml5KQt1rhiu TmiEXG5UoEQukEm2l/I47G7TLrnvp8NKFnXhu6Ak/3L11ck7j7Ow+3UbFDkbUNop A/dkS3xviddYNTcv39DQ6nuUPz/XsbXBkQxKCnxuMKnGYniTTRwCv2uuZ3CgAqE+ y71TFkRUiVt2Npou9yCi65YPB0fvQNlqGVdSsuxRVWjmNpiybFWW4bTCuPF51FNQ uUDlK1zYNCc= =FheJ -END PGP SIGNATURE- -- gentoo-dev@gentoo.org mailing list
