Adding -dev to CC: in case someone has any meaningful input or has
already tackled this problem...
On Aug 14, 2005, at 8:21 AM, Grobian wrote:
Introduction
Recently, once again we were confronted with a package marked as
ppc-macos stable, while it didn't compile at all, let alone run.
It is
believed more of these packages are in portage, and need to be
found and
fixed. Keeping the cause of why they are marked stable up to another
discussion, and out of the scope of this discussion, I will focus
on how
to track these packages down and report them to us.
In the secondary line, all 'unstable' packages, marked ~ppc-macos
should
be tested as well, since they can be faulty as well. Since for OSX
much
is in ~ppc-macos, many users consider it a normal procedure to
switch to
the unstable side of portage, hence some extra need for careful
testing
of ~ppc-macos also.
Proposed Global Structure
=
Testing should be done on a regular basis, both push and pull based.
This means that the testing machine would start testing packages
itself
if it is out of work, and on the other hand starts testing packages as
soon as they are being added/changed in CVS. It may need no great
imagination to see that the latter 'push-based' activity has priority
over the 'pull-based' work.
I'm not sure direct interaction with CVS would be needed, usually
only takes a short time for cvs commits to hit the rsync mirrors
(hence the volatile nature of the tree)
Starting over, will for the test machine mean that it starts cleaning
out its world file. Cleaning this file out to a bare minimum is an
important aspect of getting a test environment that reflects the
situation on new user's machines. If an ebuild uses a package without
having it in it's DEPEND, this may get noticed only when starting on a
clean machine. This, however, will add a big delay in testing as many
packages will need to be built prior the right package can be
installed.
The testing machine will have a queue file, which it reads packages to
emerge from. If the queue file is empty, i.e. when there is no push
based work, the machine will generate work by starting to compile
uncompiled packages, or emptying the tree.
Because ~ppc-macos and ppc-macos packages interfere with each other
-- a ~ppc-macos package overwrites a ppc-macos package -- both
stable and unstable have to be dealt with separately, i.e. they
should both have their own environment either via two separate
machines, or through the use of a chroot jail.
I think seperate chroots are definitely the way to go. We can just
store a 'pristine' chroot in iso or dmg or whatever on the build
server and copy when needed.
Queues
--
In order not to drag in a full DBMS (in the end Portage already is
one) queues are just simple flat files consisting of absolute
package names, one per line. Table wise locking granularity is
handled by the OS as one process opens the file in write mode.
Consumers -- the testing box in this case -- read the first line
and delete it, while producers simply add one line (or more) to the
end of the file.
The queue itself, is more a set than a list. This means that
packages that are in the queue, should be unique. If a package is
added that is already in the queue, it is dropped such that the
original queue position of the package is maintained.
Maybe a 'proper' dbms wouldn't be such a bad idea, could also store
build logs, timestamps, etc. there and make it easier for multiple
build hosts to push/pull from a centralized server.
CVS Producer
To catch up automatically with changes made to the tree, it is
necessary to act upon any commit to the tree for an ebuild file. A
possibility to do this would be via processing of CVS commit
messages, sent out as email by the CVS server. It is a task of the
producer to find out whether the ebuild found applies to the
testing machine (ppc-macos) and add the package/ebuild to the queue.
Consumer (testing process)
--
The test machine reads a line from the queue, and basically
executes 'emerge ${PACKAGE}'. However, before doing this, first it
figures out which use flags can be used (emerge -pv) and which
dependencies will be pulled (emerge -pt). If portage returns the
message all ebuilds that could satisfy X have been masked, the
emerge is cancelled, the line is removed from the queue and an
email message will be sent out.
All dependencies are put in the right order and emerged as normal
packages, that is: all dependencies are pushed at the front of the
queue, thereby keeping uniqueness of the queue and removing
duplicates that appear later on in the queue. After this, the
consumer is restarted and reads again from the queue. This should
result in usually merging only one package at a time, and as such
quite isolated cases, which should improve
