Re: [gentoo-portage-dev] virtuals and dependencies dispaly

2006-10-25 Thread Jason Stubbs
On Tuesday 24 October 2006 16:36, Brian wrote:
 def get_virtual_dep(atom):
 returns a resolved virtual dependency.
 contributed by Jason Stubbs, with a little adaptation
 # Thanks Jason
 non_virtual_atom = portage.dep_virtual([atom], portage.settings)[0]
 if atom == non_virtual_atom:
 # atom,is a 'new style' virtual (aka regular package)
 return atom
 else:
 # atom,is an 'old style' virtual that resolves to:
 non_virtual_atom
 return non_virtual_atom

This could be simplified to:

def get_virtual_dep(atom):
returns a resolved virtual dependency.
return portage.dep_virtual([atom], portage.settings)[0]

Hmm.. Actually, there is one problem here.

 import portage
 portage.dep_virtual(['virtual/mda'], portage.settings)
[['||', 'mail-mta/postfix', 'mail-filter/procmail']]

The values of portage.settings.virtuals are arranged in such a way
that the first value in a key's list is either an installed package
or the package that would be chosen (if no other package in the list
or new package PROVIDEing the virtual was brought in for other
reasons) so you'd probably be better off doing something like:

def get_virtual_dep(atom):
returns a resolved virtual dependency.
key = portage.dep_getkey(atom)
virtuals = portage.settings.getvirtuals()
if key in virtuals:
return atom.replace(key, virtuals[key][0])
else:
return atom

Displaying all known packages for any specific virtual is also a
possibility... I'll leave it up to you on that. ;)

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] virtuals and dependencies dispaly

2006-10-23 Thread Jason Stubbs
On Monday 23 October 2006 16:26, Brian wrote:
 I've been improving portholes dependency display, adding more info to
 it.  I've run into a minor snag when virtuals are concerned.  There also
 seems to be 2 sources of virtuals.  Both are not identical to each
 other:

 /usr/portage/virtual -- which seem to be packages.

They are just regular packages.

 virtuals = portage.db[portage.root][virtuals]  from
 gentoolkit.__init__.py

I'm not sure exactly what this returns, but it's probably equivalent to 
portage.settings.virtuals.

 I've found virtuals in the tree that are not in portages virtuals and
 vice-versa.
 Is there an order to their use by portage?

atom = virtual/portage
non_virtual_atom = portage.dep_virtual([atom], portage.settings)[0]

if atom == non_virtual_atom:
print atom,is a 'new style' virtual (aka regular package)
else:
print atom,is an 'old style' virtual that resolves to',
print non_virtual_atom

 Can the ones in the tree be 
 treated as a category/package for listings, etc..  The ones in the tree
 do seem to be returned by:

 portage.db['/']['porttree'].getallnodes()[:] # copy

 The main reason I am looking at this is for indicating the dependency
 installed/needed for the virtual.  If an installed one is found I use
 it's package name. If not  where should I get it?  From the virtuals in
 portage or the tree? both? - then which order?

The ones in the tree are just regular packages - but may also exist in the 
PROVIDE attribute of associated ebuilds. As for ordering, packages with 
PROVIDE override identically named packages in the tree. If you use something 
similar to the above, it should all be taken care of though.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] accessing portage updates through it's data structures

2006-04-18 Thread Jason Stubbs
On Tuesday 18 April 2006 13:20, Tom Hosiawa wrote:
 Anybody have any ideas how I can basically do 'emerge -puDNv' directly
 from my superkaramba python program, and get the updates available
 directly from the portage data structures?

What is needed for -uDNv is mostly locked away in emerge, unfortunately.
The following couple of lines should get you everything other than --newuse
though. As you only seem to be interested in what packages are available
for updating, the ordering shouldn't really matter.


import portage

pdb = portage.db[/][porttree].dbapi
vdb = portage.db[/][vartree].dbapi

installed_pkgs = dict([(key, vdb.match(key)) for key in vdb.cp_all()])
latest_pkgs = dict([(key, portage.best(pdb.match(key))) for key in 
installed_pkgs])
updatable_pkgs = [latest_pkgs[key] for key in latest_pkgs if latest_pkgs[key] 
not in installed_pkgs[key]]
updatable_pkgs = [pkg for pkg in updatable_pkgs if pkg]


That last line there is to kill off the None elements that end up in
updatable_pkgs when there is a package installed that has no versions
available in the rsync tree. Other than that, portage.catpkgsplit() will
split a package identifier into [cat, pkg, ver, rev].

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] 2.1 release candidate soon?

2006-04-14 Thread Jason Stubbs
On Saturday 15 April 2006 03:31, Brian Harring wrote:
 Sidenote, why is userfetch a feature?  That seems like something that 
 should be userpriv by default to me...

It broke somebody's ftp setup.

http://bugs.gentoo.org/show_bug.cgi?id=92960

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] 2.1 release candidate soon?

2006-04-14 Thread Jason Stubbs
s/ftp/nfs/ in the mail that I just sent.

--
Jason Stubbsw
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] 2.1 release candidate soon?

2006-04-14 Thread Jason Stubbs
On Saturday 15 April 2006 03:31, Brian Harring wrote:
 cache backend selection (failed import == defaults to sys default)

This is incorrect. It displays an error message and quits.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] 2.1 release candidate soon?

2006-04-10 Thread Jason Stubbs
On Saturday 08 April 2006 21:48, Ned Ludd wrote:
 On Sat, 2006-04-08 at 11:18 +0900, Jason Stubbs wrote:
  On Saturday 08 April 2006 07:36, Ned Ludd wrote:
   On Fri, 2006-04-07 at 14:19 -0400, solar wrote:
FEATURES=buildpkg ROOT=/ emerge gcc
rm -rf /dev/shm/foo

ROOT=/dev/shm/foo emerge gcc -pvK

Notice how it selects the incorrect deps? 
IE: eselect cuz it's the first listed dep in the || ( ) vs the
gcc-config
   
   + When you already have a copy of gcc-config installed on / and in 
   .tbz2 format in ${PKGDIR}/All and no eselect anywhere.
  
  This should work. I believed I had fixed it by adding the use_binaries
  parameter and code paths to dep_zapdeps. If it's not working then there must
  be a bug left somewhere.
 
 Must be a bug left somewhere then. I just tested with 
 Portage 2.1_pre7-r4 and the result is the same.

Got it. The bug was in bindbapi. For consistency's sake, I changed most of the
db[/] to db[myroot] except where db[/] is specifically required. However,
db[myroot][bintree].dbapi.* were all working with an empty repository as
bintree holds all the data and uses lazy initialization which bindbapi wasn't
triggering.

I've fixed the current use case and covered aux_get as well, but this bug could
easily come back if another method of bindbapi is used.

  Having a quick look at the dep_zapdeps function, I can't see what but I 
  think
  I've discovered another bug. If use_binaries is true, porttree isn't checked
  for matches which means that it'll fall through to the last resort code
  when there's no matching binaries which could end up selecting an atom that
  only has masked porttree matches.
 
 yikes.

This is and isn't the case. use_binaries is only enabled with -K so masking
doesn't take affect anyway.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] should we add userpriv and usersandox to make.globals FEATURES?

2006-04-10 Thread Jason Stubbs
On Tuesday 11 April 2006 04:28, Zac Medico wrote:
 Simon Stelling wrote:
  Zac Medico wrote:
  What do people think about adding userpriv and usersandox to
  make.globals FEATURES?  I've been using these for a long time and
  haven't had any trouble with them.  Are there any arguments against
  making them default?
 
  I didn't verify this personally, but a few days ago mkay came to
  #g-portage and asked whether FEATURES='usersandbox -sandbox' resulting
  in sandbox enabled is expected behaviour or not. Before we add
  usersandbox to the default FEATURES we should make sure that -sandbox
  always disables sandbox.
 
 Yeah, we should fix that.  In fact, usersandbox seems like a redundant
 feature to me.  Can we deprecate usersandbox and recommend sandbox as
 the sole means of toggling sandbox on and off (whether userpriv is
 enabled or not)?   

sandbox userpriv thus far has meant to prefer userpriv and fallback to
sandbox when the ebuild doesn't work with userpriv. When those two are
combined with usersandbox on the other hand, it has meant to throw
everything possible at the ebuild. I personally prefer to not use
usersandbox as the sandbox gives a sometimes-not-small performance hit
and I'm a ricer. :P

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] tree dependency check

2006-03-30 Thread Jason Stubbs
On Thursday 30 March 2006 11:40, Marius Mauch wrote:
 On Thu, 30 Mar 2006 08:30:17 +0900
 Jason Stubbs [EMAIL PROTECTED] wrote:
 
  On Thursday 30 March 2006 01:21, Marius Mauch wrote:
   Marius Mauch schrieb:
So after manifest2 is in, I'll revive the other issue that IMO is
a requirement for 2.1: enforcing dependencies needed to use the
tree (see old threads or glep44 for reasoning).
  
  Can you summarise the reasoning again please?
 
 a) avoid massive breakage when certain new features are introduced
 (past examples being cascading profiles or new-style virtuals)

I can't see how massive breakage can be avoided. With your patch the only
difference is instead of partial breakage it's something like your system
is likely broken so go and visit some page to find out if it really is and
how you can fix it.

 b) similar to a) allow people to use new features without having to
 wait for a year or two

Why not just always force portage to be upgraded to at least the latest
stable version? The user can override it by masking (although there isn't
ever any need to as portage doesn't affect other software). If the ebuild
environment is properly documented, things such as new bash features can
be tied to EAPI. Using EAPI on portage ebuilds themselves, a clear upgrade
path can be designated and maintained within the tree.

With EAPI and forced portage upgrades, the only thing that isn't covered
is profile masking. However, your patch doesn't really cover that either.
Unless I'm missing how that would be covered?

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Re: User created package lists

2006-03-23 Thread Jason Stubbs
On Thursday 23 March 2006 23:43, Brian wrote:
 On Thu, 2006-23-03 at 22:14 +0900, Jason Stubbs wrote:
  On Thursday 23 March 2006 16:23, Brian wrote:
   /etc/portage/lists/userlist1
   
   format:
   
   net-www/apache 
   www-apache/mod_perl
   ...
  
  If you make that /etc/portage/sets and support any package atom (rather
  than only cat/pkg) then I you'd pretty much have what is planned (afaik).
  
  --
  Jason Stubbs
 
 
 What about adding the other update config similar to package.keywords
 
 quoting me:
 Where updates could be one or more of M manual, A automatic, N
 never, K binary packages only.  Also L for license
 
 
 these examples may not be normally sane to actually do.  I'm not a
 server admin.  They are just for demonstration
 eg.
 
 /etc/portage/sets/server
 
 =net-www/apache-2.* M,K   # use my pre-built binaries only when I say
 www-servers/pound A # stay up to date
 ...
 
 /etc/portage/sets/games
 
 games-action/bombermaze 
 games-action/descent3 L # automatically accept it's license 


It seems like too much overloading to me. L is done with ACCEPT_LICENSE.
K could be done with separate repositories (not overlays). M can already
be done with profiles/masking. I can understand your motivation in
tying parameters together, but I honestly think this is not the way to
go about it.

--
Jason Stubbs


-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] kudoos to all

2006-03-15 Thread Jason Stubbs
On Thursday 16 March 2006 00:24, Brian wrote:
 I just want to say that all the speedup work everyone has been doing is
 really noticeable.
 
 Porthole now can reload the entire database 10763 packages (includes
 some in PORTDIR_OVERLAY) in about 3 seconds on my system.
 
 Athlon-xp 2000 based.
 
 the cache update after a sync no longer takes as long as compiling the
 mozilla suite .)  good work! :)
 
 currently @ portage-2.1_pre6

Most of that work can be attributed to Brian Harring. For a real kicker
though, throw portdbapi.auxdbmodule = cache.metadata_overlay.database
into /etc/portage/modules and FEATURES=-metadata-transfer into make.conf.
Then rm -rf /var/cache/edb/dep. This one goes to Zac Medico, the current
release coordinator, and builds on Brian's work to cut out cache updating
altogether. New in pre6 (pre5?) and in need of testing - especially when
local modifications are made - but very promising.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Move PORTAGE_INST_UID and PORTAGE_INST_GID to make.globals?

2006-03-10 Thread Jason Stubbs
On Saturday 11 March 2006 06:48, Kito wrote:
 
 On Mar 10, 2006, at 4:26 AM, Zac Medico wrote:
 
  -BEGIN PGP SIGNED MESSAGE-
  Hash: SHA1
 
  Hello,
 
  What do people think about moving PORTAGE_INST_UID and  
  PORTAGE_INST_GID to make.globals?
 
 Would the profiles not be a more logical place to set these? If not,  
 can we twist the logic to make it so? :p

The profiles would be able to override it. In the case of non-incrementals
the first definition found is the winner in the order of:

env - make.conf - make.defaults (profile) - make.globals

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] vdb-update script (for global updates) with job progress framework

2006-02-28 Thread Jason Stubbs
On Tuesday 28 February 2006 22:28, Zac Medico wrote:
 vdb-update has the beginnings of a job progress framework that portage can 
 use to run various time consuming tasks, display progress, and interrupt 
 tasks when necessary.  When vdb-update and fixpackages are ready, I'd like 
 to make both of them into emaint modules.   

It's looking nice and clean at the moment. You might want to think twice about 
integrating it into emaint, though. Integrating the couple of things that 
emaint does into what you're working on would be much better. :)

Seriously, emaint was created in about 30 minutes to counter the regression 
of emerge warning on unsatisfiable world file entries. It was/is not meant to 
stand the test of time in its current state. Why would you want to muddy up 
your code with it? ;)

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Config Cleanup Last Call

2006-02-24 Thread Jason Stubbs
On Saturday 25 February 2006 10:40, Alec Warner wrote:
 Please review for badness, otherwise I'll commit this soon ;)

$ grep ^\- portage-config-cleanup.patch | wc -l
58
$ grep ^\+ portage-config-cleanup.patch | wc -l
41

The unrelated PORTAGE_BINHOST chunk is -15/+1 which makes the relevant
part of the patch -43/+42.  What is the goal?

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Config Cleanup Last Call

2006-02-24 Thread Jason Stubbs
On Saturday 25 February 2006 10:40, Alec Warner wrote:
 Please review for badness, otherwise I'll commit this soon ;)

 - self.backupenv = os.environ.copy()
 - self.configlist.append(self.backupenv) # XXX Why though?
 - self.configdict[backupenv]=self.configlist[-1]
 + self.configdict[backupenv]=os.environ.copy() # XXX Why though? ( ferringb? 
)

Kill this comment altogether. Reading it, my first thought is why what?
It doesn't make the code clearer and so shouldn't be there.

 - self.configlist.append(os.environ.copy())
 - self.configdict[env]=self.configlist[-1]
 + ### backupenv maybe required to be a clone, and not just a reference
 + ### the old code did value copy we do a reference here
 + self.configdict[env]=self.configdict[backupenv]

Again, this comment doesn't make the code clearer and so shouldn't be there.
Changing functionality in a cleanup patch is also bad form. As for the
change itself, it's broken. Look at __setitem__() and reset() and then look
at the usage of those two throughout the code.

 - mydbs=self.configlist[:-1]
 + #Abuse the order of lookuplist to emulate old behavior
 + mydbs=self.lookuplist[:-1]

Same deal with this comment. A person seeing the code for the first time has
no idea what the old behaviour was. In the old code, configlist starts with
globals and ends with env. In the new (and old) code, lookuplist is the
reverse of that. I don't see any further changes to account for this.

 - if 0 and match and mykey in [PORTAGE_BINHOST]:
 - # These require HTTP Encoding
 ...

This shouldn't be in a cleanup patch either.

--
Jason Stubbs

-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Deprecating 'emerge action' syntax

2006-02-16 Thread Jason Stubbs
On Thursday 16 February 2006 20:31, Marius Mauch wrote:
 Right now 'emerge action' and 'emerge --action' are both supported. But
 as we learned with the rsync case 'emerge action' has potential
 namespace conflicts with 'emerge package' I'd propose to deprecate
 'emerge action' before we hit another real conflict.
 (The alternative would be to deprecate 'emerge package' in favor of a
 to-be-written 'emerge install package', but that's even more
 problematic)
 Technically it's a no-brainer, only potential problem would be user
 confusion.
 Any objections against this for pre5?

If by deprecate you mean to detect when '--' hasn't been prepended and
either go ahead with the action or notify that the package doesn't exist
then I have no objections. Might be better to go with the latter so that
users adjust quickly.


Actions specified without a '--' prefix is no longer supported.
Please use --update instead.

emerge: there are no ebuilds to satisfy update.


Doing it that way will show exactly why it's being dropped without the
need for a written explanation (and hopefully no bug about how it's a
terrible usability regression).

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Deprecating 'emerge action' syntax

2006-02-16 Thread Jason Stubbs
On Friday 17 February 2006 05:30, solar wrote:
 On Thu, 2006-02-16 at 19:04 +0100, Marius Mauch wrote:
 
 [snip]
 
   If by deprecate you mean to detect when '--' hasn't been prepended
   and either go ahead with the action or notify that the package
   doesn't exist then I have no objections. Might be better to go with
   the latter so that users adjust quickly.
  
  You saw the attached one-line patch?

Heh. Actually I completely missed it.

  Atm it just throws a warning when an action without -- is used.
  I'm divided on ignoring the action then, on one hand it would be nice to
  get rid of this, OTOH it would be kinda rude to not have a transition
  period for people. Anyone else having an opinion on this?
 
 I agree it would be kinda rude to just deprecate a behavior and
 introduce another all in one shot. I'd give it a few releases till
 people retrain themselves.
 Maybe when emerge --world/--system/--action has worked it's way into
 at least 2 stable releases before deprecating 'action'

This brings up an interesting point. emerge --world really should fail as
not being valid as it is not an action but a target. With the goal of having
targets user configurable, treating them as options really doesn't work.

So, I'm also moving toward just printing warnings for the time being.
However, we should probably figure out exactly how it should work and then
print warnings whenever any deprecated syntax is used. Not so much for users
and scripts, of which I can't see there being much of a problem, but so that
we can redo that whole bunch of code without having to do:

if incorrect_syntax:
print warning
make correct syntax

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] IUSE_DEFAULTS-v0.1

2006-02-14 Thread Jason Stubbs
On Tuesday 14 February 2006 21:44, Alec Warner wrote:
 Brian Harring wrote:
  IUSE- eapi bump it, I pushed for the var for exactly crap like this ;)
  
  The only (imo) reason to add the USE_ORDER chunks is because users use 
  -* to castrate auto-use; auto-use is dead in 2.1, so the main reason 
  for using -* is gone.
  
  So... really worth adding another chunk of metadata for this?
 
 Well that problem would be, no one wants to modify everything in
 app-portage/ :).  If my portage EAPI is 1, but my tools don't support
 processing +- in IUSE, how does EAPI help me here?  The support check is
 only for portage_const, so the tool remains sensored.  Unless I'm
 missing something.

Nah, Brian's right. Tools need to follow. Backwards compatibility isn't so
important there. The important thing is that portage keeps on living.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] confcache, final chance to ixnay it

2006-02-02 Thread Jason Stubbs
On Thursday 02 February 2006 21:12, Brian Harring wrote:
 Yo...
 
 attached is a patch enabling confcache support for portage.  Lots of 
 testing, plus fixups from comments from folks prior.
 
 So... giving it a few days, nows the time to bitch if you dislike the 
 implementation (and no, I'm not rewriting all of doebuild just for 
 this :)

Happy here. If there were no other issues, may as well go ahead with it
earlier rather than later. Spread the goodness (or something like that ;)

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] confcache integration

2006-01-24 Thread Jason Stubbs
On Tuesday 24 January 2006 21:50, Brian Harring wrote:

+os.makedirs(mysettings[CONFCACHE_DIR], mode=0775)
+os.chown(mysettings[CONFCACHE_DIR], portage_uid, -1)

This will die when running as non-root, no? ebuild is now oft used by devs
running as normal users which will be broken by this. No clues on the bash
stuff; it seems there's an external confcache binary but I can't tell much
beyond that.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] portage-2.1_pre2 and new use flag showing with emerge -p

2005-12-26 Thread Jason Stubbs
On Monday 26 December 2005 07:18, Petteri Räty wrote:
 I propose we improve the emerge -pv output to be something like the
 following:

 [ebuild U ] media-sound/alsa-driver-1.0.10-r1 [1.0.10] NEW=-debug
 OLD=-doc oss 0 kB

 This would keep the functionality with --verbose.

The NEW and OLD variables have no meaning. USE have been put there because any 
USE_EXPAND flags found in IUSE are represented separately. For example:

[ebuild   R   ] x11-base/xorg-x11-6.8.2-r6  USE=-bitmap-fonts cjk -debug 
-dlloader -dmx -doc font-server -insecure-drivers -ipv6 -minimal -nls -nocxx 
opengl -pam -sdk -static truetype-fonts -type1-fonts -xprint xv 
INPUT_DEVICES=-synaptics -wacom 44,520 kB

I doubt that the NEW and OLD would really be visible in the --verbose output 
in the general case anyway. How about just making added flags green to match 
the output of changed flags?

--
Jason Stubbs

-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] per ebuild distdir symlinking

2005-12-25 Thread Jason Stubbs
On Monday 26 December 2005 07:12, Brian Harring wrote:
 Hola.

 Continuing the tradition of being a lazy bugger and raiding from my
 previous work instead of doing something new, attached is a patch that
 forces unpack to go through a level of indirection.
...
 Either way, it's attached.  Comments?

No problems on my part. As Mike said, it'll only catch the standard unpack 
usage but that's not really an issue as far as I can see.

By the way, now that we've got -commit mail, confirming with the ML isn't 
really necessary. Of course, if it's something you want to confirm...

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [RFC] making the tree depend on portage

2005-12-20 Thread Jason Stubbs
On Tuesday 20 December 2005 01:49, Marius Mauch wrote:
 Also not talking about implementation details yet, just after comments
 about the general idea of forced portage updates.

I gave it a go anyway... ;)

# emerge -up kde-base/kde
Checking for mandatory system updates...  Done.

The following packages can't be merged until your system is updated:

   kde-base/kde

The following important packages are either not up to date or not installed:

   virtual/baselayout sys-apps/coreutils sys-apps/debianutils
   sys-apps/diffutils sys-apps/file sys-apps/findutils sys-apps/gawk 
   sys-apps/grep sys-apps/groff sys-apps/kbd sys-apps/man sys-apps/man-pages 
   sys-apps/net-tools =sys-apps/portage-2.0.51.22 sys-apps/sed 
   sys-apps/shadow sys-apps/texinfo sys-apps/which virtual/modutils 
   virtual/pager sys-apps/busybox sys-apps/hdparm sys-apps/util-linux 
   sys-apps/linux32 =sys-apps/baselayout-1.9.4-r7

Please retry your command after updating your system.


These are the packages that I would merge, in order:

Calculating system dependencies ...done!

[ebuild  N] sys-apps/sandbox-1.2.17
[ebuild  N] sys-apps/sed-4.1.4  USE=-bootstrap -build -nls -static
[ebuild  N] sys-apps/debianutils-2.15  USE=-build -static
[ebuild  N] sys-apps/portage-2.1_pre1  USE=-build
*** Portage will stop merging at this point and reload itself,
recalculate dependencies, and complete the merge.
You may avoid the remerging of packages by updating portage on its own.
[ebuild  N] sys-apps/help2man-1.35.1  USE=-nls
[ebuild  N] sys-apps/coreutils-5.3.0-r2  USE=-acl -build -nls -static
[ebuild  N] sys-apps/sysvinit-2.86-r3  USE=-bootstrap -build -ibm 
-static
[ebuild U ] sys-libs/readline-5.1 [5.0-r2]
[ebuild  N] sys-apps/baselayout-1.12.0_pre11-r3  USE=unicode -bootstrap 
-build -static
[ebuild  N] sys-apps/diffutils-2.8.7-r1  USE=-nls -static
[ebuild  N] sys-apps/file-4.16  USE=python -build
...
...


* Portage is moved to the top of both system and world
* All system atoms are checked that they are matched by installed packages
  and emerge refuses to do anything until unsatisfied atoms are installed 
  and/or updated.
* EMERGE_INSANITY_OK is provided for bootstrap.sh (and emerge itself) to 
  bypass the checks when the system is known to be in an incomplete state.

Reasoning on checking all system atoms is that other groups are just as likely 
to need the functionality as we are. Combining that with how rarely versions 
are actually updated for system packages, it shouldn't cause any more bother 
to users than it needs to.

--
Jason Stubbs
Index: bin/emerge
===
--- bin/emerge	(revision 2414)
+++ bin/emerge	(working copy)
@@ -1418,6 +1418,14 @@
 
 			mylist = sysdict.keys()
 
+		newlist = []
+		for atom in mylist:
+			if portage.dep_getkey(atom).split(/)[-1] == portage:
+newlist.insert(0, atom)
+			else:
+newlist.append(atom)
+		mylist = newlist
+		
 		for mydep in mylist:
 			try:
 if not self.select_dep(portage.root, mydep, raise_on_missing=True):
@@ -2501,6 +2509,73 @@
 if --debug in myopts:
 	edebug=1
 
+
+def check_and_update_system():
+	if os.environ.get(EMERGE_INSANITY_OK) or myaction == system:
+		return
+	rsync_timestamp = open(portage.settings[PORTDIR]+/metadata/timestamp).read().strip()
+	last_check = portage.mtimedb.get(last_system_check, )
+	if rsync_timestamp == last_check:
+		return
+	print Checking for mandatory system updates... ,
+	system_atoms = getlist(system)
+	vdb = portage.db[/][vartree].dbapi  # This check only applies to /
+	unsatisfied_atoms = [atom for atom in system_atoms if not vdb.match(atom)]
+	print Done.
+	if unsatisfied_atoms:
+		if myfiles:
+			allowed_keys = [portage.dep_getkey(atom) for atom in system_atoms]
+			virtuals = portage.settings.virtuals
+			for key in allowed_keys:
+if key in virtuals:
+	allowed_keys.extend(virtuals[key])
+			allowed_files = []
+			for myfile in myfiles:
+key = portage.dep_getkey(myfile)
+if key in allowed_keys:
+	allowed_files.append(myfile)
+	continue
+if / in myfile:
+	continue
+for category in portage.settings.categories:
+	if category+/+key in allowed_keys:
+		allowed_files.append(myfile)
+		break
+			disallowed_files = [myfile for myfile in myfiles if myfile not in allowed_files]
+			if disallowed_files:
+print red(\nThe following packages can't be merged until your system is updated:)
+width = 80
+for myfile in disallowed_files:
+	width += len(myfile) + 1
+	if width = 76:
+		width = 2
+		print \n  ,
+	print myfile,
+	myfiles.remove(myfile)
+print
+			if myfiles:
+return
+		print red(\nThe following important packages are either not up to date or not installed:)
+		width = 80
+		for atom in unsatisfied_atoms:
+			width += len(atom) + 1
+			if width = 76:
+width = 2
+print \n  ,
+			print atom,
+		print yellow(\n\nPlease retry your command

Re: [gentoo-portage-dev] [RFC] making the tree depend on portage

2005-12-20 Thread Jason Stubbs
On Tuesday 20 December 2005 23:15, Jason Stubbs wrote:
 On Tuesday 20 December 2005 01:49, Marius Mauch wrote:
  Also not talking about implementation details yet, just after comments
  about the general idea of forced portage updates.

 I gave it a go anyway... ;)

Also needed:

Index: portage.py
===
--- portage.py  (revision 2414)
+++ portage.py  (working copy)
@@ -6742,7 +6742,7 @@
 mtimedbkeys=[
 updates, info,
 version, starttime,
-resume, ldpath
+resume, ldpath, last_system_check
 ]
 mtimedbfile=root+var/cache/edb/mtimedb
 try:

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] PATCH: parallel-fetch

2005-12-16 Thread Jason Stubbs
On Friday 16 December 2005 19:01, Brian Harring wrote:
 On Fri, Dec 16, 2005 at 12:09:10PM +0900, Jason Stubbs wrote:
  On Thursday 15 December 2005 20:06, Brian Harring wrote:
   This is the only blocker for merging parallel-fetch as far as I can
   tell- so... my vote is nuking the wait out of portage.portageexit
   (it's exec subsystem crap, belong in exec's atexit (where it exists)).
   Assuming no complaints there, issues with parallel-fetch going into
   svn?
 
  Nuking the wait is fine if things will still work the same.. I'm kind of
  wondering if ctrl-c'ing behaviour will change - how gets the ctrl-c
  first?

 ctrl-c doesn't come in play here- just triggers exithandler, which
 calls portageexit.  What's stupid about this code is that exithandler
 has it's *own* set of kill calls in it (nothing like 101 duplicated
 bits).

 Either way... exit, or nothing left to execute, python will then
 trigger atexit registered callbacks in a lifo manner.  So all
 exithandler really _should_ have to do is just sys.exit.

I mean for when a sub-process is running such as ebuild or rsync. Will portage 
get the SIGINT before or at the same time as the sub-process causing portage 
to then SIGKILL it while it's trying to exit cleanly? Given that portage's 
atexit handler is registered after portage_exec's, the subprocess reaping 
code would not have been doing anything in the past. I guess the question is 
more a general one rather than specifically related to background fetching.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] DepSet

2005-12-08 Thread Jason Stubbs
On Thursday 08 December 2005 16:44, Zac Medico wrote:
 The middle hunk fixes a problem with block atoms that do not match any 
 packages.  Previously, these atoms would not make it into the okay_atoms set 
 which caused unresolved dependencies.  

Are you sure about this? For reference:

@@ -58,12 +58,19 @@
all_atoms.union_update(atomset)
okay_atoms = sets.Set()
for atom in all_atoms:
+   have_blocker=False
for child in self.pkgs:
if atom.key != child.key:
continue
-   if atom.match(child) ^ atom.blocks:
-   okay_atoms.add(atom)
+   if atom.match(child):
+   if atom.blocks:
+   have_blocker=True
+   else:
+   okay_atoms.add(atom)
break
+   if atom.blocks and not have_blocker:
+   # block atom that does not match any 
packages
+   okay_atoms.add(atom)
for choice in self.pkgs[pkg][0]:
if choice.issubset(okay_atoms):
break


if atom.match(child) ^ atom.blocks:
okay_atoms.add(atom)
break

is the same as:

if (atom.match(child) and not atom.blocks) or \
(not atom.match(child) or atom.blocks):
okay_atoms.add(atom)
break


have_blocker = False
...
if atom.match(child):
if atom.blocks:
have_blocker=True
else:
okay_atoms.add(atom)
break
if atom.blocks and not have_blocker:
okay_atoms.add(atom)

Therefore:

have_blocker = atom.blocks and atom.match(child)

which makes the last check:

if atom.blocks and not (atom.blocks and atom.match(child)):
okay_atoms.add(atom)

which is equivalent to:

if atom.blocks and (not atom.blocks or not atom.match(child)):
okay_atoms.add(atom)

Removing the impossibility of atom.blocks and not atom.blocks:

if atom.blocks and not atom.match(child):
okay_atoms.add(atom)

Within the loop you have:

if atom.match(child):
if atom.blocks:
...
else:
okay_atoms.add(atom)
break

Which could be represented as:

if atom.match(child) and not atom.blocks:
okay_atoms.add(atom)

Combining those two gives:

if atom.blocks and not atom.match(child) or \
atom.match(child) and not atom.blocks:
okay_atoms.add(atom)

Which is exactly the same thing as the original except seven lines longer...

--
Jason Stubbs

-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] DepSet

2005-12-08 Thread Jason Stubbs
On Friday 09 December 2005 04:03, Zac Medico wrote:
 Jason Stubbs wrote:
  On Thursday 08 December 2005 16:44, Zac Medico wrote:
 The middle hunk fixes a problem with block atoms that do not match any
 packages.  Previously, these atoms would not make it into the okay_atoms
  set which caused unresolved dependencies.
 
  Are you sure about this?

 Well, I'm pretty sure. You're analysis seems to be perfectly correct except
 for 2 points that you haven't accounted for:

 1) The atom.key != child.key optimization prevents the atom.match(child) ^
 atom.blocks bit from working in the case that my patch handles (block atom
 that does not match any package).

 2) Without the atom.key != child.key optimization, the original algorithm
 would bail out early, before all packages have been checked.  We need to
 ensure that the atom does not block _any_ of the packages before we add it
 to okay_atoms, otherwise, we risk choosing the wrong atomset/combination. 
 Note that checking all the packages _does_ introduce a performance penalty
 for block atoms.

Damn optimizations. :|

Both points are correct.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-12-07 Thread Jason Stubbs
On Wednesday 07 December 2005 11:57, Marius Mauch wrote:
 On Wed, 7 Dec 2005 08:41:27 +0900

 Jason Stubbs [EMAIL PROTECTED] wrote:
  On Wednesday 07 December 2005 01:01, Marius Mauch wrote:
   On Tue, 6 Dec 2005 23:19:38 +0900
  
   Jason Stubbs [EMAIL PROTECTED] wrote:
If there's no solid opposition, Saturday I will put current trunk
into ~arch as 2.1_beta20051210.
  
   Well, I've already stated several times that IMO using a 2.1 branch
   is wrong (use 2.2 instead), but if I'm overvoted, so it shall be.
 
  As Brian stated, 2.2 being a version higher than 2.1 will have all
  the same expectations placed on it. From what I can see, 1% of users
  know anything about 2.1 so 99% would be wondering why there was a
  jump from 2.0 to 2.2. Do you have anything against 2.1 other than
  fearing that people will expect more from it than it will turn out to
  be?

 It isn't about expectations.

Ok, I misunderstood your previous posts on this topic then.

 I just think it's bad engineering to use the same version prefix for
 two rather different codebases. ... After all, wasn't engineering the reason 
 why we're going to increase the minor?

I don't understand where the conflict comes in between the two. Internally, 
the old 2.1 has been known as HEAD, trunk and now 2.1-experimental. 
Externally, it's been known as 2.1.0_alpha20050718. The set of new features 
available in 2.1.0_alpha20050718 are pretty much all available in current 
trunk as far as I know... You'll need to explain the issue in a little bit 
more detail.

  Really, the bottom line is that regardless of what the response was
  when you asked about portage keywording, if all the arch teams had
  confidence in what we thought 2.0.53 would have been stable a long
  time ago. On the surface the only benefit is extra testing (which has
  already payed off) but it also allows others to take an active hand
  in the quality of portage as well as strengthens the communication
  channels.

 Ok, but it still doesn't really have anything to do with arch teams,
 just general QA.

True, but currently there's no general QA team for coordinating the stability 
of the tree in general. Instead, this has been left up to the individual arch 
teams (which is a little strange/disorganized) so 


 Also I didn't mean to criticize you, just stating that this option exists.

Isn't it your responsibility to? ;)

  I can't tell if you followed what I said in my last email so I'll
  reiterate. Trunk will go into ~arch on Saturday. 2.0.54 will go out
  (also in ~arch) two weeks after that with the two fixes and include
  the cache rewrite based on the opinion of a broad range of users
  (rather than just the noise makers). SHA1 will of course also go in
  based on how it is voted.

 Ehm, what's the point of having .54 in ~arch after trunk is in
 ~arch? You won't get much testing that way as ~arch users would
 already use trunk and stable users likely won't know about .54 ...
 (typical visibility problem)

The visibility problem is exactly why I'm suggesting it be done that way. 
Neither ~arch users nor arch users get needless bumps and testing of trunk 
doesn't get held up at all. .54 would be .53 + selective patches from trunk 
hence all its parts will have had extensive testing. The whole would only 
need a minimal amount of testing by those marking it stable before doing so.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-12-06 Thread Jason Stubbs
On Tuesday 06 December 2005 11:29, Zac Medico wrote:
 Ned Ludd wrote:
  On Mon, 2005-12-05 at 23:06 +0900, Jason Stubbs wrote:
 Okay, new suggestion.
 
 Postpone the cache rewrite from above. Have only the minimal mods
  necessary to fix the PORT_LOGDIR/tee bug. Include the other two as is.
  That would be 2.0.54 as per the attached patch. Get that out soon and
  get trunk out masked at around the same time. As soon as 2.0.54 goes
  stable put trunk into ~arch. However, instead of ~arch meaning
  regression fixes only we could just limit it to minor changes only
  (ie. no big refactorings, rewrites or similar high risk changes) until
  it is time to stable it.
 
  I think it would be wise to reconsider the cache fixes. I know you have
  been away from irc for a while now and have missed the daily events,
  but most of the people we have interacted with are expecting the cache
  updates in .54 (alot of people complaining about the hanging at 50%)
 
  The code has been pretty well tested and seems safe on the surface. I
  think ferringb's testing has shown that the cache updates use about 14M
  of ram where the existing code (as of .52.x) uses about 80M of ram.

 But still, it's annoying to be stuck with only 2 tiers.  Why not put a
 snapshot of trunk in the tree and package.mask it?  Wouldn't that make
 everyone happy?

That's what I'm thinking. Given that the people that are being asked to stable 
2.0.53 are complaining that the ldconfig fix and exec/tee fix aren't in it, 
I'm certain that 2.0.54 would likely go stable *before* 2.0.53 if it is 
pushed out soon. If the SHA1 stuff is dropped, I can pretty much guarantee 
it. And as soon as it goes stable, trunk can be pushed into ~arch.

Doing it this way, the cache rewrite gets into ~arch at pretty much the same 
time but all the other bits get added along with it. Sure it will mean that 
there will be a bit of a delay before it gets into stable but if we can speed 
up the release cycle the delay won't be by very much.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-12-06 Thread Jason Stubbs
On Tuesday 06 December 2005 11:17, Ned Ludd wrote:
 On Mon, 2005-12-05 at 23:06 +0900, Jason Stubbs wrote:
  Okay, new suggestion.
 
  Postpone the cache rewrite from above. Have only the minimal mods
  necessary to fix the PORT_LOGDIR/tee bug. Include the other two as is.
  That would be 2.0.54 as per the attached patch. Get that out soon and get
  trunk out masked at around the same time. As soon as 2.0.54 goes stable
  put trunk into ~arch. However, instead of ~arch meaning regression fixes
  only we could just limit it to minor changes only (ie. no big
  refactorings, rewrites or similar high risk changes) until it is time to
  stable it.

 I think it would be wise to reconsider the cache fixes. I know you have
 been away from irc for a while now and have missed the daily events,
 but most of the people we have interacted with are expecting the cache
 updates in .54 (alot of people complaining about the hanging at 50%)

Call me wrong, but I'm feeling that the constant pulling and pushing on IRC 
causes many misjudgements.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-12-06 Thread Jason Stubbs
On Tuesday 06 December 2005 21:37, Alec Warner wrote:
 Jason Stubbs wrote:
  On Tuesday 06 December 2005 11:17, Ned Ludd wrote:
   On Mon, 2005-12-05 at 23:06 +0900, Jason Stubbs wrote:
Okay, new suggestion.
   
Postpone the cache rewrite from above. Have only the minimal mods
necessary to fix the PORT_LOGDIR/tee bug. Include the other two as is.
That would be 2.0.54 as per the attached patch. Get that out soon and
get trunk out masked at around the same time. As soon as 2.0.54 goes
stable put trunk into ~arch. However, instead of ~arch meaning
regression fixes only we could just limit it to minor changes only
(ie. no big refactorings, rewrites or similar high risk changes) until
it is time to stable it.
  
   I think it would be wise to reconsider the cache fixes. I know you have
   been away from irc for a while now and have missed the daily events,
   but most of the people we have interacted with are expecting the cache
   updates in .54 (alot of people complaining about the hanging at 50%)
 
  Call me wrong, but I'm feeling that the constant pulling and pushing on
  IRC causes many misjudgements.

 No I agree with you here, I just wanted reasoning because now I have
 this ML thread to refer people to :p

Of course, there's enough pushing and pulling on the ML here to provide food 
for thought. :)

Ok, I've come to a conclusion and that conclusion is: We have way too much 
indecisiveness. I'm not sure if that's my fault or not - am I meant to be 
decisive? My guess is that if I were to seriously propose that question 
there'd be a lot of indecision about it. ;)

So I'm going to make a decision and offer until Friday (Saturday in my time) 
for opposers to solidify and state any opposition. If there's no solid 
opposition, Saturday I will put current trunk into ~arch as 2.1_beta20051210. 
I will also post on the 2.0.53 bug that fixes are available for the ldconfig 
bug and the tee bug stating that we'd like to also add trunk's cache 
subsystem to 2.0.54 and that dependening on the next council meeting(?) the 
SHA1 enabling as well. Doing it this way will make the ~arch users happy 
straight away. If we look at it as our responsibility to get fixes and new 
functionality into ~arch then our jobs done and we can get back to business.

As for stable users? If arch teams are willing to selectively choose what 
fixes they want backported to stable (when they're not prepared to move the 
~arch version into stable) things will go much smoother. Of course, it would 
still be our responsibility to get those things backported and assert some 
confidence that it is working. However, once the requested fixes are 
backported all that needs to be done is put out the patched stable version 
with ~arch keywords and then leave it up to the arch teams again. Except for 
the slight extra burden on (which I believe many would actually find to be a 
blessing), it should be a win-win situation.

Cross-posting to -dev@ so that some arch people can comment.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-12-06 Thread Jason Stubbs
On Wednesday 07 December 2005 01:01, Marius Mauch wrote:
 On Tue, 6 Dec 2005 23:19:38 +0900
 Jason Stubbs [EMAIL PROTECTED] wrote:
  If there's no solid opposition, Saturday I will put current trunk into
  ~arch as 2.1_beta20051210.

 Well, I've already stated several times that IMO using a 2.1 branch is
 wrong (use 2.2 instead), but if I'm overvoted, so it shall be.

As Brian stated, 2.2 being a version higher than 2.1 will have all the same 
expectations placed on it. From what I can see, 1% of users know anything 
about 2.1 so 99% would be wondering why there was a jump from 2.0 to 2.2.
Do you have anything against 2.1 other than fearing that people will expect 
more from it than it will turn out to be?

  As for stable users? If arch teams are willing to selectively choose
  what fixes they want backported to stable (when they're not prepared
  to move the ~arch version into stable) things will go much smoother.
  Of course, it would still be our responsibility to get those things
  backported and assert some confidence that it is working. However,
  once the requested fixes are backported all that needs to be done is
  put out the patched stable version with ~arch keywords and then leave
  it up to the arch teams again. Except for the slight extra burden on
  (which I believe many would actually find to be a blessing), it
  should be a win-win situation.

 Just in case you forgot and also for general reference, when I asked
 the arch teams about the portage keywording policy a few months ago
 (wether we should stable even without testing on all archs or to
 delegate that to arch teams) everyone seemed to be happy with the old
 policy, at least nobody voted for a change. As portage doesn't really
 have any arch specific code and a rather short dep list IMO it also
 doesn't yield any real benefit other than more people testing it (which
 is of course always a good thing).

Really, the bottom line is that regardless of what the response was when you 
asked about portage keywording, if all the arch teams had confidence in what 
we thought 2.0.53 would have been stable a long time ago. On the surface the 
only benefit is extra testing (which has already payed off) but it also 
allows others to take an active hand in the quality of portage as well as 
strengthens the communication channels. It's these auxillary benefits as well 
as the benefit of being able to focus on trunk more (which will yield faster 
rollout of features) that make me think it is the best way to go.

On Wednesday 07 December 2005 01:21, Alec Warner wrote:
 I spoke with Brian today ( no clue if he will be sending mail or not )
 but he stressed that he would like the cache rewrite in ~arch.

Any reason why you're speaking on his behalf?

 I would prefer that it be in .54, the code itself is old, 6+ months.  It is 
 a straight backport from the 2.1 branch (the dead one) and the interface 
 code to make it fit with 2.0 is small compared to the patch size ( Brian
 estimated 100-150 lines ).

These are all reasons that I myself have given already.

 I don't have a problem with releasing 2 ebuilds, one with the patch and one 
 without ( or a use flag ) although the question that raises is will the 
 cache rewrite actually end up in .54 final, or will it be put off :)  

This, I do have a problem with. You're essentially suggesting that three lines 
be maintained rather than the current two.

I can't tell if you followed what I said in my last email so I'll reiterate.
Trunk will go into ~arch on Saturday. 2.0.54 will go out (also in ~arch) two 
weeks after that with the two fixes and include the cache rewrite based on 
the opinion of a broad range of users (rather than just the noise makers). 
SHA1 will of course also go in based on how it is voted.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-12-05 Thread Jason Stubbs
On Thursday 01 December 2005 22:28, Jason Stubbs wrote:
 On Monday 28 November 2005 03:49, Marius Mauch wrote:
  Jason Stubbs wrote:
   On Sunday 27 November 2005 00:09, Marius Mauch wrote:
  Jason Stubbs wrote:
  Well, the vote was more for the SHA1 change actually as that's the one
  triggering the size increase. The pycrypto stuff itself doesn't do
  anything really, it would just make the size increase more apparent.
  
   Hmm.. I thought it was for hashes supported by pycrypto being added
   into Manifest before Manifest2 comes along. If it was with regard to
   SHA1 then I take back my vote to delay.
 
  Yeah, I guess the mail could be read both ways.
 
  Don't think so given that offhand I don't even know what getlist() does
   ...
  
   getlist() is defined in emerge and is used to access the system and
   world sets. It wouldn't be too hard to customize it to handle user sets
   and modify other code to support them but the can't combine sets and
   atoms rule would get a bit messy.
 
  So gutting of emerge ... nope, tried that two times already, but gave up
  after hitting too many direct references to system and world.
 
  Oh, btw, two things that are in trunk but weren't listed in your
  original mail:
  - the rewritten versioning code (including the cvs and mult-suffix
  enhancements)
  - finally killing of the stupid masked by -* message
  
   That makes the current list for .54:
  
   * autouse death
   * cache rewrite
   * dyn_install cleanup
   * einfo logging
   * exec cleanup
   * flattened vdb *DEPENDs
   * hash support via pycrypto
   * ldconfig fix
   * metascan/auxget
   * postsync hooks
   * recursive grab*
   * RRDEPEND/LDEPEND
   * sha1 enabling
   * splitdebug
   * vdb empty file culling
  
   Are we about there yet? Also, what does this mean for 2.1/2.2?
 
  Well, if that featurelist is .54 then I really don't see a point for
  making a 2.1 or 2.2 release line. Before your mail starting this thread
  I assumed that .54 would just contain the ldconfig fix, the hash stuff
  and maybe a few other minor things, while trunk would become 2.2.
  But if things like elog, the new cache subsystem, splitdebug or the
  *DEPEND changes don't qualify for a minor version bump, then I can't
  think of anything that would.

 The ldconfig bug and the exec cleanup are the only urgent ones among them.
 The exec cleanup could be postponed and the existing code twiddled in a
 couple of places to fix the logging bug. However, the biggest issue that
 users are complaining about at present is the caching. The biggest issue
 for devs is security. Hence my original list:

 * cache rewrite
 * exec cleanup
 * ldconfig fix
 * sha1 enabling

Okay, new suggestion.

Postpone the cache rewrite from above. Have only the minimal mods necessary to 
fix the PORT_LOGDIR/tee bug. Include the other two as is. That would be 
2.0.54 as per the attached patch. Get that out soon and get trunk out masked 
at around the same time. As soon as 2.0.54 goes stable put trunk into ~arch. 
However, instead of ~arch meaning regression fixes only we could just limit 
it to minor changes only (ie. no big refactorings, rewrites or similar high 
risk changes) until it is time to stable it.

However, with the trunk's target (2.1?) rather than letting the arch teams 
decide when we call it stable, I think it would be a better idea to move it 
to the .0 version when we feel it is ready leaving it in ~arch. As 
regressions are fixed the .0 can be bumped to .1, .2 or whatever, but the 
focus can move to what happens beyond that rather than waiting...

First paragraph is more important right now.

--
Jason Stubbs
diff -uNr portage-2.0.53/pym/portage.py portage-2.0.54/pym/portage.py
--- portage-2.0.53/pym/portage.py	2005-12-01 22:04:17.0 +0900
+++ portage-2.0.54/pym/portage.py	2005-12-05 22:54:53.0 +0900
@@ -2039,11 +2039,6 @@
 			myline +=  +mysum
 			myline +=  +myarchive
 			myline +=  +str(mysize)
-			if sumName != MD5:
-#  This cannot be used!
-# Older portage make very dumb assumptions about the formats.
-# We need a lead-in period before we break everything.
-continue
 			mylines.append(myline)
 	return mylines
 
@@ -6430,6 +6425,9 @@
 writemsg(!!! FAILED postrm: +str(a)+\n)
 sys.exit(123)
 
+		#update environment settings, library paths. Change symlinks.
+		env_update(makelinks=1)
+		
 		self.unlockdb()
 
 	def isowner(self,filename,destroot):
@@ -6672,13 +6670,10 @@
 			writemsg(!!! FAILED postinst: +str(a)+\n)
 			sys.exit(123)
 
-		downgrade = False
-		for v in otherversions:
-			if pkgcmp(catpkgsplit(self.pkg)[1:], catpkgsplit(v)[1:])  0:
-downgrade = True
-
 		#update environment settings, library paths. DO NOT change symlinks.
-		env_update(makelinks=(not downgrade))
+		#only needed if we did not already run unmerge
+		if not (oldcontents):
+			env_update(makelinks=0)
 		#dircache may break autoclean because it remembers the -MERGING-pkg file
 		global dircache

Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-12-05 Thread Jason Stubbs
On Tuesday 06 December 2005 00:21, Alec Warner wrote:
 Jason Stubbs wrote:
  Okay, new suggestion.
 
  Postpone the cache rewrite from above. Have only the minimal mods
  necessary to fix the PORT_LOGDIR/tee bug. Include the other two as is.
  That would be 2.0.54 as per the attached patch. Get that out soon and get
  trunk out masked at around the same time. As soon as 2.0.54 goes stable
  put trunk into ~arch. However, instead of ~arch meaning regression fixes
  only we could just limit it to minor changes only (ie. no big
  refactorings, rewrites or similar high risk changes) until it is time to
  stable it.

 Postponing the cache rewrite is going to piss a lot of poeple off, just
 FYI :)  I realize it's a large patch, but it has been tested by plenty
 of people, and many of them are waiting for this fix to hit stable
 (don't want to patch portage on a production system).  Any particular
 reason you want it held off (besides it's immensity?)

The delay it would cause for other things waiting in the pipeline.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] VDB double reading

2005-12-03 Thread Jason Stubbs
On Sunday 04 December 2005 11:08, Zac Medico wrote:
 Ned Ludd wrote:
  With the size of the vdb effecting the speed of portage itself and with
  all the desktop splitting up packages into massively large sets of many
  ebuilds it's becoming apparent that that some optimizations need to be
  done.
 
  strace -o woof -eopen -f python -c 'import portage'
  grep '/var/db/pkg/sys-apps/portage-' woof
 
  You will notice that on a simple import that it's reading everything
  twice. If anybody is up for the challenge of addressing this double
  reading I think the entire user community would be thankful
  (or atleast I would be)

 sandboxshell
 adddeny /var/db/pkg
 python -c 'import portage'

 From the resulting traceback(s), I was able to see that the /var/db/pkg
 reads originate in calls to a getvirtuals() method, which ultimately leads
 to a get_all_provides() method.  Assuming that the results from
 get_all_provides() do not change during the course of the initial 'import
 portage', it should be safe to cache the results (until the end of the
 import).

Following that through a bit further, get_all_provides() is only called by 
loadVirtuals() which is only called by config.__init__(). getvirtuals() 
itself is called outside of config.__init() once during initialization as 
well as by the scripts pkgmerge and repoman. In all three cases, the result 
is passed to portagetree.__init__(), binarytree.__init() or 
vartree.__init__(). However, none of those three classes use the passed 
virtuals anymore. Hence, all calls to getvirtuals() external to the config 
class could theoretically be dropped and replaced with an empty dictionary. 
The only possibility of breakage in doing that is if other parts of portage 
are accessing *tree's virtual member directly.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-12-01 Thread Jason Stubbs
On Monday 28 November 2005 03:49, Marius Mauch wrote:
 Jason Stubbs wrote:
  On Sunday 27 November 2005 00:09, Marius Mauch wrote:
 Jason Stubbs wrote:
 Well, the vote was more for the SHA1 change actually as that's the one
 triggering the size increase. The pycrypto stuff itself doesn't do
 anything really, it would just make the size increase more apparent.
 
  Hmm.. I thought it was for hashes supported by pycrypto being added into
  Manifest before Manifest2 comes along. If it was with regard to SHA1 then
  I take back my vote to delay.

 Yeah, I guess the mail could be read both ways.

 Don't think so given that offhand I don't even know what getlist() does
  ...
 
  getlist() is defined in emerge and is used to access the system and world
  sets. It wouldn't be too hard to customize it to handle user sets and
  modify other code to support them but the can't combine sets and atoms
  rule would get a bit messy.

 So gutting of emerge ... nope, tried that two times already, but gave up
 after hitting too many direct references to system and world.

 Oh, btw, two things that are in trunk but weren't listed in your
 original mail:
 - the rewritten versioning code (including the cvs and mult-suffix
 enhancements)
 - finally killing of the stupid masked by -* message
 
  That makes the current list for .54:
 
  * autouse death
  * cache rewrite
  * dyn_install cleanup
  * einfo logging
  * exec cleanup
  * flattened vdb *DEPENDs
  * hash support via pycrypto
  * ldconfig fix
  * metascan/auxget
  * postsync hooks
  * recursive grab*
  * RRDEPEND/LDEPEND
  * sha1 enabling
  * splitdebug
  * vdb empty file culling
 
  Are we about there yet? Also, what does this mean for 2.1/2.2?

 Well, if that featurelist is .54 then I really don't see a point for
 making a 2.1 or 2.2 release line. Before your mail starting this thread
 I assumed that .54 would just contain the ldconfig fix, the hash stuff
 and maybe a few other minor things, while trunk would become 2.2.
 But if things like elog, the new cache subsystem, splitdebug or the
 *DEPEND changes don't qualify for a minor version bump, then I can't
 think of anything that would.

The ldconfig bug and the exec cleanup are the only urgent ones among them. The 
exec cleanup could be postponed and the existing code twiddled in a couple of 
places to fix the logging bug. However, the biggest issue that users are 
complaining about at present is the caching. The biggest issue for devs is 
security. Hence my original list:

* cache rewrite
* exec cleanup
* ldconfig fix
* sha1 enabling

The cache rewrite has existing for many months and has gone through a fair bit 
of testing so I personally don't have any issues with it going into stable. 
As I said, the exec cleanup could be postponed and the existing code twiddled 
only where necessary...

This is more what I was hoping for 2.0.54. It should really follow hot on the 
heals of 2.0.53 and be in stable hopefully by the end of the year. If 
everybody's happy with that, the remainder is:

* autouse death
* dyn_install cleanup
* einfo logging
* exec cleanup
* flattened vdb *DEPENDs
* hash support via pycrypto
* metascan/auxget
* postsync hooks
* recursive grab*
* RRDEPEND/LDEPEND
* splitdebug
* vdb empty file culling

When I said shortterm goals in the original email, I was actually referring 
to after 2.0.54. ;)

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-11-27 Thread Jason Stubbs
On Sunday 27 November 2005 02:03, Ned Ludd wrote:
 On Sat, 2005-11-26 at 13:15 +0900, Jason Stubbs wrote:
  On Saturday 26 November 2005 02:05, Ned Ludd wrote:
   On Sat, 2005-11-26 at 00:51 +0900, Jason Stubbs wrote:
On Saturday 26 November 2005 00:31, Ned Ludd wrote:
 * post_sync action hook (.53/.54 )
 * VDB prevention of single byte NULL entries being created. ( .54 )
   
Doable for .54.

 

   Yeah and from the sounds of it we may want more than 1 set of postsync
   hooks or the addition of a postsync.d/
   (dev thread on getting vital info to users)
 
  Heh.. that was my suggestion. ;)

 Yeah it seems doable for .53 but if you want to wait till .54 or
 a .53-rXX thats fine. I'd prefer to see it in .53 unless
 that's going out the door today.

2.0.53 is pretty much closed. After it hits 2.0.53, the only -r releases will 
be for major regressions and security fixes. The only reason I think killing 
of CDEPEND for 2.0.53 is okay is because nothing uses CDEPEND (I killed off 
emerge scanning it already) so there's nothing to regress.

Should generating internal
debug info still be offered?
  
   When FEATURES=nostrip is enabled we should not split off
   any debug info or touch the executable.
 
  FEATURES=splitdebug|stripdebug and do nothing if neither is set?

 If feature based it seems logical to me to have it as either
 splitdebug || nosplitdebug

That would be splitdebug or -splitdebug. I was suggesting stripdebug as a 
replacement for -nostrip to get rid of another no* option and so that portage 
could still be configured to give the current behaviour. As far as I can 
tell, there are three possibilities:

* Do nothing
* Split of the debug info
* Strip any debug info

Have I got that right?

 I know some people hate no* functions or rather they hate no* USE
 flags. But it would seem fitting if we decided to make it a default vs
 sticking in splitdebug in all profiles.

There's no need to touch the profiles. /etc/make.globals defines portage 
defaults and is controlled by portage.

 * flattened vdb {P,R,}DEPEND (.54 )
   
I might be wrong but I can't really see this being done cleanly. At
best, only USE flags could be gotten rid of which would still leave
|| () constructs. This leads me to question of what use it would
really be. If it can only do a half-arsed job and in a messy way at
that I'd personally prefer it to be done later on.
  
   It will indeed still leave you with || ( foo bar ) or =cpv which you
   can be parsed just fine. Yeah it would be nice if it could be reduced
   more but later on or now I don't see how it can be reduced anymore than
   to the requirements that the ebuild requested.
 
  Again, if it can be done cleanly code-wise no issues with resolving the
  USE flags out.

 Should only be a few lines of code. (I hope)
 Something like hand off the env varable from dyn_compile() in ebuild.sh
 to python and python passes it back to ebuild.sh in the next phase for
 dyn_install() which gets recorded flattened

There's no passing back of variables at the moment except through temporary 
files. Perhaps USE and *DEPEND could be read in and *DEPEND rewritten out 
after dyn_compile() completes?

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [PATCH] CDEPEND removal

2005-11-27 Thread Jason Stubbs
On Monday 28 November 2005 00:01, Ned Ludd wrote:
 The following untested attached patch removes CDEPEND from ebuild.sh
 A quick grep -i shows that there is one case of CDEPEND left in
 pym/portage.py after applying the patch. It's in the auxdbkeys around
 line 5130. Not sure if that is safe to remove or not.
 The cache entry for CDEP has been replaced with a blank line to keep
 the cache format compatible.

The CDEPEND in auxdbkeys is needed, but it won't cause any harm other than the 
extra 10 or 11 bytes added to portage.py. This patch is pretty much perfect 
as it is. :)

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [PATCH] CDEPEND removal

2005-11-27 Thread Jason Stubbs
On Monday 28 November 2005 00:20, Ned Ludd wrote:
 On Mon, 2005-11-28 at 00:15 +0900, Jason Stubbs wrote:
  On Monday 28 November 2005 00:01, Ned Ludd wrote:
   The following untested attached patch removes CDEPEND from ebuild.sh
   A quick grep -i shows that there is one case of CDEPEND left in
   pym/portage.py after applying the patch. It's in the auxdbkeys around
   line 5130. Not sure if that is safe to remove or not.
   The cache entry for CDEP has been replaced with a blank line to keep
   the cache format compatible.
 
  The CDEPEND in auxdbkeys is needed, but it won't cause any harm other
  than the extra 10 or 11 bytes added to portage.py. This patch is pretty
  much perfect as it is. :)

 Great.
 So it's all in your hands now then I can assume and I don't need to file
 a bug? ;-)

Committed.

 Anything else pending before .53 hits stable?

I don't believe so. I'll give it until Wednesday 12:00 UTC and put it out then 
if nothing comes up. I don't expect anything to, but I'm having a couple of 
whiskeys atm before bed and will be heading out to a drinking establishment 
tomorrow... ;)

 You know I can be somewhat impatient and I'm ready to get started on
 making patches for .54

As are we all. However, the wait seems to be doing good. While trunk is has 
been reopened for a while, the only things that have gone in have been in 
testing outside of trunk for a while which has improved quality greatly. 

Given people's expectations of what 2.0.54 should be I'm not sure of what will 
happen with 2.1/2.2. However, given the quality that trunk currently has it 
might be best to split off a branch and lock down the changes in 2.0.54 as 
soon as the first _pre is released. As far as I know, everything I listed in 
my last email to the .53, .54.. thread have existing tested patches so we 
can probably get a 2.0.54 up to where 2.0.53 is now in half the time...

--
Jason Stubbs

-- 
gentoo-portage-dev@gentoo.org mailing list



[gentoo-portage-dev] .53, .54 and beyond...

2005-11-25 Thread Jason Stubbs
Hi all,

I don't think there's really anything else that can be done for 2.0.53 so am 
thinking that we should probably push _rc7 + docs out and let the arch teams 
mark it stable when they're ready (or stick with 2.0.51.22-r3 if it pleaseth 
them).

We should put out a 2.0.54_pre1 out soon after that. What I'm wondering in is 
what will be going in? So far I'm thinking along the lines of:

* cache rewrite
* exec cleanup
* ldconfig fix
* sha1 enabling

The only other new thing in trunk that I know of is logging but there's still 
a question mark over the ordering of messages... Can that be resolved soon? 
Anything else missing? Any reasons against any of the above?

What's on the map for after that? There's a few things listed on the new 
(still unreleased?) project index and I'm looking to get the dependency stuff 
refactored and moved out of emerge.. What are the shortterm goals?

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-11-25 Thread Jason Stubbs
On Saturday 26 November 2005 00:31, Ned Ludd wrote:
 On Sat, 2005-11-26 at 00:01 +0900, Jason Stubbs wrote:
  Hi all,
 
  I don't think there's really anything else that can be done for 2.0.53 so
  am thinking that we should probably push _rc7 + docs out and let the arch
  teams mark it stable when they're ready (or stick with 2.0.51.22-r3 if it
  pleaseth them).

 [snip]

  There's a few things listed on the new
  (still unreleased?) project index and I'm looking to get the dependency
  stuff refactored and moved out of emerge.. What are the shortterm goals?

 For me my short term goals are to see these things happen

 * pax-utils depends ( .53 )
 * seeing CDEPEND stop being created for the VDB ( .53 )

Definitely doable.

 * post_sync action hook (.53/.54 )
 * VDB prevention of single byte NULL entries being created. ( .54 )

Doable for .54.

 * new prepstrip offering splitdebug ( .53/.54 )

Need to work out exactly what will be offered when on this one, but at the 
earliest it will be .54. Perhaps go with your patch for .54 and leave 
Olivier's fancy bits for later? There are a few other questions too... Should 
the default be to generate external debug info? Should generating internal 
debug info still be offered? What platforms is it supported on?..

 * misc cleanups of dyn_install (.54 )

Need more info.

 * flattened vdb {P,R,}DEPEND (.54 )

I might be wrong but I can't really see this being done cleanly. At best, only 
USE flags could be gotten rid of which would still leave || () constructs. 
This leads me to question of what use it would really be. If it can only do a 
half-arsed job and in a messy way at that I'd personally prefer it to be done 
later on.

 * introduction of RRDEPEND to the VDB ( .54 )

What is this again?

 --
 Ned Ludd [EMAIL PROTECTED]
 Gentoo Linux
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-11-25 Thread Jason Stubbs
On Saturday 26 November 2005 11:07, Marius Mauch wrote:
 On Sat, 26 Nov 2005 00:01:15 +0900
 Jason Stubbs [EMAIL PROTECTED] wrote:
  The only other new thing in trunk that I know of is logging but
  there's still a question mark over the ordering of messages... Can
  that be resolved soon? Anything else missing? Any reasons against any
  of the above?

 Resolved how? I'm not really sure I understood the original problem
 (other than listdir being underdeterministic in theory).

TGL suggested that all the messages go into a single file with some sort of 
prefix that would then be parsed on the python side. The original order of 
messages could then be maintained. However, seeing as there needs to be no 
compatibility with the temporary files it could wait for later anyway.

  What's on the map for after that? There's a few things listed on the
  new (still unreleased?) project index and I'm looking to get the
  dependency stuff refactored and moved out of emerge.. What are the
  shortterm goals?

 - the pycrypto hash additions (for .54)

This is only useful if the vote goes in favour of adding further hash types to 
Manfiest, right? If the vote goes that way I've got no issues with it, but if 
it doesn't it would essentially be dead code.

 - Manifest2 verification support (need the GLEP first so the format is
 sanctioned). Technically it's complete, just generation is still
 unfinished. (so a maybe for .54 depending on the timeframe)

Again, depends on -dev.

 - Killing off auto-use+USE_ORDER?

Yep, I'd really like to see this one gone too. We should probably state the 
intention on -dev first though as there might be a lot of people against it.

 - the recursive grab* functions I just posted (for .54)

Needs a small amount of work (/etc/portage/package.mask/foo/bar would break 
it) but I like the general idea.

 - addition of auxget/metascan tools (could be for .54)

No qualms.

 - integration of set modules, either as emerge targets (requires
 serious gutting of emerge) or a first-class atoms (semantically
 tricky, no clue about implementation yet)

I'm working on this with my refactoring. Defininately a post-.54 thing unless 
you want to quickly hack it into getlist()?

--
Jason Stubbs

-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-11-25 Thread Jason Stubbs
On Saturday 26 November 2005 02:05, Ned Ludd wrote:
 On Sat, 2005-11-26 at 00:51 +0900, Jason Stubbs wrote:
  On Saturday 26 November 2005 00:31, Ned Ludd wrote:
   * post_sync action hook (.53/.54 )
   * VDB prevention of single byte NULL entries being created. ( .54 )
 
  Doable for .54.

 Yeah and from the sounds of it we may want more than 1 set of postsync
 hooks or the addition of a postsync.d/
 (dev thread on getting vital info to users)

Heh.. that was my suggestion. ;)

   * new prepstrip offering splitdebug ( .53/.54 )
 
  Need to work out exactly what will be offered when on this one, but at
  the earliest it will be .54. Perhaps go with your patch for .54 and leave
  Olivier's fancy bits for later?

 I can only assure you the code I wrote will function properly.
 So that's the only thing I'm trying/willing to push myself.
 As long as he has those [ -x checks ] his code should be harmless,
 but I don't see the advantage in it over building with -g3.

  There are a few other questions too... Should
  the default be to generate external debug info?

 I think the security team would say yes they want it by default and
 would be willing (taviso) to write a proper debug-HOWTO.xml for gentoo.
 I think ferringb would say make it FEATURES=splitdebug if
 it's going in midstream.

 It does add some size which would make peoples $ROOT's a little bit
 bigger. But from mine and other peoples testing it's pretty damn
 minimal. I think Halcy0n @ gentoo said after doing an -e world he ended
 up with only 18M of split debug info

 I'm also fond of split packaging of debug info also (but I'm not going
 to push for that till I find a more elegant way)

 [EMAIL PROTECTED] debug $ qsize pax-utils
 app-misc/pax-utils-debug-0.1.4-r0: 3 files, 5 non-files, 16.27 KB
 app-misc/pax-utils-0.1.4: 6 files, 6 non-files, 102.485 KB

 Perhaps we should get input from gentoo-dev ml ?

Sounds good for pretty much all aspects of split debug (other than the 
capability itself).

  Should generating internal
  debug info still be offered?

 When FEATURES=nostrip is enabled we should not split off
 any debug info or touch the executable.

FEATURES=splitdebug|stripdebug and do nothing if neither is set?

   * misc cleanups of dyn_install (.54 )
 
  Need more info.

 This is just something I want todo for my own sanity,
 ie break parts of our existing dyn_install() out
 into /usr/lib/portage/bin/ scripts.
 The current function is about 209 lines of code and I
 can see it growing even more.

Code cleanups are always good. :)

   * flattened vdb {P,R,}DEPEND (.54 )
 
  I might be wrong but I can't really see this being done cleanly. At best,
  only USE flags could be gotten rid of which would still leave || ()
  constructs. This leads me to question of what use it would really be. If
  it can only do a half-arsed job and in a messy way at that I'd personally
  prefer it to be done later on.

 It will indeed still leave you with || ( foo bar ) or =cpv which you
 can be parsed just fine. Yeah it would be nice if it could be reduced
 more but later on or now I don't see how it can be reduced anymore than
 to the requirements that the ebuild requested.

Again, if it can be done cleanly code-wise no issues with resolving the USE 
flags out.

 One big advantage for me here is that virtuals would be resolved.

Resolving virtuals can't be done. Virtuals are meant to be binary compatible 
with each other which means that they can be replaced by each other 
interchangeably. However, once portage starts using info from vdb (soon) and 
starts doing proper sanity checking (not so soon) one would need to re-emerge 
all reverse dependencies in order to switch installed virtual provider.

   * introduction of RRDEPEND to the VDB ( .54 )
 
  What is this again?

 Ok I talked a little bit about this on this list the other day and a few
 months ago with you on #-portage.

 man
  RRDEPEND
   This entry is automatically created by portage. It contains a
   list of reverse dependencies that is achieved by resolving the
   DT_NEEDED entries of an ELF executable.
 /man


 Justification

 Programs such as revdep-rebuild, verify-rdepend would be able to make
 immediate use.

Yes, but an extension to emaint will be needed to create the files for 
existing installed packages. I'm not sure that Brian's plugin architecture is 
ready for .54 so it would need to be another hardcoded check/fix into emaint 
itself. Not sure I really like that idea but I'm not going to fight if the 
majority think that it should go into .54... What do the majority think?

 A little bit of a longer term goal is to see portage gain 
 the ability to request to only use RRDEPEND entries to be used for
 depgraph creation for use with embedded/mimimal systems.

 ROOT=/dev/shm/minimal emerge -KO --deps=RRDEPEND pkgfoo

 RRDEPEND will need to exist due to the RDEPEND explosion and lack of a
 clear definition when it was first introduced to portage.
 The advantage from where I'm sitting

Re: [gentoo-portage-dev] Making pax-utils a depend

2005-11-19 Thread Jason Stubbs
On Sunday 20 November 2005 12:46, Brian Harring wrote:
 On Sat, Nov 19, 2005 at 10:39:29PM -0500, Ned Ludd wrote:
  On Fri, 2005-11-18 at 10:16 -0600, Brian Harring wrote:
  [snip]
 
   Still have issues with it- revdep-rebuild for example can't rely on
   NEEDED until a repository regen script is written.
 
  # How about something like this then?
 
  cd ${ROOT}/var/db/pkg || exit 1
 
  for f in *-*/*/CONTENTS ; do
  NEEDED=${f%/*}/NEEDED
  if [[ ! -e $NEEDED ]]; then
  grep ^obj $f | awk '{print $2}' | scanelf -BF%F %n -f -  
  $NEEDED
  fi
  done

 Aside from my inherint dislike of awk, algo is there.
 Should be schlopped into emaint though imo; you game, or want someone
 to do the python bits?

emaint needs the plugin stuff for that to happen, but yeah...

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Plugin backport PATCH (1/2)/(2/2)

2005-11-18 Thread Jason Stubbs
... That
should only be used by register and deregister, right? Perhaps breaking
the unlocked version into a protected method?

+   # designed this way to minimize lock holding time.
+   if plugin_type != None:
+   ptypes = [plugin_type + PLUGINS_EXTENSION]
+   d = {}
+   if locking:
+   try:
+   plug_lock = FsLock(plugins_dir)
+   except NonExistant:
+   return {}
+   plug_lock.acquire_read_lock()
+   try:
+   if plugin_type == None:
+   ptypes = [x for x in os.listdir(plugins_dir) if 
x.endswith(PLUGINS_EXTENSION) ]

The plugin_type if/else could be combined as ptypes is not used before this.
Is [x for x in list] 2.2 safe by the way?

+   len_exten = len(PLUGINS_EXTENSION)
+   for x in ptypes:
+   c = RawConfigParser()
+   c.read(os.path.join(plugins_dir, 
x.lstrip(os.path.sep)))
+   if raw:
+   d[x[:-len_exten]] = c
+   else:
+   d[x[:-len_exten]] = dict([(y, 
dict(c.items(y))) for y in c.sections()])

Same here on the 2.2 bit... Although that's pretty hard to read either way.

+   finally:
+   if locking:
+   plug_lock.release_read_lock()
+   if plugin_type != None:
+   return d[plugin_type]
+   return d

Not sure about having different return types...

+registry = None
+from portage.util.currying import pre_curry, pretty_docs
+
+def proxy_it(method, *a, **kw):
+   global registry
+   if registry == None:
+   registry = GlobalPluginRegistry()
+   return getattr(registry, method)(*a, **kw)
+
+for x in [register, deregister, query_plugins, get_plugin]:
+   v = pre_curry(proxy_it, x)
+   doc = getattr(GlobalPluginRegistry, x).__doc__
+   if doc == None:
+   doc = ''
+   else:
+   # do this so indentation on pydoc __doc__ is sane
+   doc = \n.join(map(lambda x:x.lstrip(), doc.split(\n))) +\n
+   doc += proxied call to module level registry instances %s method % x
+   globals()[x] = pretty_docs(v, doc)
+
+del x, v, proxy_it, doc

What's this currying and pretty_docs stuff?

--
Jason Stubbs

-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Making pax-utils a depend

2005-11-18 Thread Jason Stubbs
On Wednesday 16 November 2005 04:41, solar wrote:
 For those of you that do not know we Mike Frysinger and myself have
 written a general purpose ELF Q/A tool called pax-utils

 The tool itself can be used to preform a number of tasks. What ferringb
 would like todo is take advantage of this tool for the an updated
 verify-rdepend code.

snip

 Before he does that does anybody have any objections? If so please let
 us know what and why.

The last I heard, Brian was the only one *against* having pax-utils added
to portage's RDEPEND list. ;)

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Plugin framework

2005-11-14 Thread Jason Stubbs
On Monday 14 November 2005 23:17, Marius Mauch wrote:
 On Mon, 14 Nov 2005 22:38:28 +0900

 Jason Stubbs [EMAIL PROTECTED] wrote:
  The cache and elog plugin selection(s) come from user settings but
  emaint (and repoman whenever that happens (and possibly even emerge
  itself one day?)) needs to automatically detect what's available and
  use it.

 The last part I consider questionable, while they shouldn't come from
 the user config directly I'd be very careful with a use everything
 possible policy. Not saying that it's flat wrong or that I have a
 better plan right now, but having this as a primary design goal seems
 wrong.

That's why there's the issubclass() check. That guarantees that modules found 
in a certain path (of a certain plugin type) provide a prespecified API. 
Utilizing that API, it's then possible to inspect the plugin in any way 
necessary. My goal at this level is just to provide an easy way to enumerate 
what plugins are available and have some assurance that a certain API is 
available.

If you're talking with regard to emaint specifically. The goal is to have 
plugins detected at runtime and available as extra targets beyond the current 
world. That would allow things such as revdep-rebuild to be integrated 
without the need for special handling on the emaint side.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] going to need a 2.0.53-rc8

2005-11-14 Thread Jason Stubbs
On Monday 14 November 2005 00:46, Jason Stubbs wrote:
 On Sunday 13 November 2005 11:52, Brian Harring wrote:
  On Sun, Nov 13, 2005 at 09:19:55AM +0900, Jason Stubbs wrote:
   On Sunday 13 November 2005 04:00, Brian Harring wrote:
*cough* that's that funky _p1 you're using there? :)
  
   patchlevel... I think it gives a stronger impression that 2.0.53 is
   distinct from 2.0.54. Is distinct the right word? I mean that it kind
   of shows that 2.0.53 is done but there was something that needed to be
   fixed quickly.
 
  2.0.53.1 vs 2.0.53_p1 vs 2.0.53.p1 ... either of the three indicates
  2.0.53 had minor fix tagged onto the base 2.0.53 release...
 
   Given
   portage's history of using lots of dots, 2.0.53.1 doesn't have as much
   impact. Is the *cough* a complaint of sorts?
 
  Well, knowing what you mean by pN, I'm just going to gesture wildly at
  my earlier email of lets fix the whacked out versioning now. ;)

 So then my suggested 2.0.53_p1 should be 2.0.54 and what is currently
 referred to as 2.0.54 should be 2.1.0?

Any thoughts on this? If we use 2.0.54 for the fix for this, that can go into 
~arch before 2.0.53_pre7 goes to .53 and arch without versioning getting 
screwed up. I'm still pretty sure 2.0.53 will be stable (at least on some 
arch) in under 48 hours and the fix for this should really go out at the same 
time or before...

--
Jason Stubbs

-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] going to need a 2.0.53-rc8

2005-11-14 Thread Jason Stubbs
On Tuesday 15 November 2005 00:32, Marius Mauch wrote:
 On Tue, 15 Nov 2005 00:24:02 +0900

 Jason Stubbs [EMAIL PROTECTED] wrote:
  On Monday 14 November 2005 00:46, Jason Stubbs wrote:
   On Sunday 13 November 2005 11:52, Brian Harring wrote:
On Sun, Nov 13, 2005 at 09:19:55AM +0900, Jason Stubbs wrote:
 On Sunday 13 November 2005 04:00, Brian Harring wrote:
  *cough* that's that funky _p1 you're using there? :)

 patchlevel... I think it gives a stronger impression that
 2.0.53 is distinct from 2.0.54. Is distinct the right word? I
 mean that it kind of shows that 2.0.53 is done but there was
 something that needed to be fixed quickly.
   
2.0.53.1 vs 2.0.53_p1 vs 2.0.53.p1 ... either of the three
indicates 2.0.53 had minor fix tagged onto the base 2.0.53
release...
   
 Given
 portage's history of using lots of dots, 2.0.53.1 doesn't have
 as much impact. Is the *cough* a complaint of sorts?
   
Well, knowing what you mean by pN, I'm just going to gesture
wildly at my earlier email of lets fix the whacked out
versioning now. ;)
  
   So then my suggested 2.0.53_p1 should be 2.0.54 and what is
   currently referred to as 2.0.54 should be 2.1.0?
 
  Any thoughts on this? If we use 2.0.54 for the fix for this, that can
  go into ~arch before 2.0.53_pre7 goes to .53 and arch without
  versioning getting screwed up. I'm still pretty sure 2.0.53 will be
  stable (at least on some arch) in under 48 hours and the fix for this
  should really go out at the same time or before...

 Replace 2.1.0 with 2.2.0 and I'll agree.

Brian? Others?

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] going to need a 2.0.53-rc8

2005-11-11 Thread Jason Stubbs
On Friday 11 November 2005 22:53, Brian Harring wrote:
 Hola,

 Short version is that via bug 112082 plus glibc upstream doing
 something stupid, a lovely bug has reared it's head.  Basically,
 users upgrading from glibc-2.3.5.200*.ebuild have libc.so-2.3.90.so,
 and glibc-2.3.6.ebuild has libc.so-2.3.6.so .

 ldconfig views 2.3.90 as greater then 2.3.6; during merge, portage
 views 2.3.5.200* - 2.3.6 as an upgrade, and triggers an ldconfig
 call.  Said call, and said upstream weird lib versioning results in
 ld.so.6 being reset from libc-2.3.6.so to libc-2.3.90.so , which
 obviously gets a bit screwed up when unmerge comes around and yanks
 the lib.

 Az split off a patch for it (after lots of fun digging) to correct it;
 we're going to need a rc8 covering this one offhand, since it's
 invalid linking in certain cases.

 Thoughts/complaints/issues/further testing?

Short answer: No.

Long answer: This is not a regression; you'll find the same problem in stable. 
It's an area where a slight error can break lots of things that are even 
slightly non-standard. This case where the bug is occurring is very 
non-standard and has not cropped up before (or at least hasn't been brought 
to anybody's attention) in the time since I (and you) have been with the 
project. Lastly, 2.3.5.200* is/was hard masked.

My preference would be to put the patch into trunk and release .54_pre1 within 
the next 24 hours.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [ferri...@lark.gentoo.org: r2267 - main/trunk/bin] (pre/post hooks)

2005-11-09 Thread Jason Stubbs
On Thursday 10 November 2005 05:08, Brian Harring wrote:
 If people are after having the commit mail dumped in their mbox,
 please contact either jstubbs, genone, or myself.

Or if you're a dev, just add yourself. ;)

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] release versioning meaning

2005-11-08 Thread Jason Stubbs
On Tuesday 08 November 2005 17:29, Brian Harring wrote:
 On Sun, Nov 06, 2005 at 11:54:11AM +0900, Jason Stubbs wrote:
  On Sunday 06 November 2005 06:09, Brian Harring wrote:
   Yes we'll run aground of the dead 2.1 release (not incredibly happy
   about that), but I'd like to see if we can get bug fixes out a bit
   quicker, with some semblence of a gurantee we're not tagging in stuff
   an admin isn't going to care about.
 
  What sort of bug fixes are you looking to get out quicker? While the EAPI
  stuff drew out .53 a little longer than originally expected it was still
  only 30 days from first rc to final (assuming _rc7 is final). I can't
  really see the necessity for getting non-regression fixes out quicker.
  At the moment, a lot of fixes go out all at once rather than in lots of
  small bumps. I doubt the overall speed would change very much.

 Question is how will it scale for non-bugfixes, disruptive changes
 like cache backport, elog backporting, confcache, etc?  What I'm
 concerned about is what's going to occur with .5x when large changes
 start sliding into it (or into a minor)- basically the territory we're
 wandering into right now with cache/exec refactoring for .54.

If the changes are reviewed roughly in proportion to the number of hunks, we 
should be okay. At minimum, we should at least see how .54 turns out as there 
will be a few major changes in there already. I kind of expect .54 to do 
better than .53 actually.

Remember also that if we go the 2.1.x route, 2.0.x bugfixes can be pushed out 
quicker but the fixes all have to be committed twice. I for one am terrible 
at that. ;)

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] release versioning meaning

2005-11-08 Thread Jason Stubbs
On Tuesday 08 November 2005 19:42, Brian Harring wrote:
 On Tue, Nov 08, 2005 at 07:39:01PM +0900, Jason Stubbs wrote:
  If the changes are reviewed roughly in proportion to the number of hunks,
  we should be okay. At minimum, we should at least see how .54 turns out
  as there will be a few major changes in there already. I kind of expect
  .54 to do better than .53 actually.

 roughly in proportion ?

Lot of sparse changes should get more review, or something to that affect...

 Reminds me... portage-commits.  Alias right now sort of does the job,
 but I want it to be an ml so we don't have to manage adding new people
 in- what was the end result infra wise for that?

The initial bug was to request an alias like gentoo-x86-commits. Not sure if 
that's available to the general public or not, but it's likely easiest to 
just maintain it manually for those that request it.

  Remember also that if we go the 2.1.x route, 2.0.x bugfixes can be pushed
  out quicker but the fixes all have to be committed twice. I for one am
  terrible at that. ;)

 That's why we're using svn, and why merge rules. ;)

http://svnbook.red-bean.com/en/1.0/ch04s03.html

This has given me an idea of what you are talking about but, by the sounds of 
it, it still won't be a walk in the park. I guess if we only merge changes 
from the stable branch into trunk whenever a release is made from stable it 
shouln't be to hard to keep track of though.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [Bug 44796] Per package environment variables

2005-11-04 Thread Jason Stubbs
On Friday 04 November 2005 04:30, Thomas de Grenier de Latour wrote:
 On Fri, 4 Nov 2005 01:19:35 +0900

 Jason Stubbs [EMAIL PROTECTED] wrote:
  package.env would be a list of atom file [file ...]

 ...

  With a couple of small modifications to emerge to check FEATURES
  for buildpkg after the call to setcpv() is done rather than
  doing it once globally, this would also cover TGL's BUILD_PKGS
  addition too.

 Not if env files are only selectable by strict depatoms. I mean,
 sure this syntax is perfect for *DEPEND strings, and is fine for
 package.{use,mask,unmask} (although the randomness of
 best_match_to_list() is rather annoying). But for package.keywords,
 it is already sub-optimal imho (i run ~arch so i don't care
 much, but for instance i often see people on forums who list
 some whole categories there), and it would be too for the generic
 package.env.  For instance, people who develop gnome stuffs might
 want to use the debug env for gnome-*/* packages.  As for
 the buildpkg FEATURES flag, it would be a real pita if i had to list
 all packages matched by my current BUILD_PKGS spec (just look at
 the examples i've put in my email on that topic to get the idea).

gnome-*/* for package.keywords is already not supported. There is no 
difference between what I/pclouds is suggesting for package.env and 
package.keywords. What you are really after is for extending the 
configuration syntax to support more than just the DEPEND atom. I was
going to reply to that earlier but this patch came up in between me
reading it and, umm, now.

If the configuration syntax is going to be extended, it needs to be
extended across the board. There is already a (closed) bug asking for
regex atom support. This is essentially what you are asking for with the 
BUILD_PKGS patch. The difference is that you are completely breaking
away from the mostly standard configuration mechanisms portage currently 
supports.

The extension to per-package configuration beyond basic atoms is fine,
but it needs to apply everywhere. If it can remain quick all the better,
but the most important thing is that whatever syntax is used can be used 
whereever an atom can be used at the moment.

 Since being able to list several env file on a same line doesn't
 sounds like a must have feature to me, i would much prefer a
 package.env format of that kind:
   rule [rule ...]envfile
 where rule would be similar to what i've defined for BUILD_PKGS
 (with addition of full versioned dep atoms, which is a trivial
 change to my code).  And if a package happens to match the rules
 lists of several lines, then the corresponding env files would all
 be sourced, in the order of the said lines.  I can try to implement
 that if you agree on the idea.

I don't see the difference in the end result. I can only see a difference in 
perspective. Perhaps you could enlighten me on this point?

 My second concern is about unsupported variables (some of the
 FEATURES flags, some of the *DIR locations, etc.): i think they
 should be somehow blacklisted, to avoid crazy breakages/bugreports
 (btw, a quick look on the variables listed in make.conf.example
 made me realize it was not that easy to write an accurate
 blacklist, which tends to confirm it's really needed).

Rather than blacklist, I'd think that whitelisting is easier. Anything unknown 
to portage (and hence only used on the bash side) will work.

Stuff known to portage that is okay to per-package
--
ACCEPT_KEYWORDS
FEATURES=buildpkg ccache distcc keeptemp keepwork noauto noclean nodoc noinfo 
noman nostrip sandbox sfperms suidctl test userpriv usersandbox
USE

Stuff documented but unknown to portage that will work per-package
--
MAKE_OPTS
CFLAGS
CHOST
CTARGET
EBEEP_IGNORE
EPAUSE_IGNORE
NOCOLOR

Stuff known to portage but is portage configuration
---
FEATURES=autoaddcvs cvs digest distlocks fixpackages getbinpkg gpg mirror 
notitles severe sign strict
FETCHCOMMAND
GENTOO_MIRRORS
PKGDIR
PORT_LOGDIR
PORTAGE_BINHOST
PORTAGE_NICENESS
PORTAGE_TMPDIR
PORTDIR
PORTDIR_OVERLAY
RESUMECOMMAND
ROOT
RSYNC_EXCLUDEFROM
RSYNC_RETRIES
RSYNC_TIMEOUT
RPMDIR
SYNC
USE_ORDER

* Some of those FEATURES could probably be moved to the supported list

Stuff unknown to portage but is only documented as it affects the default 
FETCH_COMMEND
-
http_proxy
ftp_proxy


The question of facing the supported vs unsupported variables is one that we 
have been facing for some time. Just looking at the short list of those that 
are known to portage and can be supported, it's a hell of a lot of options to 
implement as separate configuration files. Thinking of not support per-
package env in some way or another at some point down the track is pure 
fantasy.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] emerge -pv and masked dependencies

2005-11-04 Thread Jason Stubbs
On Saturday 05 November 2005 04:19, [EMAIL PROTECTED] wrote:
 On Sat, Nov 05, 2005 at 03:44:30AM +0900, Jason Stubbs wrote:
  Nobody else has shown a real example, why should I?
  ...
  I am focusing on what it could do. I stated all the options in my
  previous email.
  ...
  To restate: How often is it there is exactly one masked version
  available? What to do when there are two?

 How old are you?  You sound like some crotchety old fart on a rocking
 chair on his porch.

26; almost 27.

 Good god.  Probably once or twice a month I read about some program
 that sounds interesting, run emerge -p on my amd64, it complains that
 some dependency is masked, I edit /etc/portage/packages.keywords,
 emerge -p again, get another complaint about some other top level
 dependency, rinse, lather, repeat, until I have a half dozen additions
 or give up in disgust.  If you have never had this happen, then I feel
 sorry for you for being so unadventurous.

I run ~amd64 but always run whatever the latest packaged KDE is available.
Unmasking all of KDE would fit in your category I guess, but I think the 
discussion with TGL tied with package.unmask would solve that. The only
other times I run into masked packages are those that are missing an amd64
keyword.

 And the only way I can provide a real world example is wait til it
 comes around again on the gitar, to quote Arlo, and I am not going to
 waste my time trying to remember to come back to this dicsussion then,
 it will be quite cold in its grave, obviously where you want it to go.

I don't really want a real world example. You seemed to want one.

 I swear you have got to be just about the most negative pessimistic
 whining poster on this list.

Heh. This made me laugh. If negative means that I try to be pro-active by 
searching for problems, pessimistic means that I do find a lot of problems, 
and whining means that I keep going back to points that haven't been 
addressed, then why sure I am! ;)

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



[gentoo-portage-dev] branches/2.0 reopened

2005-11-02 Thread Jason Stubbs
Hi all,

As per previous discussion, 2.0.53_rc7 is pretty much ready to go stable. 
We're now just waiting on it lasting a couple of weeks without bugs. Seeing 
as we're all wanting to move ahead with 2.0.54 (and that 2.0.54 has already 
been committed to the branch), we may as well get on with it. ;)

Reminder: Anybody who commits to portage should be on this list and 
following/participating in what's going on.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [Bug 110386] Unable to remerge any package with -K (rc5 and rc6)

2005-10-29 Thread Jason Stubbs
On Saturday 29 October 2005 15:38, Zac Medico wrote:
 Jason Stubbs wrote:
  On Saturday 29 October 2005 13:20, Jason Stubbs wrote:
 I've adjusted the patch a bit to make all method signature changes into
 keyword arguments. I've also change the default tree to None and added a
 warning message to doebuild when None is passed and defaulting it to tree
 there. With the EAPI changes, passing a tree is really a requirement. If
  a tree isn't passed, it'd be better to alert people to that rather than
  trying to backtrack to it as the cause of the bugs that will inevitably
  pop up.
 
  Many usages of doebuild in emerge - one of which wasn't meant to go to
  porttree. The warning has helped already! ;)
 
  Cleaned up those instances and also fixed a typo (of mine) in
  portage.unmerge().

 I'm using your new patch an it looks good.  Nice idea to place that warning
 in there.  I did a recursive grep and found a couple doebuild calls that
 will generate a warning (see patch).  It looks like you're about ready to
 package up an _rc7 that hopefully will become stable soon. :)

Yep, I got the repoman one after sending the patch (but didn't want to spam 
everyone too much ;) but missed the aux_get() one. Here's the final patch 
against 2.0.53_rc6. If there's no other issues, I'll put it in and do an
_rc7 tomorrow.

--
Jason Stubbs
Index: pym/portage_util.py
===
--- pym/portage_util.py	(revision 2198)
+++ pym/portage_util.py	(working copy)
@@ -455,5 +455,15 @@
 	return mya
 
 
+def dump_traceback(msg):
+	try:
+		import sys, traceback
+		writemsg(\n\n)
+		writemsg(Warning: %s\n % msg)
+		for line in traceback.format_list(traceback.extract_stack()[:-1]):
+			writemsg(line)
+		writemsg(Please file a bug for %s\n % sys.argv[0])
+		writemsg(\n\n)
+	except:
+		pass
 
-
Index: pym/portage.py
===
--- pym/portage.py	(revision 2198)
+++ pym/portage.py	(working copy)
@@ -89,7 +89,7 @@
 	import portage_util
 	from portage_util import grab_multiple, grabdict, grabdict_package, grabfile, grabfile_package, \
 		grabints, map_dictlist_vals, pickle_read, pickle_write, stack_dictlist, stack_dicts, stack_lists, \
-		unique_array, varexpand, writedict, writeints, writemsg, getconfig
+		unique_array, varexpand, writedict, writeints, writemsg, getconfig, dump_traceback
 	import portage_exception
 	import portage_gpg
 	import portage_locks
@@ -2352,9 +2352,13 @@
 	return str(eapi).strip() == str(portage_const.EAPI).strip()
 
 
-def doebuild(myebuild,mydo,myroot,mysettings,debug=0,listonly=0,fetchonly=0,cleanup=0,dbkey=None,use_cache=1,fetchall=0,tree=porttree):
+def doebuild(myebuild,mydo,myroot,mysettings,debug=0,listonly=0,fetchonly=0,cleanup=0,dbkey=None,use_cache=1,fetchall=0,tree=None):
 	global db, actionmap_deps
 
+	if not tree:
+		dump_traceback(tree not specified to doebuild)
+		tree = porttree
+
 	ebuild_path = os.path.abspath(myebuild)
 	pkg_dir = os.path.dirname(ebuild_path)
 
@@ -2759,12 +2763,12 @@
 			print !!! mydo=qmerge, but install phase hasn't been ran
 			sys.exit(1)
 		#qmerge is specifically not supposed to do a runtime dep check
-		return merge(mysettings[CATEGORY],mysettings[PF],mysettings[D],mysettings[BUILDDIR]+/build-info,myroot,mysettings,myebuild=mysettings[EBUILD])
+		return merge(mysettings[CATEGORY],mysettings[PF],mysettings[D],mysettings[BUILDDIR]+/build-info,myroot,mysettings,myebuild=mysettings[EBUILD],mytree=tree)
 	elif mydo==merge:
 		retval=spawnebuild(install,actionmap,mysettings,debug,alwaysdep=1,logfile=logfile)
 		if retval:
 			return retval
-		return merge(mysettings[CATEGORY],mysettings[PF],mysettings[D],mysettings[BUILDDIR]+/build-info,myroot,mysettings,myebuild=mysettings[EBUILD])
+		return merge(mysettings[CATEGORY],mysettings[PF],mysettings[D],mysettings[BUILDDIR]+/build-info,myroot,mysettings,myebuild=mysettings[EBUILD],mytree=tree)
 	else:
 		print !!! Unknown mydo:,mydo
 		sys.exit(1)
@@ -2937,12 +2941,12 @@
 
 	return newmtime
 
-def merge(mycat,mypkg,pkgloc,infloc,myroot,mysettings,myebuild=None):
-	mylink=dblink(mycat,mypkg,myroot,mysettings)
+def merge(mycat,mypkg,pkgloc,infloc,myroot,mysettings,myebuild=None,mytree=None):
+	mylink=dblink(mycat,mypkg,myroot,mysettings,treetype=mytree)
 	return mylink.merge(pkgloc,infloc,myroot,myebuild)
 
 def unmerge(cat,pkg,myroot,mysettings,mytrimworld=1):
-	mylink=dblink(cat,pkg,myroot,mysettings)
+	mylink=dblink(cat,pkg,myroot,mysettings,treetype=vartree)
 	if mylink.exists():
 		mylink.unmerge(trimworld=mytrimworld,cleanup=1)
 	mylink.delete()
@@ -5393,7 +5397,7 @@
 	writemsg(Uncaught handled exception: %(exception)s\n % {exception:str(e)})
 	raise
 
-			myret=doebuild(myebuild,depend,/,self.mysettings,dbkey=mydbkey)
+			myret=doebuild(myebuild,depend,/,self.mysettings,dbkey=mydbkey,tree=porttree)
 			if myret

Re: [gentoo-portage-dev] [5/7] portage_exec cleanups

2005-10-29 Thread Jason Stubbs
On Saturday 29 October 2005 19:56, Jason Stubbs wrote:
 All file descriptors opened by spawn() are now closed in the correct
 places. The code that closes all unnecessary descriptors via brute force is
 no longer required.

 I found that a file descriptor was being left open and passed around.
 However, it was definately not created in spawn(). It seems to me though
 that code outside of spawn should take care of closing (or telling spawn to
 close) any unnecessary FDs.


On Saturday 29 October 2005 23:22, Brian Harring wrote:
 fd_unused doesn't cut it either also, or at the very least introduces
 an ass biter hidden behaviour into spawn- consider

 fd_pipes={}

 Previous code *would* close all fd's, and execute the process as such.
 With fd_unused addition, it won't anymore- meaning at the very least a
 chunk of the regen code will need updating (closes stdin last I
 looked).  Very least, all callers of spawn with fd_pipes != {0:0, 1:1,
 2:2} need verification, since the handles they were having explicitly
 closed prior, are now left open.

Not explicitly closed. Implicitly closed.

 With the fd_unused route, there is no way clean way to have all fd's
 closed.  You don't know what fd's are open when you spawn (nor is
 there any way to know, beyond testing each potential fd, or overriding
 the file builtin); going the fd_unused route means fd's the process
 shouldn't have access to, will have access to.

This is a difference of viewpoint, I think. Having unknown fds left open by 
the calling code seems unclean to me. Iterating through 100s or 1000s of fds 
and attempting to close them just in case the caller was sloppy seems equally 
unclean.

By the way, this method does not allow having a pipe open to a child process 
other than stdin, stdout or stderr.

 Mind you the fd_unused comments are about this patch and the one +2
 down the set.

 So... regressions, both in fd_pipes handling, and in open handles
 leaking to the spawned processes

fd_pipes handling is now addressed. The open handles leaking is an issue, but 
where the issue stems from is yet to be resolved.

 The original code is ugly, and knowing a bit more this time around I'd
 write it a bit differently, that said the protections/behaviour
 originally in it should be left- it's more robust, and is expected
 now.

I disagree here. Presently, other than the execve call, every try/except block 
in portage_exec.py has bugs that are hidden by the protections.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [5/7] portage_exec cleanups

2005-10-29 Thread Jason Stubbs
On Sunday 30 October 2005 01:04, Brian Harring wrote:
 On Sun, Oct 30, 2005 at 12:25:23AM +0900, Jason Stubbs wrote:
  By the way, this method does not allow having a pipe open to a child
  process other than stdin, stdout or stderr.

 It does; the portage_exec that is in stable is the modified version I
 created for ebd, which uses this exact functionality.

 line #153 of trunk/pym/ebuild.py, example call-

 self.pid = spawn_func(self.ebd+ daemonize, fd_pipes={0:0, 1:1, 2:2,
   3:cread, 4:dwrite}, returnpid=True,env=env, *args, **spawn_opts)[0]

Yep. Oversight on my part.

   The original code is ugly, and knowing a bit more this time around I'd
   write it a bit differently, that said the protections/behaviour
   originally in it should be left- it's more robust, and is expected
   now.
 
  I disagree here. Presently, other than the execve call, every try/except
  block in portage_exec.py has bugs that are hidden by the protections.

 Original in this case was referencing the fd_pipes additions, not the
 true original spawn func :)

Even with the fd_pipes, the try/except block in there covers a bug that is hit 
every time it is entered.

 Looking through the 2.4 subprocess module (which came along after the
 mods in question), adding an option controlling the fd cleansing would
 make sense, as long as the default option is to close the fds.

Given the threading comment, I'd be agreeable to this. However, the spawn of 
tee would have to specifically close the write side of the pipe when unknown 
FDs aren't closed. Is having two extra paramaters, close_fds (a list) and 
close_all_fds (a bool), to spawn okay?

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [5/7] portage_exec cleanups

2005-10-29 Thread Jason Stubbs
Trying to alter patches that have been split up is a PITA. Missed a var rename 
in the patch just sent. :/

--
Jason Stubbs
--- portage_exec.py.orig	2005-10-30 02:05:50.0 +0900
+++ portage_exec.py	2005-10-30 02:18:24.0 +0900
@@ -11,11 +11,8 @@
 
 try:
 	import resource
-	max_fd_limit=resource.getrlimit(RLIMIT_NOFILE)
-except SystemExit, e:
-	raise
-except:
-	# hokay, no resource module.
+	max_fd_limit=resource.getrlimit(resource.RLIMIT_NOFILE)[1]
+except ImportError:
 	max_fd_limit=256
 
 spawned_pids = []
@@ -83,7 +80,7 @@
 	mypid=[]
 	if logfile:
 		pr,pw=os.pipe()
-		mypid.extend(spawn(('tee','-i','-a',logfile),returnpid=True,fd_pipes={0:pr,1:fd_pipes[1],2:fd_pipes[2]},fd_unused=[pw]))
+		mypid.extend(spawn(('tee','-i','-a',logfile),returnpid=True,fd_pipes={0:pr,1:fd_pipes[1],2:fd_pipes[2]}))
 		os.close(pr)
 		fd_pipes[1]=pw
 		fd_pipes[2]=pw
@@ -95,22 +92,15 @@
 	mypid.append(os.fork())
 	if mypid[-1] == 0:
 		my_pipes = {}
-		close_pipes = {}
 		for x in fd_pipes:
 			my_pipes[x] = os.dup(fd_pipes[x])
-			close_pipes[fd_pipes[x]] = True
-		for x in close_pipes:
-			os.close(x)
 		for x in my_pipes:
 			os.dup2(my_pipes[x], x)
-			os.close(my_pipes[x])
-		for x in range(0,max_fd_limit):
-			if x not in trg_fd:
-try: 
+		for x in range(max_fd_limit):
+			if x not in my_pipes:
+try:
 	os.close(x)
-except SystemExit, e:
-	raise
-except:
+except OSError:
 	pass
 		# note this order must be preserved- can't change gid/groups if you change uid first.
 		if gid:


Re: [gentoo-portage-dev] [5/7] portage_exec cleanups

2005-10-29 Thread Jason Stubbs
On Sunday 30 October 2005 02:17, Brian Harring wrote:
 On Sun, Oct 30, 2005 at 01:40:41AM +0900, Jason Stubbs wrote:
  Even with the fd_pipes, the try/except block in there covers a bug that
  is hit every time it is entered.

 *cough* really doesn't surprise me. :)

   Looking through the 2.4 subprocess module (which came along after the
   mods in question), adding an option controlling the fd cleansing would
   make sense, as long as the default option is to close the fds.
 
  Given the threading comment, I'd be agreeable to this. However, the spawn
  of tee would have to specifically close the write side of the pipe when
  unknown FDs aren't closed. Is having two extra paramaters, close_fds (a
  list) and close_all_fds (a bool), to spawn okay?

 It's needed for any logged calls, doesn't mean I like it though.  How
 about this,
 def spawn(...close_fds=True...):
   if close_fds is True:
   # kill all that aren't marked for open
   elif close_fds:
   # must be an iterable of what to kill
   else:
   #whatever code required to keep it from nuking file handles

 Don't really like that one much myself, but it would allow for the
 logging to use the existing structure, and one less option on the
 overgrown spawn* prototype(s).

Yep, this sort of messyness is why I chose to back down. ;)

 sidenote, since max_fd_limit needs to survive, tag
  try:
   import resource
 - max_fd_limit=resource.getrlimit(RLIMIT_NOFILE)
 + max_fd_limit=resource.getrlimit(resource.RLIMIT_NOFILE)
  except SystemExit, e:
   raise

 in while you're at it, since that's obviously broke.  Current code
 would default to range(256), but default is 1024 fds for gentoo boxes
 (so that _is_ a hole).

Got that one already, but I didn't want to point out too many bugs. ;)
(There's still one more you missed there, btw; getrlimit() returns a tuple)

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [0/7] portage_exec cleanups (WAS: [Bug 104705] emerge doesn't print complete error message)

2005-10-29 Thread Jason Stubbs
On Saturday 29 October 2005 19:34, Jason Stubbs wrote:
 On Sunday 23 October 2005 15:45, Jason Stubbs wrote:
  Commented on the bug due to reasoning behind this patch. Essentially,
  SIGTERM is sent to tee, a WNOHANG waitpid() is performed followed by
  SIGKILL if it hasn't exited. So if tee doesn't exit immediately upon
  getting the SIGTERM, its buffers won't get a chance to get to disk due to
  it being killed. This patch simply adds a 1 second window between the
  SIGTERM and the SIGKILL.

 Per discussion on the bug, the problem is actually that tee shouldn't be
 sent a SIGTERM or SIGKILL at all. The decided solution was to wait on tee
 rather ebuild. I went a bit further than that though and cleaned up most of
 spawn().

Here's the full patch after incorporating feedback. It should be fairly easy 
to follow given the other explanations. The only change beyond what's already 
been gone over is that spawn() is now removing pids from spawned_pids as it 
cleans them up.

--
Jason Stubbs
Index: portage_util.py
===
--- portage_util.py	(revision 2199)
+++ portage_util.py	(working copy)
@@ -455,5 +455,15 @@
 	return mya
 
 
+def dump_traceback(msg):
+	try:
+		import sys, traceback
+		writemsg(\n\n)
+		writemsg(Warning: %s\n % msg)
+		for line in traceback.format_list(traceback.extract_stack()[:-1]):
+			writemsg(line)
+		writemsg(Please file a bug for %s\n % sys.argv[0])
+		writemsg(\n\n)
+	except:
+		pass
 
-
Index: portage.py
===
--- portage.py	(revision 2199)
+++ portage.py	(working copy)
@@ -89,7 +89,7 @@
 	import portage_util
 	from portage_util import grab_multiple, grabdict, grabdict_package, grabfile, grabfile_package, \
 		grabints, map_dictlist_vals, pickle_read, pickle_write, stack_dictlist, stack_dicts, stack_lists, \
-		unique_array, varexpand, writedict, writeints, writemsg, getconfig
+		unique_array, varexpand, writedict, writeints, writemsg, getconfig, dump_traceback
 	import portage_exception
 	import portage_gpg
 	import portage_locks
@@ -2352,9 +2352,13 @@
 	return str(eapi).strip() == str(portage_const.EAPI).strip()
 
 
-def doebuild(myebuild,mydo,myroot,mysettings,debug=0,listonly=0,fetchonly=0,cleanup=0,dbkey=None,use_cache=1,fetchall=0,tree=porttree):
+def doebuild(myebuild,mydo,myroot,mysettings,debug=0,listonly=0,fetchonly=0,cleanup=0,dbkey=None,use_cache=1,fetchall=0,tree=None):
 	global db, actionmap_deps
 
+	if not tree:
+		dump_traceback(tree not specified to doebuild)
+		tree = porttree
+
 	ebuild_path = os.path.abspath(myebuild)
 	pkg_dir = os.path.dirname(ebuild_path)
 
@@ -2759,12 +2763,12 @@
 			print !!! mydo=qmerge, but install phase hasn't been ran
 			sys.exit(1)
 		#qmerge is specifically not supposed to do a runtime dep check
-		return merge(mysettings[CATEGORY],mysettings[PF],mysettings[D],mysettings[BUILDDIR]+/build-info,myroot,mysettings,myebuild=mysettings[EBUILD])
+		return merge(mysettings[CATEGORY],mysettings[PF],mysettings[D],mysettings[BUILDDIR]+/build-info,myroot,mysettings,myebuild=mysettings[EBUILD],mytree=tree)
 	elif mydo==merge:
 		retval=spawnebuild(install,actionmap,mysettings,debug,alwaysdep=1,logfile=logfile)
 		if retval:
 			return retval
-		return merge(mysettings[CATEGORY],mysettings[PF],mysettings[D],mysettings[BUILDDIR]+/build-info,myroot,mysettings,myebuild=mysettings[EBUILD])
+		return merge(mysettings[CATEGORY],mysettings[PF],mysettings[D],mysettings[BUILDDIR]+/build-info,myroot,mysettings,myebuild=mysettings[EBUILD],mytree=tree)
 	else:
 		print !!! Unknown mydo:,mydo
 		sys.exit(1)
@@ -2937,12 +2941,12 @@
 
 	return newmtime
 
-def merge(mycat,mypkg,pkgloc,infloc,myroot,mysettings,myebuild=None):
-	mylink=dblink(mycat,mypkg,myroot,mysettings)
+def merge(mycat,mypkg,pkgloc,infloc,myroot,mysettings,myebuild=None,mytree=None):
+	mylink=dblink(mycat,mypkg,myroot,mysettings,treetype=mytree)
 	return mylink.merge(pkgloc,infloc,myroot,myebuild)
 
 def unmerge(cat,pkg,myroot,mysettings,mytrimworld=1):
-	mylink=dblink(cat,pkg,myroot,mysettings)
+	mylink=dblink(cat,pkg,myroot,mysettings,treetype=vartree)
 	if mylink.exists():
 		mylink.unmerge(trimworld=mytrimworld,cleanup=1)
 	mylink.delete()
@@ -5393,7 +5397,7 @@
 	writemsg(Uncaught handled exception: %(exception)s\n % {exception:str(e)})
 	raise
 
-			myret=doebuild(myebuild,depend,/,self.mysettings,dbkey=mydbkey)
+			myret=doebuild(myebuild,depend,/,self.mysettings,dbkey=mydbkey,tree=porttree)
 			if myret:
 portage_locks.unlockfile(mylock)
 self.lock_held = 0
@@ -6027,12 +6031,13 @@
 
 class dblink:
 	this class provides an interface to the standard text package database
-	def __init__(self,cat,pkg,myroot,mysettings):
+	def __init__(self,cat,pkg,myroot,mysettings,treetype=None):
 		create a dblink object for cat/pkg.  This dblink entry may or may not exist
 		self.cat = cat

Re: [gentoo-portage-dev] [0/7] portage_exec cleanups (WAS: [Bug 104705] emerge doesn't print complete error message)

2005-10-29 Thread Jason Stubbs
On Sunday 30 October 2005 02:32, Jason Stubbs wrote:
 On Saturday 29 October 2005 19:34, Jason Stubbs wrote:
  On Sunday 23 October 2005 15:45, Jason Stubbs wrote:
   Commented on the bug due to reasoning behind this patch. Essentially,
   SIGTERM is sent to tee, a WNOHANG waitpid() is performed followed by
   SIGKILL if it hasn't exited. So if tee doesn't exit immediately upon
   getting the SIGTERM, its buffers won't get a chance to get to disk due
   to it being killed. This patch simply adds a 1 second window between
   the SIGTERM and the SIGKILL.
 
  Per discussion on the bug, the problem is actually that tee shouldn't be
  sent a SIGTERM or SIGKILL at all. The decided solution was to wait on tee
  rather ebuild. I went a bit further than that though and cleaned up most
  of spawn().

 Here's the full patch after incorporating feedback. It should be fairly
 easy to follow given the other explanations. The only change beyond what's
 already been gone over is that spawn() is now removing pids from
 spawned_pids as it cleans them up.

02:30.. Brain is starting to stop working. Here's a patch of _just_ 
portage_exec.py. With regard to splitting up patches, I agree it's much 
easier to parse a complete patch when familiar with the code. When one isn't 
familiar with the code however... Splitting it up a bit makes explaining the 
whys easier too - as well as keeping discussion threads seperate.

--
Jason Stubbs
Index: pym/portage_exec.py
===
--- pym/portage_exec.py	(revision 2199)
+++ pym/portage_exec.py	(working copy)
@@ -11,11 +11,8 @@
 
 try:
 	import resource
-	max_fd_limit=resource.getrlimit(RLIMIT_NOFILE)
-except SystemExit, e:
-	raise
-except:
-	# hokay, no resource module.
+	max_fd_limit=resource.getrlimit(resource.RLIMIT_NOFILE)[1]
+except ImportError:
 	max_fd_limit=256
 
 spawned_pids = []
@@ -23,12 +20,9 @@
 	global spawned_pids
 	while spawned_pids:
 		pid = spawned_pids.pop()
-		try:
-			os.kill(pid,SIGKILL)
-		except SystemExit, e:
-			raise
-		except:
-			pass
+		if os.waitpid(pid,os.WNOHANG) == (0,0):
+			os.kill(pid,signal.SIGKILL)
+			os.waitpid(pid,0)
 atexit.register(cleanup)
 
 from portage_const import BASH_BINARY,SANDBOX_BINARY,SANDBOX_PIDS_FILE
@@ -67,6 +61,7 @@
 
 # base spawn function
 def spawn(mycommand,env={},opt_name=None,fd_pipes=None,returnpid=False,uid=None,gid=None,groups=None,umask=None,logfile=None,path_lookup=True):
+	global spawned_pids
 	if type(mycommand)==types.StringType:
 		mycommand=mycommand.split()
 	myc = mycommand[0]
@@ -76,21 +71,15 @@
 		myc = find_binary(myc)
 		if myc == None:
 			return None
-		
+
+	if fd_pipes is None:
+		fd_pipes = {0:0, 1:1, 2:2}
+
 	mypid=[]
 	if logfile:
 		pr,pw=os.pipe()
-		mypid.extend(spawn(('tee','-i','-a',logfile),returnpid=True,fd_pipes={0:pr,1:1,2:2}))
-		retval=os.waitpid(mypid[-1],os.WNOHANG)[1]
-		if retval != 0:
-			# he's dead jim.
-			if (retval  0xff)==0:
-return (retval  8) # exit code
-			else:
-return ((retval  0xff)  8) # signal
-		if not fd_pipes:
-			fd_pipes={}
-			fd_pipes[0] = 0
+		mypid.extend(spawn(('tee','-i','-a',logfile),returnpid=True,fd_pipes={0:pr,1:fd_pipes[1],2:fd_pipes[2]}))
+		os.close(pr)
 		fd_pipes[1]=pw
 		fd_pipes[2]=pw
 		
@@ -100,41 +89,16 @@
 	myargs.extend(mycommand[1:])
 	mypid.append(os.fork())
 	if mypid[-1] == 0:
-		# this may look ugly, but basically it moves file descriptors around to ensure no
-		# handles that are needed are accidentally closed during the final dup2 calls.
-		trg_fd=[]
-		if type(fd_pipes)==types.DictType:
-			src_fd=[]
-			k=fd_pipes.keys()
-			k.sort()
-			for x in k:
-trg_fd.append(x)
-src_fd.append(fd_pipes[x])
-			for x in range(0,len(trg_fd)):
-if trg_fd[x] == src_fd[x]:
-	continue
-if trg_fd[x] in src_fd[x+1:]:
-	new=os.dup2(trg_fd[x],max(src_fd) + 1)
-	os.close(trg_fd[x])
-	try:
-		while True: 
-			src_fd[s.index(trg_fd[x])]=new
-	except SystemExit, e:
-		raise
-	except:
-		pass
-			for x in range(0,len(trg_fd)):
-if trg_fd[x] != src_fd[x]:
-	os.dup2(src_fd[x], trg_fd[x])
-		else:
-			trg_fd=[0,1,2]
-		for x in range(0,max_fd_limit):
-			if x not in trg_fd:
-try: 
+		my_pipes = {}
+		for x in fd_pipes:
+			my_pipes[x] = os.dup(fd_pipes[x])
+		for x in my_pipes:
+			os.dup2(my_pipes[x], x)
+		for x in range(max_fd_limit):
+			if x not in my_pipes:
+try:
 	os.close(x)
-except SystemExit, e:
-	raise
-except:
+except OSError:
 	pass
 		# note this order must be preserved- can't change gid/groups if you change uid first.
 		if gid:
@@ -145,50 +109,29 @@
 			os.setuid(uid)
 		if umask:
 			os.umask(umask)
-		try:
-			# XXX: We would do this to stop ebuild.sh from getting any
-			# XXX: output, and consequently, we'd get to handle the sigINT.
-			#os.close(sys.stdin.fileno())
-			pass
-		except SystemExit, e:
-			raise
-		except:
-			pass
 
 		try:
-			#print execing, myc, myargs
 			os.execve

Re: [gentoo-portage-dev] decoupling .tbz'2 from bzip2

2005-10-29 Thread Jason Stubbs
On Saturday 29 October 2005 02:54, Jason Pepas wrote:
 Hi,

 I would like to work on making the compression program used by portage a
 configurable option.

 I would like to start with binary packages, but eventually make all of
 portage more flexible (man, info, doc, etc).

Support for info and friends is supported in current trunk. It is configurable 
by the var PORTAGE_COMPRESS and PORTAGE_COMPRESS_FLAGS. Mike Frysinger is the 
author so it's his call, but it could probably be put into 2.0.54.

 What would be the best route to pursue this if I wish to keep the gentoo
 dev's receptive to my changes?

 I have already started hacking my way through portage :)

Even though there's a lot of mess, try not to clean any of it up if you're 
posting a patch for something else. That is, clean ups should be separate to 
functionality additions/modifications. Other than that, try not to make more 
mess. ;)

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [PATCH] post_sync actions

2005-10-29 Thread Jason Stubbs
On Sunday 30 October 2005 01:03, Ned Ludd wrote:
 The following simple patch adds the ability for to run a userside
 post_sync script, right now it's only used with rsync but the idea could
 be adopted rather easy for other methods of transfer that portage may
 use. The basic idea is to be able to preform a set of maintenance tasks
 right after sync up with an rsync server. Be that adding a call to svn
 co, cvs co, ebuild manipulation, updating search caches or other. I've
 been using this method on and off sense about portage-2.0.51.16 without
 any problems. Any comments before I file a bug for inclusion?

This really goes hand in hand with the pre/post phase hooks patch. That patch, 
however, implements hooks as bash functions rather than external executables. 
The discrepancy there should probably be worked out before either are 
included. Personally, I'm for the external executable method but I'm sure 
that there will be alternative opinions...

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] bug 104000 - package.masked message from overlay

2005-10-29 Thread Jason Stubbs
On Thursday 20 October 2005 01:45, Thomas Matthijs wrote:
 Hey,

 I made a patch for http://bugs.gentoo.org/show_bug.cgi?id=104000
 I'm not really familiar with portage internals, but i was bored, and
 though i'd try and see if i could fix any of the bugs, and this one
 seemed easy enough :)

 http://dev.gentoo.org/~axxo/portage_bug_104000.patch

 Ferringb didn't seem to like the change to locations at first sight, and
 adviced to use settings.profiles, but that only contians the actual
 profiles, and not the top level {PORDIR,OVERLAY}/profile, where
 package.mask can aslo be.

It looks okay to me... All that's changed is that locations has become an 
instance var rather than a local var. What was the issue?

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [Bug 110386] Unable to remerge any package with -K (rc5 and rc6)

2005-10-28 Thread Jason Stubbs
On Friday 28 October 2005 19:35, Zac Medico wrote:
 Brian Harring wrote:
  On Thu, Oct 27, 2005 at 11:50:02PM +0900, Jason Stubbs wrote:
 On Wednesday 26 October 2005 23:56, Jason Stubbs wrote:
 I've attached a quickly thrown together patch for it, which works here,
  but we really need to get this whole thing down pat. Perhaps usage of
  doebuild without specifying tree should be deprecated and the code
  audited for any usage without it? Making tree a keyword parameter will
  break too many external things so that's not really an option...
 
 Updated patch; I missed a call to merge(). I really don't like this patch
 though because it touches way too much stuff. Anybody have any ideas for
  a better way of doing it?
 
  Offhand, don't pass mytree around like that; pass it into the
  constructor, assign to the obj's namespace, and have treelink pass the
  saved value into doebuild.
 
  That's how I did it in 2.1 at least; been a long while, but don't
  recall any major issues with that route.  Plus side, heck of a lot
  less modification to methods signatures, just the objects behaviour.
  ~harring

 I adapted Jason's patch to go along with the dblink.treetype bit from 2.1
 and it seems to work alright.  The only notable difference from Jason's
 patch is that tree=self.treetype is passed into doebuild in any case for
 preinst and postinst (following the lead of 2.1).

If tree wasn't being passed to doebuild in my earlier patch, it was only 
because I missed those instances. Using an instance member is much better 
though. I should have picked up that most of the changes were in dblink...

I've adjusted the patch a bit to make all method signature changes into 
keyword arguments. I've also change the default tree to None and added a 
warning message to doebuild when None is passed and defaulting it to tree 
there. With the EAPI changes, passing a tree is really a requirement. If a 
tree isn't passed, it'd be better to alert people to that rather than trying 
to backtrack to it as the cause of the bugs that will inevitably pop up.

So are we just about ready on this one?

--
Jason Stubbs
Index: pym/portage_util.py
===
--- pym/portage_util.py	(revision 2198)
+++ pym/portage_util.py	(working copy)
@@ -455,5 +455,15 @@
 	return mya
 
 
+def dump_traceback(msg):
+	try:
+		import sys, traceback
+		writemsg(\n\n)
+		writemsg(Warning: %s\n % msg)
+		for line in traceback.format_list(traceback.extract_stack()[:-1]):
+			writemsg(line)
+		writemsg(Please file a bug for %s\n % sys.argv[0])
+		writemsg(\n\n)
+	except:
+		pass
 
-
Index: pym/portage.py
===
--- pym/portage.py	(revision 2198)
+++ pym/portage.py	(working copy)
@@ -89,7 +89,7 @@
 	import portage_util
 	from portage_util import grab_multiple, grabdict, grabdict_package, grabfile, grabfile_package, \
 		grabints, map_dictlist_vals, pickle_read, pickle_write, stack_dictlist, stack_dicts, stack_lists, \
-		unique_array, varexpand, writedict, writeints, writemsg, getconfig
+		unique_array, varexpand, writedict, writeints, writemsg, getconfig, dump_traceback
 	import portage_exception
 	import portage_gpg
 	import portage_locks
@@ -2352,9 +2352,13 @@
 	return str(eapi).strip() == str(portage_const.EAPI).strip()
 
 
-def doebuild(myebuild,mydo,myroot,mysettings,debug=0,listonly=0,fetchonly=0,cleanup=0,dbkey=None,use_cache=1,fetchall=0,tree=porttree):
+def doebuild(myebuild,mydo,myroot,mysettings,debug=0,listonly=0,fetchonly=0,cleanup=0,dbkey=None,use_cache=1,fetchall=0,tree=None):
 	global db, actionmap_deps
 
+	if not tree:
+		dump_traceback(tree not specified to doebuild)
+		tree = porttree
+
 	ebuild_path = os.path.abspath(myebuild)
 	pkg_dir = os.path.dirname(ebuild_path)
 
@@ -2759,12 +2763,12 @@
 			print !!! mydo=qmerge, but install phase hasn't been ran
 			sys.exit(1)
 		#qmerge is specifically not supposed to do a runtime dep check
-		return merge(mysettings[CATEGORY],mysettings[PF],mysettings[D],mysettings[BUILDDIR]+/build-info,myroot,mysettings,myebuild=mysettings[EBUILD])
+		return merge(mysettings[CATEGORY],mysettings[PF],mysettings[D],mysettings[BUILDDIR]+/build-info,myroot,mysettings,myebuild=mysettings[EBUILD],mytree=tree)
 	elif mydo==merge:
 		retval=spawnebuild(install,actionmap,mysettings,debug,alwaysdep=1,logfile=logfile)
 		if retval:
 			return retval
-		return merge(mysettings[CATEGORY],mysettings[PF],mysettings[D],mysettings[BUILDDIR]+/build-info,myroot,mysettings,myebuild=mysettings[EBUILD])
+		return merge(mysettings[CATEGORY],mysettings[PF],mysettings[D],mysettings[BUILDDIR]+/build-info,myroot,mysettings,myebuild=mysettings[EBUILD],mytree=tree)
 	else:
 		print !!! Unknown mydo:,mydo
 		sys.exit(1)
@@ -2937,12 +2941,12 @@
 
 	return newmtime
 
-def merge(mycat,mypkg,pkgloc,infloc,myroot,mysettings,myebuild=None):
-	mylink=dblink(mycat

Re: [gentoo-portage-dev] [Bug 110386] Unable to remerge any package with -K (rc5 and rc6)

2005-10-28 Thread Jason Stubbs
On Saturday 29 October 2005 13:20, Jason Stubbs wrote:
 I've adjusted the patch a bit to make all method signature changes into
 keyword arguments. I've also change the default tree to None and added a
 warning message to doebuild when None is passed and defaulting it to tree
 there. With the EAPI changes, passing a tree is really a requirement. If a
 tree isn't passed, it'd be better to alert people to that rather than
 trying to backtrack to it as the cause of the bugs that will inevitably pop
 up.

Many usages of doebuild in emerge - one of which wasn't meant to go to 
porttree. The warning has helped already! ;)

Cleaned up those instances and also fixed a typo (of mine) in 
portage.unmerge().

--
Jason Stubbs
Index: pym/portage_util.py
===
--- pym/portage_util.py	(revision 2198)
+++ pym/portage_util.py	(working copy)
@@ -455,5 +455,15 @@
 	return mya
 
 
+def dump_traceback(msg):
+	try:
+		import sys, traceback
+		writemsg(\n\n)
+		writemsg(Warning: %s\n % msg)
+		for line in traceback.format_list(traceback.extract_stack()[:-1]):
+			writemsg(line)
+		writemsg(Please file a bug for %s\n % sys.argv[0])
+		writemsg(\n\n)
+	except:
+		pass
 
-
Index: pym/portage.py
===
--- pym/portage.py	(revision 2198)
+++ pym/portage.py	(working copy)
@@ -89,7 +89,7 @@
 	import portage_util
 	from portage_util import grab_multiple, grabdict, grabdict_package, grabfile, grabfile_package, \
 		grabints, map_dictlist_vals, pickle_read, pickle_write, stack_dictlist, stack_dicts, stack_lists, \
-		unique_array, varexpand, writedict, writeints, writemsg, getconfig
+		unique_array, varexpand, writedict, writeints, writemsg, getconfig, dump_traceback
 	import portage_exception
 	import portage_gpg
 	import portage_locks
@@ -2352,9 +2352,13 @@
 	return str(eapi).strip() == str(portage_const.EAPI).strip()
 
 
-def doebuild(myebuild,mydo,myroot,mysettings,debug=0,listonly=0,fetchonly=0,cleanup=0,dbkey=None,use_cache=1,fetchall=0,tree=porttree):
+def doebuild(myebuild,mydo,myroot,mysettings,debug=0,listonly=0,fetchonly=0,cleanup=0,dbkey=None,use_cache=1,fetchall=0,tree=None):
 	global db, actionmap_deps
 
+	if not tree:
+		dump_traceback(tree not specified to doebuild)
+		tree = porttree
+
 	ebuild_path = os.path.abspath(myebuild)
 	pkg_dir = os.path.dirname(ebuild_path)
 
@@ -2759,12 +2763,12 @@
 			print !!! mydo=qmerge, but install phase hasn't been ran
 			sys.exit(1)
 		#qmerge is specifically not supposed to do a runtime dep check
-		return merge(mysettings[CATEGORY],mysettings[PF],mysettings[D],mysettings[BUILDDIR]+/build-info,myroot,mysettings,myebuild=mysettings[EBUILD])
+		return merge(mysettings[CATEGORY],mysettings[PF],mysettings[D],mysettings[BUILDDIR]+/build-info,myroot,mysettings,myebuild=mysettings[EBUILD],mytree=tree)
 	elif mydo==merge:
 		retval=spawnebuild(install,actionmap,mysettings,debug,alwaysdep=1,logfile=logfile)
 		if retval:
 			return retval
-		return merge(mysettings[CATEGORY],mysettings[PF],mysettings[D],mysettings[BUILDDIR]+/build-info,myroot,mysettings,myebuild=mysettings[EBUILD])
+		return merge(mysettings[CATEGORY],mysettings[PF],mysettings[D],mysettings[BUILDDIR]+/build-info,myroot,mysettings,myebuild=mysettings[EBUILD],mytree=tree)
 	else:
 		print !!! Unknown mydo:,mydo
 		sys.exit(1)
@@ -2937,12 +2941,12 @@
 
 	return newmtime
 
-def merge(mycat,mypkg,pkgloc,infloc,myroot,mysettings,myebuild=None):
-	mylink=dblink(mycat,mypkg,myroot,mysettings)
+def merge(mycat,mypkg,pkgloc,infloc,myroot,mysettings,myebuild=None,mytree=None):
+	mylink=dblink(mycat,mypkg,myroot,mysettings,treetype=mytree)
 	return mylink.merge(pkgloc,infloc,myroot,myebuild)
 
 def unmerge(cat,pkg,myroot,mysettings,mytrimworld=1):
-	mylink=dblink(cat,pkg,myroot,mysettings)
+	mylink=dblink(cat,pkg,myroot,mysettings,treetype=vartree)
 	if mylink.exists():
 		mylink.unmerge(trimworld=mytrimworld,cleanup=1)
 	mylink.delete()
@@ -6027,12 +6031,13 @@
 
 class dblink:
 	this class provides an interface to the standard text package database
-	def __init__(self,cat,pkg,myroot,mysettings):
+	def __init__(self,cat,pkg,myroot,mysettings,treetype=None):
 		create a dblink object for cat/pkg.  This dblink entry may or may not exist
 		self.cat = cat
 		self.pkg = pkg
 		self.mycpv   = self.cat+/+self.pkg
 		self.mysplit = pkgsplit(self.mycpv)
+		self.treetype = treetype
 
 		self.dbroot   = os.path.normpath(myroot+VDB_PATH)
 		self.dbcatdir = self.dbroot+/+cat
@@ -6213,7 +6218,7 @@
 
 		#do prerm script
 		if myebuildpath and os.path.exists(myebuildpath):
-			a=doebuild(myebuildpath,prerm,self.myroot,self.settings,cleanup=cleanup,use_cache=0,tree=vartree)
+			a=doebuild(myebuildpath,prerm,self.myroot,self.settings,cleanup=cleanup,use_cache=0,tree=self.treetype)
 			# XXX: Decide how to handle failures here.
 			if a != 0

Re: [gentoo-portage-dev] [PATCH] elog-modules

2005-10-24 Thread Jason Stubbs
mod_mail.py

An exception will be thrown if PORTAGE_ELOG_MAILSUBJECT is defined but does 
not contain exactly one %s.

Any reason to not have a PORTAGE_ELOG_MAILRECIPIENT?


mod_syslog.py

The definition of pri should be moved outside of the loops and probably 
outside of the function altogether.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] The road ahead...

2005-10-24 Thread Jason Stubbs
On Friday 21 October 2005 19:06, Marius Mauch wrote:
 Jason Stubbs wrote:
  After thinking about it, incremental feature creep does seem like the
  best way to go at this late stage in 2.0's life. The problem is how to
  guage what is and what is not more trouble than worth. Perhaps adhering
  to the kernel's rule of Separate each logical change into its own patch
  would help to ease the possible impact of larger changes?

 Probably the best solution.

So we're all agreed then?

* 2.x will never go beyond 2.0
* New features are considered on a case by case basis
* New features can be integrated whenever they're ready (and we're not 
  stabalizing or the feature doesn't aid stabalizing)
* Major changes need to be split up

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



[gentoo-portage-dev] [Bug 104705] emerge doesn't print complete error message

2005-10-24 Thread Jason Stubbs
Commented on the bug due to reasoning behind this patch. Essentially, SIGTERM 
is sent to tee, a WNOHANG waitpid() is performed followed by SIGKILL if it 
hasn't exited. So if tee doesn't exit immediately upon getting the SIGTERM, 
its buffers won't get a chance to get to disk due to it being killed. This 
patch simply adds a 1 second window between the SIGTERM and the SIGKILL.

One question; is the waitpid(x,0) necessary in the case where SIGKILL wasn't 
sent? Is waitpid(x,os.NOHANG) enough to clean up the zombie when SIGTERM 
succeeds? If so, the waitpid(x,0) could be indented into the if not 
timeout: block.

--
Jason Stubbs
Index: pym/portage_exec.py
===
--- pym/portage_exec.py	(revision 2150)
+++ pym/portage_exec.py	(working copy)
@@ -4,7 +4,7 @@
 # $Id: /var/cvsroot/gentoo-src/portage/pym/portage_exec.py,v 1.13.2.4 2005/04/17 09:01:56 jstubbs Exp $
 
 
-import os,types,atexit,string,stat
+import os,types,atexit,string,stat,time
 import signal
 import portage_data
 import portage_util
@@ -182,7 +182,13 @@
 			for x in mypid[0:-1]:
 try:
 	os.kill(x,signal.SIGTERM)
-	if os.waitpid(x,os.WNOHANG)[1] == 0:
+	timeout = 100
+	while timeout:
+		if os.waitpid(x,os.WNOHANG)[1] != 0:
+			break
+		time.sleep(0.01)
+		timeout -= 1
+	if not timeout:
 		# feisty bugger, still alive.
 		os.kill(x,signal.SIGKILL)
 	os.waitpid(x,0)


Re: [gentoo-portage-dev] The road ahead...

2005-10-21 Thread Jason Stubbs
On Friday 21 October 2005 19:06, Marius Mauch wrote:
 Jason Stubbs wrote:
  After thinking about it, incremental feature creep does seem like the
  best way to go at this late stage in 2.0's life. The problem is how to
  guage what is and what is not more trouble than worth. Perhaps adhering
  to the kernel's rule of Separate each logical change into its own patch
  would help to ease the possible impact of larger changes?

 Probably the best solution.

Brian, you agree on this? It'll mean splitting up the cache patch...

  Speaking of which, if something does crop up with 2.0.53 while the arch
  guys are deciding if it's stable, how should that be dealt with in
  subversion? Continue development under branches/2.0 and, should an issue
  crop up, `svn cp tags/2.0.53_rc6 tags/2.0.53_rc7` and make any required
  change directly under the tag?

 Never commit to a tag, make a branch (branches are cheap in svn), and if
 the branch is finished make a tag.

The cheapness is exactly why I was questioning. Consider:

# svn cp tags/2.0.53 branches/2.0.53-branch
# cd branches/2.0.53-branch
# patch  something-that-needs-fixing-now.patch
# svn ci
# cd ../..
# svn cp branches/2.0.53-branch tags/2.0.53-r1
# svn rm branches/2.0.53-branch
# svn ci

compared to:

# svn cp tags/2.0.53 tags/2.0.53-r1
# cd tags/2.0.53-r1
# patch  something-that-needs-fixing-now.patch
# svn ci

With the way subversion works, I would have thought the end result would be 
identical...

 Btw, anyone object to swap branches/2.0 with trunk (seeing that 2.1 is 
 dead)? 

No objections here.

  Another by the way, `svn -v log  ChangeLog` for 2.0.53_rc6. I'm open
  to anything better if anybody has a good format for autogeneration. The
  quality of the commit messages themselves isn't really useful though
  without knowing their context, so this might be a bit of a catch 22.. For
  2.0.54, viewsvn should be available so it might be better to just use the
  tracker bug to manually create a summary of notable changes.

 Hmm, so you want to change the ChangeLog into release notes? IMO we
 should have both (a detailed technical ChangeLog and user friendly
 release notes).

I'm not much for the ChangeLog at all really. At least not without going over 
what makes a good commit message and setting up some guidelines. I'm 
definitely for any ChangeLog being autogenerated though.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] The road ahead...

2005-10-21 Thread Jason Stubbs
On Saturday 22 October 2005 10:08, Brian Harring wrote:
 On Sat, Oct 22, 2005 at 12:14:40AM +0900, Jason Stubbs wrote:
  On Friday 21 October 2005 19:06, Marius Mauch wrote:
   Jason Stubbs wrote:
After thinking about it, incremental feature creep does seem like
the best way to go at this late stage in 2.0's life. The problem is
how to guage what is and what is not more trouble than worth. Perhaps
adhering to the kernel's rule of Separate each logical change into
its own patch would help to ease the possible impact of larger
changes?
  
   Probably the best solution.
 
  Brian, you agree on this? It'll mean splitting up the cache patch...

 Would be curious how it would be chunked up...
 patch of the new subsystem, patch of portage.py modifications?

 Depends on def. of logical changes I guess; in the case of the cache
 patch, it's kind of all or none with the changes.
 Suggestions/preferences?

Something like:
* Add base class(es) for new cache framework
* Add cache backend for XYZ database
* Switch portdbapi to the new framework
* Remove old framework

  I'm not much for the ChangeLog at all really. At least not without going
  over what makes a good commit message and setting up some guidelines. I'm
  definitely for any ChangeLog being autogenerated though.

 No ChangeLog will piss off users; dev's already poke about it on each
 release.

Nothing at all will(/has) piss(ed) of users. I can't see people not being 
happy with a concise set of major changes though. I don't mind much either 
way.

 So... guideliness. ?

Should start a new thread about it later. I'd like to get this one finalized 
first. :)

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [PATCH] pre/post phase hooks for users

2005-10-20 Thread Jason Stubbs
On Saturday 15 October 2005 07:05, Brian Harring wrote:
 On Fri, Oct 14, 2005 at 05:02:02PM -0500, Brian Harring wrote:
  Jason, your thoughts on this 53 wise?

 Bleh, pardon, meant .54 for inclussion

Just to be sure it's clear to everybody (although I think Brian knows 
already), my job is not to approve or disapprove of any particular change to 
any particular release. If you want to put a title on it, it'd simply be 
called release executor. Hence, the answer to the above question really 
lies in the outcome of the .54 thread. The only small perk is that any 
suggestions I might have on the release process are quick to be 
integrated. ;)

So, if it's just my input that is being sought after I do have a couple of 
concerns, specifically related to this bit below:

On Tuesday 11 October 2005 17:05, Brian Harring wrote:
 That said, their will be an exemption for java ebuilds due to the fact
 that they're blocked by ebuild.sh env handling- they need ebd for
 things to work properly, and in the meantime this gives them a method
 to have things work properly.  Downside is that the pre/post hooks are
 not available for users for java ebuilds.

Why exactly would their be an exemption for java ebuilds? Are the hooks 
intended to be used with ebuild packaging as well as by users? Wouldn't new 
or altered phases serve ebuild packaging better? If it is for ebuild 
packaging, wouldn't the EAPI need to change? If it's not for ebuild 
packaging, again why the exemption?

On the user side of things, will the hooks continue on into later versions? 
Specifically, with 3.0 supporting hooks on the python side will the bash 
hooks be deprecated? It seems reasonable that both can coexist nicely, so 
this is more just confirmation then anything.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [PATCH] pre/post phase hooks for users

2005-10-20 Thread Jason Stubbs
On Friday 21 October 2005 08:07, Brian Harring wrote:
 Bash hooks would exist in 3.0; they're user specific hooks only, hence
 the bit about java being an evil exception till 3.0 comes to town.  I
 intend to lock down the pre/post hooks prior to ebuild sourcing under
 ebd, so ebuilds/eclasses trying to use those hooks won't be able to.

 In the meantime, it's a nice abuse of a user feature that makes java
 1.4-1.5 stuff work, and works fine when 3.0 autodisables it.

I don't get why it's needed by java ebuilds? Is it a fasttrack to getting 
unsandboxed root access?

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] The road ahead...

2005-10-19 Thread Jason Stubbs
On Monday 17 October 2005 08:25, Zac Medico wrote:
 Jason Stubbs wrote:
  It will likely be that some of the bugs marked against 108262 won't be
  fixed in time. Perhaps it would have been better to just open a metabug
  when the branch is opened and mark bugs against it as they are fixed.

 It's nice to have a list of open bugs against the release where everyone
 can see it.  I suppose this mailing list can serve that purpose though.

That's the thing though.. Most of the bugs we should be looking at fixing are 
15 minute jobs - maybe an hour on the big ones. If bugs are marked as 
blocking as patches are made and committed and then marked as fixed when a 
release is made, it should still be as easy to follow the progress.

Looking closer at some of the bugs I've marked as possible targets for .54, a 
few will likely have to be deferred. That just means more junk mail from 
[EMAIL PROTECTED] and broken hopes for those that had activity on their 
bugs.

 It should be possible to integrate some refactorings+features without too
 much slowdown of the easy bugfix release pace (call it the EBRP for now).
 IMO the timing of such merges should be limited to times that everyone
 (people with commit access) agrees will have a minimal impact on the
  EBRP.
 
  Stabalizing 2.0.53 took 2 weeks - assuming all the regressions are now
  worked out. Perhaps something like a 6 week cycle would be good? 2 weeks
  for any change, 2 weeks for small changes only, 2 weeks stabalizing...

 I'm not so sure about the 2 weeks for *any* change part. We need to be
 very selective about the larger changes.

After thinking about it, incremental feature creep does seem like the best 
way to go at this late stage in 2.0's life. The problem is how to guage what 
is and what is not more trouble than worth. Perhaps adhering to the kernel's 
rule of Separate each logical change into its own patch would help to ease 
the possible impact of larger changes?

  To clarify: Treat backports as regular bugs and go through cycles of open
  the 2.0 branch for fixes for a while followed by stabalizing for a while?

So where shall we head next? 2.0.53_rc6 is tagged and a tarball is making it's 
way to the mirrors. I'm pretty confident that this is the last - that is to 
say, 2.0.53 will essentially just be a renamed tarball - so time's pretty 
much up for deciding on the above. I know I'm itching to start knocking of 
some bugs again...

Speaking of which, if something does crop up with 2.0.53 while the arch guys 
are deciding if it's stable, how should that be dealt with in subversion? 
Continue development under branches/2.0 and, should an issue crop up, `svn cp 
tags/2.0.53_rc6 tags/2.0.53_rc7` and make any required change directly under 
the tag?

Another by the way, `svn -v log  ChangeLog` for 2.0.53_rc6. I'm open to 
anything better if anybody has a good format for autogeneration. The quality 
of the commit messages themselves isn't really useful though without knowing 
their context, so this might be a bit of a catch 22.. For 2.0.54, viewsvn 
should be available so it might be better to just use the tracker bug to 
manually create a summary of notable changes.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] PATCH: Bumping portage to top of dependancy list (bug 48531)

2005-10-18 Thread Jason Stubbs
On Tuesday 18 October 2005 16:49, Zac Medico wrote:
 http://bugs.gentoo.org/show_bug.cgi?id=48531

 This simple patch automatically bumps portage to the top of the merge list.
  I've always wanted this feature and it is a dependency of bug 108262. 
 Feedback please. :)

No good. ;)

What if portage's dependencies aren't satisfied? Something like the below 
(possibly combined with the pprovided stuff) would be better...

--- emerge  (revision 2138)
+++ emerge  (working copy)
@@ -875,8 +875,15 @@
mynewlines.remove(atom)
break

-   return mynewlines
+   mynewlist = []
+   for atom in mynewlines:
+   if portage.dep_getkey(atom) == sys-apps/portage:
+   mynewlist.insert(0, atom)
+   else:
+   mynewlist.append(atom)

+   return mynewlist
+
 def genericdict(mylist):
mynewdict={}
for x in mylist:
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] PATCH: Bumping portage to top of dependancy list (bug 48531)

2005-10-18 Thread Jason Stubbs
On Tuesday 18 October 2005 19:52, Marius Mauch wrote:
 On Tue, 18 Oct 2005 19:32:26 +0900

 Jason Stubbs [EMAIL PROTECTED] wrote:
  On Tuesday 18 October 2005 16:49, Zac Medico wrote:
   http://bugs.gentoo.org/show_bug.cgi?id=48531
  
   This simple patch automatically bumps portage to the top of the
   merge list. I've always wanted this feature and it is a dependency
   of bug 108262. Feedback please. :)
 
  No good. ;)
 
  What if portage's dependencies aren't satisfied? Something like the
  below (possibly combined with the pprovided stuff) would be better...

 Hmm, how would that work with virtual/portage != sys-apps/portage?
 (rhetoric question)

That's where the something like comes into it. What do you expect for less 
than one minute's work? (rhetoric question)

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] The road ahead...

2005-10-16 Thread Jason Stubbs
On Sunday 16 October 2005 14:20, Zac Medico wrote:
 I am very impressed with the number of bugs closed in preparation for the
 2.0.53 release (dependencies of bug 108082) and it seems like it is in
 everyone's best interest to keep up this pace so that all the easy bugs are
 closed ASAP.  There is already a large list of *open* dependencies for the
 2.0.54 (bug 108262) that will hopefully be closed and released soon.  :)

It will likely be that some of the bugs marked against 108262 won't be fixed 
in time. Perhaps it would have been better to just open a metabug when the 
branch is opened and mark bugs against it as they are fixed.

 It should be possible to integrate some refactorings+features without too
 much slowdown of the easy bugfix release pace (call it the EBRP for now). 
 IMO the timing of such merges should be limited to times that everyone
 (people with commit access) agrees will have a minimal impact on the EBRP.

Stabalizing 2.0.53 took 2 weeks - assuming all the regressions are now worked 
out. Perhaps something like a 6 week cycle would be good? 2 weeks for any 
change, 2 weeks for small changes only, 2 weeks stabalizing...

On Saturday 15 October 2005 14:16, Brian Harring wrote:
 Really not much for 2.1 (existing ebd based) being brought up as a
 release line N months down the line, since 2 branches of development
 is a pita enough, let alone 3 with fairly radical differences between
 each branch.  Not saying the path can't be tweaked as we're
 proceeding, but would *really* like to know that as a group/community,
 this is what we're going towards rather then in N different
 directions.

I don't really like the idea of trunk being stabalized either. Too much work 
to bring forward all the changes in stable as well as fixing its own bugs.

On Saturday 15 October 2005 13:59, Brian Harring wrote:
 On Sat, Oct 15, 2005 at 01:45:42PM +0900, Jason Stubbs wrote:
  So, there's pretty much three ways we can go:
 
  1) Backport refactorings+features and release.
  2) Fix more bugs, backport refactorings+features and release.
  3) Fix more bugs, release, backport refactorings+features and release.

 Aside from the 2.1 name being already slightly abused, prefer option
 4, bug/release work, integrating chunks in as they're ready and
 releasing when things are stable.  Basically... when the chunks are
 ready to be integrated, they've been tested (ala cache patch + some
 more time), yank the suckers in, and continue with stabilising towards
 a release.

To clarify: Treat backports as regular bugs and go through cycles of open the 
2.0 branch for fixes for a while followed by stabalizing for a while?

 On the subject of versions, which ever version the chunks get included
 under, they're going to need integration testing, so versioning isn't
 as much an issue to me as time.

 The delta between 2.1 and 2.0, last time I generated it was half a
 meg; pulling chunks from 2.1 into 2.0 requires rewriting a chunk of
 glue, so minimizing the time/delta is something of a concern- eg,
 doing 2.0.* for a while, then a 2.1 I'm not totally much for.

The thing I'm concerned about is the backports and bigger bugs that will 
require a longer stabalizing period. That and the fact that a few of them are 
also quite major in terms of what the user sees. It makes sense to me to 
group these together, bump the version to 2.1 and finally make the version 
numbers meaningful.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



[gentoo-portage-dev] The road ahead...

2005-10-14 Thread Jason Stubbs
Since I've fallen into the terrible pit of trying to make everybody happy 
and since IRC sucks for making decisions due to lack of continuity, let's 
battle it out here. ;)

Where we're at:

branches/2.0/
Branch from which releases are being made. Very stable as far as portage goes.

branches/savior/
Rewrite which will eventually become 3.0. Still in the early stages.

trunk/
I think Brian summed it up quite well. A lot of experimentation, some 
haphazard refactoring, some well done refactoring, some incomplete features 
and some completed features. Currently quite buggy.

At an estimate, trunk would take at least 3 months of dedicated work to 
stabilize. Given the fact that it will be obsoleted by savior, it would not 
be economical to focus our efforts there. That leaves us with savior, 2.0 and 
a set of refactorings and features. Savior is easy enough. Keep working on it 
until it's ready. ;)

Which leaves us with 2.0 and a set of refactorings and features. I think it's 
pretty much decided that these will be backported to 2.0. The only question 
at this stage is when. The only complicating factor here is the current 225 
open bug reports. That and what to call 2.0+refactorings+features.

So, there's pretty much three ways we can go:

1) Backport refactorings+features and release.
2) Fix more bugs, backport refactorings+features and release.
3) Fix more bugs, release, backport refactorings+features and release.

There's still a lot of bugs that can be fixed without too much work, so I'd 
like to go with 2) or 3). I was thinking to go with 3) with the backported 
stuff being named 2.1.0, which is how we arrived at this thread.

Anyway, flame-war time. :D

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Questions about CVS locations and GID...

2005-10-06 Thread Jason Stubbs
On Thursday 06 October 2005 20:51, Ciaran McCreesh wrote:
 On Wed, 5 Oct 2005 21:39:49 -0500 Brian Harring [EMAIL PROTECTED]

 wrote:
 | So bluntly, shut up and let those who you think are being retarded,
 | be retarded.  Discussions on this list regarding those attempts
 | shouldn't be heckled unless you're contributing to those efforts (and
 | I truly mean *contributing*, not trying to punch holes in embryonic
 | efforts that are trying to get off the ground addressing the major
 | issues up front).

 Pfff. By contributing to the design by showing what *won't* work, I'm
 saving a heck of a lot of wasted man hours that will otherwise be spent
 writing a non-working solution.

I must admit that I've simply ignored this thread for the most part, but 
glancing through your emails I can't find any list of reasons why it won't 
work. I can only find statements that it won't work and that the creative 
process should follow a different path (=plan everything first).

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



[gentoo-portage-dev] [PATCH] Use vdb categories when calling vartree.dep_match()

2005-10-03 Thread Jason Stubbs
This (and the last patch both) relates to bug 107982. Essentially, an 
installed package has a category that no longer exists in $PORTDIR's 
categories file. Hence, not specifying the category on the command line won't 
find the package. This patch fixes that.

Note, under my own rules this one should wait until .54. People's thoughts on 
that? :)
Index: portage.py
===
--- portage.py	(revision 2082)
+++ portage.py	(working copy)
@@ -3660,7 +3660,7 @@
 return virts[mykey][0]
 		return mykey
 
-def cpv_expand(mycpv,mydb=None,use_cache=1):
+def cpv_expand(mycpv,mydb=None,use_cache=1,catlist=None):
 	Given a string (packagename or virtual) expand it into a valid
 	cat/package string. Virtuals use the mydb to determine which provided
 	virtual is a valid choice and defaults to the first element when there
@@ -3701,7 +3701,9 @@
 		mykey=None
 		matches=[]
 		if mydb:
-			for x in settings.categories:
+			if not catlist:
+catlist = settings.categories
+			for x in catlist:
 if mydb.cp_list(x+/+myp,use_cache=use_cache):
 	matches.append(x+/+myp)
 		if (len(matches)1):
@@ -3745,7 +3747,7 @@
 	else:
 		return origdep
 
-def dep_expand(mydep,mydb=None,use_cache=1):
+def dep_expand(mydep,mydb=None,use_cache=1,catlist=None):
 	if not len(mydep):
 		return mydep
 	if mydep[0]==*:
@@ -3761,7 +3763,7 @@
 	elif mydep[:1] in =~!:
 		prefix=mydep[:1]
 		mydep=mydep[1:]
-	return prefix+cpv_expand(mydep,mydb=mydb,use_cache=use_cache)+postfix
+	return prefix+cpv_expand(mydep,mydb=mydb,use_cache=use_cache,catlist=catlist)+postfix
 
 def dep_check(depstring,mydbapi,mysettings,use=yes,mode=None,myuse=None,use_cache=1,use_binaries=0):
 	Takes a depend string and parses the condition.
@@ -4823,7 +4825,8 @@
 
 	def match(self,origdep,use_cache=1):
 		caching match function
-		mydep=dep_expand(origdep,mydb=self,use_cache=use_cache)
+		catlist = listdir(self.root+VDB_PATH, dirsonly=True)
+		mydep=dep_expand(origdep,mydb=self,use_cache=use_cache,catlist=catlist)
 		mykey=dep_getkey(mydep)
 		mycat=mykey.split(/)[0]
 		if not use_cache:


[gentoo-portage-dev] [CLEANUP] ebuild

2005-10-03 Thread Jason Stubbs
I've attached the whole file because the patch was pretty much nonsensical.

* Cleaned up initialization
* Check that the ebuild specified is the ebuild that portage will use and
  adjust paths if it isn't.
* Alter clean behavior when FEATURES=noauto to bring it into line with
  Brian's changes to FEATURES=-noauto
* Modified the remainder of the script slightly to match the above changes.


Again, the first two parts (cleanup and ebuild path check) can be dropped if 
people think it is better that they should. However, keeping the devs happy 
is perhaps more important than keeping users happy (or else it'll never go 
stable ;)

Brian, also need you to confirm I haven't altered at all on the -noauto path 
and none incorrectly on the noauto path.


ebuild
Description: application/python


Re: [gentoo-portage-dev] [PATCH] EAPI cleanups and fixes

2005-10-03 Thread Jason Stubbs
On Tuesday 04 October 2005 03:30, Brian Harring wrote:
 On Tue, Oct 04, 2005 at 01:06:35AM +0900, Jason Stubbs wrote:
  Don't like the size of this patch, but it's quite repetitive so...

 Wouldn't worry on the repetitive, it's repetitive due to the fact the
 *dbapi classes don't (ab|)use inheritance...

  * Make all aux_get() functions return a list of strings again

 Why is this a good thing for EAPI?

Consistency. There is no mapping of names to types so any tool that uses 
aux_get to enumerate values and assumes that they are strings (as they have 
always been and still are for every other key) would break.

  * Move the EAPI validity check into a separate function
  * Raise a specific UnsupportedAPIException instead of plain Exception

No problem with these two?

  * Mark metadata of unsupported EAPIs as INVALID rather than -1

 This doesn't really fly imo. You mark it as invalid, and _no_ portage
 version (regardless of it's ability to handle that EAPI) will _ever_
 regenerate that entry.

 An old portage version updating the cache would make certain nodes
 never usable.

 The -1 is wrong, should be -EAPI. 

So negative numbers signals invalid cache entries in the scheme?

 This however is getting back into 
 the eapi should be freeform, not just ints, which I thought I
 clarified why it should be ints (or people shut up instead of
 listening to me argue it). :)

The only reasoning that I recall without checking is that it wouldn't 
complicate certain code paths. That's not a valid reason, in my opinion.

Can leave that debate alone though. s/INVALID/\-1/ in the patch works for me.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Improved user information and communication

2005-10-01 Thread Jason Stubbs
On Sunday 02 October 2005 01:08, Daniel Stiefelmaier wrote:
 On Saturday 01 October 2005 14:17, Jason Stubbs wrote:
 This should go to [EMAIL PROTECTED] as ebuild developers are the ones that
  decide what USE flags are available and how they're documented.

 Of course they decide what flags are available, but how do they decide
 how they are documented? I'm sorry if i did not yet discover gentoos
 ebuild-feature-documentation system.

I meant the actual strings themselves. However, any change that affects or 
enables work for ebuild devs really needs to be discussed with them.

 I thought of a command like emerge mozilla-firefox --useinfo
 that prints what each flag is good for. Maybe some are explained in 5
 words, other may need 5 lines.

Personally, I think that this has only become necessary because USE flags are 
overloaded too much and usually encompass an unintuitive set of 
functionality. In other words, I think the flaw is with the USE flags that 
have been created rather than the USE system itself.

Take the USE flag perl, for example. It has the description Adds 
support/bindings for the Perl language. but for mysql setting it enables the 
installation of some support scripts that just happen to be written in perl.

I'd be much more inclined to push for making the USE flags themselves more 
intuitive rather than adding another layer of documentation to exlain the 
unintuitiveness (which would likely be poorly written anyway).

 also, the advanced could provide
 - links to howto's (like configuring apache)
 
 This is out of the domain of a package in any package management system
  IMO.

 You are right, there may be no pms out there that has this feature.
 I refuse to accept this as an argument to not change that.
 You may be a pro who does not need any HOWTOs, i am not.

HOWTOs can usually be found by navigating from the package's home page.
If not, a quick google will find any that exist.

 Portage saves a lot of work for people who use it. But in some cases,
 ebuilds don't do everything they could or should.

If ebuilds aren't doing what they could or should be, you can open specific 
bugs regarding those packages.

 BTW, if This is out of the domain of a package in any package
 management system, then why do some packages print additional
 information after emerging, like what files should be updated manually?

Usually it's because configuration file layout differs from upstream's 
defaults or some other Gentoo-specific information.

 THIS is not really the solution. if you batch-emerge or the package that
 wants to tell you something is just a dependancy, then you may probably
 miss that note.

http://bugs.gentoo.org/show_bug.cgi?id=11359

 Another reason to inform the user before emerging maybe this:
 I emerged a package recently, think it was amule, that first emerged
 some dependancies, including wxGTK.
 Then, after all dependancies were emerged, the package itself quitted
 saying that wxGTK needs to be emerged with flag wxgtk1.
 I thought the ebuild could manage flags of dependancies automatically,
 but that is not the point.
 The user could be informed before starting to emerge.

http://bugs.gentoo.org/show_bug.cgi?id=2272

 - list packages that are of use for this (plugins or utilized apps like
 cervisia that integrates in quanta plus)
 
 Do you mean finding packages that depend on that package in question?

 Not necessarily. Cervisia, Kompare and Tidy are standalone applications,
 but Quanta integrates them automatically if they are installed.

A related field? Could be useful. This is also a point for discussion on 
[EMAIL PROTECTED] though.

 Personally, I hate most wikis. 9 times out of 10, they are full of
 half-correct information. This makes them all the more dangerous as the
 average Joe can't tell what's correct and what isn't.

 My wiki experiences are all good, but i also wrote, that it could only
 be maintained by the developers. Only the discussion page attached to
 each page is open for comments. This also prevents defacements.

More work for ebuild devs, then. Guess where this discussion should be? ;)

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Improved user information and communication

2005-09-30 Thread Jason Stubbs
On Saturday 01 October 2005 14:01, Daniel Stiefelmaier wrote:
 I'd like to introduce two ideas i had to improve portage.

I haven't had an idea introduced to me that was new for at least a year.. 
Patches are more interesting.

 sometimes i'd like to know something more about a package before i
 emerge it.

 1. release notes. (for the ebuild, not the application)
 as a release notes file exists, a parameter could be used to show the
 most recent entry.
 maybe this fits eix better than emerge

emerge --changelog ?

 2. advanced package info
 most important here is an explanation of the available use flags. i know
 there is a list of most flags (http://www.gentoo.org/dyn/use-index.xml)
 but it is incomplete and doesnt explain some flags well. some have
 different behaviour in different packages.
 some flags are self-explanatory, but some are not. for example, the
 xprint comment should say, that this is not necessary to print, and
 under what circumstances it is needed.

This should go to [EMAIL PROTECTED] as ebuild developers are the ones that 
decide 
what USE flags are available and how they're documented.

 also, the advanced could provide
 - links to howto's (like configuring apache)

This is out of the domain of a package in any package management system IMO.

 - list packages that are of use for this (plugins or utilized apps like
 cervisia that integrates in quanta plus)

Do you mean finding packages that depend on that package in question?

 - tell how to contact the ebuild maintainer

metadata.xml. This information is printed out when using -vv in 
portage-2.1_alpha.

 that led my to another idea: a wiki for all ebuilds in portage.
 all the information could then be in the wiki, and eix/emerge just print
 the wiki-link.
 i'd suggest mediawiki, as it has a discussion page attached.
 maybe the wiki itself can only be changed by developers, and feedback is
 given on the discussion page.

Personally, I hate most wikis. 9 times out of 10, they are full of 
half-correct information. This makes them all the more dangerous as the 
average Joe can't tell what's correct and what isn't.

 just some ideas, maybe needing some development, i hope you find them
 useful. :)

Ideas are a dime, a dozen.

--
Jason Stubbs
-- 
gentoo-portage-dev@gentoo.org mailing list



[gentoo-portage-dev] [PATCH] Friendly up depclean a bit

2005-09-29 Thread Jason Stubbs
Get's rid of all the yelling from the depclean message and shortens it a bit 
too. Also got rid of the compulsory long delay (CLEAN_DELAY still takes 
affect on non-pretend run). Lastly, made the new sanity checks a long delay 
rather than immediately dieing - what if one wanted to take an existing 
system back to a stage3?
diff -uNr 2.0-original/bin/emerge 2.0/bin/emerge
--- 2.0-original/bin/emerge	2005-09-29 22:28:38.0 +0900
+++ 2.0/bin/emerge	2005-09-29 23:14:28.0 +0900
@@ -2986,33 +2986,37 @@
 	# dependency of another package. World file is explicit.
 
 	print
-	print red(*** WARNING ***)+ : DEPCLEAN CAN  SERIOUSLY  IMPAIR YOUR SYSTEM. USE CAUTION.
-	print red(*** WARNING ***)+ : (Cancel: CONTROL-C) -- ALWAYS VERIFY ALL PACKAGES IN THE
-	print red(*** WARNING ***)+ : CANDIDATE LIST FOR  SANITY  BEFORE  ALLOWING DEPCLEAN TO
-	print red(*** WARNING ***)+ : UNMERGE ANY PACKAGES.
-	print red(*** WARNING ***)+ :
-	print red(*** WARNING ***)+ : USE FLAGS MAY HAVE AN EXTREME EFFECT ON THE OUTPUT.
-	print red(*** WARNING ***)+ : SOME LIBRARIES MAY BE USED BY PACKAGES BUT ARE NOT
-	print red(*** WARNING ***)+ : CONSIDERED TO BE A DEPEND DUE TO USE FLAG SETTINGS.
-	print red(*** WARNING ***)+ : emerge --update --deep --newuse world TO VERIFY
-	print red(*** WARNING ***)+ : SANITY IN THIS REGARD.
-	print red(*** WARNING ***)+ :
-	print red(*** WARNING ***)+ : Packages  in the list  that are  desired  may be added
-	print red(*** WARNING ***)+ : directly to the world file to cause them to be ignored
-	print red(*** WARNING ***)+ : by depclean and maintained in the future. BREAKAGES DUE
-	print red(*** WARNING ***)+ : TO UNMERGING AN  ==IN-USE LIBRARY==  MAY BE REPAIRED BY
-	print red(*** WARNING ***)+ : MERGING  *** THE PACKAGE THAT COMPLAINS ***  ABOUT THE
-	print red(*** WARNING ***)+ : MISSING LIBRARY.
-	print
-	if (--pretend not in myopts) and (--ask not in myopts):
-		countdown(EMERGE_WARNING_DELAY,  Depclean)
-		emergelog(  depclean)
+	print red(*** WARNING ***)+  --depclean is known to be broken. It is highly recommended
+	print red(*** WARNING ***)+  that +green(`emerge --update --newuse --deep world`)+ be ran before
+	print red(*** WARNING ***)+  commencing. However, using --depclean may still break link
+	print red(*** WARNING ***)+  level consistency within your system. +green(`revdep-rebuild`)
+	print red(*** WARNING ***)+  from app-portage/gentoolkit can help to detect breakage.
+	print red(*** WARNING ***)
+	print red(*** WARNING ***)+  Also study the list of packages to be cleaned for any
+	print red(*** WARNING ***)+  obvious mistakes. Packages can be manually added to the
+	print red(*** WARNING ***)+  world list by running +green(`emerge --noreplace atom`)+.
+	print red(*** WARNING ***)
+	print red(*** WARNING ***)+  +bold(Make sure you have a backup.)
 
-	mydepgraph=depgraph(myaction,myopts)
 	syslist=getlist(system)
 	worldlist=getlist(world)
+	myvarlist=portage.vardbapi(portage.root).cp_all()
+
+	if not syslist:
+		print \n!!! You have no system list.,
+	if not worldlist:
+		print \n!!! You have no world file.,
+	if not myvarlist:
+		print \n!!! You have no installed package database (%s). % portage.VDB_PATH,
+
+	if not (syslist and worldlist and myvarlist):
+		print \n!!! Proceeding will break your installation.\n
+		countdown(EMERGE_WARNING_DELAY,  Depclean)
+
+	emergelog(  depclean)
+	mydepgraph=depgraph(myaction,myopts)
 
-	print Calculating,myaction,dependencies  ,
+	print \nCalculating dependencies  ,
 	if not mydepgraph.xcreate(world):
 		print \n!!! Failed to create deptree.
 		sys.exit(1)
@@ -3026,19 +3030,9 @@
 		sys.exit(1)
 
 	alldeps=mydepgraph.digraph.allnodes()
-	myvarlist=portage.vardbapi(portage.root).cp_all()
 
-	if not syslist:
-		print !!! You have no system list. Cannot determine system from world.
-	if not worldlist:
-		print !!! You have no world file. Cannot determine explicit merges.
-	if not myvarlist:
-		print !!! You have no installed package tree (%s). This is a problem. % portage.VDB_PATH
 	if not alldeps:
 		print !!! You have no dependencies. Impossible. Bug.
-
-	if not (syslist and worldlist and myvarlist and alldeps):
-		print
 		sys.exit(1)
 
 	reallist=[]


[gentoo-portage-dev] [PATCH] Ignore system packages that are in package.provided

2005-09-27 Thread Jason Stubbs
Removes any atoms that are satisfied by package.provided in emerge's getlist() 
function.
diff -uNr 2.0/bin/emerge 2.0-patched/bin/emerge
--- 2.0/bin/emerge	2005-09-27 13:16:09.0 +0900
+++ 2.0-patched/bin/emerge	2005-09-27 15:00:23.0 +0900
@@ -861,6 +861,16 @@
 continue
 			myline=myline[1:]
 		mynewlines.append(myline.strip())
+
+	# Remove everything that is package.provided from our list
+	for atom in mynewlines[:]:
+		for expanded_atom in portage.flatten(portage.dep_virtual([atom], portage.settings)):
+			mykey = portage.dep_getkey(expanded_atom)
+			if portage.settings.pprovideddict.has_key(mykey) and \
+portage.match_from_list(expanded_atom, portage.settings.pprovideddict[mykey]):
+	mynewlines.remove(atom)
+	break
+
 	return mynewlines
 
 def genericdict(mylist):


[gentoo-portage-dev] [PATCH] Extra info about installed packages feature

2005-09-27 Thread Jason Stubbs
This patch is by swegener. It allows one to specify atoms after `emerge info` 
that will be matched to installed packages. Any installed packages found that 
have settings differing to the current settings will have those settings 
printed out along with the global info.
diff -u -r1.345.2.37 emerge
--- bin/emerge	5 Aug 2005 04:15:00 -	1.345.2.37
+++ bin/emerge	11 Aug 2005 14:26:16 -
@@ -2838,6 +2879,8 @@
 	unameout=commands.getstatusoutput(uname -mrp)[1]
 	print getportageversion()
 	print =
+	printSystem Settings
+	print =
 	print System uname: +unameout
 	if os.path.exists(/etc/gentoo-release):
 		os.system(cat /etc/gentoo-release)
@@ -2914,6 +2957,61 @@
 			if cvs_id_string in dir(module):
 print %s: %s % (str(x), str(module.cvs_id_string))
 
+	# See if we can find any packages installed matching the strings
+	# passed on the command line
+	mypkgs = {}
+	for x in myfiles:
+		for y in portage.db[portage.root][vartree].dbapi.match(x):
+			if y:
+mypkgs[y] = True
+	mypkgs = mypkgs.keys()
+	mypkgs.sort()
+
+	# If some packages were found...
+	if mypkgs:
+		# Get our global settings (we only print stuff if it varies from
+		# the current config)
+		mydesiredvars = [ 'CHOST', 'CFLAGS', 'CXXFLAGS', 'USE' ]
+
+		# Loop through each package
+		# Only print settings if they differ from global settings
+		header_printed = False
+		for pkg in mypkgs:
+
+			# Get the directory where the files are stored
+			prefix = os.path.join(portage.root, portage.VDB_PATH, pkg)
+
+			# Get all package specific variables
+			tmp = portage.db[portage.root][vartree].dbapi.aux_get(pkg, mydesiredvars)
+			diff_found = False
+			for i in range(len(mydesiredvars)):
+# If the package variable doesn't match the
+# current global variable, something has changed
+# so set diff_found so we know to print
+if tmp[i] != portage.settings[mydesiredvars[i]]:
+	diff_found = True
+
+			# If a difference was found, print the info for
+			# this package.
+			if diff_found:
+
+# If we have not yet printed the header, 
+# print it now
+if not header_printed:
+	print =
+	printPackage Settings
+	print =
+	header_printed = True
+
+# Print package info
+print %s was built with the following: % pkg
+for i in range(len(mydesiredvars)):
+	if tmp[i] != portage.settings[mydesiredvars[i]]:
+		print %s=\%s\ % (mydesiredvars[i], tmp[i])
+print 
+diff_found = False
+
+
 # SEARCH action
 elif search==myaction:
 	if not myfiles:


[gentoo-portage-dev] [PATCH] More info when dependencies can't be satisfied

2005-09-27 Thread Jason Stubbs
This patch has two parts. The first adds information to show what package 
brought in the offending atom. The second part clarifies that the problem 
with ebuild foo is talking about the world/system/cli package rather than 
the actual package requiring the unsatisfiable atom.
diff -u -r1.345.2.37 emerge
--- bin/emerge	5 Aug 2005 04:15:00 -	1.345.2.37
+++ bin/emerge	11 Aug 2005 14:26:16 -
@@ -1256,6 +1264,8 @@
 			print !!! Either add a suitable binary package or compile from an ebuild.
 	else:
 		print \nemerge: there are no ebuilds to satisfy +xinfo+.
+		if myparent:
+			print xfrom
 		print
 	return 0
 
@@ -1423,7 +1433,7 @@
 
 			if not self.create(myk,myuse=binpkguseflags):
 print
-print !!! Problem with,myk[0],myk[2]
+print !!! Problem with dependencies under ,myk[0],myk[2]
 print !!! Possibly a DEPEND/*DEPEND problem.
 print
 return 0


[gentoo-portage-dev] [PATCH] Ignore blockers when fetching and using --ask

2005-09-27 Thread Jason Stubbs
Subject says it all...
diff -uNr 2.0/bin/emerge 2.0-patched/bin/emerge
--- 2.0/bin/emerge	2005-09-27 13:16:09.0 +0900
+++ 2.0-patched/bin/emerge	2005-09-27 15:18:53.0 +0900
@@ -3173,7 +3173,7 @@
 if x[3]!=nomerge:
 	mergecount+=1
 #check for blocking dependencies
-if x[0]==blocks:
+if x[0]==blocks and --fetchonly not in myopts and --fetch-all-uri not in myopts:
 	print \n!!! Error: The above package list contains packages which cannot be installed
 	print   !!!on the same system.
 	print


Re: [gentoo-portage-dev] [PATCH] Ignore blockers when fetching and using --ask

2005-09-27 Thread Jason Stubbs
While not harmful, there was one issue with the previous patch; the parent of 
the blocking package would be fetched as well. This would usually mean that 
one of the packages being merged would be fetched twice. This patch fixes 
that.
diff -uNr 2.0/bin/emerge 2.0-patched/bin/emerge
--- 2.0/bin/emerge	2005-09-27 13:16:09.0 +0900
+++ 2.0-patched/bin/emerge	2005-09-27 15:34:03.0 +0900
@@ -3173,7 +3173,7 @@
 if x[3]!=nomerge:
 	mergecount+=1
 #check for blocking dependencies
-if x[0]==blocks:
+if x[0]==blocks and --fetchonly not in myopts and --fetch-all-uri not in myopts:
 	print \n!!! Error: The above package list contains packages which cannot be installed
 	print   !!!on the same system.
 	print
@@ -3224,7 +3224,14 @@
 		y=portage.portdb.findname(pkgline[2])
 		tmpsettings = portage.config(clone=portage.settings)
 		retval=portage.doebuild(y,digest,portage.root,tmpsettings,edebug,(--pretend in myopts))
-			mydepgraph.merge(mydepgraph.altlist())
+			if --fetchonly in myopts or --fetch-all-uri in myopts:
+pkglist = []
+for pkg in mydepgraph.altlist():
+	if pkg[0] != blocks:
+		pkglist.append(pkg)
+			else:
+pkglist = mydepgraph.altlist()
+			mydepgraph.merge(pkglist)
 
 		if portage.mtimedb.has_key(resume):
 			del portage.mtimedb[resume]


[gentoo-portage-dev] [PATCH] Kill the smileys!

2005-09-27 Thread Jason Stubbs
Kill 'em all!
(Original patch by solar)
diff -uNr 2.0/pym/portage.py 2.0-patched/pym/portage.py
--- 2.0/pym/portage.py	2005-09-26 11:48:15.0 +0900
+++ 2.0-patched/pym/portage.py	2005-09-27 16:06:21.0 +0900
@@ -1864,7 +1864,7 @@
 	fetched=0
 else:
 	for x_key in mydigests[myfile].keys():
-		writemsg( Previously fetched file: +str(myfile)+ +x_key+ ;-)\n)
+		writemsg( Previously fetched file: +str(myfile)+ +x_key+\n)
 	fetched=2
 	break #No need to keep looking for this file, we have it!
 	else:
@@ -1962,8 +1962,6 @@
 	os.unlink(mysettings[DISTDIR]+/+myfile)
 	fetched=0
 else:
-	for x_key in mydigests[myfile].keys():
-		writemsg( +str(myfile)+ +x_key+ ;-)\n)
 	fetched=2
 	break
 		except (OSError,IOError),e:


Re: [gentoo-portage-dev] [PATCH] Kill the smileys!

2005-09-27 Thread Jason Stubbs
On Wednesday 28 September 2005 12:15, Marius Mauch wrote:
 Jason Stubbs wrote:
  Kill 'em all!
  (Original patch by solar)

 Hmm, I actually liked them.

Even now that FEATURES=strict is default and every package results in a 
screenful of everything's alright, mate ;-) ?
-- 
gentoo-portage-dev@gentoo.org mailing list



[gentoo-portage-dev] [PATCH]es -- several of 'em

2005-09-26 Thread Jason Stubbs
Hi all,

Bunch of patches here that have been sitting on my disk for a couple of 
months. There's actually a few more beyond this, but they'll likely need a 
little discussion. These are pretty much all open and shut. Acks would still 
be nice though.


DUAL-remove-CDEPEND-and-disable-RDEPENDs-on-buildpkgonly.patch

Kills of the reading of CDEPEND as it's not used at all. The second part 
clears RDEPEND and PDEPEND when --build-pkg-only is specified as these deps 
are not required to build the package(s).


add-newuse-to-help.patch

Completes the monstrosity that is `emerge --help` by adding the short hand 
option for --newuse.


check-world-writable-files.patch

Submitted by solar. Sets o-w on any u+s or g+s files and warns on any other 
file that is o+w.


correct-sizes-in-pretend-fetch.patch

Makes fetch() use the supplied USE flags rather than blanketly overriding 
them. This was causing emerge -p output to be incorrect when package.use came 
into play.


disallow-relative-globs.patch

Present portage looks at =sys-apps/portage-2.0* as being valid. It's not.


dispatch-conf-typo-fix.patch

Harmless typo fix.


dist-mirror-update.patch

Decaptilizes Linux to match the upstream change.


ignore-pprovided-system-packages.patch

Removes anything in system that is satisfied by package.provided.


--
Jason Stubbs
diff -u -r1.201.2.40 ebuild.sh
--- bin/ebuild.sh	9 Aug 2005 11:25:44 -	1.201.2.40
+++ bin/ebuild.sh	11 Aug 2005 14:26:16 -
@@ -1017,12 +1017,24 @@
 	for i in $(find ${D}/ -type f -perm -2002); do
 		((UNSAFE++))
 		echo UNSAFE SetGID: $i
+		chmod -s,o-w $i
 	done
 	for i in $(find ${D}/ -type f -perm -4002); do
 		((UNSAFE++))
 		echo UNSAFE SetUID: $i
+		chmod -s,o-w $i
 	done
 	
+	# Now we look for all world writable files.
+	for i in $(find ${D}/ -type f -perm -2); do
+		echo -ne '\a'
+		echo QA Security Notice:
+		echo - ${i:${#D}:${#i}} will be a world writable file.
+		echo - This may or may not be a security problem, most of the time it is one.
+		echo - Please double check that $PF really needs a world writeable bit and file bugs accordingly.
+		sleep 1
+	done
+
 	if type -p scanelf  /dev/null ; then
 		# Make sure we disallow insecure RUNPATH/RPATH's
 		# Don't want paths that point to the tree where the package was built
diff -u -r1.8.2.2 emergehelp.py
--- pym/emergehelp.py	16 Jan 2005 02:35:33 -	1.8.2.2
+++ pym/emergehelp.py	11 Aug 2005 14:26:17 -
@@ -15,7 +15,7 @@
 	print+turquoise(emerge)+  +turquoise(--sync)+ | +turquoise(--metadata)+ | +turquoise(--info)+ 
 	print+turquoise(emerge)+ +turquoise(--resume)+ [ +green(--pretend)+ | +green(--ask)+ | +green(--skipfirst)+ ]
 	print+turquoise(emerge)+ +turquoise(--help)+ [ +green(system)+ | +green(config)+ | +green(sync)+ ] 
-	print bold(Options:)+ +green(-)+[+green(abcCdDefhikKlnoOpPsSuUvV)+] [+green(--oneshot)+] [+green(--newuse)+] [+green(--noconfmem)+]
+	print bold(Options:)+ +green(-)+[+green(abcCdDefhikKlnNoOpPsSuUvV)+] [+green(--oneshot)+] [+green(--newuse)+] [+green(--noconfmem)+]
 	print  [+green(--columns)+] [+green(--nospinner)+]
 	print bold(Actions:)+ [ +green(--clean)+ | +green(--depclean)+ | +green(--inject)+ | +green(--prune)+ | +green(--regen)+ | +green(--search)+ | +green(--unmerge)+ ]
 	print
@@ -185,7 +185,7 @@
 		print   downloaded from the remote server without consulting packages
 		print   existing in the packages directory.
 		print
-		print+green(--newuse)
+		print+green(--newuse)+ (+green(-N)+ short option)
 		print   Tells emerge to include installed packages where USE flags have 
 		print   changed since installation.
 		print
diff -u -r1.524.2.76 portage.py
--- pym/portage.py	29 May 2005 12:40:08 -	1.524.2.76
+++ pym/portage.py	11 Aug 2005 14:26:18 -
@@ -3055,6 +3057,8 @@
 	mycpv_cps = catpkgsplit(dep_getcpv(atom))
 	operator = get_operator(atom)
 	if operator:
+		if operator[0] in  and atom[-1] == *:
+			return 0
 		if mycpv_cps and mycpv_cps[0] != null:
 			# =cat/pkg-1.0
 			return 1
diff -u -r1.524.2.76 portage.py
--- pym/portage.py	29 May 2005 12:40:08 -	1.524.2.76
+++ pym/portage.py	11 Aug 2005 14:26:18 -
@@ -5389,7 +5398,8 @@
 			print red(getfetchlist():)+ aux_get() error reading +mypkg+; aborting.
 			sys.exit(1)
 
-		useflags = string.split(mysettings[USE])
+		if useflags is None:
+			useflags = string.split(mysettings[USE])
 
 		myurilist = portage_dep.paren_reduce(myuris)
 		myurilist = portage_dep.use_reduce(myurilist,uselist=useflags,matchall=all)
diff -u -r1.7.2.10 dispatch-conf
--- bin/dispatch-conf	12 May 2005 15:20:22 -	1.7.2.10
+++ bin/dispatch-conf	11 Aug 2005 14:26:15 -
@@ -48,7 +48,7 @@
 
 # Ensure the scratch dir is deleted
 def cleanup(mydir=SCRATCH_DIR):
-shutil.rmtree(SCRATCH_DIR)
+shutil.rmtree(mydir)
 atexit.register(cleanup)
 
 MANDATORY_OPTS  = [ 'archive-dir', 'diff', 'replace-cvs', 'replace-wscomments', 'merge' ]
diff -u

[gentoo-portage-dev] [PATCH] ignore files within categories in portdbapi.cp_all()

2005-09-26 Thread Jason Stubbs
cp_all() currently includes category metadata.xml and any other files that 
happen to be there. This patch limits cp_all() to directories.
diff -u -r1.524.2.76 portage.py
--- pym/portage.py	29 May 2005 12:40:08 -	1.524.2.76
+++ pym/portage.py	11 Aug 2005 14:26:18 -
@@ -5478,7 +5488,8 @@
 			for oroot in self.porttrees:
 for y in listdir(oroot+/+x,EmptyOnError=1,ignorecvs=1):
 	mykey=x+/+y
-	d[x+/+y] = None
+	if os.path.isdir(oroot+/+mykey):
+		d[mykey] = None
 		l = d.keys()
 		l.sort()
 		return l


Re: [gentoo-portage-dev] [PATCH] resend of 99616 overridable lchown/lchgrp

2005-09-26 Thread Jason Stubbs
Getting rid of the magic constants.
diff -uNr 2.0/bin/ebuild.sh 2.0-patched/bin/ebuild.sh
--- 2.0/bin/ebuild.sh	2005-09-26 11:48:16.0 +0900
+++ 2.0-patched/bin/ebuild.sh	2005-09-27 13:37:33.0 +0900
@@ -83,6 +83,15 @@
 	export SANDBOX_PREDICT=$SANDBOX_PREDICT:$1
 }
 
+lchown()
+{
+	chown -h $@
+}
+
+lchgrp()
+{
+	chgrp -h $@
+}
 
 # source the existing profile.bashrc's.
 save_IFS
@@ -996,6 +1005,8 @@
 }
 	
 
+PORTAGE_INST_UID=0
+PORTAGE_INST_GID=0
 
 dyn_install() {
 	trap abort_install SIGINT SIGQUIT
@@ -1133,15 +1144,17 @@
 	local count=0
 	find ${D}/ -user  portage | while read file; do
 		count=$(( $count + 1 ))
-		if [ ! -L ${file} ]; then
-			s=$(stat_perms $file)
+		if [ -L ${file} ]; then
+			lchown ${PORTAGE_INST_UID} ${file}
+		else
+			s=$(stat_perms $file)
 			if [ -z ${s} ]; then
 ewarn failed stat_perm'ing $file.  User intervention during install isn't wise...
 continue
 			fi
+			chown ${PORTAGE_INST_UID} $file
+			chmod $s $file
 		fi
-		chown root $file
-		[[ ! -L ${file} ]]  chmod $s $file
 	done
 	if (( $count  0 )); then
 		ewarn $count files were installed with user portage!
@@ -1150,15 +1163,17 @@
 	count=0
 	find ${D}/ -group portage | while read file; do
 		count=$(( $count + 1 ))
-		if [ ! -L ${file} ]; then
+		if [ -L ${file} ]; then
+			lchgrp ${PORTAGE_INST_GID} ${file}
+		else
 			s=$(stat_perms $file)
 			if [ -z ${s} ]; then
 echo failed stat_perm'ing '$file' . User intervention during install isn't wise...
 continue
 			fi
+			chgrp ${PORTAGE_INST_GID} $file
+			chmod $s $file
 		fi
-		chgrp 0 ${file}
-		[[ ! -L ${file} ]]  chmod $s $file
 	done
 	if (( $count  0 )); then
 		ewarn $count files were installed with group portage!


Re: [gentoo-portage-dev] [PATCH] ignore files within categories in portdbapi.cp_all()

2005-09-26 Thread Jason Stubbs
Kill off the extra stat call by moving the filtering into listdir.
(Brian wrote the bulk of this patch.)
diff -uNr 2.0/pym/portage.py 2.0-patched/pym/portage.py
--- 2.0/pym/portage.py	2005-09-26 11:48:15.0 +0900
+++ 2.0-patched/pym/portage.py	2005-09-27 13:52:49.0 +0900
@@ -272,7 +272,7 @@
 
 
 def listdir(mypath, recursive=False, filesonly=False, ignorecvs=False, ignorelist=[], followSymlinks=True,
-	EmptyOnError=False):
+	EmptyOnError=False, dirsonly=False):
 
 	list, ftype = cacheddir(mypath, ignorecvs, ignorelist, EmptyOnError, followSymlinks)
 
@@ -302,6 +302,11 @@
 		for x in range(0,len(ftype)):
 			if ftype[x]==0:
 rlist=rlist+[list[x]]
+	elif dirsonly:
+		rlist = []
+		for x in range(0, len(ftype)):
+			if ftype[x] == 1:
+rlist = rlist + [list[x]]	
 	else:
 		rlist=list
 
@@ -5544,8 +5549,7 @@
 		d={}
 		for x in self.mysettings.categories:
 			for oroot in self.porttrees:
-for y in listdir(oroot+/+x,EmptyOnError=1,ignorecvs=1):
-	mykey=x+/+y
+for y in listdir(oroot+/+x,EmptyOnError=1,ignorecvs=1,dirsonly=1):
 	d[x+/+y] = None
 		l = d.keys()
 		l.sort()


[gentoo-portage-dev] [PATCH] Typo in PORT_LOGDIR validity check

2005-09-26 Thread Jason Stubbs
Changes ValueError into OSError so that the except: block can do it's thing.
diff -u -r1.524.2.76 portage.py
--- pym/portage.py	29 May 2005 12:40:08 -	1.524.2.76
+++ pym/portage.py	11 Aug 2005 14:26:18 -
@@ -2595,7 +2597,7 @@
 		mysettings[LOG_PF]=mysettings[PF]
 		mysettings[LOG_COUNTER]=str(db[myroot][vartree].dbapi.get_counter_tick_core(/))
 	logfile=%s/%s-%s.log % (mysettings[PORT_LOGDIR],mysettings[LOG_COUNTER],mysettings[LOG_PF])
-except ValueError, e:
+except OSError, e:
 	mysettings[PORT_LOGDIR]=
 	print !!! Unable to chown/chmod PORT_LOGDIR. Disabling logging.
 	print !!!,e


[gentoo-portage-dev] [PATCH] 105352: ldconfig needs an -i option on FreeBSD

2005-09-26 Thread Jason Stubbs
Subject says it all...
diff -u -r1.524.2.76 portage.py
--- pym/portage.py	29 May 2005 12:40:08 -	1.524.2.76
+++ pym/portage.py	11 Aug 2005 14:26:18 -
@@ -609,7 +609,7 @@
 	elif ostype == FreeBSD:
 		if (ld_cache_update):
 			writemsg( Regenerating +str(root)+var/run/ld-elf.so.hints...\n)
-			commands.getstatusoutput(cd / ; /sbin/ldconfig -elf -f +str(root)+var/run/ld-elf.so.hints +str(root)+etc/ld.so.conf)
+			commands.getstatusoutput(cd / ; /sbin/ldconfig -i -elf -f +str(root)+var/run/ld-elf.so.hints +str(root)+etc/ld.so.conf)
 
 	del specials[LDPATH]
 


  1   2   >