Re: [gentoo-portage-dev] Try to specify how to get that a USE flag is present in current ebuild

2012-09-21 Thread Brian Harring
On Fri, Sep 21, 2012 at 12:45:30PM -0700, Zac Medico wrote:
 On 09/21/2012 12:08 PM, Pacho Ramos wrote:
  Hello
  
  This comes from this gentoo-dev thread:
  http://www.gossamer-threads.com/lists/gentoo/dev/260536
  
  In that one, we try to use the following:
  has vala ${IUSE//+/}  ! use vala  return 0 
  
  as already done in many eclasses/ebuilds. The problem is that Ciaran
  wants to forbid it because he says it's not specified in PMS. My
  suggestion was to simply specify it as it's currently implemented in
  portage because that functionality is (apart of needed) being used for a
  long time in the tree by numerous eclasses/ebuilds, then, from my point
  of view, wouldn't be any sense on lose time for moving them to current
  functionality to a worse one, wait for the next eapi and, finally,
  revert them back to current behavior.
  
  The problem is that I cannot find any doc about how this is currently
  handled in portage. Could you help me on it please?
 
 That `has vala ${IUSE//+/}` thing should work for all versions of

*cough* negated defaults; you need a - in addition.

~harring



Re: [gentoo-portage-dev] Is there any short syntax for REQUIRED_USE when a lot of USE flags need another one enabled?

2011-12-17 Thread Brian Harring
On Sat, Dec 17, 2011 at 11:24:37AM +0100, Pacho Ramos wrote:
 I am referring in this case to abiword, it has a plugins USE flag that
 enables some minimal set of plugins and, then, a lot of USE flags for
 building extra plugins (with extra dependencies). All of this extra
 plugins need plugins USE flag to be enabled. Is there any way to write
 a REQUIRED_USE flag variable without needing to list all USE flags
 depending on plugins to be set?

I think the better question is why you have a plugin use flag if 
the vast majority of interesting flags require it.

What's the gain of having plugin controllable, vs forced on by default 
(or force on by one of the flags you referenced being enabled)?

~brian



Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

2011-10-02 Thread Brian Harring
On Sun, Oct 02, 2011 at 02:10:09PM -0700, Zac Medico wrote:
 On 10/02/2011 01:54 PM, Robin H. Johnson wrote:
  On Sun, Oct 02, 2011 at 01:39:41PM -0700, Zac Medico wrote:
  On 10/02/2011 05:46 AM, Robin H. Johnson wrote:
  On Sat, Oct 01, 2011 at 09:40:13PM -0700, Zac Medico wrote:
  If we control these hashes via metadata/layout.conf, then we can toggle
  it atomically for all commiters. Otherwise, we'll have an annoying
  period of time where different committers are committing different sets
  of hashes, depending on their portage version.
  How do you suggest doing it via layout.conf? I've kept SHA256 in both
  sets for now, but if you could enforce new signatures including both
  WHIRLPOOL and SHA256, that would be great.
  How about if we put something like this in
  gentoo-x86/metadata/layout.conf now:
  Did you mean profiles/layout.conf? I just want to make sure no scripts
  that pull from CVS and expect that dir to not exist don't break.
 
 No, it's metadata/layout.conf. I didn't choose the location. We actually
 inherited it from paludis about 1.5 years ago:
 
 
 http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=f16aee82cefa95e9903fa46f448d30f6d4350f64
 
 We're also using it to control thin-manifest support, among other things
 now:
 
   https://bugs.gentoo.org/show_bug.cgi?id=333691
 
 manifest2-sha1 = true
 manifest2-whirlpool = false
  Bikeshedding slightly, but can we figure something like a list or dict
  instead? (Also gives us a chance to make the required hashes a list).
  manifest2-hashes = ['SHA1', 'SHA256', 'RMD160']
 
 Well, booleans are simpler. Also, note that I designed them to be
 removed from layout.conf eventually, which means that we will accumulate
 less bloat in layout.conf over time.

Should use a space delimited list instead named hashes instead; those 
being the hashes that should be generated, and that can be /used/.  
Not in the list, not an acceptable hash (even if a manifest2 carries 
that data).

If it's not set, then the pm defaults in a list; that default list 
should be tracked somewhere (rather than just whatever the PM author 
decides) also, although that's a seperate discussion.

Breaking it out into individual booleans isn't particularly great; we 
use lists for masters, a tristate for use-manifest, etc.  Having each 
CHF controlled by a seperate boolean adds more toggles than is worth 
it imo, and having the manifest2- prefix makes the parsing slightly 
more complex while also making the key name a bit daft if we ever 
switch to a manifest3. ;) 

~harring



Re: [gentoo-portage-dev] [GLEP59v2 2/5] Manifest2 hash: Whirlpool

2011-10-01 Thread Brian Harring
On Sat, Oct 01, 2011 at 07:40:52AM +, Robin H. Johnson wrote:
 From: Robin H. Johnson robb...@gentoo.org
 
 Provide public-domain implementation of the Whirlpool hash algorithm to
 be used as new Manifest2 hash.
 
 Signed-off-by: Robin H. Johnson robb...@gentoo.org
 ---
  pym/portage/checksum.py |8 ++--
  1 files changed, 6 insertions(+), 2 deletions(-)
 
 diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
 index e5455fa..3593686 100644
 --- a/pym/portage/checksum.py
 +++ b/pym/portage/checksum.py
 @@ -71,6 +71,10 @@ except ImportError:
  
  sha1hash = _generate_hash_function(SHA1, _new_sha1, origin=internal)
  
 +# Bundled WHIRLPOOL implementation
 +from portage.util.whirlpool import new as _new_whirlpool
 +whirlpoolhash = _generate_hash_function(WHIRLPOOL, _new_whirlpool, 
 origin=bundled)
 +

Likely should shift this to a trailing check if no whirlpool 
implementation was found; via this, we can avoid the import unless 
it's needed.
~brian

  # Use pycrypto when available, prefer it over the internal fallbacks
  try:
   from Crypto.Hash import SHA256, RIPEMD
 @@ -80,14 +84,14 @@ except ImportError as e:
   pass
  
  # Use hashlib from python-2.5 if available and prefer it over pycrypto and 
 internal fallbacks.
 -# Need special handling for RMD160 as it may not always be provided by 
 hashlib.
 +# Need special handling for RMD160/WHIRLPOOL as they may not always be 
 provided by hashlib.
  try:
   import hashlib, functools
   
   md5hash = _generate_hash_function(MD5, hashlib.md5, origin=hashlib)
   sha1hash = _generate_hash_function(SHA1, hashlib.sha1, 
 origin=hashlib)
   sha256hash = _generate_hash_function(SHA256, hashlib.sha256, 
 origin=hashlib)
 - for local_name, hash_name in ((rmd160, ripemd160), ):
 + for local_name, hash_name in ((rmd160, ripemd160), (whirlpool, 
 whirlpool)):
   try:
   hashlib.new(hash_name)
   except ValueError:
 -- 
 1.7.7
 



Re: [gentoo-portage-dev] Re: [gentoo-commits] proj/portage:master commit in: bin/

2011-09-13 Thread Brian Harring
On Tue, Sep 13, 2011 at 04:38:35AM +, Robin H. Johnson wrote:
 On Tue, Sep 13, 2011 at 03:20:35AM +, Zac Medico wrote:
  commit: 677240f7b3db66bdcd403c214e5d3fa30e31a24a
  Author: Zac Medico zmedico AT gentoo DOT org
  AuthorDate: Tue Sep 13 03:20:00 2011 +
  Commit: Zac Medico zmedico AT gentoo DOT org
  CommitDate: Tue Sep 13 03:20:00 2011 +
  URL:
  http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=677240f7
  
  repoman: don't sign thin manifests
  
  Thin manifests imply reliance on the VCS for file integrity,
  which implies that manifest signatures are not needed.
 
 This is only true after the VCS has signed commits.
 
 If the VCS does not have signed commits, then we should have this
 signature.

This really doesn't provide for much; without a chain of trust to the 
content of layout.conf, you can't toggle behaviour (making lack of 
signing a failure) for example.  So the manager has to either trust 
the VCS validity, or require accept mixed signed/unsigned (which opens 
a new branch of attacks).

Worse, what this tries to protect against is the VCS being screwed 
with- by definition, thin defers that to the VCS, instead providing 
the data of what the VCS cannot, the distfiles checksums.

If an attack on the VCS can be done (whether SHA1 breakage, or just 
in the middle branch injection), an attacker can just as easily 
inject in a patch to files/ along w/ a modification to the ebuild to 
include that trojan.  It's a bit harsher than I'd like, but 
signed thin manifests are security theater as best I can tell.

While I grok your concerns, it would be *very* useful to know 
exactly what attacks a signed manifest precludes that a thin 
manifest doesn't; all I see is remote distfiles manipulation w/ 
a branch injection; that same injection can just as easily 
mangle profile.bashrc, the ebuild itself, or slip patches into 
files.

If it doesn't actually do anything, it should be disabled.

On the portage front, this just change portage behaviour to defaulting 
to signing, rather than configuration based- very least that deserves 
a PSA notice...

~brian



[gentoo-portage-dev] [PATCH 0/3] thin manifest support

2011-09-01 Thread Brian Harring
This patch series adds thin manifest support.  While mini-manifest feature
exists in funtoo, that functionality is a global toggle- either all are mini,
or none.

This series makes thin controllable per repository via layout.conf; if
the file exists and has 'thin-manifest = true' in it, the manifest rules
switch to thin- *just* for that repository.

This should allow git vcs overlays to use thin, while the mainline rsync
repo continues to use full manifest2.

Finally, some of the api work here is kind of ugly.  I went for consistancy
with the surrounding code; that said, refactoring of the RepoConfig api's
likely would be beneficial (that's outside the scope of my intent however).

Brian Harring (3):
  Bind all manifest access through repoconfigs
  add thin manifest support to the Manifest class
  add layout.conf awareness of thin-manifests

 bin/ebuild|5 +-
 bin/repoman   |8 +-
 pym/_emerge/EbuildFetcher.py  |6 +-
 pym/_emerge/search.py |4 +-
 pym/portage/dbapi/porttree.py |8 +-
 pym/portage/manifest.py   |  149 ++--
 pym/portage/package/ebuild/digestcheck.py |6 +-
 pym/portage/package/ebuild/digestgen.py   |3 +-
 pym/portage/package/ebuild/doebuild.py|4 +-
 pym/portage/package/ebuild/fetch.py   |3 +-
 pym/portage/repository/config.py  |   14 +++-
 11 files changed, 142 insertions(+), 68 deletions(-)

-- 
1.7.6.1




[gentoo-portage-dev] [PATCH 2/3] add thin manifest support to the Manifest class

2011-09-01 Thread Brian Harring
'thin' is just distfiles.  This is primarily useful when the ebuild
lives in a vcs- git for example, which already has it's own checksums
to rely on.
---
 pym/portage/manifest.py   |  149 ++--
 pym/portage/package/ebuild/digestcheck.py |2 +-
 2 files changed, 97 insertions(+), 54 deletions(-)

diff --git a/pym/portage/manifest.py b/pym/portage/manifest.py
index 13efab7..ef1a552 100644
--- a/pym/portage/manifest.py
+++ b/pym/portage/manifest.py
@@ -49,6 +49,12 @@ def guessManifestFileType(filename):
else:
return DIST
 
+def guessThinManifestFileType(filename):
+   type = guessManifestFileType(filename)
+   if type != DIST:
+   return None
+   return DIST
+
 def parseManifest2(mysplit):
myentry = None
if len(mysplit)  4 and mysplit[0] in 
portage.const.MANIFEST2_IDENTIFIERS:
@@ -93,12 +99,14 @@ class Manifest2Entry(ManifestEntry):
 class Manifest(object):
parsers = (parseManifest2,)
def __init__(self, pkgdir, distdir, fetchlist_dict=None,
-   manifest1_compat=False, from_scratch=False):
+   manifest1_compat=False, from_scratch=False, thin=False):
 create new Manifest instance for package in pkgdir
and add compability entries for old portage versions if 
manifest1_compat == True.
Do not parse Manifest file if from_scratch == True (only 
for internal use)
The fetchlist_dict parameter is required only for 
generation of
-   a Manifest (not needed for parsing and checking 
sums).
+   a Manifest (not needed for parsing and checking sums).
+   If thin is specified, then the manifest carries only 
info for
+   distfiles.
self.pkgdir = _unicode_decode(pkgdir).rstrip(os.sep) + os.sep
self.fhashdict = {}
self.hashes = set()
@@ -120,7 +128,11 @@ class Manifest(object):
else:
self.fetchlist_dict = {}
self.distdir = distdir
-   self.guessType = guessManifestFileType
+   self.thin = thin
+   if thin:
+   self.guessType = guessThinManifestFileType
+   else:
+   self.guessType = guessManifestFileType
 
def getFullname(self):
 Returns the absolute path to the Manifest file for this 
instance 
@@ -313,64 +325,20 @@ class Manifest(object):
distfilehashes = {}
self.__init__(self.pkgdir, self.distdir,
fetchlist_dict=self.fetchlist_dict, from_scratch=True,
-   manifest1_compat=False)
-   cpvlist = []
+   manifest1_compat=False, thin=self.thin)
pn = os.path.basename(self.pkgdir.rstrip(os.path.sep))
cat = self._pkgdir_category()
 
pkgdir = self.pkgdir
+   if self.thin:
+   cpvlist = self._update_thin_pkgdir(cat, pn, pkgdir)
+   else:
+   cpvlist = self._update_thick_pkgdir(cat, pn, pkgdir)
 
-   for pkgdir, pkgdir_dirs, pkgdir_files in os.walk(pkgdir):
-   break
-   for f in pkgdir_files:
-   try:
-   f = _unicode_decode(f,
-   encoding=_encodings['fs'], 
errors='strict')
-   except UnicodeDecodeError:
-   continue
-   if f[:1] == .:
-   continue
-   pf = None
-   if f[-7:] == '.ebuild':
-   pf = f[:-7]
-   if pf is not None:
-   mytype = EBUILD
-   ps = portage.versions._pkgsplit(pf)
-   cpv = %s/%s % (cat, pf)
-   if not ps:
-   raise PortagePackageException(
-   _(Invalid package name: '%s') 
% cpv)
-   if ps[0] != pn:
-   raise PortagePackageException(
-   _(Package name does not 
-   match directory name: '%s') % 
cpv)
-   cpvlist.append(cpv)
-   elif manifest2MiscfileFilter(f):
-   mytype = MISC
-   else:
-   continue
-   self.fhashdict[mytype][f] = 
perform_multiple_checksums(self.pkgdir+f, self.hashes)
-   recursive_files = []
-
-   pkgdir = self.pkgdir
-   cut_len 

[gentoo-portage-dev] [PATCH 1/3] Bind all manifest access through repoconfigs

2011-09-01 Thread Brian Harring
This enables controling the behaviour (creation and validation) per
repo, and while mildly ugly, refactors in the right direction.
---
 bin/ebuild|5 +++--
 bin/repoman   |8 ++--
 pym/_emerge/EbuildFetcher.py  |6 --
 pym/_emerge/search.py |4 +++-
 pym/portage/dbapi/porttree.py |8 ++--
 pym/portage/package/ebuild/digestcheck.py |4 +++-
 pym/portage/package/ebuild/digestgen.py   |3 ++-
 pym/portage/package/ebuild/doebuild.py|4 +++-
 pym/portage/package/ebuild/fetch.py   |3 ++-
 pym/portage/repository/config.py  |7 ++-
 10 files changed, 38 insertions(+), 14 deletions(-)

diff --git a/bin/ebuild b/bin/ebuild
index db7e5e3..92105bb 100755
--- a/bin/ebuild
+++ b/bin/ebuild
@@ -200,8 +200,9 @@ def discard_digests(myebuild, mysettings, mydbapi):
portage._doebuild_manifest_exempt_depend += 1
pkgdir = os.path.dirname(myebuild)
fetchlist_dict = portage.FetchlistDict(pkgdir, mysettings, 
mydbapi)
-   from portage.manifest import Manifest
-   mf = Manifest(pkgdir, mysettings[DISTDIR],
+   mf = mysettings.repositories.get_repo_for_location(
+   os.path.dirname(os.path.dirname(pkgdir)))
+   mf = mf.load_manifest(pkgdir, mysettings[DISTDIR],
fetchlist_dict=fetchlist_dict, manifest1_compat=False)
mf.create(requiredDistfiles=None,
assumeDistHashesSometimes=True, 
assumeDistHashesAlways=True)
diff --git a/bin/repoman b/bin/repoman
index 6ec84e5..5f81a0f 100755
--- a/bin/repoman
+++ b/bin/repoman
@@ -1104,7 +1104,9 @@ for x in scanlist:
portage._doebuild_manifest_exempt_depend += 1
try:
distdir = repoman_settings['DISTDIR']
-   mf = portage.manifest.Manifest(checkdir, 
distdir,
+   mf = 
repoman_settings.repositories.get_repo_for_location(
+   
os.path.dirname(os.path.dirname(checkdir)))
+   mf = mf.load_manifest(checkdir, distdir,
fetchlist_dict=fetchlist_dict)
mf.create(requiredDistfiles=None,
assumeDistHashesAlways=True)
@@ -1314,7 +1316,9 @@ for x in scanlist:
raise
continue
 
-   mf = Manifest(checkdir, repoman_settings[DISTDIR])
+   mf = repoman_settings.repositories.get_repo_for_location(
+   os.path.dirname(os.path.dirname(checkdir)))
+   mf = mf.load_manifest(checkdir, repoman_settings[DISTDIR])
mydigests=mf.getTypeDigests(DIST)
 
fetchlist_dict = portage.FetchlistDict(checkdir, repoman_settings, 
portdb)
diff --git a/pym/_emerge/EbuildFetcher.py b/pym/_emerge/EbuildFetcher.py
index feb68d0..4389f84 100644
--- a/pym/_emerge/EbuildFetcher.py
+++ b/pym/_emerge/EbuildFetcher.py
@@ -206,8 +206,10 @@ class EbuildFetcher(SpawnProcess):
def _get_digests(self):
if self._digests is not None:
return self._digests
-   self._digests = portage.Manifest(os.path.dirname(
-   self._get_ebuild_path()), None).getTypeDigests(DIST)
+   pkgdir = os.path.dirname(self._get_ebuild_path())
+   mf = 
self.pkg.root_config.settings.repositories.get_repo_for_location(
+   os.path.dirname(os.path.dirname(pkgdir)))
+   self._digests = mf.load_manifest(pkgdir, 
None).getTypeDigests(DIST)
return self._digests
 
def _get_uri_map(self):
diff --git a/pym/_emerge/search.py b/pym/_emerge/search.py
index 096b384..4a4183d 100644
--- a/pym/_emerge/search.py
+++ b/pym/_emerge/search.py
@@ -317,7 +317,9 @@ class search(object):
installed=False, 
metadata=metadata,

root_config=self.root_config, type_name=ebuild)
pkgdir = 
os.path.dirname(myebuild)
-   mf = Manifest(
+   mf = 
self.settings.repositories.get_repo_for_location(
+   
os.path.dirname(os.path.dirname(pkgdir)))
+   mf = mf.load_manifest(
pkgdir, 
self.settings[DISTDIR])
try:
uri_map = 
_parse_uri_map(mycpv, metadata,
diff --git a/pym/portage/dbapi/porttree.py b/pym/portage/dbapi/porttree.py
index 

Re: [gentoo-portage-dev] [API] First steps for creating an API for portage

2010-06-18 Thread Brian Harring
On Fri, Jun 18, 2010 at 08:35:13AM +0200, Fabian Groffen wrote:
 On 18-06-2010 02:08:04 +0200, René 'Necoro' Neumann wrote:
  In parallel (or thereafter), we build the C-bindings. The API for these
  bindings probably look different -- but I guess they should be
  implemented in terms of the library created above.
  
  By example:
  
  - Operation: get the list of categories
  - Python-API: portage.api.categories() : Category list
  - Implementation: def categories(): return 
  - C-API: category * categories()
  - C-Implementation: some wrapper around portage.api.categories
 
 If you want to deliver a C implementation, I'd wonder why you wouldn't
 just make a full C implementation and create Python wrappers for them
 instead of the other way around.  Might accidentially speed up Portage,
 and make tools like portage-utils happy.

Got a couple of years of developer time you can spare? ;)

Portage's slowness ain't python, if in doubt look at pkgcore.  Even 
w/out our extensions we blow it's lid off.  So kindly skip the it'll 
be faster in c bits... that joke is beyond dead at this point ;)

back to the subject at hand...

While I'm not generally a fan of embedding python, in this case it's 
what makes sense.  That said I'm not hugely convinced the proposal on 
the table is accurate- knocking out a public portage API needs to 
occur, but a c-api is a very large and seperate beast from a public 
python API- namely due to crossing the vm, having to do FFI 
conversions, etc.  It's not the simplest thing.

For pkgkit, what I'd suggest is basically pushing into pkgkit just cpy 
stubs that know to import a specific namespace, and grab functions 
from there.  It's been a while since I've gone through that code, but 
if memory serves this shouldn't be too hard to do- the reasons for 
doing this pretty much come down to allowing portage to minimize the 
area it needs to provide a stable API for.

If all it has to provide is a stable set of functors for pkgkit, 
that leaves open fixing portage internals/architecture.

My two cents either way.  I know for pkgcore/pkgkit, this is my 
intention also...

~harring


pgpwOmmDdPb1V.pgp
Description: PGP signature


Re: [gentoo-portage-dev] [API] First steps for creating an API for portage

2010-06-18 Thread Brian Harring
On Fri, Jun 18, 2010 at 12:55:37PM +0200, Rennn 'Necoro' Neumann wrote:
 Am 18.06.2010 09:55, schrieb Brian Harring:
  While I'm not generally a fan of embedding python, in this case it's 
  what makes sense.  That said I'm not hugely convinced the proposal on 
  the table is accurate- knocking out a public portage API needs to 
  occur, but a c-api is a very large and seperate beast from a public 
  python API- namely due to crossing the vm, having to do FFI 
  conversions, etc.  It's not the simplest thing.
 
 Well - there a tools like Cython which will take large parts of this
 burden from your shoulder.

Cython is for extending python, embedding python.  Cython doesn't do 
jack for embedding... frankly very little does.


  For pkgkit, what I'd suggest is basically pushing into pkgkit just cpy 
  stubs that know to import a specific namespace, and grab functions 
  from there.  It's been a while since I've gone through that code, but 
  if memory serves this shouldn't be too hard to do- the reasons for 
  doing this pretty much come down to allowing portage to minimize the 
  area it needs to provide a stable API for.
  
  If all it has to provide is a stable set of functors for pkgkit, 
  that leaves open fixing portage internals/architecture.
 
 I thought this was the whole idea of having such an API - providing a
 stable set of higher level functions, whose implementation can change if
 needed.

 Just pulling random portage functions into a new namespace is not a goal
 imho.

Everything I've seen of the pkgkit hooks required for this, it's not a 
good api.  It's a good API for *pkgkit* which targets LCD, but that 
doesn't mean it's a good *portage* api to expose and maintain.

As such a portage provided/bundled shim is the best notion, at least 
until portage locks down a stable api it's willing to support rather 
than sure, what we've got we'll call stable!.

~harring


pgpJfGRjIRwx1.pgp
Description: PGP signature


Re: [gentoo-portage-dev] [API] First steps for creating an API for portage

2010-06-18 Thread Brian Harring
On Fri, Jun 18, 2010 at 10:08 AM, Brian Harring ferri...@gmail.com wrote:

 On Fri, Jun 18, 2010 at 12:55:37PM +0200, Rennn 'Necoro' Neumann wrote:
  Am 18.06.2010 09:55, schrieb Brian Harring:
   While I'm not generally a fan of embedding python, in this case it's
   what makes sense.  That said I'm not hugely convinced the proposal on
   the table is accurate- knocking out a public portage API needs to
   occur, but a c-api is a very large and seperate beast from a public
   python API- namely due to crossing the vm, having to do FFI
   conversions, etc.  It's not the simplest thing.
 
  Well - there a tools like Cython which will take large parts of this
  burden from your shoulder.

 Cython is for extending python, embedding python.  Cython doesn't do
 jack for embedding... frankly very little does.

Pardon, just to be clear I meant Cython is for extending python, not
embedding python.

Need coffee...
~harring



Re: [gentoo-portage-dev] Package compression header for binhosts

2010-06-01 Thread Brian Harring
On Tue, Jun 1, 2010 at 1:01 PM, Ned Ludd so...@gentoo.org wrote:

 On Mon, 2010-05-31 at 22:16 -0700, Brian Harring wrote:
  On Mon, May 31, 2010 at 08:32:34PM -0700, Zac Medico wrote:
   Hi,
  
   In order to support alternative compression types for binhost
   packages, I was thinking about adding support for a header field in
   the Packages index file. For example, a header line like
   PACKAGE_EXTENSION: txz could be used to indicate that clients
   should download files with txz extensions instead of tbz2
   extensions. I'm planning to add support for both tgz [1] and txz
   extensions.
  
   [1] http://bugs.gentoo.org/show_bug.cgi?id=142579
 
  1) requires a version header bump

 Agreed. But there were some other pending changes for VERSION: 1

 Any planned changes to the format should be documented on
 https://bugs.gentoo.org/show_bug.cgi?id=263994


  2) a header alone isn't useful unless it's specifiable per cpv entry;
  thus it must be inheritable

 Per CPV entries is going to bloat the format and make me carry around a
 more data on a per pkg basis then I'd want to. How about we run with
 zac's idea but use tools to convert a full repo over to $EXTENTION
 This should keep the portage code fast as well as it checks for invalid
 binpkgs all the time. Having to have portage process a ton of ever
 growing extentions is just going to be slow.


Note I said 'inheritable'; one of the main flaws w/ version 0 is that it
requires quite a few entries per CPV, instead of setting a default in the
preamble and then overriding as needed at the CPV level.

What I'm suggesting is a COMPRESSOR in the preamble, and individual cpv's
override it if they're not that compressor.

As for zacs tool to try and generate new views of a repository via
hardlinking/recreating the tree... frankly it's a bit of a hack.  Via
DEFAULT_URI and relying on the hash, you can make a stable repository that
is able to be updated in place without corrupting ongoing downloads- simply
put, new additions to the repo don't perturb current DL's since the md5 is
the same (hash collision chance is low enough that I don't care about it
here).


  3) PACKAGE_EXTENSION is overly verbose and unclear it's specifying
  the compressor too; it's intention is for compression, state it as
  such (I mention this in light of URI's existance where
  PACKAGE_EXTENSION would only be a hint of compressor)
 
  Re: #1, there is a decent set of optimizations I'm kicking around in
  pkgcore for the next version- a discussion should probably be started
  there.
 
  Offhand, having a compression specific header (a simple enumeration
  of known compressors) and a DEFAULT_URI that is python string

 No go bro. The 'Packages' format should be independent of python.


  interpolation  assembled (for example,
  DEFAULT_URI=%(host)s/%(category)s/%(pf)s.txz) seems wiser.  Via
  doing what I'm suggesting, it would be possible to do binpkg
  repository 'views' w/out having to map each binpkg into the url space
  for it.


Then come up w/ an alternative w/ the same power as DEFAULT_URI that isn't
python specific; think through the potentials of it, I could very easily
centralize the binpkgs for an arch, use the hash as they're lookup value,
then use the Packages cache as a 'view' into that binpkg repository.
 Differing use flag combinations, differing license views, hell, differing
ACCEPT_KEYWORDS, all of that can have the raw pkgs stored centrally while
just providing differing views into it- DEFAULT_URI lays the groundwork for
it.


Re: [gentoo-portage-dev] Package compression header for binhosts

2010-06-01 Thread Brian Harring
On Tue, Jun 1, 2010 at 2:37 PM, Zac Medico zmed...@gentoo.org wrote:

 On 06/01/2010 02:22 PM, Brian Harring wrote:
  As for zacs tool to try and generate new views of a repository via
  hardlinking/recreating the tree... frankly it's a bit of a hack.  Via
  DEFAULT_URI and relying on the hash, you can make a stable repository
 that
  is able to be updated in place without corrupting ongoing downloads-
 simply
  put, new additions to the repo don't perturb current DL's since the md5
 is
  the same (hash collision chance is low enough that I don't care about it
  here).

 When you say hash collision are you talking about
 http://crosbug.com/3225? Maybe that behavior is acceptable for
 small-scale private use, but for large scale public repositories I'd
 say it's totally unacceptable.


That bug isn't about a collision, it's about files being replaced underneath
Packages feet.  Even with the tricks you've leveled the issue of things
changing under foot still is possible- you've just made the race less
likely.

What I was talking about was solving this issue once and for all via
restructuring, and specifically refering to the potential of an md5
collision in the URI space- specifically what I'm implementing for pkgcore
is the ability to do stupid stuff like this-

http://host/binpkg-store/$MD5.{txz,tbz2,tgz}

then have multiple views accessible just via pointing the binpkg repo remote
url at

http://host/views/license/oss-approved/
http://host/views/keywords/amd64/stable/
http://host/views/raw/ # no filtering on the view of the binpkg repo, see
everything.

Via restructuring where the binpkgs are stored and doing this approach,
multiple views can be had easily into the repo.  An additional benefit of
this approach is that via making URI able to point outside the host, you
could combine multiple seperate repositories into one just via a view.



 Eventually, I'd like to see gentoo
 officially distributing binary packages, so that we'll be able to
 get a slice of the binary distribution pie. When that happens, we're
 certainly not going to want to have race conditions like these in
 our public binhosts.


I'd suggest abandoning the current repository layout of Packages then, since
it's irrevocably flawed.  You can hack around it via jamming timestamp/md5
info into URI, but that's not a sane solution.

~harring


Re: [gentoo-portage-dev] Package compression header for binhosts

2010-06-01 Thread Brian Harring
On Tue, Jun 01, 2010 at 04:53:31PM -0700, Zac Medico wrote:
 On 06/01/2010 02:52 PM, Brian Harring wrote:
  That bug isn't about a collision, it's about files being replaced underneath
  Packages feet.  Even with the tricks you've leveled the issue of things
  changing under foot still is possible- you've just made the race less
  likely.
 
 AFAIK the race is completely eliminated by the RCU-like snapshot
 mechanism. I think I like your hash-in-the-filename idea better
 though, since it seems simpler to implement and maintain.

You're forgetting about how one actually updates the snapshot- client 
grabs the packages cache, starts pulling binpkgs.  During that time 
the snapshot is being updated- the client now has a stale view of the 
repo, and since the repo's structure is based on cpv (which doesn't 
change regardless of metadata changing like use configuration) they 
can grab a binpkg that has the wrong metadata/checksums.

It's racey in exactly the same was as before, the only difference is 
you switched it to rewrite the tbz2 in a temp file instead of directly 
to the tbz2.  Reduction, but same level of risk for any form of 
updates.

Snapshot script just duck tapes around the issue, while leaving the 
core flaw intact.

  What I was talking about was solving this issue once and for all via
  restructuring, and specifically refering to the potential of an md5
  collision in the URI space- specifically what I'm implementing for pkgcore
  is the ability to do stupid stuff like this-
  
  http://host/binpkg-store/$MD5.{txz,tbz2,tgz}
 
 That would be the MD5 of the entire file, after compression and
 having the xpak segment appended, right?

Yep.  The only potential issue here is the unlikely case of a CHF 
collision.  There is a way to resolve that one too, although it is 
outside of what I'm willing to do format wise (namely a secondary url 
fallback).



  then have multiple views accessible just via pointing the binpkg repo remote
  url at
  
  http://host/views/license/oss-approved/
  http://host/views/keywords/amd64/stable/
  http://host/views/raw/ # no filtering on the view of the binpkg repo, see
  everything.
 
 So, the default path of a package would come from looking at the MD5
 in the Packages file and then mapping that to a path?

default path would be defined in the preamble by a string interpolated 
pattern; whatever folk wanted to use.

A sane default is %(host)s/raw-pkgs/%(md5)s.%(compressor_ext)s imo.


  Via restructuring where the binpkgs are stored and doing this approach,
  multiple views can be had easily into the repo.  An additional benefit of
  this approach is that via making URI able to point outside the host, you
  could combine multiple seperate repositories into one just via a view.
 
 This might also be useful for creating per-profile views while
 allowing packages to be shared between profiles in cases when
 hosting a separate build would be redundant. It might be possible to
 save lots of build time, disk space, and testing that way.
 
 Being able to have multiple builds of the same package with
 different USE settings is also solves bug 150031 [1].

Yep and Yep.

  Eventually, I'd like to see gentoo
  officially distributing binary packages, so that we'll be able to
  get a slice of the binary distribution pie. When that happens, we're
  certainly not going to want to have race conditions like these in
  our public binhosts.
 
  
  I'd suggest abandoning the current repository layout of Packages then, since
  it's irrevocably flawed.  You can hack around it via jamming timestamp/md5
  info into URI, but that's not a sane solution.
 
 Shrug, it's a handy way to solve race conditions given the existing
 version 0 format. It's not optimal, so we'll surely want something
 better in version 1.

The problem here is that version0 still maps down to the existing 
binpkg on disk layout.  That layout is the core flaw here- as long as 
binpkgs are stored cpv orientated, version0 isn't able to do the crazy 
things I'm intending.
~harring


pgpPeqOI5lTt1.pgp
Description: PGP signature


Re: [gentoo-portage-dev] [RFC] Store [,R,P]DEPEND with unevaluated use conditionals in vdb

2010-04-24 Thread Brian Harring
On Sat, Apr 24, 2010 at 11:27:49AM -0700, Zac Medico wrote:
 On 04/24/2010 11:00 AM, Sebastian Luther wrote:
  Am 24.04.2010 13:32, schrieb Gentoo:
  On Fri, 2010-04-23 at 22:31 -0700, Zac Medico wrote:
  On 04/23/2010 05:43 AM, Sebastian Luther wrote:
  Someone might come up with some logic to detect new use flags in
  *DEPEND, but this looks like a hack to me.
 
  It doesn't seem too bad to me.
  
  It doesn't work, because it's not guaranteed, that only use flag from
  IUSE are used in use conditionals. That means you can't do it reliably
  without the unevaluated value.
 
 We can and should add a check to repoman to enforce this. It's long
 overdue. The flags already cannot be enabled unless they are in
 IUSE, since portage filters them (except for special things like
 use.force).

Offhand, the tree should be clean on this account- pcheck has scanned 
for it since near day one.  You also need to scan LICENSE, SRC_URI, 
and RESTRICT btw.

~brian


pgpbdWHuynx1g.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Composite exceptions?

2010-02-26 Thread Brian Harring
On Sat, Feb 27, 2010 at 05:02:18AM +0100, Sebastian Pipping wrote:
 On 02/27/10 04:20, Zac Medico wrote:
  Do you have an example case where you want to use this?
 
 Multiple defects in metadata.xml are such a case.
 At some point all the exceptions will have to collected, e.g. two
 invalid herds are mentioned.  In that case a single exception with a
 list of invalid herds may suffice but it gets worse when combining
 errors of slightly different types.

I'd suggest looking at pchecks design instead of trying to do 
composite exceptions- essentially pass in a reporter that is invoked 
w/ the failure/'exception' instead passed in.

Doing what you're suggesting (catching all exceptions at the top and 
trying sum them essentially) results in screwy code flow in the 
specific check- consider a check that can flag multiple issues.  
Chucking an exception means you get the first warning spotted (and 
just that).

Do the reporter/observer/tweaked visitor approach, you get all of the 
issues, and it's left to the reporter to decide what to output (and 
how to format it).

Also makes it a helluva lot easier to serialize the results into 
pickles, or go parallel (multiple processes running, the reporter 
in the subprocess just serializes the issue to a central process that 
then amalgamates the results).

~harring


pgpoMPmfDsQs1.pgp
Description: PGP signature


Re: [gentoo-portage-dev] add UUID for comparison of installed package to binary package?

2010-02-14 Thread Brian Harring
On Fri, Feb 12, 2010 at 04:24:05PM -0800, Zac Medico wrote:
 On 02/12/2010 01:38 PM, Brian Harring wrote:
  On Fri, Feb 12, 2010 at 12:54:21PM -0800, Zac Medico wrote:
  Hi,
 
  I'm thinking about adding a UUID file in /var/db/pkg, for comparing
  installed packages to binary packages. We already have BINPKGMD5,
  but the problem with that is that the MD5 of a binary package
  changes when it's updated for package moves. A UUID would be
  assigned at build time and remain constant thereafter. We can use
  python's uuid.uuid4() function to generate a random UUID. Any
  suggestions for alternative approaches?
  
  Purpose?
  ~harring
 
 If you build binary packages for installation on multiple systems,
 and periodically have to rebuild packages for various reasons
 (revdep-rebuild or whatnot), it makes it easier to know whether a
 particular system has the latest build installed. Then you can
 create a package set that reinstalls any installed packages that are
 not the latest build.
 
 It's basically a catch-all for rebuilds that aren't tracked by other
 kinds of metadata yet. Eventually, we should introduce metadata to
 indicate when things need to be reinstalled, as discussed in this bug:
 
   https://bugs.gentoo.org/show_bug.cgi?id=192319
 
 In the mean time, it's nice to have a catch-all for keeping systems
 synchronized with the latest builds. We may want to consider
 including it in the $PKGDIR/Packages file as a convenience for
 binhost users.

This gets nasty... you're basically talking about the rpm equivalent 
of EPOCH.

Not a fan of an adhoc UUID (especially since it'll become standard 
via portage doing it), but a *timestamp* for the build, labeled as 
such, gets you what you want and is usable for other things- detecting 
when to rebuild a scm package for example.

That route gets my vote, and should also address your intentions.
~harring


pgpQadYFVbGdp.pgp
Description: PGP signature


Re: [gentoo-portage-dev] add UUID for comparison of installed package to binary package?

2010-02-14 Thread Brian Harring
On Sun, Feb 14, 2010 at 06:02:28PM -0800, Ned Ludd wrote:
 On Sun, 2010-02-14 at 12:11 -0800, Zac Medico wrote:
  On 02/14/2010 04:36 AM, Brian Harring wrote:
   This gets nasty... you're basically talking about the rpm equivalent 
   of EPOCH.
   
   Not a fan of an adhoc UUID (especially since it'll become standard 
   via portage doing it), but a *timestamp* for the build, labeled as 
   such, gets you what you want and is usable for other things- detecting 
   when to rebuild a scm package for example.
   
   That route gets my vote, and should also address your intentions.
   ~harring
  
  Ok, then how about a vdb entry named CTIME that contains an integer
  number of seconds since the unix Epoch?
 
 That would be UNIXTIME vs CTIME I'd think.

How about a more descriptive name... build_time or something similar.  
Yes, CTIME means creation time, but I'd rather not overload terms that 
have specific meanings already just for the sake of saving a few chars 
in typing the filename in...
~harring


pgpTTyGjPY9W9.pgp
Description: PGP signature


Re: [gentoo-portage-dev] add UUID for comparison of installed package to binary package?

2010-02-12 Thread Brian Harring
On Fri, Feb 12, 2010 at 12:54:21PM -0800, Zac Medico wrote:
 Hi,
 
 I'm thinking about adding a UUID file in /var/db/pkg, for comparing
 installed packages to binary packages. We already have BINPKGMD5,
 but the problem with that is that the MD5 of a binary package
 changes when it's updated for package moves. A UUID would be
 assigned at build time and remain constant thereafter. We can use
 python's uuid.uuid4() function to generate a random UUID. Any
 suggestions for alternative approaches?

Purpose?
~harring


pgp5uUhnOSt6y.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Re: [PATCH] Add support to Mercurial SCM in bin/repoman

2010-01-17 Thread Brian Harring
On Sun, Jan 17, 2010 at 08:46:24PM -0200, Rafael Martins wrote:
 I'm re-sending the patch, with a small fix. Please disregard the previous 
 patch.

Might I suggest breaking classes out for each syncer rather then 
continuing the huge and nasty giant if/elif ?

Certainly would make it easier to maintain and extend.
~harring


pgp1sq0N3UIDH.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Re: forcing a USE flag if another is on

2009-12-30 Thread Brian Harring
On Wed, Dec 30, 2009 at 04:20:45PM +, Duncan wrote:
 Amit Dor-Shifer posted on Wed, 30 Dec 2009 16:45:40 +0200 as excerpted:
 
  Is there some method of specifing if USE flag X is enabled, enable USE
  flag y as-well? Something like a conditional use.force file in
  profiles/. Amit
 
 That should be doable globally using /etc/portage/bashrc or the like (
 /etc/portage/profile/bashrc).

It's not doable via bashrc (bashrc runs at build time)- intentionally 
to block abuses of this sort I might add.

Reiterating- playing w/ USE in bash doesn't change the USE state 
python/package manager side, meaning you'll not get the proper 
depedencies pulled in...

~harring


pgpaFikDS1XEq.pgp
Description: PGP signature


Re: [gentoo-portage-dev] rsync support for fetching binary packages

2009-05-21 Thread Brian Harring
On Tue, May 19, 2009 at 02:47:45PM +0300, Amit Dor-Shifer wrote:
 Hi.
 Looking at getbinpkg.py, I see that BINPKGs can be retrieved using
 http/s s/ftp. I'm wondering about rsync, as it is mostly supported
 across portage (and also in layman). Is there some design reasoning
 behind this lack of support, or is it that no-one has yet gotten around
 to implement it?
 
 I'm raising the question because I'm implementing a private repository
 and would prefer not to service it via http if I don't have to. Right
 now, the only feature which is blocking me is this lack of support for
 binary packages.

Look in the code to be sure, but I'd expect it just hasn't been 
implemented.  If I were in your shoes, I'd generalize the SYNC proto 
support into something usable for binpkgs also- I did this in pkgcore, 
works fairly well (and could be extended to fetching in general, which 
again works well enough).

Personally, I wouldn't dick around with rsync for individual file 
transfers- http/ftp seems better suited, but I may be missing 
something about your setup that warrants it.

Cheers-
~harring


pgpWLgNBHXAcF.pgp
Description: PGP signature


Re: [gentoo-portage-dev] equery: deprecate --category filtering in belongs

2009-02-07 Thread Brian Harring
On Sun, Feb 08, 2009 at 02:07:08PM +0900, Douglas Anderson wrote:
 Hi, does anyone use --category filtering in equery belongs? I want to
 get rid of it, or at least deprecate it. My reasoning:
 
 * We use 'equery belongs' when don't know to what package a file
 belongs. Even if we have a suspicion, most users would have to look up
 the category of the package before typing it in.
 * Even if you happen to know the exact category of the package that
 installed the file (why are you using belongs?), typing
 --category=app-portage takes more time than is saved by filtering by
 category (about 5 seconds more by my unscientific test).
 
 Even in a script setting, I see no use for this. The time saved is minuscule:
 
 $ time equery belongs /usr/bin/equery --category app-portage
 [ Searching for file(s) /usr/bin/equery in app-portage... ]
 app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)
 
 real0m4.002s
 user0m3.680s
 sys 0m0.076s
 $ time equery belongs /usr/bin/equery
 [ Searching for file(s) /usr/bin/equery in *... ]
 app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)
 
 real0m4.205s
 user0m3.738s
 sys 0m0.102s

You should do profiling, and *verify* category actually works; it 
should be able to make a world of difference.  Case in point via 
pkgcore-

ferri...@beast ~ $ time pquery --owns /usr/bin/equery
app-portage/gentoolkit-0.2.4.2-r1

real0m1.162s
user0m1.142s
sys 0m0.020s
ferri...@beast ~ $ time pquery --owns /usr/bin/equery 'app-portage/*'
app-portage/gentoolkit-0.2.4.2-r1

real0m0.168s
user0m0.156s
sys 0m0.011s


Yes it's not equery, but I *know* pkgcores implementation works thus 
using it to point out the speed difference if implemented correctly.

It would probably help if the category support was enabled in the 
code, also (quick check of equery shows the cat filtering is 
disabled).

~harring


pgpl7rBjX9iss.pgp
Description: PGP signature


Re: [gentoo-portage-dev] equery: deprecate --category filtering in belongs

2009-02-07 Thread Brian Harring
patch attached against 0.2.4.2-r1; rough stats follow;

full cold cache

[ Searching for file(s) /usr/bin/equery in *... ]
app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)

real0m10.320s
user0m0.733s
sys 0m0.162s

[ Searching for file(s) /usr/bin/equery in app-portage... ]
app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)

real0m8.512s
user0m0.315s
sys 0m0.124s

That particular cold cache is a *full* cold cache; not the best test 
imo since most users have at least some chunks of portage 
configuration/python cached.


Cold cache, with equery --help primer to warm the cache; CONTENTS 
(what belongs operates on) is still out of the cache however making 
this a bit more likely use scenario.

[ Searching for file(s) /usr/bin/equery in *... ]
app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)

real0m2.335s
user0m0.670s
sys 0m0.050s

[ Searching for file(s) /usr/bin/equery in app-portage... ]
app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)

real0m0.391s
user0m0.248s
sys 0m0.046s

Pretty heavy difference, no?


hotcache:
[ Searching for file(s) /usr/bin/equery in *... ]
app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)

real0m0.710s
user0m0.661s
sys 0m0.047s

[ Searching for file(s) /usr/bin/equery in app-portage... ]
app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)

real0m0.291s
user0m0.237s
sys 0m0.053s


Mind you this isn't multiple runs, so the numbers are rough 
approximations- that said they're fairly representative.

Strongly suggest y'all keep category support (although I'll keep on 
using pquery instead ;).

Cheers,
~harring
--- /usr/bin/equery 2009-02-07 13:16:51.0 -0800
+++ /home/ferringb/equery   2009-02-07 14:15:52.0 -0800
@@ -345,7 +345,7 @@
need_help = 1
break
elif x in [-c, --category]:
-   opts[category] = args[i+1]
+   opts[category] = args[i+1].rstrip(/)
skip = 1
elif x in [-e, --earlyout]:
opts[earlyOut] = 1
@@ -383,16 +383,14 @@
die(2, The query ' + pp.regexpquery(q) + ' does not 
appear to be a valid regular expression)
 
# Pick out only selected categories
-   cat = opts[category]
-   filter_fn = None
-   if cat != *:
-   filter_fn = lambda x: x.find(cat+/)==0
 
+   cat = opts[category]
if not Config[piping] and Config[verbosityLevel] = 3:
print_info(3, [ Searching for file(s)  + 
pp.regexpquery(,.join(query)) +  in  + pp.cpv(cat) + ... ])

matches = portage.db[/][vartree].dbapi.cpv_all()
-   #matches = gentoolkit.find_all_installed_packages(filter_fn)
+   if cat != *:
+   matches = (x for x in matches if x.startswith(cat))
 
found = 0
 


pgpqntNmV2sMt.pgp
Description: PGP signature


Re: [gentoo-portage-dev] How to extract the version/revision of an installed package?

2008-11-25 Thread Brian Harring
On Tue, Nov 25, 2008 at 06:05:21PM +0200, Amit Dor-Shifer wrote:
 Given the following:
 # qlist -Iv sys-apps/portage
 sys-apps/portage-2.1.4.5

 How do I safely extract the 2.1.4.5?

 (I don't necessarily need to use qlist. Just want to get the version of an 
 installed package within a bash script)

This *really* should be folded into portageq offhand- it's the initial 
step towards shifting versionator logic (yet another standalone 
parser/comparison implementation) into the PM.

Counter arguements?
~brian


pgpxv73MMnzsm.pgp
Description: PGP signature


Re: [gentoo-portage-dev] relying on vdb only

2008-02-12 Thread Brian Harring
On Mon, Feb 11, 2008 at 12:58:51PM +0100, Selckin wrote:
 On Monday 11 February 2008 12:50:39 Brian Harring wrote:
  On Mon, Feb 11, 2008 at 09:48:01AM +0100, Vlastimil Babka wrote:
 
   Well, the idea that devs will have to revbump packages just for RDEPEND
   version restrictions so that portage picks it freaks me :)
 
  Relying on the vdb is far saner then relying on the tree; so no, it's
  not particularly dangerous, the inverse (relying on the tree to have
  the same deps for vdb) is far worse imo.
 
  Solution to this is to reuse the existing update infrastructure, and
  add a new command into it that resets the depends/rdepends- haven't
  looked to see if older portage versions would behave well if they
  encounter an unknown command in profiles/updates/* however.

 This should really be [possible|done] without introducing yet another ugly 
 and 
 very difficult to maintain update/* hack?

Err... hack?  Justify that statement please.

Few things you might as well address also-
1) update/* runs once per sync; alternatives (building a mapping, or 
forcing 2x metadata lookup via hitting up the tree for new metadata) 
can't really compete from a amoritized cost standpoint
2) it's simple to maintain; exact atom (rev included), metadata key, 
metadata value.  Can't realistically rewrite eapi-1 IUSE via it 
(default iuse can change the pkgs USE configuration), same for other 
build values (CHOST), but it's powerful, and simple.
3) basic infrastructure is already there, so why not reuse it?

~brian


pgpdU8xbYXcD0.pgp
Description: PGP signature


Re: [gentoo-portage-dev] relying on vdb only

2008-02-11 Thread Brian Harring
On Mon, Feb 11, 2008 at 09:48:01AM +0100, Vlastimil Babka wrote:
 Hi,
 
 reading comments on bug 209538, I've seen this dangerous thing from Zac:
 
 Once these issues are solved it will be nice if we can rely exclusively 
 on the dependencies from /var/db/pkg.
 
 Well, the idea that devs will have to revbump packages just for RDEPEND 
 version restrictions so that portage picks it freaks me :)
 
 Then there's: I do have a tool that copies metadata from ebuilds but 
 I'd prefer to avoid doing anything like that if possible.
 
 So maybe it's time to discuss what's possible? :)
 If that discussion already happens/happened elsewhere, then sorry for 
 noise and please point me there :)

Relying on the vdb is far saner then relying on the tree; so no, it's 
not particularly dangerous, the inverse (relying on the tree to have 
the same deps for vdb) is far worse imo.

Solution to this is to reuse the existing update infrastructure, and 
add a new command into it that resets the depends/rdepends- haven't 
looked to see if older portage versions would behave well if they 
encounter an unknown command in profiles/updates/* however.

~brian


pgpiIfINfwuXf.pgp
Description: PGP signature


Re: [gentoo-portage-dev] [RFC] Properties of package sets

2007-06-30 Thread Brian Harring
On Thu, Jun 28, 2007 at 09:03:54PM -0700, Ned Ludd wrote:
 On Fri, 2007-06-29 at 05:07 +0200, Marius Mauch wrote:
  - should sets be supported everywhere, or only in selected use cases?
  (everywhere would include depstrings for example)
 
 Please NO. emerge.py should know about sets but ebuild.py should be
 oblivious to them. package sets as you are proposing imo should be
 limited to portage only. By putting package sets in ebuild depstrings we
 would be effectively forcing all other pkg managers to support the
 feature. I don't think we should do that.

Seconded on the 'no'; aside from it screwing up parsing, that's 
*system* configuration- meaning it's not part of a repository data 
set, thus the repository data set should in no way depend on it.

~harring


pgplLWf3Hqo7s.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Improvement suggestion for emerge: Not using a new connection for every file

2007-02-24 Thread Brian Harring
On Sat, Feb 24, 2007 at 05:55:47PM -0800, Robin H. Johnson wrote:
 On Sat, Feb 24, 2007 at 10:00:29PM +0100, Beginner wrote:
  I recommend not to use wget and not to reconnect to the server for every 
  single packet, but to hold the connection
  therefore spare traffic and download more fast.
 If you are doing lots of downloads, use 'emerge -pvf FOO' and feed each
 line of that output to whatever you want to do your fetching.
 
 On that, I haven't kept up with the code in recent years, is there a way
 that portage itself can hand off those entire lines to a fetching
 application, instead of putting them in one by one? (Telling the app
 about the expected size and checksums would be handy too).

Current fetch implementation... not worth trying.  No abstraction 
built into it- would suggest if you're looking to try this, either rip 
off the old EBD/saviour fetch refactoring (ick), or rip what we've got 
in pkgcore now.

EBD version had ftplib/httplib direct usage; for pkgcore, dropped the 
builtin mainly... since I was too lazy to update it.

Either way, trying it with current fetch implementation in portage, 
would suggest either gutting from codebases mentioned above, or 
refactoring fetch such that FETCHCOMMAND/RESUMECOMMAND are 
encapsulated and the fetcher functor/obj is pulled from the passed in 
settings instance.

~harring


pgp5NlrZxJJzS.pgp
Description: PGP signature


[gentoo-portage-dev] Re: r5993 - main/trunk/pym/portage/dbapi

2007-02-18 Thread Brian Harring
round two of the patch, still is missing...


On Sun, Feb 18, 2007 at 06:27:59PM +, Marius Mauch wrote:
 Author: genone
 Date: 2007-02-18 18:27:59 + (Sun, 18 Feb 2007)
 New Revision: 5993
 
 Modified:
main/trunk/pym/portage/dbapi/vartree.py
 Log:
 extend check for internal references, should remove all libs that are only 
 used internally now
 
 Modified: main/trunk/pym/portage/dbapi/vartree.py
 ===
 --- main/trunk/pym/portage/dbapi/vartree.py   2007-02-18 15:38:56 UTC (rev 
 5992)
 +++ main/trunk/pym/portage/dbapi/vartree.py   2007-02-18 18:27:59 UTC (rev 
 5993)
 @@ -1266,14 +1266,33 @@
   preserve_libs = old_libs.difference(mylibs)
  
   # ignore any libs that are only internally used by the 
 package
 - for lib in preserve_libs.copy():
 - old_contents_without_libs = [x for x in 
 old_contents if os.path.basename(x) not in preserve_libs]
 - if 
 len(set(libmap[lib]).intersection(old_contents_without_libs)) == 
 len(libmap[lib]):
 - preserve_libs.remove(lib)
 + def has_external_consumers(lib, contents, otherlibs):
 + consumers = set(libmap[lib])

# this should be done *once*.
# further, if libmap where k-set, you'd avoid repeated
# creation of sets all over the damn place.

 + contents_without_libs = [x for x in contents if 
 not os.path.basename(x) in otherlibs]

# os.path.basename invocations aren't horribly expensive, but the 
 + 
 + # just used by objects that will be autocleaned
 + if 
 len(consumers.difference(contents_without_libs)) == 0:

if not consumers.difference(contents_without_libs)):...
# no point in doing the len, thus don't.
# standard behaviour in python code to rely on bool protocol, thus do 
# so.

 + return False
 + # used by objects that are referenced as well, 
 need to check those 
 + # recursively to break any reference cycles
 + elif len(consumers.difference(contents)) == 0:

# same.  len isn't needed, thus don't use it.
# majority of builtins *do* support boolean, thus use it.

 + otherlibs = set(otherlibs)

# this will explode. you're passing None in below.

 + for ol in 
 otherlibs.intersection(consumers):
 + if has_external_consumers(ol, 
 contents, otherlibs.copy().remove(lib)):

no way that code is right.
1) you're copying a set, then removing an item from it.
2) remove is a Py_RETURN_NONE, meaning it returns None.  you're 
  literally cloning a set, poping an item from it, and then
  throwing away the set, passing None into has_external_consumers
3) algo in general could be replaced by just doing walks over the 
  target preserve libs, pruning until a walk doesn't prune anything.
  Runtime ought to be better, less daft copys, saner to read also.

 + return True
 + return False
 + # used by external objects directly
 + else:
 + return True
 +
 + for lib in list(preserve_libs):
 + if not has_external_consumers(lib, 
 old_contents, preserve_libs):
 + preserve_libs.remove(lib)   
 

# this is a bit retarded... for handling libs referring to each other, 
# code above continually recalculates.
# reverse the mapping, and work from that angle; should avoid having 
# to do this insane little dance.

   
   # get the real paths for the libs
   preserve_paths = [x for x in old_contents if 
 os.path.basename(x) in preserve_libs]
 -
 + del old_contents, old_libs, mylibs, preserve_libs
 + 
   # inject files that should be preserved into our image 
 dir
   import shutil
   for x in preserve_paths:
 @@ -1293,7 +1312,7 @@
   preserve_paths.append(linktarget)
   else:
   shutil.copy2(x, os.path.join(srcroot, 
 x.lstrip(os.sep)))
 - 
 + del preserve_paths
  
   # check for package collisions
   if collision-protect in self.settings.features:
 

Aside from my earlier suggestion that you should pop this crap out of 
svn and work it out outside of mainline, 

Re: [gentoo-portage-dev] Re: r5993 - main/trunk/pym/portage/dbapi

2007-02-18 Thread Brian Harring
Bleh, pardon; left out a bit accidentally-

On Sun, Feb 18, 2007 at 11:03:33AM -0800, Brian Harring wrote:
 On Sun, Feb 18, 2007 at 06:27:59PM +, Marius Mauch wrote:
  -   for lib in preserve_libs.copy():
  -   old_contents_without_libs = [x for x in 
  old_contents if os.path.basename(x) not in preserve_libs]
  -   if 
  len(set(libmap[lib]).intersection(old_contents_without_libs)) == 
  len(libmap[lib]):
  -   preserve_libs.remove(lib)
  +   def has_external_consumers(lib, contents, otherlibs):
  +   consumers = set(libmap[lib])
 
 # this should be done *once*.
 # further, if libmap where k-set, you'd avoid repeated
 # creation of sets all over the damn place.
 
  +   contents_without_libs = [x for x in contents if 
  not os.path.basename(x) in otherlibs]
 
 # os.path.basename invocations aren't horribly expensive, but the 
continual calls to it make the code more verbose then needed (Thus 
harder to follow), and slower.  Consider a mapping instead, see the 
first review for old_based...

~harring


pgpGvsXB6QNVY.pgp
Description: PGP signature


[gentoo-portage-dev] Re: r5975 - FEATURES=preserve-libs

2007-02-17 Thread Brian Harring
Realize you didn't want comments upon the implementation, but tough 
cookies, already reviewed it; suckers in svn mainline anyways, thus 
it's fair game.

 Modified: main/trunk/pym/portage/dbapi/vartree.py
 ===
 --- main/trunk/pym/portage/dbapi/vartree.py   2007-02-17 04:17:14 UTC (rev 
 5974)
 +++ main/trunk/pym/portage/dbapi/vartree.py   2007-02-17 08:53:35 UTC (rev 
 5975)
 @@ -523,6 +523,39 @@
   write_atomic(cpath, str(counter))
   return counter
  
 + def get_library_map(self):
 +  Read the global library-consumer map for this vdb instance 
 
 + mapfilename = self.getpath(.NEEDED)
 + if not os.path.exists(mapfilename):
 + self.update_library_map()
 + rValue = {}
 + for l in open(mapfilename, r).read().split(\n):

for l in open(mapfilename):...
# no point in building the list in memory when you're just itering.

 + mysplit = l.split()
 + if len(mysplit)  1:
 + rValue[mysplit[0]] = mysplit[1].split(,)

# testing for this is stupid; your update code doesn't write 
# empty entries, nor should it be allowed in the file.

rValue[myvalue[0]] = mysplit[1].split(',')

 + return rValue
 +
 + def update_library_map(self):
 +  Update the global library-consumer map for this vdb 
 instance. 
 + mapfilename = self.getpath(.NEEDED)
 + obj_dict = {}
 + for cpv in self.cpv_all():
 + needed_list = self.aux_get(cpv, [NEEDED])[0]
 + for l in needed_list.split(\n):
 + mysplit = l.split()
 + if len(mysplit)  2:
 + continue
 + libs = mysplit[1].split(,)
 + for lib in libs:
 + if not obj_dict.has_key(lib):

# __contains__ is your friend.
# has_key is the slower, slightly retarded older brother.
# avoid the older brother.

if lib in obj_dict:
...

 + obj_dict[lib] = [mysplit[0]]
 + else:
 + obj_dict[lib].append(mysplit[0])

# alternative, conciser form.
obj_dict.setdefault(lib, []).append(mysplit[0])


 + mapfile = open(mapfilename, w)
 + for lib in obj_dict.keys():
 + mapfile.write(lib+ +,.join(obj_dict[lib])+\n)

# per the norm, stop using keys().  You're not mutating the dict, thus 
# no reason to for list creation unless you *want* to make portage 
# slower then it is already.
# use iteritems further, instead of forcing a lookup for every single
# item when you're already itering the dict.

for lib, revs in obj_dict.iteritems():
  mapfile.write(%s %s\n % (lib, ','.join(revs)))


 + mapfile.close()
 +
  class vartree(object):
   this tree will scan a var/db/pkg database located at root (passed to 
 init)
   def __init__(self, root=/, virtual=None, clone=None, categories=None,
 @@ -952,6 +985,9 @@
   doebuild(myebuildpath, cleanrm, self.myroot, 
 self.settings,
   tree=vartree, 
 mydbapi=self.vartree.dbapi,
   vartree=self.vartree)
 + 
 + # regenerate reverse NEEDED map
 + self.vartree.dbapi.update_library_map()
  
   finally:
   if builddir_lock:
 @@ -1162,6 +1198,7 @@
   
   This function does the following:
   
 + Preserve old libraries that are still used.
   Collision Protection.
   calls doebuild(mydo=pkg_preinst)
   Merges the package to the livefs
 @@ -1197,6 +1234,10 @@
   if not os.path.exists(self.dbcatdir):
   os.makedirs(self.dbcatdir)
  
 + myfilelist = listdir(srcroot, recursive=1, filesonly=1, 
 followSymlinks=True)
 + mysymlinks = filter(os.path.islink, listdir(srcroot, 
 recursive=1, filesonly=0, followSymlinks=False))
 + myfilelist.extend(mysymlinks)
 +

# I hope this feature is really worth it, since previously, the
# two walks of ${D} occured only if FEATURES=collision-protect.
# via this change, even if FEATURES=-collision-protect -preserve-libs,
# the user now gets to enjoy the two additional walks.
#
# low mem system with large ${D}, just jacked up the runtime a fair 
# bit via that regression.
#


   otherversions = []
   for v in self.vartree.dbapi.cp_list(self.mysplit[0]):
   otherversions.append(v.split(/)[1])
 @@ -1209,17 +1250,59 @@
   catsplit(slot_matches[0])[1], destroot, 

Re: [gentoo-portage-dev] New preserve-libs feature

2007-02-17 Thread Brian Harring
On Sat, Feb 17, 2007 at 09:39:58AM -0500, Mike Frysinger wrote:
 On Saturday 17 February 2007, Brian Harring wrote:
  Security impact is from a pkg potentially dragging along old libs; if
  you've got a stable pkg that gets an update once every blue moon, it
  can hold onto the lib for a *long* time while still using the lib;
  thus if a vuln. in the lib, said pkg still is screwed.
 
 umm, no ... the package itself is updated against the new copy while the old 
 copy exists for other packages that have already been built

Suspect you're ignoring soname changes, which is about all this patch 
would address- for example, ssl's old form of breakage.  In that case, 
*yes* the package gets updated, anything recompiled should get the 
correct lib (assuming the code knows the appropriate linker args), but 
the old vuln. lib still will hang around as long as anything refs it.


  Other angle is someone intentionally forcing usage of a known bad
  library that is still dangling.  Corner case, but doable.
 
 as i said, this is the invalid syntax:
 ... /usr/lib/libfoo.so.3 ...

Not to LD_PRELOAD :)

Haven't tried anything crazy, but suspect it can be abused to override 
to the old.

  Bit curious how this is going to behave if via linked in libs, new loc
  and old get loaded alongside...
 
 this would require multiple libraries to be involved in the equation and the 
 answer is undefined behavior which most certainly result in runtime 
 failures ...

Point there was that instead of just bailing with lib is missing, 
suspect it'll manage to run, then segfault at potentially crappy 
times.

~harring


pgpTux6lyn48U.pgp
Description: PGP signature


Re: [gentoo-portage-dev] New preserve-libs feature

2007-02-17 Thread Brian Harring
On Sat, Feb 17, 2007 at 10:09:35AM -0500, Mike Frysinger wrote:
 On Saturday 17 February 2007, Brian Harring wrote:
  On Sat, Feb 17, 2007 at 09:39:58AM -0500, Mike Frysinger wrote:
   On Saturday 17 February 2007, Brian Harring wrote:
Security impact is from a pkg potentially dragging along old libs; if
you've got a stable pkg that gets an update once every blue moon, it
can hold onto the lib for a *long* time while still using the lib;
thus if a vuln. in the lib, said pkg still is screwed.
  
   umm, no ... the package itself is updated against the new copy while the
   old copy exists for other packages that have already been built
 
  Suspect you're ignoring soname changes, which is about all this patch
  would address- for example, ssl's old form of breakage.  In that case,
  *yes* the package gets updated, anything recompiled should get the
  correct lib
 
 i'm not ignoring soname changes, those are exactly what i'm talking about
 
  (assuming the code knows the appropriate linker args)
 
 there is no such thing ... it's always -lfoo

The point is that it is *not* always -lfoo.  Commenting about packages 
that collapse, split libs apart, where -lorig becomes -lnew1 -lnew2.

Non-slotted package upgrade crossing a major version (assuming they're 
not being dumb asses), that scenario *does* occur, and it's not the 
same args.

Until all packages are upgraded (assuming an ugprade is available) to 
use the new layout, the old lingers.

Also, so help me, if you respond to valid critism with it doesn't 
actually matter as a retort, you're getting wedgied considering 
this is one of the scenarios this patch is supposed to address :p


Other angle is someone intentionally forcing usage of a known bad
library that is still dangling.  Corner case, but doable.
  
   as i said, this is the invalid syntax:
   ... /usr/lib/libfoo.so.3 ...
 
  Not to LD_PRELOAD :)
 
  Haven't tried anything crazy, but suspect it can be abused to override
  to the old.
 
 again, not in any scenario that actually matters ... so this too is a 
 pointless line of thought to pursue as it has no real security impact

Eh?  setuid comes to mind, or any other attempt to finagle privs via 
forcing the bad lib.  Combined with the scenario above (where a 
crappy pkg can hold the lib around for a *long* time), tend to think 
it matters.


Bit curious how this is going to behave if via linked in libs, new loc
and old get loaded alongside...
  
   this would require multiple libraries to be involved in the equation and
   the answer is undefined behavior which most certainly result in runtime
   failures ...
 
  Point there was that instead of just bailing with lib is missing,
  suspect it'll manage to run, then segfault at potentially crappy
  times.
 
 this is really an outside case and not worth worrying over

It's a change in behaviour; previously, would just flat out stop; now 
it'll run, but probably take a poo in your shoes.

Going from the potential of it won't run to it eats itself in 
special way is *not* something I'd blistfully write off as not 
worth worring over, considering what this change intends to address.

Additional comment, introducing the change makes it so that glsa's 
aren't really as accurate- version based, rather then lib.

~harring


pgp91Hw929egi.pgp
Description: PGP signature


Re: [gentoo-portage-dev] [RFC] Depending on active version

2007-01-30 Thread Brian Harring
On Tue, Jan 30, 2007 at 05:06:51PM +0100, Marius Mauch wrote:
 Sometimes a package has to depend on a specific version of a 
 slotted package being the active one to build correctly, like in 
 the current tr1 discussion on -dev [1] or with packages that 
 depend on the running kernel.

tr1 is partially addressed via addition of a 'binding' modifier for 
rdeps, to state that ||() deps are locked down after compilation.

Doesn't gurantee the user doesn't pop back to 3.4 after compilation, 
but that's their own mess.

 The idea is to add a special category (let's call it active for 
 now) that has the following properties:
 - this category doesn't exist in portdir or vdb (= no ebuilds)
 - when portage ($pkgmanager) encounters a active/foo atom in a 
 dependency string it executes some special code (e.g. 
 $PORTDIR/scripts/active-check/foo =active/foo-1) to determine if 
 that atom is satisfied

Non deterministic resolution; previous steps in the graph can cause 
that value to flip to a different setting by the time the 'dep' is 
encountered.

That's ignoring the kick in the nads usage of this will due to 
resolution...

 (and yes, this kinda goes with multi-repo/multi-format support)

Don't really see how this enables multiple standalone repos in any 
sane way, so that one requires justification...

~harring


pgpetWeCOF0tF.pgp
Description: PGP signature


[gentoo-portage-dev] Max parallelization setting

2006-10-10 Thread Brian Harring
Resending to portage dev ml to solicit comments...

Thoughts welcome, since it's an issue that must be dealt with to 
sanely do efficient parallelization (monkeying with make.conf and 
hardcoding vals isn't much of a solution).

Zac- comments?  You seem to have totally missed the original email ;)

-
Date: Sun, 1 Oct 2006 12:27:13 -0700
To: gentoo-dev@lists.gentoo.org
From: Brian Harring [EMAIL PROTECTED]
Subject: Re: [gentoo-dev]  Setting number of parallel builds for other 
build-systems than 'make'

On Sun, Oct 01, 2006 at 09:52:14AM -0700, Donnie Berkholz wrote:
 Tiziano Müller wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 Hi everyone
 
 It seems that setting the number of parallel builds using '-jN' does not
 only work for make, but also for scons and bjam (and maybe others as well).
 Since it isn't save to assume that '-jN' is the only option in MAKEOPTS,
 some filtering is needed.
 Now, SCONSOPTS (BJAMOPTS respectively) could be added to make.conf and used
 whenever one of those build-systems is being used. But we would probably
 have to add a ...OPTS for every build-system.
 
 What do you think? Better ideas?
 
 I think adding it for each build system is probably a good idea, 
 nobody's guaranteeing option-level compatibility with make. Optionally 
 falling back to using the few valid MAKEOPTS might be a nice addition.

I might be daft (likely), but why not just introduce a var indicating 
max parallelization instead?  Tweak portage to push that setting into 
MAKEOPTS=${MAKEOPTS+${MAKEOPTS} } -j${PARALLELIZATION}.

Might sound daft, but -j is hardcoded parallelization; if you're 
trying to run 3 build processes, the original var of -j isn't all that 
useful, nor will the hardcoded -j# var set for 3 package build 
processes be useful if you're doing a single build.

Depending on number of seperate package build processes possible, the 
# of build processes allocated per build process needs to vary 
(essentially).

Now... that's a bit ahead of what portage is doing atm, but folks 
*will* parallelize portage building (see bug 147516 if in doubt), so 
its not too far out.

Getting back to the actual topic, set this var, it's a raw int that a 
scons/bjam eclass can use to easily set appropriate var with the 
parallelization requested, thus unifying this particular knob; more 
importantly, it gives them a way to do what they're after while 
exposing a knob that the pkg manager can easily fiddle with for global 
parallelization needs.

Only downside to it I see is that it requires mangling the pkg manager 
to translate the parallelization setting into MAKEOPTS+=-j#; can't 
really get around that however due to the fact MAKEOPTS is already 
forced in installed configuration though.

Thoughts?
~harring


pgpZOWWRmNkmO.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Max parallelization setting

2006-10-10 Thread Brian Harring
On Tue, Oct 10, 2006 at 03:20:55AM -0700, Zac Medico wrote:
 Brian Harring wrote:
  I might be daft (likely), but why not just introduce a var indicating 
  max parallelization instead?  Tweak portage to push that setting into 
  MAKEOPTS=${MAKEOPTS+${MAKEOPTS} } -j${PARALLELIZATION}.
 
 The idea sounds good, but I'm not clear on all the details.  It
 seems like there are several distinct parts:
 
 1) Ebuild maintainter sets a metadata variable to indicate the level
 of parallelization possible in a build.

RESTRICT moreso indicating if it's parallelizable or not.

 2) Gentoo user sets a configuration option indicating the maximum
 level of parallelization spread across multiple builds at a given time.


 3) Package manager uses the user's config to allocate an appropriate
 PARALLELIZATION at build time, based on 1 and 2 above.  Then the
 src_compile() function of the ebuild translates PARALLELIZATION into
 the appropriate build system flags (possibly with the help of an
 eclass function).

Not necessarily src_compile, but close enough; the details of how 
MAX_PARALLELIZATION gets shoved in is semi package specific, although 
it's kind of implicit at this point that -j# for MAKEOPTS would likely 
be directly fooled with.

For other build systems, rely on eclasses handling it; for MAKEOPTS, 
would be preferable to do the same imo, but that's not an easy 
transition.

Mind you since there isn't a way to adjust the allowed slices 
(essentially) while a compile is underway, this won't hit 100% 
utilization- further, src_install still abides by MAKEOPTS, but it's 
not like -jN is going to help much there.

That said, it's better then the current crapshoot required for trying 
to do parallel builds; either you have to monkey patch make.conf 
everytime, or try env overrides for it, both of which aren't 
incredibly friendly/simple if you're just trying to do an upgrade that 
abuses your duo/quad.
~harring


pgpZKzky1JHbB.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Moving ebuild-related where they belong

2006-09-07 Thread Brian Harring
On Thu, Sep 07, 2006 at 09:32:04AM -0700, Zac Medico wrote:
 Simon Stelling wrote:
  repo-level profile, we move parts of the EAPI out into the tree, which
  is a bad idea because we are unable to support multiple versions. As the
  EAPI needed for the ebuild is unknown when sourcing
  install-helpers.eclass, we're actually forced into using that one and
  only version, which is quite limiting.
 
 Well, if the metadata generation step is viewed as being separate from the 
 rest,
 and the helpers aren't needed during that step, then it's possible to get the
 EAPI from the ebuild without the helpers being in the environment.  Once the
 EAPI is known, the package manager can use that to determine what else (if
 anthing) needs to be added to the environment.  Then you'd only need a way to
 tell the package manager which EAPI levels the functions in your
 install-helpers.eclass (or whatever) apply to.
 
  So, the correct way to do it is to define an EAPI=1 that does no longer
  include these helper functions and make the eclasses/ebuilds that use it
  inherit the eclass manually. However, this will need to run through -dev
  and I'm afraid the ebuild devs wouldn't like it at all :-/
 
 They won't like it because it's expected that EAPI will provide the necessary
 information.  Why should they have to inherit some special eclass if it's
 already implied by the EAPI that they've specified?

Blubb left out the real kicker.

Make this change, and it means that all overlays that can function as 
standalone, must bundle the eapi helpers themselves.

This is ignoring the potential fun of an overlay that redefines an 
eapi locked function.

Understand initial intention of wanting to make the scripts usable by 
all pkg managers (pushed this myself shortly after I became a portage 
dev), but the 
A) requirement of trying to keep helper functionality exactly in sync 
per tree,
B) fragmentation this implicitly enables
isn't good.
~harring


pgpnCDHRvNPUM.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Moving ebuild-related where they belong

2006-09-07 Thread Brian Harring
On Thu, Sep 07, 2006 at 07:11:01PM +0200, Simon Stelling wrote:
 Brian Harring wrote:
  Make this change, and it means that all overlays that can function as 
  standalone, must bundle the eapi helpers themselves.
 
 Standalone-repos will have that
 problem, but there is none yet to my knowledge. Sure, the problem
 persists for future repos, but it's not so much of a deal to copy an
 eclass once.

Err... that's rather whacky.

You're suggesting that a fix in an eapi class must be copied into 
*every* standalone repo; lets pretend for a moment that glep19 
actually got somewhere, and subtree generation is possible.

We'll pretend that a 1k admins use it to filter down to 
lamp/mailserver/whatever, point is, they generated their own stand 
alone.

See where the we'll just copy it around starts to break down and go 
mildly apeshit?

 Also, as it stands today, portage IS connected to gentoo-x86 through
 exactly these helpers.

Not really.

Yes, portage knows the names of a few pkgs, but those pkgs are defined 
in the tree; in that respect, it's actually semi-sanely designed.

Point there is that portage can be ran against a foreign tree without 
needing gentoo-x86; frankly, claiming otherwise is a bit daft and 
requires real world examples of where it breaks down.


 Upon the needs of the portage tree the functions
 in portage are changed, and this is simply not correct.

This assertion doesn't make any sense; clarify please.


  This is ignoring the potential fun of an overlay that redefines an 
  eapi locked function.
 
 eclasses in overlays that also exist in the tree are fun anyway. If
 people are messing with eutils, we tell them that it is entirely their
 responsibility if something breaks. Same goes for install-helpers.eclass.
 That being said, this problem already exists today: A custom eclass
 could simply define a function 'dosbin' and ebuild.sh would use that
 one. While it's a possible cause for breakage, it's not an argument
 against the move.

There's a real difference however for the dosbin example; the eclass 
can wrap the functionality, not flat out replacing it.  If it's a 
func, you're boned trying to get at the original, non-wrapped dosbin.

Move it into the tree, and you enable people to accidentally cause 
mayhem on a repository wide scale via tinkering with bundled bits of 
the eapi format; folks can do it now, but this makes it *far* easier.

  A) requirement of trying to keep helper functionality exactly in sync 
  per tree,
 
 If the helpers were a part of the EAPI those trees would have to verify
 that the EAPI portage provides matches their needs.

The assertion there was that you would be stuck copying eclasses 
across all repos that exist; thus not addressing that assertion 
(should have been clearer, pardon).

Thing I think you're missing here is that the ebuild format is not 
bound to gentoo-x86, nor is portage; thus trying to move intrinsic 
bits of the format into gentoo-x86 goes pretty hardcore against the 
goal of ever trying to sanely support N standalone repos due to the 
fact each repo now can now carry it's own potential offshoot of the 
eapi spec.
~harring


pgpUQjEKtSjJO.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Moving ebuild-related where they belong

2006-09-07 Thread Brian Harring
On Thu, Sep 07, 2006 at 10:22:38AM -0700, Zac Medico wrote:
 Simon Stelling wrote:
  Zac Medico wrote:
  Well, if the metadata generation step is viewed as being separate from the 
  rest,
  and the helpers aren't needed during that step, then it's possible to get 
  the
  EAPI from the ebuild without the helpers being in the environment.  Once 
  the
  EAPI is known, the package manager can use that to determine what else (if
  anthing) needs to be added to the environment.  Then you'd only need a way 
  to
  tell the package manager which EAPI levels the functions in your
  install-helpers.eclass (or whatever) apply to.
  
  That is a workaround, but it makes a pretty hard link between the
  package manager and the functions, which is exactly what I am trying to
  cut through with the patch. Sure, it'd be a workaround, but just keeping
  them in the portage package until EAPI=1 is reached is one too...
 
 It's not intended as a workaround and it shouldn't create a hard link between
 the package manager and the functions.  The idea is that the tree should 
 provide
 the necessary information for the package manager (portage or not) to 
 determine
 how it should setup the environment for a given EAPI.  The file containing the
 functions from install-helpers.eclass would have to be labeled in some way 
 such
 that any package manager would know that those functions are required when
 EAPI=0 (and possible other EAPI levels).

Just so we're clear... if gentoo-x86 wants to define their own base 
template all ebuilds in that repo use, that's fine.  That's a 
different beast from moving the format definition into the tree 
though.

Kind of curious if either of you have thought through the implications 
of moving this functionality into the tree for the vdb and binpkgs 
also (short version, even with ebds env handling its not going to be 
pretty for backwards compatibility).

  So, the correct way to do it is to define an EAPI=1 that does no longer
  include these helper functions and make the eclasses/ebuilds that use it
  inherit the eclass manually. However, this will need to run through -dev
  and I'm afraid the ebuild devs wouldn't like it at all :-/
  They won't like it because it's expected that EAPI will provide the 
  necessary
  information.  Why should they have to inherit some special eclass if it's
  already implied by the EAPI that they've specified?
  
  It wouldn't be implied anymore. The install-helpers.eclass would
  actually be like every other eclass, like eutils fex.
 
 Fine, but the EAPI can still be used to imply that some other functions may or
 may not be available.

Not really.  Via moving parts of the format into gentoo-x86 it's 
implciitly setting it up such that whatever tree portage is working 
against now defines EAPI.

Sounds whacky, but think it through; instead of trying to verify that 
3 package managers are EAPI compliant, it's now dependant on the 
environment the tree defines, as such the tree can redefine the env 
without bumping the EAPI.

Yes that's stupid to do, but moving the format into the tree enables 
it.  Theres a reason no other package format has their actual format 
definitions (env setup in our case) shoved into the repo, and thats 
one of them.

  Actually, the reason they won't like it for will more likely be that it
  requires adding another eclass to the inherit line for ~15'000 ebuilds.
 
 See, that statement shows me that you've missed my point that EAPI can be used
 to imply which functions are implicitly available.  It would be redundant to
 inherit an eclass containing functions that are already implied by the EAPI.

Would require 24,600 (last count) mods to ebuilds to specify an elib 
import moreso; automagically trying to import an elib out of a 
repository is pretty much guranteed to not behave as desired (overlays 
again come to mind).

Eclasses are designed as crappy oop; what this change means for 
eclasses/elibs is that instead of reusing functionality, functionality 
gets copy/pasted across repositories- not a good thing.

~harring


pgpeOqGvPDsqS.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Refactoring ebuild.sh

2006-08-27 Thread Brian Harring
On Sun, Aug 27, 2006 at 10:26:28AM +0200, Simon Stelling wrote:
 Brian Harring wrote:
 diefunc()
 dump_trace()
  
  these are general utility, not debugging.
 
 Where would you stick them? 'die' to 'ebuild helpers' and 'dump_trace'
 to 'internals'?

util, same for hasq.

  If you're going to try breaking the beast up, suggest you go the step 
  further and break up what must be supplied from the ebuild.sh 
  implementation, and what is bound to the ebuild's environment.
  
  
  Internal eclass functions:
 newdepend()
 newrdepend()
 newpdepend()
 do_newdepend()
  
  These aren't eclass functions- usable without involving eclasses.
  inherit ought to be in internal eclass functions also (yes it's user 
  visible, but it's also ebuild.sh implementation specific).
 
 Where to put them, then?

...they're ebuild functions.  stick them with the other ebuild 
functions (use_with fex).

Personally, I think trying to seperate it up at this level is probably 
not a great idea; splitting the EAPI specific bits into seperate 
modules, yes, but when/if EAPI=1 arrives changing chunks, it's going 
to be a mess going through each module and ifdef'ing functions.


  I moved them into own files within a subdir in bin/, called modules, and
  source these files at the right time, so that nothing changes
  functionality wise. The global scope code I left untouched, except for a
  bunch of exports that set sane defaults for the do*/new* helpers.
 
  This cuts down ebuild.sh to 403 lines, which makes it far more readable,
  IMO.
 
  Except there are now magic vars hidden away in submodules.  If trying 
  to seperate it into seperate blocks, document 'em.
 
 Uhm, how would that documentation part work? What exactly do you want
 documented? (What do it and 'em refer to?)
 
 About the magic vars.. I wasn't sure either whether putting the defaults
 into the modules was a good idea. I thought it would be good because
 they are not used anywhere else (didn't check), but I can just as well
 stick them back into ebuild.sh.

Sticking magic vars (globals mostly, eclass_depth for example) in 
locations seperate from the functionality that uses it needs to be 
documented; moving all of the globals up to ebuild.sh means that for 
any code to reuse those modules without using ebuild.sh, it has to 
maintain it's own globals list.

So... magic vars need to stay in the module that uses them, and need 
documentating if another module relies on it; with ebuild.sh, this 
isn't an issue, everything is guranteed loaded, with submodules it's a 
bit more of an issue.


  I also set all the functions that aren't meant to be overridden by saved
  env, profile bashrc, portage bashrc or ebuilds/eclasses readonly.
  
  Don't think much thought was put into this tbh- use* and has* (all but 
  useq and hasq basically) are bound to the ebuild environment.
 
 This is actually true.
 
  Why?  Because they got mangled transitioning from 2.0.50 2.0.51, for 
  the ebuild to function properly it needs to use the original func from 
  the environment, not what the ebuild.sh implementation tries to force.
  
  Identified all of them that are affected by this?
 
 Well, as it stands now, all variables except the sandbox related ones
 are defined after env  bashrc are sourced and before the ebuild is
 sourced. That means that the env can't override them, which means that
 they are considered implementation specific in the current
 implementation. I agree that consideration is nonsense, though ;)

 So, to address the problem: What functions are bound to the ebuild
 rather than ebuild.sh?
pkg_*, src_*, *opts, *into, *depend .

Basically, think of which are implementation specific (portageq) vs 
which aren't directly reliant on implementation details.  If it's not 
implementation specific, it's likely bound to the ebuild env.

 Beside {use,has}{,v} I don't know any.
 
  Further, all of these are still overridable by *bashrc and env.
 
 How?
readonly -f .

bashrc, env, etc, all can force different definitions of functions.  

This is why ebd sets it's required functions and marks them readonly 
*before* it even goes near ebuild data; do it the other way around, 
you're not guranteeing anything.


  Drop the readonly crap; it's not actually blocking anything, just 
  attempting to catch non real issues; in the process, it actually is a 
  step in the wrong direction.
 
 I will drop it for now, but I would like to know the answers to above
 questions nevertheless.
 
  Reordering is a bit arbitrary, but fine.  Attempted env changes, would 
  avoid those.
 
 My primary goal is to cut down ebuild.sh to a managable size. I don't
 see how that can be solved with a different env handling, tbh.

Env handling affects the flow of the code; never said it reduces the 
total size of code though (just said don't go screwing with readonly).
~harring


pgp2VkSZo3f1G.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Refactoring ebuild.sh

2006-08-26 Thread Brian Harring
On Sat, Aug 26, 2006 at 07:54:41PM +0200, Simon Stelling wrote:
 Hi all,
 
 ebuild.sh is a mess. There are a lot of functions scattered to the four
 winds. Searching for a function in ebuild.sh takes a lot of time, and it
 is very tiring to scroll down huge chunks of totally unrelated functions
 just to find the next one which one is after. The current layout also
 makes it very hard to find the spots where new functions/variables are
 loaded from other files, what gets overwritten when profiles/eclasses
 are sourced etc.
 
 I would like to change that, so I tried to categorize all the functions
 we have in ebuild.sh. I ended up with this setup:

categorization is a bit whonky in a few spots-

 Sandbox:
   addread()
   addwrite()
   adddeny()
   addpredict()
   lchown()
   lchgrp()
 
 Unknown?:
   esyslog()
 
 Debugging:
   register_die_hook()

ebuild specific func, while debugging, it's not ebuild.sh debugging.

   diefunc()
   dump_trace()

these are general utility, not debugging.

   debug-print()
   debug-print-function()
   debug-print-section()
 
 Internal helpers:
   strip_duplicate_slashes()
   remove_path_entry()
 
 Action functions:
   dyn_setup()
   dyn_unpack()
   dyn_clean()
   dyn_compile()
   dyn_test()
   dyn_install()
   dyn_preinst()
   dyn_help()
 
   Abort handlers:
   abort_handler()
   abort_compile()
   abort_unpack()
   abort_test()
   abort_install()
   killparent()

internal is a better label.


 Default functions for ebuilds:
   pkg_setup()
   pkg_nofetch()
   src_unpack()
   src_compile()
   src_test()
   src_install()
   pkg_preinst()
   pkg_postinst()
   pkg_prerm()
   pkg_postrm()
   pkg_config()
 
 Ebuild helpers:
   into()
   insinto()
   exeinto()
   docinto()
   insopts()
   diropts()
   exeopts()
   libopts()
   keepdir()
   unpack()
   econf()
   einstall()
   use()
   usev()
   useq()
   has_version()
   portageq()
   best_version()
   use_with()
   use_enable()
   gen_wrapper()
   check_KV()
   inherit()
   EXPORT_FUNCTIONS()

Most of these are mislabed; gen_wrapper is an internal function that 
ebuilds shouldn't know about, nor ever access.  It's existance was 
strictly to cover gcc's ass; considering it's usage was strictly 
implementation side (rather then ebuild side), probably can be 
dropped.

Either way, ain't a helper.  has is missing also...

If you're going to try breaking the beast up, suggest you go the step 
further and break up what must be supplied from the ebuild.sh 
implementation, and what is bound to the ebuild's environment.


 Internal eclass functions:
   newdepend()
   newrdepend()
   newpdepend()
   do_newdepend()

These aren't eclass functions- usable without involving eclasses.
inherit ought to be in internal eclass functions also (yes it's user 
visible, but it's also ebuild.sh implementation specific).


 I moved them into own files within a subdir in bin/, called modules, and
 source these files at the right time, so that nothing changes
 functionality wise. The global scope code I left untouched, except for a
 bunch of exports that set sane defaults for the do*/new* helpers.
 
 This cuts down ebuild.sh to 403 lines, which makes it far more readable,
 IMO.

Except there are now magic vars hidden away in submodules.  If trying 
to seperate it into seperate blocks, document 'em.


 I also set all the functions that aren't meant to be overridden by saved
 env, profile bashrc, portage bashrc or ebuilds/eclasses readonly.

Don't think much thought was put into this tbh- use* and has* (all but 
useq and hasq basically) are bound to the ebuild environment.

Why?  Because they got mangled transitioning from 2.0.50 2.0.51, for 
the ebuild to function properly it needs to use the original func from 
the environment, not what the ebuild.sh implementation tries to force.

Identified all of them that are affected by this?

Further, all of these are still overridable by *bashrc and env.


 If an ebuild would ever try to re-define a function which is 
 internally used by portage, it would fail sourcing, which is a good 
 thing, IMO.

Actually it doesn't fail sourcing.

echo 'f() { echo grande tiza; }; :;'  t;
. t
readonly -f f
. t  echo it relies on the last exit code, which was $?


 So, if noone objects, I would like to push the attached patch into SVN.

Drop the readonly crap; it's not actually blocking anything, just 
attempting to catch non real issues; in the process, it actually is a 
step in the wrong direction.

If a func is going to marked as immutable, it's implicitly 
implementation specific- as such it has no business winding up in the 
environment file.

Reordering is a bit arbitrary, but fine.  Attempted env changes, would 
avoid those.

~harring


pgpJc9zkoGGA0.pgp

Re: [gentoo-portage-dev] [PATCH] per-package use.mask (bug 96368)

2006-08-06 Thread Brian Harring
On Fri, Aug 04, 2006 at 08:46:34PM -0700, Zac Medico wrote:
 Brian Harring wrote:
  On Fri, Aug 04, 2006 at 12:38:39PM -0700, Zac Medico wrote:
  I haven't seen a specification for use dependencies yet, so I'm not quite 
  sure how they'd work.
  cat/pkg-ver[use1,use2,-use3,use4]
  cat/pkg-ver[use]
  etc.
 
 Okay, so the only difference from package.use format is that whitespace is 
 replaced by square brackets and commas?

Yep- bracket/comma usage allows the atom and use reqs to bundled as 
one token.

  Is the existing format of of use.mask bad?  What about package.use?  The 
  implementation that I've proposed is a combination of these two formats 
  that everyone is already familiar with.
  
  No... the issue is that this _is_ basically a crappy form of 
  package.mask supporting use-deps; why use an alt syntax for it then?
 
 Well, no part of portage currently supports use-deps.  Therefore, 
 use-deps are an alternate syntax in themselves.  The implementation 
 that I've proposed uses the same type of syntax that portage already 
 uses in package.use files.

use-dep syntax is usable globally however- package.use is strictly 
config file, if/when portage gains use dep, killing off the 
package.use format for the globally usable use-dep will need to occur.

With that in mind, why not just do it from the get go?

  Just use what was originally intended, and at some point down the line 
  when either portage grows use deps, or it gets replaced, folks can 
  just copy the package.use.mask into package.mask, and wipe the file.
  
  ~harring
 
 Such a migration (if it ever takes place) could just as well be 
 performed by a simple conversion tool.
Or if just using use-dep, a cat :P
~harring


pgpEbCNxfV1Es.pgp
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] per-package use.mask (bug 96368)

2006-08-06 Thread Brian Harring
On Sun, Aug 06, 2006 at 02:54:36AM -0700, Zac Medico wrote:
 Brian Harring wrote:
  On Fri, Aug 04, 2006 at 08:46:34PM -0700, Zac Medico wrote:
  Brian Harring wrote:
  On Fri, Aug 04, 2006 at 12:38:39PM -0700, Zac Medico wrote:
  I haven't seen a specification for use dependencies yet, so I'm not 
  quite sure how they'd work.
  cat/pkg-ver[use1,use2,-use3,use4]
  cat/pkg-ver[use]
  etc.
  Okay, so the only difference from package.use format is that whitespace is 
  replaced by square brackets and commas?
  
  Yep- bracket/comma usage allows the atom and use reqs to bundled as 
  one token.
 
 Isn't there more of a difference than just in the parsing?

Not for what I'm suggesting- I'm suggesting just using use dep syntax 
for package.use.mask.

You've already got the code for the masking in your patch now, all you 
have to do is just change the parsing a bit.

 It 
 seems to me that we'd also have to implement use-dep matching in 
 order to correctly support use-dep syntax.

If you were actually supporting use deps, yes.  You're not 
however- package.use.mask is just a kludge in the (hopefully short) 
interim.

I'm suggesting that you think a bit forward- use use-dep syntax for it 
now rather then having to change it down the line.
~harring


pgpHaDIntI5xO.pgp
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] per-package use.mask (bug 96368)

2006-08-04 Thread Brian Harring
On Thu, Aug 03, 2006 at 12:23:58PM -0700, Zac Medico wrote:
 Hi everyone,
 
 I've written a patch [1] that adds support for package.use.mask in the 
 profile.  It should behave exactly as use.mask currently does except that it 
 allows USE flags to be masked for specific packages rather than for all 
 packages.
 
 In previous discussion it's been noted that package.mask + use deps would be 
 an alternative way to express this type of masking.  However, 
 package.use.mask + use deps would have the added ability to mask certain USE 
 flags based on other flags that have been selected for a package.  Either 
 way, the per-package use.mask functionality is certainly needed.  Shall we go 
 ahead with the package.use.mask implementation or not?
 
 Zac
 
 [1] 
 http://dev.gentoo.org/~zmedico/portage/branches/2.1/patches/package.use.mask.patch

Since you're sliding this in, why not slide it in using use dep 
syntax?

No, not going to fight over this not being in package.mask, what I'm 
saying is this _is_ masking of a use dep atom, just use use dep syntax 
in the file instead.

If y'all get use deps, it'll be a bit simpler for folks to support 
then the existing crappy format used imo.

Plus, parsing it's easy.
~harring


pgprlqZ7Bqnh3.pgp
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] per-package use.mask (bug 96368)

2006-08-04 Thread Brian Harring
On Fri, Aug 04, 2006 at 12:38:39PM -0700, Zac Medico wrote:
 Brian Harring wrote:
  On Thu, Aug 03, 2006 at 12:23:58PM -0700, Zac Medico wrote:
  Hi everyone,
 
  I've written a patch [1] that adds support for package.use.mask in the 
  profile.  It should behave exactly as use.mask currently does except that 
  it allows USE flags to be masked for specific packages rather than for all 
  packages.
 
  In previous discussion it's been noted that package.mask + use deps would 
  be an alternative way to express this type of masking.  However, 
  package.use.mask + use deps would have the added ability to mask certain 
  USE flags based on other flags that have been selected for a package.  
  Either way, the per-package use.mask functionality is certainly needed.  
  Shall we go ahead with the package.use.mask implementation or not?
 
  Zac
 
  [1] 
  http://dev.gentoo.org/~zmedico/portage/branches/2.1/patches/package.use.mask.patch
  
  Since you're sliding this in, why not slide it in using use dep 
  syntax?

 I haven't seen a specification for use dependencies yet, so I'm not quite 
 sure how they'd work.
cat/pkg-ver[use1,use2,-use3,use4]
cat/pkg-ver[use]
etc.

pretty simple, and was layed out in the original we need use deps 
bug.

  No, not going to fight over this not being in package.mask, what I'm 
  saying is this _is_ masking of a use dep atom, just use use dep syntax 
  in the file instead.
  
  If y'all get use deps, it'll be a bit simpler for folks to support 
  then the existing crappy format used imo.
 
  Plus, parsing it's easy.
  ~harring
 
 Is the existing format of of use.mask bad?  What about package.use?  The 
 implementation that I've proposed is a combination of these two formats that 
 everyone is already familiar with.

No... the issue is that this _is_ basically a crappy form of 
package.mask supporting use-deps; why use an alt syntax for it then?  
Just use what was originally intended, and at some point down the line 
when either portage grows use deps, or it gets replaced, folks can 
just copy the package.use.mask into package.mask, and wipe the file.

~harring


pgpxZHH0wSRkC.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Atom matching behavior

2006-07-31 Thread Brian Harring
On Mon, Jul 31, 2006 at 07:42:23PM +0200, Marius Mauch wrote:
 Was just brought to my attention that the =* operator doesn't work as I
 thought, as for example =foo-1.2* matches foo-1.20 as well as foo-1.2.3.
 This wouldn't be a bug problem if it could be used as a general glob
 operator like with =foo-1.2.*, but it's use is strictly limited to the
 above version (can only be used when a version component separator may
 appear), so atm there is no facility to reliably lock an atom at a
 specific version component when you have to account for multi-digit
 components.
 Now the question is if we want this glob-style behavior or not. From
 the code comments it seems to be intentional,

Intentional in the sense that it was originally implemented as a slice 
comparison (likely due to that being far simpler then mangling older 
versions of ver_cmp).

=2.1 ver_cmp can be mangled easy enough now though.

 but I'd suspect that many
 people share my original assumption and expect it to only match full
 version components

Hear a bit of screaming from it once every 4-6 months; personally, I 
interpret that as devs know which to use usually- additionally, once 
the (bluntly) hissy fit from the dev subsides, and they're reminded 
yes it's annoying, but if you want it changed take it to dev to get 
consensus folks promptly forget about it.

Either they're silently working around it, or it's not that much of an 
issue (I suspect the latter, but am neutral towards the change).

 (as that is the much more common use case). Doesn't
 help that the atom description in ebuild(5) doesn't specify the
 behavior for this case either, 
 
 *  means  match  any version of the package so long as the specified
 base is matched
 
 can be read both ways.
 
 Opinions?

Should be discussed on -dev, not here imo; they're the ones affected 
by it (it's essentially their standard after all).  Changing it?  
Sure, but it's a required eapi bump; portage chokes on .* now.

I'd also not bump eapi just for one change; there is a boatload of 
other stuff that's waiting for an apt time to be pushed out together.

~harring


pgpc4fKGA4uWF.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Atom matching behavior

2006-07-31 Thread Brian Harring
On Mon, Jul 31, 2006 at 11:53:42PM +0200, Simon Stelling wrote:
 Marius Mauch wrote:
  Opinions?
 
 I'd never expect it to match 1.20, so this is a bug to me.

Not a bug; it has been that way for a long tim (at least through .50 
last time I checked), iow it's standard now (since cruft in the 
tree/folks overlays may rely on it, even if it may be retarded).

As said in the other email, if after changing it, sure, but do it 
proper- don't just stick it in ;)
~harring


pgpRr9WVpVc7Q.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Re: Atom matching behavior

2006-07-31 Thread Brian Harring
On Mon, Jul 31, 2006 at 04:55:09PM -0700, Drake Wyrm wrote:
 Brian Harring [EMAIL PROTECTED] wrote:
  On Mon, Jul 31, 2006 at 07:42:23PM +0200, Marius Mauch wrote:
   Was just brought to my attention that the =* operator doesn't work
   as I thought, as for example =foo-1.2* matches foo-1.20 as well as
   foo-1.2.3.
 snip
   but I'd suspect that many people share my original assumption and
   expect it to only match full version components
  
  Hear a bit of screaming from it once every 4-6 months; personally, I 
  interpret that as devs know which to use usually- additionally, once 
  the (bluntly) hissy fit from the dev subsides, and they're reminded 
  yes it's annoying, but if you want it changed take it to dev to get 
  consensus folks promptly forget about it.
 
 You mean a consensus on -dev like the one regarding the Sunrise project?
 
  Either they're silently working around it, or it's not that much of an 
  issue (I suspect the latter, but am neutral towards the change).
 
 Or ignoring it because it's not worth the heartache. Or they feel it to
 be more likely that their input will be rejected by devs who just don't
 feel like working on it, but also don't want their babies touched by
 foreign hands. See Bug #69343 and everything marked as a dupe against it
 for a fine example of that mentality.

Either you're trolling, or your whinging (bluntly).

Either way, it's not even remotely related to an EAPI bump, so 
kindly don't thread hijack (got enough of that going on in sunrise 
thread already).

~harring


pgp7SsbyS9hEA.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Re: Atom matching behavior

2006-07-31 Thread Brian Harring
On Mon, Jul 31, 2006 at 06:12:46PM -0700, Drake Wyrm wrote:
 Brian Harring [EMAIL PROTECTED] wrote:
  On Mon, Jul 31, 2006 at 04:55:09PM -0700, Drake Wyrm wrote:
   Brian Harring [EMAIL PROTECTED] wrote:
On Mon, Jul 31, 2006 at 07:42:23PM +0200, Marius Mauch wrote:
 Was just brought to my attention that the =* operator doesn't
 work as I thought, as for example =foo-1.2* matches foo-1.20 as
 well as foo-1.2.3.
   snip
 but I'd suspect that many people share my original assumption
 and expect it to only match full version components

Hear a bit of screaming from it once every 4-6 months; personally,
I interpret that as devs know which to use usually- additionally,
once the (bluntly) hissy fit from the dev subsides, and they're
reminded yes it's annoying, but if you want it changed take it to
dev to get consensus folks promptly forget about it.
   
   You mean a consensus on -dev like the one regarding the Sunrise
   project?
   
Either they're silently working around it, or it's not that much
of an issue (I suspect the latter, but am neutral towards the
change).
   
   Or ignoring it because it's not worth the heartache. Or they feel it
   to be more likely that their input will be rejected by devs who just
   don't feel like working on it, but also don't want their babies
   touched by foreign hands. See Bug #69343 and everything marked as a
   dupe against it for a fine example of that mentality.
  
  Either you're trolling, or your whinging (bluntly).
 
 Mostly trolling, but it's a valid point. The technical issue is not
 nearly as daunting as the political one.

And not doing something because of fear of screaming/politics means 
that those using such tools get their way (one of the few cases where 
it pays to be a stubborn bastard who'll kick back).

Meanwhile, this _is_ thread hijacking, getting back to the subject is 
a better use of folks time and channels normally sane s/n ratio.

Besides...  People aren't going to bitch about this one, it's a matter 
of trying to keep people in the loop rather then portages usual modus 
operandi of just slipping changes in, with portage devs being the ones 
knowing about it (and yes, I was guilty of it in the past too). :)

This *should* require an EAPI bump, so at the very least repoman will 
need a few tweaks, and people will need to be educated a bit re: the 
fact the version op. behaves differently for EAPI0 vs EAPI1.
~harring


pgpWaeQw2fRNp.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Atom matching behavior

2006-07-31 Thread Brian Harring
On Mon, Jul 31, 2006 at 10:11:06PM -0400, Mike Frysinger wrote:
 On Monday 31 July 2006 21:30, Brian Harring wrote:
  On Mon, Jul 31, 2006 at 09:11:35PM -0400, Mike Frysinger wrote:
   On Monday 31 July 2006 18:48, Brian Harring wrote:
Should be discussed on -dev, not here imo;
  
   why ?  short term we're gaining functionality:
   simply add .*
 
  1) portage isn't the only package manager
 
 invalid reason

 until a spec exists. portage's behavior is the spec
 
  2) it's not a backwards compatible change
 
 so ?  same way any new feature gets merged: support it now, dont use it till 
 later

And EAPI exists to version those changes so they can be rolled out 
_now_, not cross your finger for six months and hope nobody uses it 
before then inadvertantly triggering some massive mess like bug 114798  
(the infamous sync ate itself due to virtual/*).

Brutal truth here is that once it's available, people *will* use it 
sooner then when they're supposed to.  Further, the realistic estimate 
of deploying it, and sitting on it for 6 months *still* will lead to 
people hitting it- this isn't some minor idiot cache change, this 
is a change in dependency parsing- rooting out why a users portage 
installation is selecting some later dep in a ||() block might be easy 
for devs once they've seen it, but it ain't going to make sense to 
users (gurantee even if a dev knows of the change, it's going to take 
a bit for it to click also).

So... two potentials here.

1) Deploy it, and pretend that it'll be unused for 6 months.  Users 
hit weirdness that is fun to track down.

2) EAPI bump it.  Portage tells you it can't play with that node, and 
that you need to upgrade.  Can use it _immediately_.

EAPI was added to specifically stop the practice of eh... just shove 
it in, we'll let it cook for a bit and hope people have upgraded by 
when it gets used- benefit is you get immediate access to the new 
functionality, only con is that you have to flip one var in the 
ebuild.
~harring


pgpVLFtFpLcs4.pgp
Description: PGP signature


Re: [gentoo-portage-dev] [RFC] Naming Conventions

2006-07-22 Thread Brian Harring
On Sat, Jul 22, 2006 at 04:25:01PM -0700, Zac Medico wrote:
 Chris White wrote:
  1) Create aliases to the new functions, then at some
  yet-to-be-determined point, kill the aliases and bomb on the scripts
  (this suffers from procrastination).
  
  2) Make an official release with the new function names and no aliases,
  as well as the soon to come docs.  I sort of like this method because
  those with official portage tools can adjust their scripts, and simply
  alter the depend atoms for = (new API versions) and = (old versions),
  effectively forcing/preventing upgrades.
 
 I vote for #1 because it's smoother and easier (which is good for me 
 especially because I do releases).  The disruptive change proposed in #2 
 seems like it would cause unnecessary problems with no practical advantage 
 over #1.  

Suggest you mark the aliases funcs in some way in the code so that 
they can be spotted via scanning.

Use a convience func; 

def alias_func(alias_name, alised_func):
def f(alias_name, aliased_func, *a, **kw):
import warnings
warnings.warn(don't use %s, use %s % (alias_name, 
alias_func.__name__)
return aliased_func(*a, **kw)
return f

pkgsplit = alias_func('pkgsplit', PkgSplit)

upshot, you can puke warnings via it, and you've got something to scan 
the actual namespace for; makes it quite a bit easier to switch off 
the aliases.
~harring


pgpCrN0nW5uiC.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Porage API Documentation Proposal

2006-07-16 Thread Brian Harring
On Mon, Jul 17, 2006 at 03:12:25AM +0900, Chris White wrote:
 This document is meant to serve as a proposal for the documentation of portage
 code using epydoc[1] and custom doc blocks.

epytext actually- that's what relies on, and is supported by 
other doc manglers.


 2. Other portage API functions used:
 
   Functions used:
   - L{pkgsplit pkgsplit}
   - L{example portage.example}

Bad idea.  doc strings rules for doc manglers, the base docstring 
bleeds through to derivative methods iff the prototype hasn't been 
mangled.  So... you state in the base method, I use blah.  Now 
you're requiring every derivative to either
1) rewrite the whole docstring
2) do replace tricks to slip in their func additions.

#1 sucks, #2 is a good way to wind up with whacky docstrings (rather 
fragile).


 Known Issues
 ===
 
 The following are known issues:
 
 1. A large increase in the code size will occur becuase of this.  Sugestions 
 were made to strip out the docstrings, but it was noted that this would make 
 it very difficult to work with tracebacks.

No reason to strip it out- file size isn't going to make a difference 
(the slow bits in terms of imports is forced execution in the module 
loadup, import lookup, and loading chunks of stdlib).
~harring


pgp9rr97M2pUh.pgp
Description: PGP signature


Re: [gentoo-portage-dev] support for registering pre/post phase hooks?

2006-07-05 Thread Brian Harring
On Tue, Jul 04, 2006 at 11:04:07PM -0700, Zac Medico wrote:
 Hi everyone,
 
 Has anyone noticed that java-pkg-2.eclass makes it difficult to use 
 pre/post phase hooks in /etc/portage/bashrc?

Yep, they were greenlighted _only_ because it's a workaround for bug 
#56408, bad environment handling by portage.

 Wouldn't it be wise to 
 provide a function for registering these types of things, so that 
 more than one can be registered without stomping out others that 
 were registered before?

No.  Pre/post are user phase hooks only- their only real reason for 
existance is because with proper environment handling (namely true 
store/restoring of the env without having to try recreating it), it's 
the only way to preserve bashrc functionality.

Strictly user level hooks, mandated that way to prevent the ebuild 
phases from going 2x what's there.  If (as you're suggesting) a 
registration func were added, ebuild devs would also run into issues 
of registration order, and needing to yank a registered func from one 
of the hooks... further mess iow.


Reiterating, for java pkgs to work properly, the environment _needs_ 
to be preserved and not stomped per phase change- portages env 
handling is broke, java usage of it is a stop gap measure rather then 
having to rewrite all java ebuilds to include per phase env restoring 
working around portage.

pre/post is, and should remain strictly user level except when 
greenlighted by QA as a bandaid (as this was)- shouldn't be expanded 
any further (especially since it was an EAPI change that wasn't 
versioned since it occured prior to EAPI existing).

~harring


pgppA7Sccz56p.pgp
Description: PGP signature


Re: [gentoo-portage-dev] QA Notice: ECLASS 'foo' inherited illegally

2006-04-24 Thread Brian Harring
On Mon, Apr 24, 2006 at 04:10:50AM -0700, Duncan wrote:
 I continue to see way more of these than I'm comfortable with. 
 Illegally implies the functionality will eventually go away, and stuff
 will quit working.  That what's making me uncomfortable.
 
 Some time ago this came up on the dev list and I asked about bugging them
 at that time.  The reply was effectively not yet.  Are the notices now
 to be considered bugs and filed as such, or is there perhaps something
 portage doesn't like in my layout?
snip
 I've notices that these warnings occur almost entirely (if not entirely)
 during the unmerge phase.  Some stray comment I saw somewhere lead me to
 believe they may be the result of FEATURES=buildpkg

bug 56408 ( http://bugs.gentoo.org/56408 ).

When it's a (99% of the time) a valid bug-
1) If you see it during unpack - install, it's a bug in the ebuild.
2) If you see it during setup when *building* a package, it's a bug in 
the ebuild.

When it's (typically) not a valid bug in the ebuild-
1) binpkg merging, setup phase.
2) (pre|post)(inst|rm)
3) config phase.
4) When you've gone and screwed with PORTAGE_TMPDIR location, or 
wiped the env file when walking through the phases.

Basically, portage doesn't always reuse the saved env properly- since 
the check relies on a proper env, it gets false positives.

Fixing up the env handling is problematic- basically, that env _needs_ 
to be reused (both the relevant portage snippets of it, and eclass).  

You can address the eclass issue by just copying the eclasses around 
with the ebuild (in affect, duplicating the eclass code when you've 
already got the env state), but that's pointless- just fix the 
env handling so it runs from the preserved state.

That leaves effectively the portage api side of it- fex, use must 
work the same regardless of portage version (or if it changes, be 
versioned by eapi fex to denote the break in compatibility).  Why does 
this matter?  Older versions of portage had use echo to stdout if it 
matched (hence [ -n $(use blah) ] tests), if you change in portage 
use's behaviour (to rely on return code instead of echo'ing), older 
ebuild/saved envs won't function properly anymore.

Sordid details are in the bug (fun sidenote, that bug holds a *really* 
old version of ebd before I daemonized it) :)

~harring


pgpMuoK9Q06Sj.pgp
Description: PGP signature


Re: [gentoo-portage-dev] 2.1 release candidate soon?

2006-04-14 Thread Brian Harring
On Fri, Apr 14, 2006 at 05:15:53PM +0200, Philipp Riegger wrote:
 On Apr 7, 2006, at 5:26 PM, Alec Warner wrote:
 
 We have a new cache format, confcache, parallel fetch, etc... The  
 bonus
 is these features are already mature and relatively old ( a year +  
 as of
 now ).
 
 Reading about confcache i have one question:
 
 When i saw, that this feature exists (in make.examples) i activated  
 it. But it did not work because i had not emerged confcache. I think  
 this check should be stricter, if i want confcache and have  
 FEATURES=confcache and confcache is not emerged, i think  
 emerge ...  should die, not just say Ok, you said you want it but  
 you don't have it, so i don't use it. What do you think about that?

Precedent is against you in this case...sandbox is the same way 
(notify instead of bailing).

Personally I prefer the if I told you to do something, bail if you 
can't approach, but for features portage has usually done the 
opposite.
~harring


pgpZNDBdMy8pf.pgp
Description: PGP signature


Re: [gentoo-portage-dev] 2.1 release candidate soon?

2006-04-14 Thread Brian Harring
On Sat, Apr 15, 2006 at 11:01:56AM +0900, Jason Stubbs wrote:
 On Saturday 15 April 2006 03:31, Brian Harring wrote:
  cache backend selection (failed import == defaults to sys default)
 
 This is incorrect. It displays an error message and quits.
Still leaves the other features then (and raises the question that 
it's not internally totally standard)...

Either way, standard for it is preferable (again, prefer failures 
myself, but I'm not a normal user)...
~harring


pgpEc8sZvtMRn.pgp
Description: PGP signature


Re: [gentoo-portage-dev] tree dependency check

2006-03-29 Thread Brian Harring
On Wed, Mar 29, 2006 at 07:21:15PM +0300, Marius Mauch wrote:
 Marius Mauch schrieb:
 So after manifest2 is in, I'll revive the other issue that IMO is a 
 requirement for 2.1: enforcing dependencies needed to use the tree (see 
 old threads or glep44 for reasoning). A patch for that is available at 
 dev.gentoo.org/~genone/patches/treedeps.diff. Unless somebody objects 
 I'll add that somewhere next week.
 
 Ok, from a discussion with Zac and a few others it seems that we should 
 add a layer of indirection, so instead of specifying atoms the tree has 
 a freeform version identifier, and portage can use it to look up the 
 required atoms by using a (remote) mapping file.
 This new approach is implemented in 
 dev.gentoo.org/~genone/pacthes/format-check.diff (it's basically a 
dev.gentoo.org/~genone/patches/format-check.diff moreso...
~harring 


pgpaNFYC7B1mG.pgp
Description: PGP signature


Re: [gentoo-portage-dev] tree dependency check

2006-03-29 Thread Brian Harring
On 3/29/06, Marius Mauch [EMAIL PROTECTED] wrote:
 On Thu, 30 Mar 2006 08:30:17 +0900
 Jason Stubbs [EMAIL PROTECTED] wrote:

  On Thursday 30 March 2006 01:21, Marius Mauch wrote:
   Marius Mauch schrieb:
So after manifest2 is in, I'll revive the other issue that IMO is
a requirement for 2.1: enforcing dependencies needed to use the
tree (see old threads or glep44 for reasoning).
 
  Can you summarise the reasoning again please?

 a) avoid massive breakage when certain new features are introduced
 (past examples being cascading profiles or new-style virtuals)
 b) similar to a) allow people to use new features without having to
 wait for a year or two

It's worth noting that the massive breakage (virtuals fex) can be
dealt with a bit saner by writing robust code.  The virtuals horkage
during cache transfer being the prime example of that...

 These docs will then tell the user about the following options
 a) use the secret override

This seems like a case where --force would be a helluva lot better
then some random nasty env var that is documented on a webpage (assume
the user has no browser due to bootstrapping the system without a web
connection).

 That should be an extreme exception though, there will be some rules
 regarding format bumps (like only use formats where that haven been
 supported by stable portage for n months).

Knowing the rules up front would be useful to evaluate this patch's
usefulness...

-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] DB and binary dependency

2006-03-24 Thread Brian Harring
On Fri, Mar 24, 2006 at 01:40:01PM +0200, tvali wrote:
 On 24/03/06, Paul de Vrieze [EMAIL PROTECTED] wrote:
  On Thursday 23 March 2006 21:38, Gustavo Sverzut Barbieri wrote:
   Cons:
- it's not the final solution to the problem, as said, interfaces
   would be better... but interfaces would demand much more effort and
   not being automatically generated, would be async and probably
   incorrect at some point
 
  Interfaces, while nice, would overcomplicate things. They are also not
  needed as we have depend atoms.
 
  Paul
 
 Interface can be made somewhat automatically checkable.
Checking the interfaces/symbols sucks however, because you can only do 
it _after_ you've built whatever you're building (packages do adjust 
the defines/typedefs/structs dependant on configure/build options).

As I stated earlier, bincompat (not binslot paul :P) is the route to 
go- it gives you up front information so a resolver can plan out what 
has to be rebuilt automatically.

~harring



pgpVQWlcq5D0D.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Wildcards in package.keywords

2006-03-16 Thread Brian Harring
On Thu, Mar 16, 2006 at 11:57:06AM +0100, Simon Stelling wrote:
 Pingveno wrote:
 kde-base/* ~x86
 
 or, to apply it to a single version, this:
 
 =kde-base/*-3.5.1 ~x86
 
 Regular expressions would, of course, work too. They might be a little
 bit of overkill, though.
 
 Bug 57153, was RESOLVED LATER
 
 http://bugs.gentoo.org/show_bug.cgi?id=57153

If anyone is bored, I'd be interested in an implementation of that for 
pkgcore (formerly bcportage, formerly saviour, formerly savior, 
formerly the portage rewrite)...

Implementation of it would require modification to 
pkgcore/config/domain.py , namely changing the package_keywords 
splitter to use 

from pkgcore.restrictions import values, packages
from pkgcore.package.atom import VersionMatch
packages.BooleanRestriction(
   packages.PackageRestriction(category, values.StrExactmatch(cat)),
   Versionmatch(ver_component, rev_component)
)

instead of just passing the str to atom, and change the DictBased 
filter used for keywords adjustment.

Bit wordy, but it's actually minor to do- 20 lines or so, just not top 
priority for me (eg I'll get it someday down the line if I remember) :)

~harring


pgpPclLxHWDYE.pgp
Description: PGP signature


Re: [gentoo-portage-dev] DB and binary dependency

2006-03-16 Thread Brian Harring
On Thu, Mar 16, 2006 at 02:58:00PM +0100, Paul de Vrieze wrote:
 On Wednesday 15 March 2006 16:13, Gustavo Sverzut Barbieri wrote:
  Hello,
 
  There is any provision for binary dependency on Gentoo/Portage? The
  way it works now is quite messy with things like revdep-rebuild.

 Solving this is not trivial. Basically suppose application X depends on 
 sys-libs/db-4.*
 
 This can be resolved by differently slotted libraries. We need to record 
 which one was actually used (not easy by itself, but more an issue of the 
 ebuild itself, if more than 1 candidate is available). But suppose that 
 we know that the application was compiled to use db-4.3.29. We must then 
 know that it is ok to replace the db-4.3.29 package with 4.3.30, but that 
 it isn't ok to replace it by 4.3.28 or 4.4.20.
 
 To make this things worse, the above example assumes that within a slot, 
 the libraries are binary compatible. There are examples of libraries that 
 are not. And what about a library whose interface is dependent on a third 
 library: B uses A, C uses B, but B exports A. So B is dependent on A, and 
 the binary package of C must record that B was compiled with A.
 
 In short, welcome to binary package hell. This is the reason that binary 
 distributions must use versions. Even debian. It is just very very hard 
 to fix these kinds of indirect dependencies.

You're forgetting we're equally screwed when upstream (*cough* 
openssl) goes and changes the soname while preserving abi.

Old solution to it was to add another metadata key to ebuilds, 
bincompat that is compared within slotting to see if a rebuild of 
rdepend consumers needs to occur.

fex...
alsa-lib-0.90
slot=0
bincompat=0

alsa-lib-1.0
slot=0
bincompat=1

The pkg manager does it's little dance to figure out what should be 
installed where (and what versions can coexist), while doing it if 
it's going to replace a pkg and bincompat differs, pull in the subset 
of the state graph at that point (including vdb) that rdeps on that 
package/slot and force a rebuild of them.

As per the norm, requires a smart resolver; for c++ would expect 
cycles to occur where the only solution is to pull in libstdc++ (fex) 
to sidestep horkage while doing the rebuilds...

Also note that (like all general solutions) it's not going to be able 
to cover all insane stuff; if the api breaks in transitioning 
bincompat is going to cause a minor explossion that is only resolvable 
by packages specifying exact dependencies (read: version ranges).
~harring


pgpFDZuONXCnW.pgp
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] Manifest2 reloaded

2006-03-15 Thread Brian Harring
On Wed, Mar 15, 2006 at 09:53:24PM -0800, Zac Medico wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 Marius Mauch wrote:
  Marius Mauch schrieb:
  The first should be delayed until there is some consensus how the gpg
  stuff should work in the future, the others I don't see the use for.
  Also I only checked portage.py for changes, so emerge/repoman/... might
  still have to be fixed.
  Last but not least: I did some basic testing with this and the
  important stuff seems to work, but I'm quite sure the code still has a
  lot of bugs/issues, and this being a core functionality it needs a
  *lot* of testing, so I'd really appreciate if you could all give it a
  spin (but do not commit anything to the tree without manually checking
  it first).
  
  Does the lack of feedback (only got a reaction from Brian so far) mean
  that noone tried it or that it doesn't have any issues?
 
 The patch applies and seems to work well.  At a quick glance the code looks 
 pretty clean and it's nice to migrate more code out of portage.py to a 
 separate module.  I've attached a refreshed version of the patch that applies 
 cleanly against current svn (I've made no changes).
 
 Zac
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1.4.2.2 (GNU/Linux)
 
 iD8DBQFEGP1S/ejvha5XGaMRAl/7AJ9cZbjhWtjCz+ac2/tjQNUoivj0twCg7xAG
 cYvDbMiqU5HtpNrVk7fs6RM=
 =Eqlo
 -END PGP SIGNATURE-

 === added file 'pym/portage_manifest.py'
 --- /dev/null 
 +++ pym/portage_manifest.py   
 @@ -0,0 +1,314 @@
 +import os, sets
 +
 +import portage, portage_exception, portage_versions, portage_const
 +from portage_checksum import *
 +from portage_exception import *
 +
 +class FileNotInManifestException(PortageException):
 + pass
 +
 +def manifest2AuxfileFilter(filename):
 + filename = filename.strip(/)
 + return not (filename in [CVS, .svn] or filename[:len(digest-)] == 
 digest-)
 +
 +def manifest2MiscfileFilter(filename):
 + filename = filename.strip(/)
 + return not (filename in [CVS, .svn, files, Manifest] or 
 filename[-7:] == .ebuild)

python -m timeit -s 's=asdf*400;s+=fdsa.ebuild' 's.endswith(.ebuild)'
100 loops, best of 3: 0.88 usec per loop

python -m timeit -s 's=asdf*400;s+=fdsa.ebuild' 's[-7:] == .ebuild'
100 loops, best of 3: 0.564 usec per loop

Use endswith

oddly, worth noting that startswith differs in this behaviour...
python -m timeit -s 's=asdf*400;s+=fdsa.ebuild' 's[:7] == .ebuild'
100 loops, best of 3: 0.592 usec per loop

python -m timeit -s 's=asdf*400;s+=fdsa.ebuild' 's.startswith(.ebuild)'
100 loops, best of 3: 0.842 usec per loop

 +class Manifest(object):
 + def __init__(self, pkgdir, db, mysettings, 
 hashes=portage_const.MANIFEST2_HASH_FUNCTIONS, manifest1_compat=True, 
 fromScratch=False):
 + self.pkgdir = pkgdir+os.sep

rstrip os.sep prior to adding it

 + self.fhashdict = {}
 + self.hashes = hashes
 + self.hashes.append(size)
 + if manifest1_compat:
 + 
 self.hashes.extend(portage_const.MANIFEST1_HASH_FUNCTIONS)
 + self.hashes = sets.Set(self.hashes)
 + for t in portage_const.MANIFEST2_IDENTIFIERS:
 + self.fhashdict[t] = {}
 + self._read()
 + self.compat = manifest1_compat
 + self.db = db
 + self.mysettings = mysettings
 + if mysettings.has_key(PORTAGE_ACTUAL_DISTDIR):
 + self.distdir = mysettings[PORTAGE_ACTUAL_DISTDIR]
 + else:
 + self.distdir = mysettings[DISTDIR]

Why pass in mysettings?  
Have the code push it in, manifest shouldn't know about the DISTDIR 
key nor PORTAGE_ACTUAL_DISTDIR, should just have a directory to look 
in.


 + def guessType(self, filename):
 + if filename.startswith(files/digest-):
 + return None
 + if filename.startswith(files/):

if you're intent on using os.sep, might want to correct the two '/' 
uses above to use os.path.join/os.path.sep

If concerned about cost, just calculate it once in the class namespace 
as a constant.

related, might I suggest converting away from internal strings to a 
class level enumeration?

int comparison is faster then string, plus it unbinds the internal 
code from the on disk symbols used (eg, just cause on disk is AUX 
doesn't mean internally it should be throwing around AUX).

 + return AUX
 + elif filename.endswith(.ebuild):
 + return EBUILD
 + elif filename in [ChangeLog, metadata.xml]:
 + return MISC
 + else:
 + return DIST
 + 
 + def getFullname(self):
 + return self.pkgdir+Manifest

Err... move that into the initializer.

If you're concerned folks will screw up the var, use a property to 
make it immutable.

Either way, func calls aren't cheap, and that's not really needed :)

 + 
 +   

esearch integration [was Re: [gentoo-portage-dev] Few things, which imho would make portage better]

2006-03-14 Thread Brian Harring
On Tue, Mar 14, 2006 at 04:33:06PM +0200, tvali wrote:
 I did think about it now and it seems to me that probably it would be
 much faster if esearch is not just another package, but part of
 portage.
 
 I mean -- functions of portage, which query db, should use esearch
 index wherever they need information, which exists in that index.
 
 As much as i can understand, /var/cache/edb/ contains esearch database
 in many files and esearchdb.py is search index as python script.

No...
esearch is a static db- only useful for 'frozen' trees, eg rsync 
distributed trees with no eclasses in overlays.  All cvs users (devs) 
run unfrozen trees (readonly/readwrite is better terminology), thus 
portage updates the cache db on the fly as needed.

If esearch was integrated into portage the result would be stale 
metadata for cvs users, and stale metadata for rsync users when 
overlays with eclasses are involved- no go.

That and esearch last I looked just generates a giant dict (thus the 
cache is in memory), which kind of blows the 25mb mem usage 2.1 
now sports :)

~harring


pgpjlCAV4I3Za.pgp
Description: PGP signature


sync suggestions [was Re: [gentoo-portage-dev] Few things, which imho would make portage better]

2006-03-14 Thread Brian Harring
On Tue, Mar 14, 2006 at 03:50:18PM +0200, tvali wrote:
 Another question now is about sync.
 
 I did read somewhere, that this is not good user behavior to sync more
 than once per day. I understand that as if this is a huge download
 even if there is nothing changed.
 
 Isnt it nice idea to have this database just optimized?
 
 I mean (assuming portage using SQL now) -- that would be really simple
 to log every change in portage tree as series of SQL queries, which
 would reproduce this change.

Pushing the delta (what you're suggesting) is only usable if it can be 
guranteed the user hasn't modified their tree at all (thus resulting 
in cache db differing from upstreams).

That right there is the brass tacks of it; You wouldn't be able to 
push just the changes, you would have to regenerate the _whole_ db 
(slow, 20k inserts assuming only one table).

Sidenote... please post seperate threads for seperate 
ideas/discussions, else it's damn hard to look back and pull the 
specific thread were something was discussed.
~harring



pgpmJPQ6iTqcg.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Questions regarding the new portage API (savior branch)

2006-03-02 Thread Brian Harring
On Wed, Mar 01, 2006 at 08:15:19PM -0800, Brian wrote:
 On Wed, 2006-01-03 at 17:39 -0800, Brian Harring wrote:
  emerge bzr
  bzr get http://gentooexperimental.org/~ferringb/bzr/saviour
  cd saviour
  bzr pull
  
  ...roughly. ;)
 
 a little too rough :)
 
 [EMAIL PROTECTED] ~ $ bzr get 
 http://gentooexperimental.org/~ferringb/bzr/saviour
 bzr: ERROR: Not a branch: http://gentooexperimental.org/~ferringb/bzr/saviour
 [EMAIL PROTECTED] ~ $ 

Try again :)

~harring


pgpJVERABoPLh.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Questions regarding the new portage API (savior branch)

2006-03-02 Thread Brian Harring
On Thu, Mar 02, 2006 at 06:54:45PM +0100, Michael Schilling wrote:
 Hi,
 
 Brian wrote on Thursday the 2nd of March 2006:
 
  [EMAIL PROTECTED] ~ $ bzr get 
  http://gentooexperimental.org/~ferringb/bzr/saviour
  bzr: ERROR: urllib2.HTTPError: HTTP Error 403: Forbidden
at /usr/lib/python2.4/urllib2.py line 480
in http_error_default
  [EMAIL PROTECTED] ~ $
  
  After getting 156 items in saviour/.bzr/revision-store (219 items in weaves)
 
 I'm getting exactly the same error after a few minutes and 285 of 384
 packages.

See... I always have a helluva time transitioning the repository since 
I keep forgetting simple things like upload the repo, not an export 
or reset perms due to umask...

Either way, bzr repo should be fine, and a tarball is up there (might 
be quickest grabbing that and just doing a pull afterwards)

~harring


pgpmkLn1TQrvP.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Questions regarding the new portage API (savior branch)

2006-03-02 Thread Brian Harring
On Thu, Mar 02, 2006 at 10:44:58PM +0200, Marius Mauch wrote:
 Brian Harring wrote:
 On 2/28/06, *Michael Schilling* [EMAIL PROTECTED] 
 mailto:[EMAIL PROTECTED] wrote:
 
 - Is one of these svn-web-repository up to date?
   * http://sources.gentoo.org/viewcvs.py/portage/main/branches/savior/
   * http://mzz.mine.nu/bzr/savior-svn/portage/
 http://mzz.mine.nu/bzr/savior-svn/portage/
 
 
 I switched over to bzr about 2 months back; svn doesn't allow for 
 offline committing, nor does gentoo's vcs allow for anon*... bzr 
 natively allows for those capabilities, so that's what I'm using. :)
 
 http://gentooexperimental.org/~ferringb/bzr/saviour
 Is where I'll be updating the code  for at least the near future.
 
 
 Does that mean we should drop the SVN branch?

Realistically... I have no intention of going back to SVN, haven't for 
a while.  Can't even access the darn thing now a days anyways, thus 
the branch has fallen further in usefulness to me :)

If y'all want to mirror it, might I suggest poking marienz for his 
tailorization knowledge?  Afaik, he had a bzr-svn push working, or at 
least has investigated it.

~harring


pgpp4WaacDO2M.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Portage-2.1_pre5

2006-02-22 Thread Brian Harring
upgrade to 0.4.2 ...

On Wed, Feb 22, 2006 at 12:32:19AM -0800, Zac Medico wrote:
 updating cache config.cache
 configure: creating ./config.status
 config.status: creating Makefile
 config.status: creating doc/Makefile
 config.status: creating doc/stylesheets/fcron-doc.dsl
 config.status: creating config.h
 
 Summary :
 ---
 
 run in debug mode by default :  no
 PAM :   yes
 SELinux :   no
 Run without root's rights : no
 fcron's user (resp. group) name :   cron (resp. cron)
 sysfcrontab :   yes (systab)
 spooldir :  /var/spool/cron
 Load average support :  yes
 compile fcrondyn :  yes
 
 You can now run 'make' to compile
 and then (as root) 'make install' to install fcron.
 
 Traceback (most recent call last):
   File /usr/bin/confcache, line 546, in ?
 sys.exit(c.run(args))
   File /usr/bin/confcache, line 207, in run
 self._update(new_loc, curdir)
   File /usr/bin/confcache, line 242, in _update
 self.file_db[f] = md5_file(f)
   File /usr/bin/confcache, line 32, in lambda
 md5_file = lambda x: fchksum.fmd5t(x)[0]
 IOError: [Errno 13] Permission denied: '/etc/shadow'
 
 !!! Please attach the following file when filing a report to bugs.gentoo.org:
 !!! 
 /mnt/storage/primary/var/tmp/portage/fcron-3.0.0/work/fcron-3.0.0/config.log

~harring


pgpp0GUtZVBEv.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Re: confcache, final chance to ixnay it

2006-02-15 Thread Brian Harring
On Tue, Feb 14, 2006 at 04:56:06PM -0600, R Hill wrote:
 agree here.  i would go as far as to maybe print a message to that effect 
 if the build fails while FEATURES=confcache.
EBUILD_DEATH_HOOKS comes to mind :)

 
 Meanwhile, if you're getting failures up the ying yang and it's not 
 tracked down, I'd state tabling the feature (or tagging massively 
 hideous warnings regarding it) is required.  Iirc, solar ran a 
 full build with confcache enabled (believe it was 0.3.*), so input 
 from him would be useful also for comparison.
 
 damn, i don't want to hold this up, especially if no one else is having 
 troubles.  i have a couple days off right now so i'll do some poking around 
 and see what i can come up with.  i'll open a bug and assign to you if i 
 figure anything out.

CC flameeyes on it please.

If ruby isn't restrict'd already, please poke the maintainers to do so 
also :)
~harring


pgpiQ3mMjvlu6.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Re: confcache, final chance to ixnay it

2006-02-15 Thread Brian Harring
On Wed, Feb 15, 2006 at 10:40:09AM +0100, Nagatoro wrote:
 Brian Harring wrote:
  On Tue, Feb 14, 2006 at 12:25:34PM +0100, Nagatoro wrote:
  Brian Harring wrote:
  Specific merge list would be wonderful... ;)
  Just had one fail on me: media-libs/libsndfile-1.0.12-r1
  It hung right after configure.
  
  Specifics...
  
  Hanging isn't much detail here, 'specially since confcache hanging is 
  a new complaint.  If it's a slow system, would wonder if it's not hung 
  as much spin it's wheels doing verification.
 
 Hung as in not doing anything noticeable and 0% CPU usage, in this
 state for  5min (might have been longer). After this first hang no
 more emerges seemed to work, after a cache clean up (rm
 /var/tmp/confcache/*) still no emerges working. After a reemerge of
 confcache (might have been a reboot too) everything seems to be
 working again (even a reemerge of libsndfile).

Yay, first locking bug.

Need to rework that anyways, just noticed a nice little note in 
flock's page regarding downgrading/upgrading releasing locks. :)
~harring


pgpFQ0HbU9D0y.pgp
Description: PGP signature


Re: [gentoo-portage-dev] [patch] emaint: check/fix package.{keywords,unmask}

2006-02-15 Thread Brian Harring
On Tue, Feb 14, 2006 at 11:26:16AM +0100, Christian Hoenig wrote:
 Hi,
 
 attached is a patch for emaint contained in the sys-apps/portage-2.1_pre4-r1 
 package that adds support for checking and fixing redundant entries in 
 package.keywords and package.unmask.
 
 Fixing those files is done by commenting out corresponding lines (as 
 suggested 
 by Simon Stelling, thanks!)
 
 I added two handlers, one for package.keywords, one for package.unmask, as I 
 like to check and fix them seperately. Though I don't think, that I shared 
 their code pretty nice. I have experience in c++ but ...
 
 CODING DISCLAIMER
 ... I am pretty new to portage and python coding, so please forgive me my 
 quirks and critizise me constructive :-).

Few things...
-(self.noneInstalled , self.noneAffected) = checkDict( cfg.punmaskdict )
+self.nonInstalled, self.noneAffected = checkDict(cfg.punmaskdict)
is a bit more normal

check should actually do the checks, rather then having 
the work done during instantiation of the checker... potentially have 
a base class those checkers inherit from instead of having the funcs 
unbound (although this is definitely debatable).

Also...
errors = []
errors += map

just do
errors = map...

map returns a list, no point instatiating an empty list then updating 
it with another list :)

~harring


pgpLK17H6lXxv.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Re: confcache, final chance to ixnay it

2006-02-14 Thread Brian Harring
On Mon, Feb 13, 2006 at 08:06:24PM -0600, R Hill wrote:
 (sorry if this double posts)
 
 Brian Harring wrote:
 Yo...
 
 attached is a patch enabling confcache support for portage.  Lots of 
 testing, plus fixups from comments from folks prior.
 
 So... giving it a few days, nows the time to bitch if you dislike the 
 implementation (and no, I'm not rewriting all of doebuild just for 
 this :)
 
 Well, i've been testing this on an x86 laptop and an x86_64 box over the 
 weekend.  Good news is that when it works, it works well.  Bad news is I've 
 yet to be able to get through an 'emerge -e world' without at least a dozen 
 build failures that resolve themselves when i clear the cache.

Specific merge list would be wonderful... ;)

 The errors 
 range between unresolved symbols to 'platform does not support (null)' to 

Err... that one I'd be very interested in.  

 compiler cannot create C executables to a simple file not found

 and everything in between.  they sometimes crop up during configure, 
 sometimes 
 during compile, and sometimes during install.

What version of sandbox are you running?  You seem to be having a 
helluva lot more problems then I do; further, 'file not found' 
(depending on the m4 leading up to it) is indicative that tracking of 
file access is failing on your machine (iow, if validiation isn't 
working properly, I'm not surprised you're hitting holy hell).

So... what features you got, and what sandbox version?


 This seems to me like it would be a nightmare for maintainers.

Potentialy, hence a buttload of work put in attempting to verify that 
confcache's validation checks are air tight.  Flat out contrary/broken 
cache snippets in a configure script (python's dlopen check) cannot be 
dealt with beyond restricting that package, and/or correcting the 
configure statement so it's inline with what is globally the norm.

Personally, I will state this- same for ccache[1], I wouldn't use 
confcache on production hardware at this stage.


[1] ccache hands off to whatever ${CCACHE_CC-${0}} it finds next in 
traversing $PATH; if whatever that is, does further expansion/mangling 
of args to gcc (insane, but does occur) it's outside of ccache's 
knowledge of the inputs, thus validation can be horked.

 There are apparently a lot of broken configures out there.

Not as many as you would think actually; just takes one to cause some 
issues though.  My personal experience has been different from yours, 
but reports re: confcache I've been watching/weary about.

One potential alternative is breaking the cache down so it's per cp 
instead of global; doing that is easy enough, but potentially 
nukes the gain since each cache is only used on re-emerge/updates to a 
cp.


Personally, any bug reports that come in with confcache enabled should 
have the confcache disabled imo; just the same as potentially whonky 
cflags, scale it back to ensure the problem is in the source, not in 
any bastardization's the users configuration has done to it.

Meanwhile, if you're getting failures up the ying yang and it's not 
tracked down, I'd state tabling the feature (or tagging massively 
hideous warnings regarding it) is required.  Iirc, solar ran a 
full build with confcache enabled (believe it was 0.3.*), so input 
from him would be useful also for comparison.

Sidenote, confcache isn't prelink aware.  Anyone running prelink will 
suffer continual invalidation everytime they run prelink -afmr... 
probably should do something about that. :)

~harring


pgpt1YwUnkFv8.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Re: confcache, final chance to ixnay it

2006-02-14 Thread Brian Harring
On Tue, Feb 14, 2006 at 12:25:34PM +0100, Nagatoro wrote:
 Brian Harring wrote:
  On Mon, Feb 13, 2006 at 08:06:24PM -0600, R Hill wrote:
  Well, i've been testing this on an x86 laptop and an x86_64 box over the 
  weekend.  Good news is that when it works, it works well.  Bad news is 
  I've 
  yet to be able to get through an 'emerge -e world' without at least a 
  dozen 
  build failures that resolve themselves when i clear the cache.
  
  Specific merge list would be wonderful... ;)
 
 Just had one fail on me: media-libs/libsndfile-1.0.12-r1
 It hung right after configure.

Specifics...

Hanging isn't much detail here, 'specially since confcache hanging is 
a new complaint.  If it's a slow system, would wonder if it's not hung 
as much spin it's wheels doing verification.
~harring


pgpHFbwHcXRwg.pgp
Description: PGP signature


Re: [gentoo-portage-dev] Re: [gentoo-dev-portage] [PATCH] prevent world file corruption by writing atomically

2006-02-05 Thread Brian Harring
On Mon, Jan 30, 2006 at 10:21:22AM -0800, Zac Medico wrote:
 Zac Medico wrote:
  Okay, I've created a file-like class called atomic_ostream and it is now 
  used for both write_atomic() and writedict().
 
 I've been using this patch locally with no problems.  Do we have 
 any more feedback or are people satisfied with it?   IMO we need 
 something like this, if not for (unsupported) parallel merges, at 
 least to prevent loss of an important file when it is being 
 overwritten and an IO error occurs (see bug 114133).

Meh you're not supposed to call me on being a slacker for not 
commenting. :)

No complaints with this going into svn for upcoming 2.1_pre*, but I'd 
like to see the class rewritten actually- the file object lacks a lot 
of capabilities that a normal file object has.

I'd suggest deriving straight from the file class, and just doing 
changes in close and open.  Should be a much smaller class also ;)

~harring


pgpiD5Co9VAKj.pgp
Description: PGP signature


Re: [gentoo-portage-dev] should CATEGORY be properly documented in ebuild.5 and declared readonly in ebuild.sh?

2006-02-05 Thread Brian Harring
On Mon, Jan 30, 2006 at 12:36:42PM -0800, Zac Medico wrote:
 Hi everyone,
 
 The subject says it all.  What do y'all think?

Go for it.

~harring


pgpJ5TEihqeyZ.pgp
Description: PGP signature


Re: [gentoo-portage-dev] emerge-webrsync patch

2006-02-02 Thread Brian Harring
On Thu, Feb 02, 2006 at 08:16:14AM +0100, Johannes Fahrenkrug wrote:
 Brian,
 
 I just want to make sure this is still on your agenda :)

InSVN, and in the tree... :)

~harring


pgpcmhNYyu2R8.pgp
Description: PGP signature


[gentoo-portage-dev] confcache, final chance to ixnay it

2006-02-02 Thread Brian Harring
Yo...

attached is a patch enabling confcache support for portage.  Lots of 
testing, plus fixups from comments from folks prior.

So... giving it a few days, nows the time to bitch if you dislike the 
implementation (and no, I'm not rewriting all of doebuild just for 
this :)
~harring
Index: pym/portage.py
===
--- pym/portage.py  (revision 2622)
+++ pym/portage.py  (working copy)
@@ -2666,7 +2666,38 @@
print !!! Perhaps: rm -Rf,mysettings[BUILD_PREFIX]
print !!!,str(e)
return 1
+   try:
+   if confcache in features:
+   if not mysettings.has_key(CONFCACHE_DIR):
+   mysettings[CONFCACHE_DIR] = 
os.path.join(mysettings[PORTAGE_TMPDIR], confcache)
+   if not 
os.path.exists(mysettings[CONFCACHE_DIR]):
+   if not os.getuid() == 0:
+   # we're boned.
+   features.remove(confcache)
+   mysettings[FEATURES] =  
.join(features)
+   else:
+   
os.makedirs(mysettings[CONFCACHE_DIR], mode=0775)
+   
os.chown(mysettings[CONFCACHE_DIR], -1, portage_gid)
+   else:
+   st = 
os.stat(mysettings[CONFCACHE_DIR])
+   if not (st.st_mode  0)  == 0775:
+   
os.chmod(mysettings[CONFCACHE_DIR], 0775)
+   if not st.st_gid == portage_gid:
+   
os.chown(mysettings[CONFCACHE_DIR], -1, portage_gid)
 
+   # check again, since it may have been disabled.
+   if confcache in features:
+   for x in listdir(mysettings[CONFCACHE_DIR]):
+   p = 
os.path.join(mysettings[CONFCACHE_DIR], x)
+   st = os.stat(p)
+   if not (st.st_mode  0)  07600 == 
0600:
+   os.chmod(p, (st.st_mode  0777) 
| 0600)
+   if not st.st_gid == portage_gid:
+   os.chown(p, -1, portage_gid)
+   
+   except OSError, e:
+   print !!! Failed resetting perms on confcachedir %s % 
mysettings[CONFCACHE_DIR]
+   return 1
#try:
#   mystat=os.stat(mysettings[CCACHE_DIR])
#   if (mystat[stat.ST_GID]!=portage_gid) or 
((mystat[stat.ST_MODE]02070)!=02070):
Index: bin/ebuild.sh
===
--- bin/ebuild.sh   (revision 2622)
+++ bin/ebuild.sh   (working copy)
@@ -493,7 +493,32 @@
LOCAL_EXTRA_ECONF=--libdir=${CONF_LIBDIR_RESULT} 
${LOCAL_EXTRA_ECONF}
fi
 
-   echo ${ECONF_SOURCE}/configure \
+   local TMP_CONFCACHE_DIR CONFCACHE_ARG
+   if hasq confcache $FEATURES  ! hasq confcache $RESTRICT; then
+   CONFCACHE=$(type -p confcache)
+   if [ -z ${CONFCACHE} ]; then
+   ewarn disabling confcache, binary cannot be 
found
+   else
+   CONFCACHE=${CONFCACHE/ /\ }
+   
TMP_CONFCACHE_DIR=${CONFCACHE:+${CONFCACHE_DIR:-${PORTAGE_TMPDIR}/confcache}}
+   TMP_CONFCACHE_DIR=${TMP_CONFCACHE_DIR/ /\ }
+   CONFCACHE_ARG=--confcache-dir
+   local s
+   if [ -n $CCACHE_DIR ]; then
+   s=$CCACHE_DIR
+   fi
+   if [ -n $DISTCC_DIR ]; then
+   s=${s:+${s}:}$DISTCC_DIR
+   fi
+   if [ -n $s ]; then
+   CONFCACHE_ARG=--confcache-ignore $s 
$CONFCACHE_ARG
+   fi
+   fi
+   else
+   CONFCACHE=
+   fi
+
+   echo ${CONFCACHE} ${CONFCACHE_ARG} ${TMP_CONFCACHE_DIR} 
${ECONF_SOURCE}/configure \
--prefix=/usr \
--host=${CHOST} \
--mandir=/usr/share/man \
@@ -504,7 +529,7 @@

Re: [gentoo-portage-dev] emerge-webrsync patch

2006-02-02 Thread Brian Harring
On Thu, Feb 02, 2006 at 01:30:58PM +0100, Johannes Fahrenkrug wrote:
 Brian Harring wrote:
 
 On Thu, Feb 02, 2006 at 08:16:14AM +0100, Johannes Fahrenkrug wrote:
  
 
 Brian,
 
 I just want to make sure this is still on your agenda :)

 
 
 InSVN, and in the tree... :)
  
 
 Great! Thanks :)
 I was just wondering: Will I be mentioned anywhere?
Bah, having it fixed doesn't count?

see rev2624; the changes are only in trunk (2.1), won't be in 2.0.54 
most likely since 2.0.54 shuld be bug fixes only...
~harring


pgp7CaImWy9Bh.pgp
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] rsync metadata cache patch (obsoletes metadata transfer on sync)

2006-01-28 Thread Brian Harring
On Sat, Jan 28, 2006 at 02:39:42AM -0800, Zac Medico wrote:
   def _delitem(self, cpv):
   try:
   del self.db_rw[cpv]
   except KeyError, ke:
   if not self.db_ro.has_key(cpv):
   raise ke

You need white outs here (lifting a unionfs term); this won't actually 
change state in any way if you're trying to delete a bad entry from 
the underlying metadata cache.

 
   def has_key(self, cpv):
   return self.db_rw.has_key(cpv) or self.db_ro.has_key(cpv)
 
   def iterkeys(self):
   s = set()
   for db in (self.db_rw, self.db_ro):
   for cpv in db.iterkeys():
   if cpv not in s:
   yield cpv
   s.add(cpv)
Should just do 
s=set()
for cpv in self.db_rw:
yield cpv
s.add(cpv)
for cpv in self.db_ro:
if cpv not in s:
yield cpv


Slightly faster (1us per yield for the set lookup), but mainly less 
memory usage.

Aside from that, the label trick is icky. ;)


pgpKDjdHMWon3.pgp
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] rsync metadata cache patch (obsoletes metadata transfer on sync)

2006-01-28 Thread Brian Harring
On Sat, Jan 28, 2006 at 11:24:18AM -0600, Andrew Gaffney wrote:
 Zac Medico wrote:
 I have reimplemented the previous patch as a normal cache module that adds 
 a writable layer on top of the pre-generated metadata.  If you'd like to 
 try this out (with portage-2.1_preX), simply copy metadata_overlay.py into 
 /usr/lib/portage/pym/cache/ and add portdbapi.auxdbmodule = 
 cache.metadata_overlay.database to /etc/portage/modules.
 
 I haven't tested this much, but like the previous patch, this shouldn't 
 cause any harm. Feedback would be appreciated.
 
 Portage didn't explode and dep calculation is pretty damn quick with an 
 empty /var/cache/edb/dep/,

Shouldn't be any difference here, if you are seeing a difference 
please time it and post the stats here- it should actually be slightly 
slower then a straight flat_hash cache since it's 2 lookups worst case 
instead of single lookup.

If you're seeing a difference, indicative of one of the 
following
1) I'm a moron and am missing something, 
2) innefficiency in flat_hash pulls (since you should be using 
metadata for most of the queries, the actual underlying ro cache)
3) something is funky.

 Is it normal for /var/cache/edb/dep/${PORTDIR}/ to be re-created 
 (it's still empty) when this new module is in use?

Flat_hash does that up front so it doesn't have to do tests for the 
directory on __setitem__ calls; so yes, normal :)


Additional issue here zach pointed out is that the current flat_list 
cache format used for rsync metadata/cache doesn't actually bundle 
eclass mtimes, so stale detection for eclasses won't work via this.

Part of the reason the metadata class exists (and can handle 
flat_list/flat_hash internally on it's own)- until flat_hash is the 
rsync tree format, staleness detection won't work for running straight 
from $PORTDIR/metadata/cache

~harring


pgpDodbkhjTKl.pgp
Description: PGP signature


Re: [gentoo-portage-dev] http-replicator: error: invalid directory '/var/cache/http-replicator' [ ok ]

2006-01-27 Thread Brian Harring
On Fri, Jan 27, 2006 at 12:11:13PM -0600, Dan Sheffner wrote:
I'm trying to get the http replicator working.
Wrong ml- try gentoo-user ml or bugs.gentoo.org.

gentoo-portage-dev == sys-apps/portage development only, ebuilds 
within the distributed gentoo tree is a seperate issue :)

~harring


pgpgd56n1gMaw.pgp
Description: PGP signature


Re: [gentoo-portage-dev] making aux_get more usable for vardbapi

2006-01-27 Thread Brian Harring
On Fri, Jan 27, 2006 at 10:19:50AM -0800, Brian Harring wrote:
 #!/bin/sh
 eval $(bzcat environment.bz2 | filter-env -f '.*' -v 'BASH.*' )
 for __x in $@; do
   echo __x=$(echo ${__x} | tr '\n,\r,\t' ' , , ')
echo ${__x}=$(echo ${!__x} | tr '\n,\r,\t' ' , , ')
 done

Change above would result in a slightly more useful output ;)
~harring


pgp5kZXG1tfAv.pgp
Description: PGP signature


Re: [gentoo-portage-dev] http-replicator: error: invalid directory '/var/cache/http-replicator' [ ok ]

2006-01-27 Thread Brian Harring
On Fri, Jan 27, 2006 at 01:35:25PM -0600, Dan Sheffner wrote:
which group should I send this to?
If you're asking about bugs.g.o, just file it without a group, the 
wrangler will redirect it (although if you look in the metadata.xml, 
it'll list the group to assign it to if there is a specific one).

If asking whether to do -user or bugs, well, I'd do bugs.
~harring


pgprlZ7kXcegd.pgp
Description: PGP signature


Re: [gentoo-portage-dev] update portage cache progress patch

2006-01-26 Thread Brian Harring
On Wed, Jan 25, 2006 at 10:05:31PM -0600, Jason Pepas wrote:
 Hi Guys,
 
 I cobbled together a quick little hack to have a little bit more 
 interactivity 
 during the Updating Portage cache phase.  It prints out the name of the 
 package it is updating, along with the percentage progress.

Err... might want to split that against 2.1.

~harring


pgpwJHvL9axjU.pgp
Description: PGP signature


Re: [gentoo-portage-dev] confcache integration

2006-01-25 Thread Brian Harring
On Tue, Jan 24, 2006 at 03:37:07PM -0500, solar wrote:
 On Wed, 2006-01-25 at 00:30 +0900, Jason Stubbs wrote:
  On Tuesday 24 January 2006 21:50, Brian Harring wrote:
  
  +os.makedirs(mysettings[CONFCACHE_DIR], mode=0775)
  +os.chown(mysettings[CONFCACHE_DIR], portage_uid, -1)
  
  This will die when running as non-root, no? ebuild is now oft used by devs
  running as normal users which will be broken by this. 
 
 Your right. I've hit that bug myself in the past switching 
 between root merges and being a real life dev and running ebuild 
 foo.build clean unpack compile;
Meh.  should've told me :P

Attached is round 2, including man/make.conf mods (ignore the fact 
it's a single make.conf, if accepted the alt make.conf's will be 
updated with the same text).
~harring
Index: pym/portage.py
===
--- pym/portage.py  (revision 2583)
+++ pym/portage.py  (working copy)
@@ -2669,7 +2669,38 @@
print !!! Perhaps: rm -Rf,mysettings[BUILD_PREFIX]
print !!!,str(e)
return 1
+   try:
+   if confcache in features:
+   if not mysettings.has_key(CONFCACHE_DIR):
+   mysettings[CONFCACHE_DIR] = 
os.path.join(mysettings[PORTAGE_TMPDIR], confcache)
+   if not 
os.path.exists(mysettings[CONFCACHE_DIR]):
+   if not os.getuid() == 0:
+   # we're boned.
+   features.remove(confcache)
+   mysettings[FEATURES] =  
.join(features)
+   else:
+   
os.makedirs(mysettings[CONFCACHE_DIR], mode=0775)
+   
os.chown(mysettings[CONFCACHE_DIR], -1, portage_gid)
+   else:
+   st = 
os.stat(mysettings[CONFCACHE_DIR])
+   if not (st.st_mode  0)  == 0775:
+   
os.chmod(mysettings[CONFCACHE_DIR], 0775)
+   if not st.st_gid == portage_gid:
+   
os.chown(mysettings[CONFCACHE_DIR], -1, portage_gid)
 
+   # check again, since it may have been disabled.
+   if confcache in features:
+   for x in listdir(mysettings[CONFCACHE_DIR]):
+   p = 
os.path.join(mysettings[CONFCACHE_DIR], x)
+   st = os.stat(p)
+   if not (st.st_mode  0)  07600 == 
0600:
+   os.chmod(p, (st.st_mode  0777) 
| 0600)
+   if not st.st_gid == portage_gid:
+   os.chown(p, -1, portage_gid)
+   
+   except OSError, e:
+   print !!! Failed resetting perms on confcachedir %s % 
mysettings[CONFCACHE_DIR]
+   return 1
#try:
#   mystat=os.stat(mysettings[CCACHE_DIR])
#   if (mystat[stat.ST_GID]!=portage_gid) or 
((mystat[stat.ST_MODE]02070)!=02070):
Index: cnf/make.conf
===
--- cnf/make.conf   (revision 2583)
+++ cnf/make.conf   (working copy)
@@ -259,6 +259,8 @@
 #  'buildpkg'causes binary packages to be created of all packages that 
 #are being merged.
 #  'ccache'  enables ccache support via CC.
+#  'confcache'   enable confcache support; speeds up autotool based configure
+#calls
 #  'collision-protect'
 #prevents packages from overwriting files that are owned by
 #another package or by no package at all.
Index: bin/ebuild.sh
===
--- bin/ebuild.sh   (revision 2583)
+++ bin/ebuild.sh   (working copy)
@@ -493,7 +493,22 @@
LOCAL_EXTRA_ECONF=--libdir=${CONF_LIBDIR_RESULT} 
${LOCAL_EXTRA_ECONF}
fi
 
-   echo ${ECONF_SOURCE}/configure \
+   local TMP_CONFCACHE_DIR CONFCACHE_ARG
+   if hasq confcache $FEATURES  ! hasq confcache $RESTRICT; then
+   CONFCACHE=$(type -p confcache)
+   if [ -z ${CONFCACHE} ]; then
+   ewarn disabling confcache, binary cannot be 
found
+   else
+   CONFCACHE=${CONFCACHE

Re: [gentoo-portage-dev] confcache integration

2006-01-25 Thread Brian Harring
On Wed, Jan 25, 2006 at 12:30:22AM +0900, Jason Stubbs wrote:
 it seems there's an external confcache binary but I can't tell much
 beyond that.
Yes, it's external (standalone) now- dev-util/confcache.
~harring


pgp4K5t0iJcYM.pgp
Description: PGP signature


[gentoo-portage-dev] confcache integration

2006-01-24 Thread Brian Harring
Yo.

Looking to integrate confcache support into trunk some time in the 
near future- had users testing it for about 2 months (give or take), 
so far it's behaved pretty decently.  A few packages eat themselves 
when ran with --cache (bad autotooling), hence the addition of 
restrict=confcache being added to the patch.

Either way, patch is attached, looking for any complaints regarding it 
(or integrating confcache)- will be sending an email off to gentoo-dev 
in the next few days to get feedback from the general dev populace,, 
but sending this email to get feedback on the implementation in the 
meantime.

~harring
Index: pym/portage.py
===
--- pym/portage.py  (revision 2490)
+++ pym/portage.py  (working copy)
@@ -2660,7 +2660,30 @@
print !!! Perhaps: rm -Rf,mysettings[BUILD_PREFIX]
print !!!,str(e)
return 1
-
+   try:
+   if confcache in features:
+   if not mysettings.has_key(CONFCACHE_DIR):
+   mysettings[CONFCACHE_DIR] = 
os.path.join(mysettings[PORTAGE_TMPDIR], confcache)
+   if not 
os.path.exists(mysettings[CONFCACHE_DIR]):
+   
os.makedirs(mysettings[CONFCACHE_DIR], mode=0775)
+   os.chown(mysettings[CONFCACHE_DIR], 
portage_uid, -1)
+   else:
+   st = 
os.stat(mysettings[CONFCACHE_DIR])
+   if not (st.st_mode  0)  == 0775:
+   
os.chmod(mysettings[CONFCACHE_DIR], 0775)
+   if not st.st_uid == portage_uid:
+   
os.chown(mysettings[CONFCACHE_DIR], portage_uid, -1)
+   for x in listdir(mysettings[CONFCACHE_DIR]):
+   p = 
os.path.join(mysettings[CONFCACHE_DIR], x)
+   st = os.stat(p)
+   if not (st.st_mode  0)  07600 == 
0600:
+   os.chmod(p, (st.st_mode  0777) 
| 0600)
+   if not st.st_uid == portage_uid:
+   os.chown(p, portage_uid, -1)
+   
+   except OSError, e:
+   print !!! Failed resetting perms on confcachedir %s % 
mysettings[CONFCACHE_DIR]
+   return 1
#try:
#   mystat=os.stat(mysettings[CCACHE_DIR])
#   if (mystat[stat.ST_GID]!=portage_gid) or 
((mystat[stat.ST_MODE]02070)!=02070):
Index: bin/ebuild.sh
===
--- bin/ebuild.sh   (revision 2490)
+++ bin/ebuild.sh   (working copy)
@@ -459,7 +459,22 @@
LOCAL_EXTRA_ECONF=--libdir=${CONF_LIBDIR_RESULT} 
${LOCAL_EXTRA_ECONF}
fi
 
-   echo ${ECONF_SOURCE}/configure \
+   local TMP_CONFCACHE_DIR CONFCACHE_ARG
+   if hasq confcache $FEATURES  ! hasq confcache $RESTRICT; then
+   CONFCACHE=$(type -p confcache)
+   if [ -z ${CONFCACHE} ]; then
+   ewarn disabling confcache, binary cannot be 
found
+   else
+   CONFCACHE=${CONFCACHE/ /\ }
+   
TMP_CONFCACHE_DIR=${CONFCACHE:+${CONFCACHE_DIR:-${PORTAGE_TMPDIR}/confcache}}
+   TMP_CONFCACHE_DIR=${TMP_CONFCACHE_DIR/ /\ }
+   CONFCACHE_ARG=--confcache-dir
+   fi
+   else
+   CONFCACHE=
+   fi
+
+   echo ${CONFCACHE} ${CONFCACHE_ARG} ${TMP_CONFCACHE_DIR} 
${ECONF_SOURCE}/configure \
--prefix=/usr \
--host=${CHOST} \
--mandir=/usr/share/man \
@@ -470,7 +485,7 @@
$@ \
${LOCAL_EXTRA_ECONF}
 
-   if ! ${ECONF_SOURCE}/configure \
+   if ! ${CONFCACHE} ${CONFCACHE_ARG} ${TMP_CONFCACHE_DIR} 
${ECONF_SOURCE}/configure \
--prefix=/usr \
--host=${CHOST} \
--mandir=/usr/share/man \


pgpu5EBoUtgFS.pgp
Description: PGP signature


Re: [gentoo-portage-dev] making aux_get more usable for vardbapi

2006-01-23 Thread Brian Harring
On Mon, Jan 23, 2006 at 11:16:03AM +0100, Marius Mauch wrote:
 On Wed, 11 Jan 2006 12:39:03 -0800
 Brian Harring [EMAIL PROTECTED] wrote:
 
  Regex you've got there allows for pulling the wrong text- recall, ebd 
  originally was doing grep based filtering (regex).  Had to rewrite 
  that in a major hurry since bash syntax (specifically here ops)
  forces you to track state/constructs rather then just a regex...
 
 Not really an issue in this case. First the code bails out if more than
 one match is found, so unless the metadata assignment is NOT found by
 it we don't get the wrong info. 
 Also a mismatch in this special is so
 extremely unlikely that honestly I don't really care about it,
 especially as this is a one time conversion (might be different if I'd
 have added the on the fly extraction).

Re-read that statement.  It's a one time conversion- meaning we better 
get it right the first time, else the user's data is effectively 
corrupted.  Forcing a full regen from the saved environment is not a 
solution for fixing past corruptions either.

If it were on the fly extraction, I wouldn't care quite as much- but 
the fact this is an untracked change to the users data means we *do* 
need to cover corner cases.

I know you want this in, but it has to be done *right* covering all 
known corner cases for it- I already wrote the tool that handles the 
corner cases properly, use it, don't adhoc a solution that 
re-introduces the potential gaps.

If we're not going to do it right, we really shouldn't do it when it 
comes to upgrades of the vdb.

Aside from that, if the code is in debate (as this is), I really 
don't think it should get slid into svn 2 weeks later effectively 
unchanged- didn't write that original email just for the hell of it :)

~harring


pgp8DlaOCY2aR.pgp
Description: PGP signature


Re: [gentoo-portage-dev] SQLite backend?

2006-01-18 Thread Brian Harring
On Tue, Jan 17, 2006 at 02:01:29AM -0200, Gustavo Sverzut Barbieri wrote:
 Hello,
 
 I admit I have not followed last threads about cache and new
 infrastructure (plugins and stuff).

Might suggest you take a look at the cache rewrite- it already has a 
sqlite backend in it, although that's not coded against pysqlite2 
(thus it needs some param work).

 However I followed the template and coded a SQLite3 (pysqlite2)
 backend: http://www.gustavobarbieri.com.br/gentoo/portage_db_sqlite.py

Really should fix the 403 if you're looking for comments.

Guessing this is against stable... in which case I'd point you at the 
2.1 line, since the stable cache is now dead from a release standpoint 
(it'll exist in 2.0.*, but everything beyond is the new cache).

 The main problem is that it's really slow to search and calculare
 dependencies. Could some developer take a look and tell me where I can
 improve it?

That's due to portage access patterns- select should be slower then 
flat_list/flat_hash, and the way portage pulls from the db (namely, 
key by key) doesn't work incredibly well for an rdbms backend- 
basically, the api is such that instead of doing a single select, 
it'll do multiple.

 In an ideal world, I wouldn't have to pickle/unpickle data but have
 its data in columns instead... one would re-encode the version strings
 using a pattern that is comparable by string, so you should be able to
 search for a given string or higher versions.

Cache rewrite is intended to work towards that- still requires the 
restriction framework I've got in savior, but gradually working 
towards that.

~harring


pgpx0u8c3voRz.pgp
Description: PGP signature


Re: [gentoo-portage-dev] making aux_get more usable for vardbapi

2006-01-11 Thread Brian Harring
On Tue, Jan 10, 2006 at 07:53:04PM +0100, Marius Mauch wrote:
 Currently vardbapi.aux_get only works for a subset of all auxdbkeys, as
 some like KEYWORDS or DESCRIPTIOn aren't stored in vdb directly.
 They are however stored in environment.bz2, but not accessible
 there.
 This is unintuitive and limits tools like equery or my own auxget
 and metascan tools in their usability.
 
 There are two solutions to this problem:
 a) enhance vardbapi.aux_get so it can use environment.bz2
 b) store more keys in vdb
 
 Now there is a tradeoff to made: a) doesn't need space but is slow
 while b) is fast but needs space, both in non-trivial amounts (runtime
 increased from 1s to 9s for a metascan -i run and from 0.5s to 0.8s
 for auxget -i, haven't actually checked the size increase, expect
 somewhere between 1 and 10 megabytes on a typical install).
 
 I'm attaching a patch that implements both (each in it's own hunk) as
 well as a new emaint option to create the missing entries offline.
 
 A not so obvious issue with a) is that due to the recent
 storage optimizations (empty entries not being stored) it's worse than
 I originally expected, as any entry missing a file will be looked up in
 env.bz2 instead. Only way to avoid that would be to add special casing
 in aux_get which I really dislike.
 
 Opinions?
Use filter-env to pull the var out.

Regex you've got there allows for pulling the wrong text- recall, ebd 
originally was doing grep based filtering (regex).  Had to rewrite 
that in a major hurry since bash syntax (specifically here ops) forces 
you to track state/constructs rather then just a regex...

Aside from that, while it's annoying, should do a one time pull- if a 
var is in a whitelist of can be pulled from env, we pull all missing 
vars *once*.  My opinion, at least.
~harring


pgpPpYlagEdYF.pgp
Description: PGP signature


Re: [gentoo-portage-dev] emerge-webrsync patch

2006-01-11 Thread Brian Harring
On Wed, Jan 11, 2006 at 10:15:00AM +0100, Johannes Fahrenkrug wrote:
 if [[ -n $PORTAGE_NICENESS ]]  ! [[ -z $WE_ARE_NICED ]]; then

Haven't looked at the patch yet, but a bit of bash fu for ya-

[[ -n $VAR ]] == ! [[ -z $VAR ]]

-z is zero length or unset, -n is length = 1 (thus must be set).

~harring


pgpYYLVtoPlnG.pgp
Description: PGP signature


Re: [gentoo-portage-dev] r2522 commit

2006-01-04 Thread Brian Harring
On Wed, Jan 04, 2006 at 09:33:07PM -0500, Alec Warner wrote:
 Author: ferringb
 Date: 2006-01-04 08:57:07 + (Wed, 04 Jan 2006)
 New Revision: 2522
 
 Modified:
main/trunk/pym/portage_dep.py
 Log:
 el buggo pointed out via spyderous.
 || ( a ( x? ( b ) y? ( c ) ) )
 -x -y , was resulting in || ( a () )
 
 the main consumer of this, portage.dep_check is stupid, and was assuming
 () was valid.
 It's not, obviously.
 
 Long term bug, around in at least .51 .  Should correct dep_check
 handling of it also, but no reason to be
 handing () in the result lists also.
 
 Sadly this breaks something else we apparently need and use ( or at
 least Ciaran tells me so :) )
 
 DEPEND=|| ( ) which should evaluate to DEPEND= doesn't with this
 patch.  It dies, which is bad, because while stupid, it should work.

Corrected with a nasty recursive filter tagged into dep_check.

I really hate that code on a side note.

~harring


pgpsy0XVePl1v.pgp
Description: PGP signature


Re: [gentoo-portage-dev] emerge-webrsync patch

2005-12-29 Thread Brian Harring
On Thu, Dec 29, 2005 at 03:51:03PM +0100, Johannes Fahrenkrug wrote:
 Brian Harring wrote:
 
 snip
 
 I'd also raid the tarsync call- this is something I was intending on 
 doing but have't yet.  
 
 /snip
 
 I don't have a very deep knowledge of all the portage internals.
 Are you suggesting that I should rather leave this up to you since you 
 were planning
 on changing these things anyway?

Nah, I'm suggesting you do it since I don't have the time right now :)

Mods involved are pretty straightforward- pretty much just flip 
through emerge-delta-webrsync, look at the python -c call, and tarsync 
call.  Integrating PORTAGE_NICENESS in, well, you already know how to 
do it. :)

I'll get if others don't, but it'll be next week before I have time to 
take a stab at it.
~harring


pgpPUncGUdFN5.pgp
Description: PGP signature


Re: [gentoo-portage-dev] emerge-webrsync patch

2005-12-28 Thread Brian Harring
On Wed, Dec 28, 2005 at 05:38:02PM +0100, Johannes Fahrenkrug wrote:
 Paul Varner wrote:
 
 
 Instead of hardcoding the nice value, use PORTAGE_NICENESS.  Here is how
 it is done in revdep-rebuild
 
 # Obey PORTAGE_NICENESS
 PORTAGE_NICENESS=$(portageq envvar PORTAGE_NICENESS)
 [ ! -z $PORTAGE_NICENESS ]  renice $PORTAGE_NICENESS $$  /dev/null
 
  
 
 Good point. Is this patch better? Or should it rather be _exactly_ as it 
 is in revdep-rebuild?

I'd suggest raiding from emerge-delta-webrsync for the portageq call; 
it's a bit nasty, but it's a single call rather then multiple.

I'd also raid the tarsync call- this is something I was intending on 
doing but have't yet.  It will cut out the untarring/rsyncing call to 
2 read throughs of the tarball, and single run through the tree.

Fair bit faster, especially if the user's box doesn't have the ram to 
buffer the tree/tarball in memory.  Tagging portage_niceness into it, 
just create a var with the appropriate nice call- if no 
PORTAGE_NICENESS, then the var is empty.

~harring


pgpNUrhTiBo25.pgp
Description: PGP signature


[gentoo-portage-dev] minimal python version required

2005-12-26 Thread Brian Harring
Yo.

So I'm getting antsy, and looking to start using fun features like 
sets, generator expressions, etc.

Not a 2.2 thing however.  So the question is, when are we going to 
give the finger to 2.2 and move forward? :)
~harring


pgpjw5zAtLPH9.pgp
Description: PGP signature


Re: [gentoo-portage-dev] making dodoc and dohtml die when they fail and stricter is on

2005-12-25 Thread Brian Harring
On Mon, Dec 26, 2005 at 12:54:04AM +0200, Petteri Räty wrote:
 Currently there are quite a few ebuilds in the tree that execute dodoc
 or dohtml for files that do not exist. I think it would be nice to have
 ebuilds die if this is the case. To not break current ebuilds this would
 only happen with FEATURES=stricter. This is what I currently do in my
 bashrc. Obviously when integreted to portage one can use helper
 functions like hasq which are not available in bashrc.
 
 
 if [[ ${FEATURES/stricter} != ${FEATURES} ]]; then
 
 _makefail() {
   bin=/usr/lib/portage/bin/${1}
   shift 1
   ${bin} [EMAIL PROTECTED] || die ${bin} [EMAIL PROTECTED] failed
 }
 
 dodoc() { _makefail ${FUNCNAME} [EMAIL PROTECTED]; }
 dohtml() {_makefail ${FUNCNAME} [EMAIL PROTECTED]; }
Seems like more of a -dev discussion imo, since they're the ones 
affected by it (for us it's just an api change).
~harring


pgpxdjyp8NzBA.pgp
Description: PGP signature


Re: [gentoo-portage-dev] per ebuild distdir symlinking

2005-12-25 Thread Brian Harring
On Sun, Dec 25, 2005 at 06:48:02PM -0500, Mike Frysinger wrote:
 On Sunday 25 December 2005 17:12, Brian Harring wrote:
  Can be defeated by a unpack ${DISTDIR}/file call, but that's invalid
  anyways.
 
 and what about things that do `cp ${DISTDIR}/aadsfasdf ${S}/` ?  those are 
 going to fail to wont they ?
Had originally thought about resetting DISTDIR to the symlink dir; it 
would address this concern.

Yay/nay on that one?
~harring


pgpumVdqz5mCm.pgp
Description: PGP signature


[gentoo-portage-dev] dep_opconvert in portage.py and portage_dep

2005-12-18 Thread Brian Harring
Yo.

New one from the looks of it- someone who has time (say, person who 
commited it), please iron out the portage.dep_opconvert and 
portage_dep.dep_opconvert.

Naming sucks at the very least, but they look to be a candidate for 
refactoring/elimination.
~harring


pgp22lOw8dnts.pgp
Description: PGP signature


Re: [gentoo-portage-dev] chunking up portage

2005-12-16 Thread Brian Harring
On Thu, Dec 15, 2005 at 01:54:06PM +0200, Marius Mauch wrote:
 Brian Harring wrote:
 So... thoughts?  I'm not much for making portage depend on tarsync 
 just for emerge-webrsync improvements, would rather chunk the bugger 
 out.
 
 How about runtime detection?
runtime detection is questionable from my standpoint, since while 
coding for it is good, without hard dep pulling it in the only folks 
who will ever have a faster emerge-webrsync are those who happen to 
know the hidden trick to merge tarsync.

Originally, did runtime detection while I was _testing_ it- once 
things proved stable enough, made it a hard coded dep instead of an 
internal optimization it'll do if it finds the binary.

With emerge-delta-webrsync, I could contact the folks who were using 
it about merging tarsync (plus I tagged it into the blog)- for 
emerge-webrsync, I don't think this is the route to go however.
~harring


pgpI46B0rPZ9r.pgp
Description: PGP signature


  1   2   >