Re: [gentoo-portage-dev] Re: equery refactorization

2008-12-06 Thread Marius Mauch
On Sun, 7 Dec 2008 12:44:25 +0900
Douglas Anderson [EMAIL PROTECTED] wrote:

 I also thought about renaming the list(l) option as search,
 because if you look at the help output, almost every module lists
 something. equery's list is actually a search, I don't see why we
 shouldn't name it that. I think maybe list was used because there were
 already two s options, stats and size. Stats is not implemented so
 I'm taking it out of help for now. Size can use the short z, becaues
 that's quite unique. That would free up s for search and it would be
 a whole lot clearer.
 
 Yes? No?

No. search (if used at all) should be reserved for a more
comprehensive search framework (though IMO a separate tool for that is
more appropriate), not just a simple name match. list makes sense if
you consider that the pkgspec argument is optional, and one of the main
tasks of it is to simply list the packages in the given repository
(that's why vardb is also the default for it) without further filtering.

Also one of the main goals of equery (according to karltk, the original
author) was to have a stable user interface, compared to the deprecated
qpkg and etcat scripts. And while the equery interface isn't exactly
the best I've seen it has been stable, so you might want to think twice
before renaming options and eventually pissing off users or breaking
third-party scripts.

Marius



[gentoo-portage-dev] Time to say goodbye

2008-11-30 Thread Marius Mauch
So, time has come for me to realize that my time with Gentoo is over. I
haven't actually been doing much Gentoo work over the last months due
to personal reasons (nothing Gentoo related), and I don't see that
situation changing in the near future. In fact I've already reassigned
or dropped most of my responsibilites in Gentoo a while ago, so there
are just a few pet projects left to give away:
- my gentoo-stats project (in the portage/gentoo-stats svn repository).
I know quite a few people are interested in the idea of collecting
various statistic data from gentoo user systems, and I'd encourage
everyone who wants to implement such a system to at least look at it (I
may have even finished it if I wouldn't have wasted my time focusing on
the wrong problems). There is quite a bit of documentation also that
should help to get you started
- a graphical security update tool (see bug #190397)

So if anyone wants to adopt those, complete or just parts, just take
them. As for Portage, Zac has practically already filled my role.

So I guess that wraps it up. It's been a nice ride most of the time,
but now it's time for me to leave the Gentoo train.

Marius




Re: [gentoo-portage-dev] search functionality in emerge

2008-11-23 Thread Marius Mauch
On Sun, 23 Nov 2008 07:17:40 -0500
Emma Strubell [EMAIL PROTECTED] wrote:

 However, I've started looking at the code, and I must admit I'm pretty
 overwhelmed! I don't know where to start. I was wondering if anyone
 on here could give me a quick overview of how the search function
 currently works, an idea as to what could be modified or implemented
 in order to improve the running time of this code, or any tip really
 as to where I should start or what I should start looking at. I'd
 really appreciate any help or advice!!

Well, it depends how much effort you want to put into this. The current
interface doesn't actually provide a search interface, but merely
functions to
1) list all package names - dbapi.cp_all()
2) list all package names and versions - dbapi.cpv_all()
3) list all versions for a given package name - dbapi.cp_list()
4) read metadata (like DESCRIPTION) for a given package name and
version - dbapi.aux_get()

One of the main performance problems of --search is that there is no
persistent cache for functions 1, 2 and 3, so if you're just
interested in performance aspects you might want to look into that.
The issue with implementing a persistent cache is that you have to
consider both cold and hot filesystem cache cases: Loading an index
file with package names and versions might improve the cold-cache case,
but slow things down when the filesystem cache is populated.
As has been mentioned, keeping the index updated is the other major
issue, especially as it has to be portable and should require little or
no configuration/setup for the user (so no extra daemons or special
filesystems running permanently in the background). The obvious
solution would be to generate the cache after `emerge --sync` (and other
sync implementations) and hope that people don't modify their tree and
search for the changes in between (that's what all the external tools
do). I don't know if there is actually a way to do online updates while
still improving performance and not relying on custom system daemons
running in the background.

As for --searchdesc, one problem is that dbapi.aux_get() can only
operate on a single package-version on each call (though it can read
multiple metadata variables). So for description searches the control
flow is like this (obviously simplified):

result = []
# iterate over all packages
for package in dbapi.cp_all():
# determine the current version of each package, this is 
# another performance issue.
version = get_current_version(package)
# read package description from metadata cache
description = dbapi.aux_get(version, [DESCRIPTION])[0]
# check if the description matches
if matches(description, searchkey):
result.append(package)

There you see the three bottlenecks: the lack of a pregenerated package
list, the version lookup for *each* package and the actual metadata
read. I've already talked about the first, so lets look at the other
two. The core problem there is that DESCRIPTION (like all standard
metadata variables) is version specific, so to access it you need to
determine a version to use, even though in almost all cases the
description is the same (or very similar) for all versions. So the
proper solution would be to make the description a property of the
package name instead of the package version, but that's a _huge_ task
you're probably not interested in. What _might_ work here is to add
support for an optional package-name-description cache that can be
generated offline and includes those packages where all versions have
the same description, and fall back to the current method if the
package is not included in the cache. (Don't think about caching the
version lookup, that's system dependent and therefore not suitable for
caching).

Hope it has become clear that while the actual search algorithm might
be simple and not very efficient, the real problem lies in getting the
data to operate on. 

That and the somewhat limited dbapi interface.

Disclaimer: The stuff below involves extending and redesigning some
core portage APIs. This isn't something you can do on a weekend, only
work on this if you want to commit yourself to portage development
for a long time.

The functions listed above are the bare minimum to
perform queries on the package repositories, but they're very
low-level. That means that whenever you want to select packages by
name, description, license, dependencies or other variables you need
quite a bit of custom code, more if you want to combine multiple
searches, and much more if you want to do it efficient and flexible.
See http://dev.gentoo.org/~genone/scripts/metalib.py and
http://dev.gentoo.org/~genone/scripts/metascan for a somewhat flexible,
but very inefficient search tool (might not work anymore due to old
age).

Ideally repository searches could be done without writing any
application code using some kind of query language, similar to how SQL
works for generic database searches (obviously not that complex).
But 

Re: [gentoo-portage-dev] search functionality in emerge

2008-11-23 Thread Marius Mauch
On Sun, 23 Nov 2008 21:01:40 -0800 (PST)
devsk [EMAIL PROTECTED] wrote:

  not relying on custom system daemonsrunning in the background.
 
 Why is a portage daemon such a bad thing? Or hard to do? I would very
 much like a daemon running on my system which I can configure to sync
 the portage tree once a week (or month if I am lazy), give me a
 summary of hot fixes, security fixes in a nice email, push important
 announcements and of course, sync caches on detecting changes (which
 should be trivial with notify daemons all over the place) etc. Why is
 it such a bad thing?

Well, as an opt-in solution it might work (though most of what you
described is IMO just stuff for cron, no need to reinvent the wheel).

What I was saying is that _relying_ on custom system
daemons/filesystems for a _core subsystem_ of portage is the wrong
way, simply because it adds a substantial amount of complexity to the
whole package management architecture. It's one more thing that can
(and will) break, one more layer to take into account for any design
decisions, one more component that has to be secured, one more obstacle
to overcome when you want to analyze/debug things.
And special care must be taken if it requires special kernel support
and/or external packages. Do you want to make inotify support mandatory
to use portage efficiently? (btw, looks like inotify doesn't really
work with NFS mounts, which would already make such a daemon completely
useless for people using a NFS-shared repository)

And finally, if you look at the use cases, a daemon is simply overkill
for most cases, as the vast majority of people only use emerge
--sync (or wrappers) and maybe layman to change the tree, usually once
per day or less often. Do you really want to push another system daemon
on users that isn't of use to them?

 Its crazy to think that security updates need to be pulled in Linux.

That's IMO better be handled via an applet (bug #190397 has some code),
or just check for updates after a sync (as syncing is the only
way for updates to become available at this time). Maybe a message
could be added after sync if there are pending GLSAs, now that the glsa
support code is in portage.

Marius



Re: [gentoo-portage-dev] [RFC] portage-2.1.6 release plans

2008-11-07 Thread Marius Mauch
On Fri, 07 Nov 2008 15:38:48 -0800
Zac Medico [EMAIL PROTECTED] wrote:

 Since portage-2.2 isn't quite ready yet due to ongoing work in
 package sets and preserve-libs, and I don't want ongoing work to
 hold back other features that are stable, I'm planning to split a
 2.1.6 branch from trunk. This branch will have package sets and
 preserve-libs support disabled.

Disabled as in FEATURES=-preserve-libs and an almost empty
sets.conf, or disabled as in remove all traces from the code?

Marius



Re: [gentoo-portage-dev] [2/4] proto-GLEPS for Tree-signing

2008-07-29 Thread Marius Mauch
On Tue, 29 Jul 2008 20:51:45 +0100
Mike Auty [EMAIL PROTECTED] wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 Sorry,
   I lost my notes from when I last looked these over several
 months ago, and only just found them again.  I haven't copied this to
 [EMAIL PROTECTED], so let me know if I should do that.  I just had a quick
 couple of things I was thinking about, and one of them I figured out
 during my re-read, so it's only really the following...
 
 In this Glep (xx+1), in the section discussing the procedure for
 creating a MetaManifest file, in step 3.3, does that include
 verification of the manifest's signature if it has one?  It would seem
 odd to ignore the signature if it's wrong (I'm not sure about the case
 if a signature isn't present).  I also don't know how this would then
 be handled (a complete abort, or ignoring the latest changeset to that
 ebuild?).

I don't think that verification at this stage would be a good idea. The
only sane way to respond to a failed check would be to either exclude
the whole package from the sync (keeping the state from the last run),
leading to various problems (what if it's a critical bugfix/security
bump, or breaks a the deptree of many packages?), or not record the
Manifest in the Metamanifest, which hasn't any benefits over reyling on
the client doing the verification.

Marius



Re: [gentoo-portage-dev] portage-2.2-rc3 parallel merges quit being parallel

2008-07-26 Thread Marius Mauch
On Sat, 26 Jul 2008 16:56:20 -0500
Andrew Gaffney [EMAIL PROTECTED] wrote:

 Duncan wrote:
  --jobs=10 --keep-going --load-average=15
 
 For a dual-dual-core setup, a load average of 4.0 is fully loaded.

Only in ideal cases, when you have long-running processes
hammering the cpu and little or no context switches. This isn't the case
with builds, the actual compile processes that need cpu time usually
terminate very quickly, which increases the load average.
I did some benchmarking a while ago with different combinations of -j
and -l in MAKEOPTS, using the kernel as testcase, and IIRC the fastest
builds were around -l6.0 (on a dual-core system) and high (or
unlimited) values for -j (sidenote here: some ebuilds like openoffice
parse MAKEOPTS to get parameters for their own build systems, but only
recognize/support -j and ignore -l, so one should still be careful with
the value for -j).

Marius



Re: [gentoo-portage-dev] A seemingly bug found when emerging @preserved-rebuild

2008-07-01 Thread Marius Mauch
On Wed, 2 Jul 2008 02:41:14 +0800
Zhang Le [EMAIL PROTECTED] wrote:

 Is this a bug? or did I miss something here?
 Thanks for your time!

The 'bug' here is that USE=multislot shouldn't exist. People using it
should be able to deal with resulting breakages on their own

Marius
-- 
gentoo-portage-dev@lists.gentoo.org mailing list



Re: [gentoo-portage-dev] Portage persistence structures :: information about ports tree

2008-06-09 Thread Marius Mauch
On Mon, 09 Jun 2008 17:36:14 -0300
João Macaíba [EMAIL PROTECTED] wrote:

 May someone give me some help on this ? How does portage do the
 searchs ? Walk into the ports tree and build some structure or store
 this info on some embedded database like berkeley db or sqlite ?

You're probably looking for the portdbapi class defined in
pym/portage.py (or pym/portage/dbapi/porttree.py in 2.2), in particular
the cp_all(), cpv_all(), cp_list() and aux_get() methods.

Marius
--
gentoo-portage-dev@lists.gentoo.org mailing list



Re: [gentoo-portage-dev] [PATCH] Repoman subversion support

2008-05-13 Thread Marius Mauch
On Mon, 12 May 2008 20:01:34 +0200
Fabian Groffen [EMAIL PROTECTED] wrote:

 After a short discussion on #-portage, here a feature patch from the
 prefix branch, diffed to the trunk, irrelevant hunks removed.
 
 Antarus once was working on factoring out the vcs bits of repoman into
 pym/repoman/utilities.py, but he never finished it.  I'm willing to
 give that some more time, provided it'll end up in the trunk.  Sofar,
 here's the svn patch that we tested extensively in Prefix ;)  I still
 believe that it has a significant value for non-Prefix users
 (overlays...), even though the code doesn't deserve any price for its
 beauty.  I'd say it fits nicely with large chunks of Portage :p

Merged in r10325 with some minor changes (removed the 'svn update' bit
until someone remembers why exactly it's there, skip the commit
optimization for svn regarding Manifest regeneration, unify isVcs and
vcs variables)

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] BNF for DEPEND in EAPI0 and EAPI1

2008-04-14 Thread Marius Mauch
On Sun, 13 Apr 2008 20:04:41 +0200
Björn Michaelsen [EMAIL PROTECTED] wrote:

 Hi there,
 
 I worked through the current PMS and tried to formalize the grammar
 for DEPEND a little better. Maybe someone could have a look at it and
 check it for correctness. It could even become part of the gentoo
 development docs or be an appendix to the PMS. The other grammars
 (SRC_URI etc.) should be fairly easy.
 
 Any comments?

Offhand the 'package name' part is wrong/incomplete, pretty sure even
PMS says that they can't start/end with underscores or dashes (it's a
real PITA to get it right).
Haven't really read the non-terminals section as it's hard to read in
this form, using EBNF or some markup language might help with that.

Marius
--
gentoo-portage-dev@lists.gentoo.org mailing list



Re: [gentoo-portage-dev] Re: querying whether pkg is installed with version X or upwards (atom syntax) (re-post)

2008-03-26 Thread Marius Mauch
On Wed, 26 Mar 2008 12:07:13 +0200
Amit Dor-Shifer [EMAIL PROTECTED] wrote:

 reposting, hoping for a reply this time.
 Thanks,
 Amit
 
 Amit Dor-Shifer wrote:
  emerge features syntax for installing a package according to certain
  bounds. This is referred to in the manpage as atom specification.
 
  I'm wondering whether equery has the same feature for querying.
  E.G: I'd like to be able to ask:
 
  equery l '=app-misc/my-pkg-1.1'
 
  I'd expect the listing to be empty when my-pkg is not installed, or
  is installed by of version less than 1.1.

Well, most equery subcommands support the same atom syntax, just 'list'
and 'size' apparently don't implement it, as those support regular
expressions instead.

Marius
-- 
gentoo-portage-dev@lists.gentoo.org mailing list



Re: [gentoo-portage-dev] Re: [gentoo-qa] splitting up package.mask

2008-03-15 Thread Marius Mauch
On Sat, 15 Mar 2008 07:45:19 -0400
Mike Frysinger [EMAIL PROTECTED] wrote:

 i dont see it being any different from an ebuild declaring its own
 KEYWORDS. if you want to be lazy, then have a magic KEYWORD: +pmask.

What's wrong with KEYWORDS= ? (I'm not saying it's the perfect
solution, just looking for specific requirements)

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Re: [gentoo-qa] splitting up package.mask

2008-03-15 Thread Marius Mauch
On Sat, 15 Mar 2008 01:34:24 -0700
Alec Warner [EMAIL PROTECTED] wrote:

 On 3/14/08, Mike Frysinger [EMAIL PROTECTED] wrote:
  On Friday 14 March 2008, Alec Warner wrote:
On 3/14/08, Mike Frysinger [EMAIL PROTECTED] wrote:
 i think the real solution here is allowing masking in a package
   
 
   You want to add a metadata key and cache it you mean?
 
 
  i dont care terribly much about the logistics, just the results.
  as long as an ebuild can declare itself masked, it sounds good to
  me.
 
   this doesnt preclude the other ideas as there are often times
  where you want to have 1 global package mask piece (like large
  package set bumps ... so X or KDE or GNOME or ...).
 
  -mike
 
 
 
 [-gentoo-qa, +gentoo-portage-dev]
 
 Original thread was splitting up package.mask entries.
 
 Genone notes the code to do this already is basically in already (we
 just don't invoke it for $PORTDIR/profiles afaik).
 
 Genone, do we use existing code for package.mask (ie if we switch from
 a file to  a dir will it break existing versions?  I am unsure if we
 used the directory code for $PORTDIR/profiles/*


The handling code for all package.mask files is identical, in
particular they all support recursion (just search for package.mask
in portage/__init__.py).

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] please explain use of hooks

2008-03-02 Thread Marius Mauch
On Sun, 2 Mar 2008 11:18:39 +0100
Jonas Bernoulli [EMAIL PROTECTED] wrote:

 And if not the handbook this should really go into the portage
 manpage.
 
 bashrc
 
 If  needed, this  file can  be  used to  set up  a
 special  environment for  ebuilds,  different from
 the standard root  environment.  The syntax is the
 same as for any other bash script.
 + One thing you can do here is define phase hooks.
 + A phase hook function name begins with a pre_ or post_ prefix to
 indicate that it will be called before or after one of the ebuild
 phases. The prefix is followed by the name of the ebuild function that
 the hook will be associated with. + For example, a hook named
 pre_src_compile will be called before src_compile, and a hook named
 post_src_compile will be called after src_compile.

Care to write a patch? (nobody likes to edit manpages)

 postsync.d should also be documented there.

No, as it isn't used by portage itself.

 All in all now that I have looked at the documentation installed with
 portage I think it is obsolete. Not in the sense that it is outdated
 but everything in it is documented elsewhere already or should be
 moved there.

Well, it's horribly incomplete and a work in progress. Eventually it
will replace the manpages (in the sense that manpages will be generated
from the same source). And authorative documentation (which isn't
exactly the same as a specification) should be kept with the
implementation, and that's not the case with the dev handbook/manual.

 If some developers an the list agree with me and welcome me to do so
 then I could file a bug(s) for those changes.

No.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


[gentoo-portage-dev] New maintainers needed

2007-12-03 Thread Marius Mauch
For various reasons I can no longer maintain the following packages (in
fact I haven't really maintained them for a while already), so they
need a new maintainer:

* app-editors/zoink - simple gtk based editor, little to no maintenance
required
* dev-util/gambas - Visual Basic look-alike for Linux, fairly active
development (2.0 is about to come out), doesn't work on 64bit archs
(only known to work on x86), has a number of open bugs

Also I'm going to remove myself from the tools-portage team as I
haven't done much there either, which effectively only leaves
Paul (fuzzyray) on it, who is also somewhat inactive recently, so we
could use a few new people there as well.

Finally at the beginning of this year I started to convert packages
from einfo to elog where appropriate, unfortuntely I haven't been able
to complete that job, if anyone is interested in completing it please
let me know (I guess it's currently about 40-60% done).

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] use* cleanup

2007-10-30 Thread Marius Mauch
On Tue, 30 Oct 2007 19:32:52 +0100
Marijn Schouten (hkBst) [EMAIL PROTECTED] wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 The purpose of this patch is to expose a generic function, namely
 _use, which can be used to build your own use* variant if you need
 that. I reimplemented all other current use function using _use (and
 _if) to cut out duplicate and verbose code. Comments expected. I
 didn't test this code.

Please don't post patches inline, there are these things called
attachments that work so much better for such things.

Marius
-- 
[EMAIL PROTECTED] mailing list



Re: [gentoo-portage-dev] RFC: replacing packages

2007-10-26 Thread Marius Mauch
On Thu, 25 Oct 2007 22:14:40 -0700
Alec Warner [EMAIL PROTECTED] wrote:

  Another issue that isn't directly related, but covered by the
  proposed solution below, is the so called implicit system
  dependency in ebuilds, which doesn't really exist. It only an
  assumption by people that the system set is satisfied before an
  ebuild is processed, so its contents don't have to be listed in
  *DEPEND. If a user breaks that assumption by not regulary ensuring
  that system is satisfied bugs can occur. Another issue is the
  informal exception when system items are allowed to/must be listed
  in *DEPEND, IMO that should be formalized.
 
 I'm going to discuss a bit your implementation below, since you
 address what you are trying to solve here.  My main problem is that
 given a tree (such as gentoo-x86) it may have one or more profiles
 with one or more sub-profiles (children).  These profiles may contain
 a set of packages that make up 'system'.  The ebuilds DEPEND (or
 RDEPEND or whatever) on this set of packages existing.  The problem I
 see with your solution is that it doesn't address different system
 sets in different profiles.  Now in practice this isn't much of a
 problem within gentoo-x86 (most profiles have similar system sets).
 However it places a certain onus on any profile in a given tree, it
 must have a system package 'similar' enough to other sets in
 functioning profiles; otherwise breakage is likely to occur.  How
 would you address this in your system?

How is the problem solved now? Remember that I'm just proposing to
formalize existing assumptions (packages in system should state all
their dependencies explicitly). The answer is that there is no real
solution to that problem, at least not without creating a huge mess by
enumerating all available profiles.
The real question is what system is supposed to be. Is it just a more
or less arbitrary list of packages chosen by the profile maintainer, or
does it have a global, predefined purpose, like defining the minimum
requirements for packages in the associated tree? In the first case the
second part of my proposal obviously doesn't work, as you can't make
any valid assumptions about system. In the second case however the
purpose already mandates that the content of system is effectively
identical across different profiles.

Marius
-- 
[EMAIL PROTECTED] mailing list



Re: [gentoo-portage-dev] RFC: replacing packages

2007-10-25 Thread Marius Mauch
On Wed, 24 Oct 2007 13:34:44 -0500
Andrew Gaffney [EMAIL PROTECTED] wrote:

 For packages that are in the system set, wouldn't adding the
 contents of system.{,r}depend to {,R}DEPEND cause problems in dep
 resolution? Would there be a way to prevent the contents of these
 files from being added for packages that are themselves in these
 files?

Did you miss the part about RESTRICT=system-deps?

 How is the system set constructed from these files? Would they
 simply be concatenated (minus duplicates) and treated like the old
 packages file (minus the magic visibility filtering)?

Maybe, we'd have to see what works best.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


[gentoo-portage-dev] RFC: replacing packages

2007-10-24 Thread Marius Mauch
As package sets are mostly done now, I'm starting to think about
something else. One of my pet peeves with portage is the packages
file in profiles, for several reasons:
1) it has two completely independent purposes
2) it implements a redundant visibility filter as package.mask is also
available in profiles
3) the syntax for defining the system set plain sucks
4) the name doesn't really say anything about the purpose

Another issue that isn't directly related, but covered by the proposed
solution below, is the so called implicit system dependency in
ebuilds, which doesn't really exist. It only an assumption by people
that the system set is satisfied before an ebuild is processed, so
its contents don't have to be listed in *DEPEND. If a user breaks that
assumption by not regulary ensuring that system is satisfied bugs can
occur. Another issue is the informal exception when system items are
allowed to/must be listed in *DEPEND, IMO that should be formalized.

To solve both issues I propose the following system:
Profiles can contain the files system.depend and system.rdepend
whose combined contents make up the system target. If any of those
two files exists the packages file is ignored. The syntax should
either be the same as package.mask or, if desired, also allow complex
atoms (use-conditionals, any-of constructs).
The second part would be that portage implicitly adds the content of
the files to the DEPEND or RDEPEND setting of each ebuild unless it
contains RESTRICT=system-deps. Obviously this would have to be tied to
a new EAPI version. (an undefined cornercase here is if a profile does
not contain the new files, not sure how that should be handled)

Benefits:
- we get rid of all the issues outlined at the top
- should make it easier long-term to setup a no-compile system that only
installs binary packages due to separation of build- and runtime deps
in system

Problems:
- obviously it has to be implemented first
- no obvious solution for stacking rules (e.g. child uses system.*, but
parent profile only has packages)
- some people might rely on the packages file

So, comments?

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Re: About system and world

2007-10-23 Thread Marius Mauch
On Mon, 22 Oct 2007 21:29:02 -0700
Zac Medico [EMAIL PROTECTED] wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 Ryan Hill wrote:
  Zac Medico wrote:
  implement greedy atoms for the world set. I've been pondering the
  idea of making world non-greedy for slots by default [1], since you
  can add slot atoms to the world file to pull in any specific
  slot(s) that you want.
  
  That would rock.  Portage always insisting on pulling in the highest
  SLOT whether needed or not is one of my biggest pet-peeves.
 
 Well, you can already use SLOT atoms in your world file if you don't
 want the highest available. Packages that pull in =foo are a
 different story though. I suppose we can add something like a
 - --upgrade=minimal option that prevents pulling in new slots if they
 aren't required.

Don't restrict it to SLOTs though. minimal implies that only upgrades
required to satisfy the depgraph are performed. The described
behavior should be another value, e.g. no-slot-change.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] About system and world

2007-10-23 Thread Marius Mauch
On Sun, 21 Oct 2007 15:12:31 +0200
Marius Mauch [EMAIL PROTECTED] wrote:

 Well, the primary goal is to make all sets behave in a consistent way.
 And some sets have the explicit purpose to rebuild stuff, so making
 sets selective by default also has issues.
 The proposed change would also make sets behave in the same way as
 packages which is IMO another benefit.

This actually has more consequences: selective is a global setting,
you can't enable it just for some arguments. Therefore if packages and
sets are treated differently it requires that they can't be mixed on
the commandline (and if we'd make the setting configurable for each set
then only one set can be used at any time). And right now I can't think
of another reason why that restriction would be necessary.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Re: About system and world

2007-10-23 Thread Marius Mauch
On Tue, 23 Oct 2007 21:11:48 +0200
Thomas de Grenier de Latour [EMAIL PROTECTED] wrote:

 On 2007/10/23, Marius Mauch [EMAIL PROTECTED] wrote:
 
  On Mon, 22 Oct 2007 21:29:02 -0700
  Zac Medico [EMAIL PROTECTED] wrote:
  
   Well, you can already use SLOT atoms in your world file if you
   don't want the highest available. Packages that pull in =foo are
   a different story though. I suppose we can add something like a
   - --upgrade=minimal option that prevents pulling in new slots if
   they aren't required.
  
  Don't restrict it to SLOTs though. minimal implies that only
  upgrades required to satisfy the depgraph are performed. 
 
 Couldn't this minimal behavior just be triggered by lack of the
 --upgrade option (emerge -D set)?

--upgrade != the current --update (at least that's what I assumed)

 Actually, the current --upgrade behavior (with or without --deep)
 is a bit weird imho, i would prefer something like this (with foo 
 being either a set or an atom):

Yes, --update has a history of being a mess of random behavior.
Personally I'd drop it completely, or alias it to --noreplace.

  * emerge foo: do the minimum upgrades or new installs required to
 get foo satisfied.  Only its direct deps are checked (and direct
 deps of the unsatisfied ones, etc.), with the minimal heuristic
 (upgrade them only if stricly required).
 
  * emerge -u foo: do all required new installs and possible upgrades
 of foo.  Only its direct deps are checked (and direct deps of the
 unsatisfied ones, etc.), with the minimal heuristic (upgrade them
 only if strictly needed).
 
  * emerge -D foo: do the minimal upgrades or new installs required
 to get foo satisfied.  Also, check all of its direct and indirect
 deps, with the minimal heuristic (upgrade only if strictly needed).
 
  * emerge -uD foo: do all required new installs and possible
 upgrades of foo.  Also, all its direct and indirect deps are checked,
 and upgraded where possible.
 
Maybe --upgrade as name was an unfortunate choice, --resolver might be
better. What I'd like ideally (as a user) is to replace --update,
--deep, --noreplace and --emptytree with the following options:

--resolver={minimal,noslotchange,leastchange,default}
minimal: select the lowest possible version
noslotchange: like default, but try to avoid slotchanges
leastchange: select the version with the smallest version delta
compared to the installed version, otherwise like default
default: current behavior

--rebuild={never,always,changeduse,newuse,changeddeps,selected}
never: never rebuild packages
always: always rebuild packages (like current --emtpytree)
changeduse: rebuild packages if $USE has changed
newuse: rebuild packages if $USE or $IUSE has changed
changeddeps: rebuild packages if any of its direct dependencies were
updated
selected: rebuild the selected arguments (root nodes)

--deplevel=n
check n level of dependencies for updates (with -1 == infinite)
of course all dependencies have to be satisfied, this controls only
what deps should be checked for optional updates

current defaults would be --resolver=default --rebuild=selected
--deplevel=0

--update would be --rebuild=never --deplevel=1
--deep would be --deplevel=-1
--emptytree would be --rebuild=always --deplevel=-1

But that's just me dreaming of a better world, where we don't have
to deal with legacies ;)

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Re: About system and world

2007-10-23 Thread Marius Mauch
On Wed, 24 Oct 2007 00:03:47 +0200
Thomas de Grenier de Latour [EMAIL PROTECTED] wrote:

 Bah, you're introducing a new options set, and have (i think) ways to
 emulate all of the old ones with them, so what is stopping you?  You
 would just have to forbid mixes of old style and new style on the same
 command line.

Mainly the work required to implement it ;) And I'd rather finish
dynoparse first so it can be used here.
Oh, and ideally the mentioned values wouldn't be hardcoded lists, but
come from a pluggable resolver system, but that probably counts as
overengineering.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] localization.py

2007-10-22 Thread Marius Mauch
On Mon, 22 Oct 2007 04:54:59 +0200
Arfrever Frehtes Taifersar Arahesis [EMAIL PROTECTED] wrote:

 2007-10-21 22:49:10 Marius Mauch napisał(a):
  On Sun, 21 Oct 2007 12:23:59 -0700
  Zac Medico [EMAIL PROTECTED] wrote:
  
   -BEGIN PGP SIGNED MESSAGE-
   Hash: SHA1
   
   Arfrever Frehtes Taifersar Arahesis wrote:
Hello,

Does localization.py exist for a reason?

   
   Over the years we've had a few people express a desire for
   internationalization and localization support in portage (bugs
   #13618, #32630, and #191189 and possible others).
   
   It seems like a nice idea to me, but I recall Marius being opposed
   to the idea due to some potential problems that he was concerned
   about. Unfortunately I don't have a link to the previous
   discussion but maybe somebody else does.
  
  Probably not the one you meant, but you can find a related
  discussion on the forums:
  http://forums.gentoo.org/viewtopic-t-500209.html
 
 It seems that you just don't like German translation, so you could set
 LC_ALL=C in make.conf.

I don't have an issue with localizations on my systems (I don't
use them anyway), the problem is with other people reporting bugs or
asking for support posting messages we can't understand, the
questionable quality of translations (as we can't really control them)
potentially leading to bogus bug reports and the general added
complexity. It's just my opinion that system tools simply shouldn't be
localized (best example: gcc error messages).

Marius
-- 
[EMAIL PROTECTED] mailing list



Re: [gentoo-portage-dev] About system and world

2007-10-21 Thread Marius Mauch
On Sun, 21 Oct 2007 05:23:45 -0700
Ned Ludd [EMAIL PROTECTED] wrote:

 On Sun, 2007-10-21 at 13:01 +0200, Marius Mauch wrote:
  So, what do people think about removing (some) of the special
  treatment for the system and world targets?
  Mainly I'm interested in removing the selective parameter that's
  currently enabled for them (so for example without that parameter
  `emerge world` would default to remerging packages, unless --update
  or --noreplace are specified). That change has already been
  requested a few times in the past by users, but OTOH it could
  probably upset people who don't use --update with world/system.
 
 What would such a disruptive change really gain us?

The goal is to make package sets behave in a consistent way.

 I personally feel our users need consistency from Gentoo. If they 
 grew up doing 'emerge world' and have come to expect that behavior 
 and all of the sudden we change behavior on them.. Yeah I can see 
 how ppl would get upset.

As do I, which is why I haven't simply changed it.

 Perhaps a less intrusive way would be to introduce another
 flag to get the specified behavior you are after.

Well, the primary goal is to make all sets behave in a consistent way.
And some sets have the explicit purpose to rebuild stuff, so making
sets selective by default also has issues.
The proposed change would also make sets behave in the same way as
packages which is IMO another benefit.
But as I said, I can see that it could upset people.

One possible solution that I've thought about would be to remove the
hardcoded selective parameter and let the set configuration determine
if a set is selective or not (with missing = false), and then enable
it for world and system in our default config, e.g.

[world]
class = portage.sets.files.WorldSet
selective = True

and extend the --rebuild option with new arguments always and never.
Don't really like the additional complexity though (and I haven't
checked how much work it would be).

Marius
-- 
[EMAIL PROTECTED] mailing list



Re: [gentoo-portage-dev] localization.py

2007-10-21 Thread Marius Mauch
On Sun, 21 Oct 2007 12:23:59 -0700
Zac Medico [EMAIL PROTECTED] wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 Arfrever Frehtes Taifersar Arahesis wrote:
  Hello,
  
  Does localization.py exist for a reason?
  
 
 Over the years we've had a few people express a desire for
 internationalization and localization support in portage (bugs
 #13618, #32630, and #191189 and possible others).
 
 It seems like a nice idea to me, but I recall Marius being opposed
 to the idea due to some potential problems that he was concerned
 about. Unfortunately I don't have a link to the previous discussion
 but maybe somebody else does.

Probably not the one you meant, but you can find a related
discussion on the forums:
http://forums.gentoo.org/viewtopic-t-500209.html
-- 
[EMAIL PROTECTED] mailing list



[gentoo-portage-dev] Re: [gentoo-dev] [RFC] Properties of package sets

2007-07-05 Thread Marius Mauch
On Fri, 29 Jun 2007 05:07:28 +0200
Marius Mauch [EMAIL PROTECTED] wrote:

 Please reply on gentoo-portage-dev, _not_ on gentoo-dev, thanks.
 
 One missing feature in portage is the lack of package sets. Before we
 (re)start working on that however I'd like to get some feedback about
 what properties/features people would expect from portage package set
 support.
 Some key questions:
 
 - should they simply act like aliases for multiple packages? E.g.
 should `emerge -C sets/kde` be equivalent to `emerge -C kdepkg1
 kdepkg2 kdepkg3 ...`? Or does the behavior need to be smarter in
 some ways?
 
 - what kind of atoms should be supported in sets? Simple and versioned
 atoms for sure, but what about complex atoms (use-conditional, any-of,
 blockers)?
 
 - should sets be supported everywhere, or only in selected use cases?
 (everywhere would include depstrings for example)
 
 - what use cases are there for package sets? Other than the
 established system and world, and the planned all and
 security sets.
 
 - how/where should sets be stored/distributed?

Forgot one question:

- should sets have metadata? (e.g. a description for searching)

There hasn't been much feedback yet, so if you want to add anything now
is your chance, otherwise I'll implement things the way that works
best/is easiest for me, which might be different from what you expect.

Marius

PS: I also accept off-list replies

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


[gentoo-portage-dev] [RFC] Properties of package sets

2007-06-28 Thread Marius Mauch
Please reply on gentoo-portage-dev, _not_ on gentoo-dev, thanks.

One missing feature in portage is the lack of package sets. Before we
(re)start working on that however I'd like to get some feedback about
what properties/features people would expect from portage package set
support.
Some key questions:

- should they simply act like aliases for multiple packages? E.g.
should `emerge -C sets/kde` be equivalent to `emerge -C kdepkg1 kdepkg2
kdepkg3 ...`? Or does the behavior need to be smarter in some ways?

- what kind of atoms should be supported in sets? Simple and versioned
atoms for sure, but what about complex atoms (use-conditional, any-of,
blockers)?

- should sets be supported everywhere, or only in selected use cases?
(everywhere would include depstrings for example)

- what use cases are there for package sets? Other than the established
system and world, and the planned all and security sets.

- how/where should sets be stored/distributed?

Please reply on gentoo-portage-dev, _not_ on gentoo-dev, thanks.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [RFC] Properties of package sets

2007-06-28 Thread Marius Mauch
On Thu, 28 Jun 2007 21:03:54 -0700
Ned Ludd [EMAIL PROTECTED] wrote:

 On Fri, 2007-06-29 at 05:07 +0200, Marius Mauch wrote:
  Please reply on gentoo-portage-dev, _not_ on gentoo-dev, thanks.
  
  One missing feature in portage is the lack of package sets. Before
  we (re)start working on that however I'd like to get some feedback
  about what properties/features people would expect from portage
  package set support.
  Some key questions:
  
  - should they simply act like aliases for multiple packages? E.g.
  should `emerge -C sets/kde` be equivalent to `emerge -C kdepkg1
  kdepkg2 kdepkg3 ...`? Or does the behavior need to be smarter in
  some ways?
 
 I like the alias way acting simply as a metapkg.

Well, mind that a very simple alias implementation as outlined above
would for example unmerge kdepkg3 even if it was merged explicitly.

  - should sets be supported everywhere, or only in selected use
  cases? (everywhere would include depstrings for example)
 
 Please NO. emerge.py should know about sets but ebuild.py should be
 oblivious to them. package sets as you are proposing imo should be
 limited to portage only. By putting package sets in ebuild depstrings
 we would be effectively forcing all other pkg managers to support the
 feature. I don't think we should do that.

What about config files (package.*) or other sets?

  - what use cases are there for package sets? Other than the
  established system and world, and the planned all and
  security sets.
 
 Assuming you guys run with with the simple alias method I don't see
 how 'security' can fit into this. 

Simple: the list of atoms would be assembled dynamically by checking
the available GLSAs, similar to how the affected target works in
glsa-check. But lets keep the backend implementation details out for
now.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] LC_ALL and friends in make.conf

2007-03-08 Thread Marius Mauch
On Thu, 8 Mar 2007 20:14:26 +0100
Kevin F. Quinn [EMAIL PROTECTED] wrote:

 This is an old issue, but I want to suggest a re-visit :)
 
 As we all know, setting LC_ALL and friends can cause all sorts of
 trouble in package builds.  However, many users really appreciate
 being able to set it so that errors from the compiler etc are in their
 own language.

Actually most people I've talked to prefer to have buildsystem messages
in english even if the rest of their system is localized. Reasons
include:
- many users don't understand the messages anyway, so to them language
is irrelevant
- often the translations are harder to understand than the english
version (IMO)
- when reporting a bug or asking questions on forums/irc/lists/...
they'll usually need the english messages anyway
- when searching for error messages in bugzilla/google/... the english
version will usually get more and/or better results

So going with Mike I'd invert your proposal and have portage (un)set
LC_MESSAGES to get english buildsystem messages, e.g. by adding

export LC_MESSAGES=${PORTAGE_LC_MESSAGES:-en_US}

to ebuild.sh.

Marius
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] dep resolution weirdness

2007-02-17 Thread Marius Mauch
On Sat, 17 Feb 2007 12:30:31 +0100
George Shapovalov [EMAIL PROTECTED] wrote:

 Hi guys.
 
 I am quite confused by the following:
 
 aldar ~ # emerge -puD world --tree
 These are the packages that would be merged, in reverse order:
 Calculating world dependencies... done!
 [nomerge  ] dev-ada/cbind-6.0
 [nomerge  ]  virtual/gnat-4.1
 [ebuild UD]   dev-lang/gnat-gcc-4.1.1 [4.1.2]
 
 aldar ~ # emerge -puD cbind --tree
 These are the packages that would be merged, in reverse order:
 Calculating dependencies... done!
 
 (or, for that matter:
 aldar gnat # e -puD virtual/gnat
 These are the packages that would be merged, in order:
 Calculating dependencies... done!

 The relevant dependency lines are here:
 dev-ada/cbind:   DEPEND=virtual/gnat
 
 virtual/gnat-4.1: 
 RDEPEND==dev-lang/gnat-gcc-4.1*
 DEPEND=
 
 (and virtual/gnat-3.4:
 RDEPEND=|| ( =dev-lang/gnat-gcc-3.4*
 =dev-lang/gnat-gpl-3.4* )
 DEPEND=
 although it looks like this one is not considered in this particular
 case)
 
 As you see emerge -uD world wants to downgrade gnat for some reason
 (which is wrong), while emerge -uD cbind (or any other Ada library
 that has similar dependencies) does not (which is right).

Is there maybe a different package in world that depends on
=gnat-gcc-4.1.1? Anyway, you may want to look at --debug output to get
a clue why it wants to downgrade.

 Oh, and ~dev-lang/gnat-gcc-4.1 as RDEPEND in virtual/gnat does not
 work at all (just, as I understand, it should not)).

It should work, but it's not the same as the dep you stated above:
it would match gnat-gcc-4.1-r1, gnat-gcc-4.1-r2 and so on, but not
gnat-gcc-4.1.1, gnat-gcc-4.1.2, ...

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


[gentoo-portage-dev] New preserve-libs feature

2007-02-17 Thread Marius Mauch
If you haven't noticed, I just added a new 'preserve-libs' feature for
bug 62207 that moves shared object that are still used but would be
removed on an update into the new package instance (so on a update from
expat-1 to expat-2 the user would still have libexpat.so.0, owned by
expat-2). The actual match criteria are a bit stricter, but you get the
idea. I think this is an long overdue bugfix, but given past
discussions not everyone has the same opinion, though the last
discussion about it ended without a conclusion (at least none that I
could see).
So everyone who has valid objections to the _general idea_ of this
implementation (preserving old libraries to avoid some runtime linker
errors) speak up now. 

Just keep the following things in mind:
- I'm not claiming that it's a silver bullet to all possible runtime
linker problems, it's supposed to cover some of the common cases (like
the expat problem)
- I'm not claiming that the implementation is perfect yet
- This feature is currently optional, so I'm not forcing it down on
anyone (in fact it's completely hidden for now). Of course if people
start using it ebuild devs will have to deal with it sooner or later,
but lets discuss it here first.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] dep resolution weirdness

2007-02-17 Thread Marius Mauch
On Sat, 17 Feb 2007 13:42:49 +0100
George Shapovalov [EMAIL PROTECTED] wrote:

 Ok, found it.
 Thanks Marius! (for the debug hint)
 
 It was indeed forcing gnat-gcc-4.1.1 by asis-gcc-4.1.1 which has 
 =dev-lang/gnat-4.1.1 (this is an extension to compiler and has to
 match it 1 for 1, forgot to bump that one :().
 However, shouldn't --tree have reported the relevant package as
 requiring this downgrade instead of some other, pretty much arbitrary
 (from dependants of gnat-gcc)? 

In an ideal world: yes, but portage is far from being ideal (I won't
pretend to know the internals of the dep resolver, so can't give a good
explanation for this behavior).

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] New preserve-libs feature

2007-02-17 Thread Marius Mauch
On Sat, 17 Feb 2007 14:55:26 +0100
Simon Stelling [EMAIL PROTECTED] wrote:

 Marius Mauch wrote:
  So everyone who has valid objections to the _general idea_ of this
  implementation (preserving old libraries to avoid some runtime
  linker errors) speak up now. 
 
 For how long are these libraries preserved? This might have a security
 impact in cases like the recent openssl-case where you had to upgrade
 to an incompatible ABI because the version using the old one was
 vulnerable. Using preserve-libs it would leave the old lib around,
 making it possible for programs to link against the wrong version and
 ending up being vulnerable. I realize that the feature is meant to
 help the transitional phase until all apps are built against the new
 ABI, but how would you find these vulnerable apps currently?
 revdep-rebuild wouldn't rebuild them since they are still functional.

Currently they are around as long as they are referenced by other
packages or until the package is unmerged. And yes, there should be a
way to tell revdep-rebuild/the user which packages should/need to be
rebuilt, but I haven't made my mind up yet on how to accomplish that
(in fact atm there is no separation between native and imported
libs in vdb, I'm aware that needs to be added).

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


[gentoo-portage-dev] Storing origin repository in VDB

2007-02-16 Thread Marius Mauch
There are several cases where the information from which repository
(portdir, overlays, ...) a package originally came from when it was
installed. Currently that information isn't really available, at most
you can use some heuristics on environment.bz2 or comparing ebuilds
directly, but those methods aren't very reliable (may return both false
positives and false negatives). So storing the origin repo seems like a
good idea, qustion is how:

1) Save (part of) the value of $EBUILD
2) Save the repository name
3) ???

Option 1) has the benefit that it would be useful from the start, but
the drawback that if the path of a repository changes the information
becomes invalid. Option 2) avoids that but would require that all
repositories are named to be useful. Don't know if there is another
option that avoids both problems.
If I'd had to choose I'd probably go for option 2) and encourage people
to name their repositories, but I'm open for suggestions to use option
1) or for a third option.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [RFC] Depending on active version

2007-01-31 Thread Marius Mauch
On Wed, 31 Jan 2007 17:47:26 +
Stephen Bennett [EMAIL PROTECTED] wrote:

 On Tue, 30 Jan 2007 17:06:51 +0100
 Marius Mauch [EMAIL PROTECTED] wrote:
 
  The idea is to add a special category (let's call it active for
  now) that has the following properties:
  - this category doesn't exist in portdir or vdb (= no ebuilds)
  - when portage ($pkgmanager) encounters a active/foo atom in a
  dependency string it executes some special code (e.g.
  $PORTDIR/scripts/active-check/foo =active/foo-1) to determine if
  that atom is satisfied
 
 Given that in the general case the package manager can't change the
 active provider and will have to bail with an appropriate message that
 the user needs to change it themselves, the obvious solution to this is
 the previously-discussed-somewhere-I-can't-remember ebuild function,
 called on depgraph creation, to check that it will be able to compile
 in the current system environment. It's considerably simpler and more
 generally useful than subverting DEPEND to add weird special-case hacks
 to it.

Yeah, thinking about it I have to agree tat generic dep_check() support (to 
give the thing a name) would be a better solution here.

Marius
-- 
gentoo-portage-dev@gentoo.org mailing list



[gentoo-portage-dev] [RFC] Depending on active version

2007-01-30 Thread Marius Mauch
Sometimes a package has to depend on a specific version of a slotted package 
being the active one to build correctly, like in the current tr1 discussion 
on -dev [1] or with packages that depend on the running kernel.

Currently this isn't really possible, however I while ago I got an idea how to 
solve this. Keep in mind this is just a rough idea and I'm pretty sure some 
people can/will point out why it is a stupid idea, but anyway:

The idea is to add a special category (let's call it active for now) that has 
the following properties:
- this category doesn't exist in portdir or vdb (= no ebuilds)
- when portage ($pkgmanager) encounters a active/foo atom in a dependency 
string it executes some special code (e.g. $PORTDIR/scripts/active-check/foo 
=active/foo-1) to determine if that atom is satisfied

(and yes, this kinda goes with multi-repo/multi-format support)

Marius
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [RFC] Depending on active version

2007-01-30 Thread Marius Mauch
On Tue, 30 Jan 2007 08:25:31 -0800
Brian Harring [EMAIL PROTECTED] wrote:

 On Tue, Jan 30, 2007 at 05:06:51PM +0100, Marius Mauch wrote:
  Sometimes a package has to depend on a specific version of a 
  slotted package being the active one to build correctly, like in 
  the current tr1 discussion on -dev [1] or with packages that 
  depend on the running kernel.
 
 tr1 is partially addressed via addition of a 'binding' modifier for 
 rdeps, to state that ||() deps are locked down after compilation.

And how would that solve the actual issue of expressing I need /usr/bin/gcc to 
run gcc-4.1 and not gcc-3.4?
The lockdown of || deps is a completely separate issue, unless I'm missing 
something.

  The idea is to add a special category (let's call it active for 
  now) that has the following properties:
  - this category doesn't exist in portdir or vdb (= no ebuilds)
  - when portage ($pkgmanager) encounters a active/foo atom in a 
  dependency string it executes some special code (e.g. 
  $PORTDIR/scripts/active-check/foo =active/foo-1) to determine if 
  that atom is satisfied
 
 Non deterministic resolution; previous steps in the graph can cause 
 that value to flip to a different setting by the time the 'dep' is 
 encountered.

Ok, that's a problem, though for the use cases at hand (gcc and kernel) it 
would be mostly irrelevant.

 That's ignoring the kick in the nads usage of this will due to 
 resolution...

Neglectable IMO, it's not such a common use case anyway, and I don't think I 
have to compare it to the current solution (die in setup or compile).

  (and yes, this kinda goes with multi-repo/multi-format support)
 
 Don't really see how this enables multiple standalone repos in any 
 sane way, so that one requires justification...

Where did I say anything about enabling? It would need more or less a separate 
repository (dbapi) instance, so it would require such support.

Marius
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [RFC] Depending on active version

2007-01-30 Thread Marius Mauch
On Tue, 30 Jan 2007 13:01:47 -0500
Alec Warner [EMAIL PROTECTED] wrote:

 Marius Mauch wrote:
  Sometimes a package has to depend on a specific version of a slotted 
  package being the active one to build correctly, like in the current 
  tr1 discussion on -dev [1] or with packages that depend on the running 
  kernel.
  
  Currently this isn't really possible, however I while ago I got an idea how 
  to solve this. Keep in mind this is just a rough idea and I'm pretty sure 
  some people can/will point out why it is a stupid idea, but anyway:
  
  The idea is to add a special category (let's call it active for now) that 
  has the following properties:
  - this category doesn't exist in portdir or vdb (= no ebuilds)
  - when portage ($pkgmanager) encounters a active/foo atom in a dependency 
  string it executes some special code (e.g. 
  $PORTDIR/scripts/active-check/foo =active/foo-1) to determine if that 
  atom is satisfied
  
  (and yes, this kinda goes with multi-repo/multi-format support)
  
  Marius
 
 I don't see why how this is any less complicated than just adding more
 functionality to || deps (runtime versus compile time)

Please explain. If a package can only be built if the currently running kernel 
is higher than 2.6.19 for example, how would you possibly express that in 
*DEPEND syntax?

Marius
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [RFC] Depending on active version

2007-01-30 Thread Marius Mauch
On Tue, 30 Jan 2007 18:04:41 +0200
Petteri Räty [EMAIL PROTECTED] wrote:

 Marius Mauch wrote:
  Sometimes a package has to depend on a specific version of a slotted 
  package being the active one to build correctly, like in the current 
  tr1 discussion on -dev [1] or with packages that depend on the running 
  kernel.
  
  Currently this isn't really possible, however I while ago I got an idea how 
  to solve this. Keep in mind this is just a rough idea and I'm pretty sure 
  some people can/will point out why it is a stupid idea, but anyway:
  
  The idea is to add a special category (let's call it active for now) that 
  has the following properties:
  - this category doesn't exist in portdir or vdb (= no ebuilds)
  - when portage ($pkgmanager) encounters a active/foo atom in a dependency 
  string it executes some special code (e.g. 
  $PORTDIR/scripts/active-check/foo =active/foo-1) to determine if that 
  atom is satisfied
  
  (and yes, this kinda goes with multi-repo/multi-format support)
  
  Marius
 
 To work this would need code in the tools like gcc-config. Otherwise
 people could easily break their systems without anything giving them a
 notice.

Not sure I can follow. If we had such a check script for gcc why would it need 
additional code in gcc-config?

Marius

-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [RFC] Mask packages that don't cross-compile

2006-11-22 Thread Marius Mauch
On Wed, 22 Nov 2006 03:10:45 -0500 (EST)
Daniel Barkalow [EMAIL PROTECTED] wrote:

 There are packages (such as perl) which work fine on both x86 and arm, 
 but don't build on x86 cross-compiling for arm. Furthermore, pam-0.78 can 
 be cross-compiled (with a patch available in bug comments), but pam-0.99 
 will require more work to get to cross-compile. It would be useful to be 
 able to have 0.99 use a keyword (or something) such that it is excluded 
 from cross-compiles and these builds get 0.78 instead.
 
 I've got a working patch that extends the KEYWORDS semantics slightly, 
 such that, if KEYWORDS includes -foo, and ACCEPT_KEYWORDS includes foo but 
 not -foo, the package is masked, even if KEYWORDS also includes bar and 
 ACCEPT_KEYWORDS also includes bar.
 
 Then you can have the ACCEPT_KEYWORDS value $ARCH cross and packages 
 with KEYWORDS including -cross will be masked, unless you have -cross 
 in packages.keywords.
 
 This probably horribly mangles the concept of KEYWORDS, but I can't really 
 tell from reading the documentation that it doesn't work this way. (Of 
 course, I know it doesn't from trying things and reading the source, 
 but...) It doesn't specify whether foo or -bar on a package takes 
 precedence if foo and bar both apply to the system. (Of course, it 
 does say that keywords are always ARCHes, which would imply that only one 
 could apply.) I think this all works as expected, however, except that a 
 package having cross means that it is known to automatically work if 
 you're cross-compiling, rather than simply meaning that it is known 
 to work if you're cross-compiling and it would work if you were building 
 natively on your target.

No. -foo is reserved for incremental negation. Maybe that isn't widely used in 
ACCEPT_KEYWORDS, but it has valid uses there and there can't be repurposed. Is 
there a particular reason why you don't use the normal keyword logic 
(KEYWORDS=cross and setting -cross in p.keywords to mask a package), or simply 
a package.mask entry?

Marius
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [PATCH] --config-root command-line option

2006-11-16 Thread Marius Mauch
On Thu, 16 Nov 2006 09:33:05 -0800
Zac Medico [EMAIL PROTECTED] wrote:

 Sure, that's one way to solve that particular problem.  However, I feel
 that command line options are more aesthetically appealing than environment
 variables (maybe it's just me).
 
 Zac

It's just you ;)

Marius
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [PATCH] --config-root command-line option

2006-11-16 Thread Marius Mauch
On Thu, 16 Nov 2006 15:43:48 -0800
Zac Medico [EMAIL PROTECTED] wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 Marius Mauch wrote:
  On Thu, 16 Nov 2006 09:33:05 -0800
  Zac Medico [EMAIL PROTECTED] wrote:
  
  Sure, that's one way to solve that particular problem.  However, I feel
  that command line options are more aesthetically appealing than environment
  variables (maybe it's just me).
 
  Zac
  
  It's just you ;)
  
  Marius
 
 Do you honestly prefer environment variables over command line options?  I
 prefer command line options because they are part of a clearly defined
 interface.  With environment variables, the interface may not be clearly
 defined and it's easy for variables to accidentally leak in, affecting
 internals in unexpected ways.
 
 Zac

I prefer to have one way to do things, and the system for variables worked 
quite well so far. Adding CLI overrides on top of that seems not only redundant 
to me but also potentially confusing (which vars have CLI overrides? where in 
the incremental stack does the override fit in?). Add that to the already 
overloaded option system.
Besides, if you need this feature on a somewhat regular base it seems easier to 
me to just export the var once instead of specifying the option on the command 
line all the time (ok, using DEFAULT_OPTS avoids that, but then you have the 
same issues you listed above).
And last but not least a CLI option is bound to emerge, but this feature can 
also be useful for other tools. Checking an env var in the config class would 
enable it implicitly for all users of portage.py, without it everyone would 
have to basically duplicate the relevant code from emerge.
I'm not generally against using CLI options for passing in arguments, but in 
this case I think it's wrong.

Marius
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] GPG support for mod_mail

2006-10-28 Thread Marius Mauch
On Sat, 28 Oct 2006 15:42:50 +0300
Ali Polatel [EMAIL PROTECTED] wrote:

 Hi everyone,
I wonder if GPG support for mod_mail would be useful.I wrote some
 basic code that works.The user is required to export his public key 
 and put the path to PORTAGE_ELOG_MAILGPG.
Attached is the patch for mod_mail.py.It won't work with armored
 key files because gpg can't directly encrypt messages with ascii
 armored keys , they need to be --dearmor'ed first.
Thoughts?

Well, I don't really see a reason why one would want to encrypt those
mails. The only potentially sensitive information (package
name/version) is already contained in the subject and therefore
wouldn't be encrypted. And gpg support can be a bitch to maintain, so
I'd like to see the benefit before adding that feature.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.
-- 
gentoo-portage-dev@gentoo.org mailing list



[gentoo-portage-dev] RFC: new virtual metadata variable to list combined deps

2006-10-26 Thread Marius Mauch
Ok, I have to admit the subject may be a bit confusing, so let me get
the motivation first:
Often people want to know the dependencies of a package. Not much of a
problem when they know how to use auxget or the portageq metadata
commands. However these all work on the specific DEPEND, RDEPEND and
PDEPEND vars directly, so to get all dependencies of a package one has
to read all three variables. And as in general some entries exist in
both DEPEND and RDEPEND one gets even more info than wanted. At least
this me this manual parsing gets annoying over time.
So in my auxget script[1] I added support for a new virtual metadata
variable '*DEPEND' that basically reads and concatenates the three dep
variables, then eliminates duplicates (of course only on the top level,
complex atoms are treated as a single unit). This allows a quick check
what dependencies a package has ignoring if it's buildtime, runtime or
postbuild (or however you want translate the P in PDEPEND).

So now I was wondering a) if I'm the only one who finds this
feature useful and b) if adding it at the dbapi level (in dbapi.aux_get)
would be considered a good idea, so it could be used by other tools?

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Max parallelization setting

2006-10-10 Thread Marius Mauch
On Tue, 10 Oct 2006 00:04:57 -0700
Brian Harring [EMAIL PROTECTED] wrote:

 I might be daft (likely), but why not just introduce a var indicating 
 max parallelization instead?  Tweak portage to push that setting into 
 MAKEOPTS=${MAKEOPTS+${MAKEOPTS} } -j${PARALLELIZATION}.
 
 Might sound daft, but -j is hardcoded parallelization; if you're 
 trying to run 3 build processes, the original var of -j isn't all
 that useful, nor will the hardcoded -j# var set for 3 package build 
 processes be useful if you're doing a single build.
 
 Depending on number of seperate package build processes possible, the 
 # of build processes allocated per build process needs to vary 
 (essentially).

Seems useful as *alternative* to the low level vars, but shouldn't
replace them (so if the low level vars are set they override the global
setting). So the order in your MAKEOPTS assignment above would have to
be reversed (assuming sequential option parsing).

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [PATCH] use.force and package.use.force (bug #142853)

2006-08-07 Thread Marius Mauch
On Mon, 7 Aug 2006 14:14:11 -0700
Brian Harring [EMAIL PROTECTED] wrote:

 On Mon, Aug 07, 2006 at 01:13:34PM -0700, Zac Medico wrote:
  Brian Harring wrote:
   Semantics of USE=-gtk not working on a package that has gtk
   forced doesn't sound all that nice btw;
  
  Which is why the flag shouldn't be forced unless it's almost 
  certain that the flag shouldn't be disabled.  The gtk flag might
  not ever fall into that category, but something like cxx might
  (nocxx inverted).
 
 Flag shouldn't be forced, period imo.

What about masked flags, should those exist? Because use.force and
use.mask are pretty much the same: accounting for configurations
(=profiles) where support for a flag is not available/not optional
(IIRC one example was USE=hardened on hardened profiles, solar may
correct me here).
So if you want to deny one, you should deny both.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Atom matching behavior

2006-08-01 Thread Marius Mauch
On Mon, 31 Jul 2006 15:48:23 -0700
Brian Harring [EMAIL PROTECTED] wrote:

  but I'd suspect that many
  people share my original assumption and expect it to only match full
  version components
 
 Hear a bit of screaming from it once every 4-6 months; personally, I 
 interpret that as devs know which to use usually- additionally, once 
 the (bluntly) hissy fit from the dev subsides, and they're reminded 
 yes it's annoying, but if you want it changed take it to dev to get 
 consensus folks promptly forget about it.
 
 Either they're silently working around it, or it's not that much of
 an issue (I suspect the latter, but am neutral towards the change).

Well, practically it's not a big issue as in 99% of all cases it works
like people expect.

 Should be discussed on -dev, not here imo; they're the ones affected 
 by it (it's essentially their standard after all).  Changing it?  
 Sure, but it's a required eapi bump; portage chokes on .* now.
 
 I'd also not bump eapi just for one change; there is a boatload of 
 other stuff that's waiting for an apt time to be pushed out together.

Sure going to sned this to -dev, just usually going for input from a
smaller audience first.
Haven't yet thought about details in case of changing it, first gonna
decide what the correct behavior is.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Portage phase hooks patch

2006-07-22 Thread Marius Mauch
On Sat, 22 Jul 2006 16:52:38 -0700
Zac Medico [EMAIL PROTECTED] wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 Mike Kelly wrote:

  In my case, I feel this functionality would be very useful as it
  allows for me to integrate my GLEP 27 implementation into portage
  without portage needing to worry about my implementation specifics,
  which may well change in later versions. I am sure there are other
  practical ways in which these hooks could be used.
  
  Patch at:
  http://soc.gentoo.org/viewcvs.py/*checkout*/glep0027/patches/portage-hooks.patch
  
  See bug# 141215.
  
 
 I like the idea.  My main concern is that, like /etc/portage/bashrc,
 this creates lots of potential for foreign code to interfere with the
 internal workings of the ebuild environment.  At a minimum, I think
 there should be something in the `emerge --info` output that
 indicates whether or not any phase hooks exist.
 
 Traditionally, configurable files that affect portage behavior go
 mostly in /etc/portage.  I see that your intention is to
 make /var/libexec/portage/hooks/ a location for third-party packages
 to install hooks, so it makes sense to keep those separate from hooks
 that the user might install.  I know that portage-utils currently
 installs a post-sync hook in /etc/portage/postsync.d/q-reinitialize,
 so there is some inconsistency there.  We need to develop a
 consistent policy for appropriate locations of files like these.

I still think that those hooks should go into the tree rather than
being installed by packages. That's mainly so they get increased
visibility and gives us (us being Gentoo, not just Portage) more
access and control over them, like we have with eclasses.

This also moves responsibility from hook authors to pkgmanager authors
(the package mamanger has to support the hook format in the tree, not
the hook has to support (all of) the specific package manager hook
formats). I know Mike has taken care of Paludis support already, but
look at a larger scale:
a) n hook authors supporting m package manager interfaces: O(n*m)
b) n hook authors and m package managers supoprting one repository
interface: O(n+m)
Also with a) you have to play what I call the catch up game, e.g. a
new package manager gets out and all hooks have to account for a new
interface (even if it's just a new path).
Of course a) has the drawback that it requires a solid spec of the
interface, but that's something we want anyway.

However, *if* they are going to be installed by packages they should go
either into /etc/portage/foobar or /usr/lib/portage/foobar (like any
3rd party elog or cache modules), foobar being something we'd have to
decide on.
The suggested /var/libexec/portage has several issues:
- /var/libexec is AFAIK not defined by FHS
- executable data doesn't belong in /var unless it's application data
which isn't the case here
- libexec generally sucks (even in /usr), especially as we already
use /usr/lib for what generally goes there (support scripts)

Independent on the location I'd like to see the ability to blacklist
individual hooks for testing, troubleshooting, debugging and probably a
few other reasons. And no, this should not be done with $FEATURES ;)

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [RFC] Naming Conventions

2006-07-22 Thread Marius Mauch
On Sat, 22 Jul 2006 16:25:01 -0700
Zac Medico [EMAIL PROTECTED] wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 Chris White wrote:
  1) Create aliases to the new functions, then at some
  yet-to-be-determined point, kill the aliases and bomb on the scripts
  (this suffers from procrastination).
  
  2) Make an official release with the new function names and no
  aliases, as well as the soon to come docs.  I sort of like this
  method because those with official portage tools can adjust their
  scripts, and simply alter the depend atoms for = (new API
  versions) and = (old versions), effectively forcing/preventing
  upgrades.
 
 I vote for #1 because it's smoother and easier (which is good for me
 especially because I do releases).  The disruptive change proposed in
 #2 seems like it would cause unnecessary problems with no practical
 advantage over #1.  

combination of both.

Phase one: change function names and add aliases
Phase two: make the aliases generate warnings
Phase three: drop the aliases

Maybe combine phases one and two.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] 'emerge' request: make --tree not imply --pretend

2006-06-24 Thread Marius Mauch
On Sat, 24 Jun 2006 18:02:19 -0700
Dan Corson [EMAIL PROTECTED] wrote:

 I am sorry, you are indeed correct.  What I should have requested is:
 a way to have the package list output in non-interactive,
 package-installing (that is, not --pretend) execution.  It may be
 that --pretend is a little over-loaded currently.  (I was confused,
 at any rate, as you can see.) Thanks, --Dan

I think this is stuff for a local wrapper script and not yet another
emerge option that is almost identical to --ask in this case (from a
user POV).
Also don't see much use for this feature, the overhead for calling
emerge twice isn't that big, especially when talking about automated
stuff (lie your case seems to be).

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Manifest verification

2006-06-18 Thread Marius Mauch
On Sun, 18 Jun 2006 04:43:28 -0700 (PDT)
Andrei Slavoiu [EMAIL PROTECTED] wrote:

 Hello, I want to raise a issue with the way that
 package integrity verification is done by portage. For
 an example of such an issue see bug 136742.
 The idea is that portage now checks the Manifest as a
 whole, not allowing a version of the package to be
 emerged if any of the files mentioned by the Manifest
 fail the verification. This makes unstable ebuilds
 wich change often and have a tendency to break from
 time to time prevent the emerge of a stable version of
 the same package.
 Do you think there is any way to fix this in portage?

This isn't exactly new behavior with FEATURES=strict (which is the
default setting).

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] GWN -portage-2.1 announcement.

2006-06-12 Thread Marius Mauch
On Mon, 12 Jun 2006 17:18:44 -0700
Brian [EMAIL PROTECTED] wrote:

 On Mon, 2006-12-06 at 18:58 -0500, Andrew Gaffney wrote:
  Brian wrote:
   Did I understand it wrong or did they get this mixed up in the GWN
   announcement:
   
   /etc/portage/package.unmask/kde, /etc/portage/package.unmask/xorg
   
   Shouldn't it be:
   
   /etc/portage/kde/package.*, /etc/portage/xorg/package.*
  
  You understood it wrong. /etc/portage/package.unmask/foo is correct.
  
  -- 
  Andrew Gaffney  
 
 Thanks.  I'm glad now I didn't have the time to add that to porthole
 yet.  Might have been a quick fix anyway though.

Haven't looked at porthole code, but if you use portage_util.grab* the
fix is trivial (just add a recursive=1 parameter).

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] GWN -portage-2.1 announcement.

2006-06-12 Thread Marius Mauch
On Mon, 12 Jun 2006 18:20:16 -0700
Brian [EMAIL PROTECTED] wrote:

 I also just did a man portage and the man page did not appear to be
 updated for this change.

I guess quite a few docs aren't up to date in 2.1. Anyone wanna work on
that?

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] default postsync

2006-05-04 Thread Marius Mauch

Zac Medico schrieb:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Ned Ludd wrote:

How do you think we should handle it?
Should we install a post_sync in a postinst phase outside of portage's 
handling if and only if not post_sync already exists?

Should we change it to handled a postsync.d by default?
Should we do both? I'm open as heck but would like to start to 
finalize then document it's behavior. I feel it could be one of the 
next untapped really useful features of portage. glsa-checking, news 
handling, search db updating, and stuff etc..



Given the existing support for /etc/portage/bin/post_sync, the user has the 
freedom to do anything they want.  If a program needs to make use of the 
post_sync trigger, it's documentation can simply state that a certain command 
needs the be run and the user can add that command to their post_sync script.  
Is that not easy and flexible enough?


Well, it requires user intervention, pretty much the same argument as 
for any other foo.d change in any package (I don't have a personal 
preference here, though using bashrc for this definitely isn't a good idea).


Marius
--
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Perl, sort, and locale

2006-05-03 Thread Marius Mauch

[EMAIL PROTECTED] schrieb:

I doubt this is the right place to bring this up, but maybe some one
can tell me where to go :-)


Yeah, this is definitely the wrong place, not really sure where the 
appropriate place for this is though. Probably a perl or coreutils 
related list/forum might help you.



I have discovered an inconsistency in how perl and sort handle locale.
Here are two commands you can run in a shell ...

(echo '/'; echo '?') | sort

(echo '/'; echo '?') | perl -e '@x = ; print $_ foreach sort @x'

With LANG=en_US.UTF-8, sort says ? comes before /, perl says the
opposite.  Setting LC_COLLATE=C switches the sort behavior.  I have no
idea what else changes or where else perl and sort disagree, or what
other programs do.

Seems like something is wrong, but I don't know what.  This happens on
other versions of Linux.  It is not a gentoo specific problem.


--
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] glsa implemented as a special set

2006-04-24 Thread Marius Mauch
On Sun, 23 Apr 2006 08:55:58 -0700
Brian [EMAIL PROTECTED] wrote:

 I was thinking that /etc/portage/sets/glsa could be a symlink to set
 list in the current metadata/glsa directory of the portage tree.  That
 file should be relatively easy to auto-generate from the existing
 glsa*.xml files there already.  Perhaps a FEATURES=GLSA_SET would
 generate that file on completion of an emerge --sync  I could also
 then put a GLSA field into porthole's package Summary view as well as
 a GLSA notebook page(s) to display the appropriate glsa?.xml file(s).

Too complicated. First you currently need gentoolkit for glsa.py, and
portage shouldn't depend on gentoolkit. Also you can't store
system-specific files in the tree. And finally using an intermediate
file creates some additional issues (check for IO/FS problems, checking
permissions, etc).
Any reason you need a real file for this instead of just generating the
list on the fly? Just check glsa-check how to do that, it's not much
more than a wrapper for glsa.py.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] glsa implemented as a special set

2006-04-24 Thread Marius Mauch
On Mon, 24 Apr 2006 07:04:13 -0700
Brian [EMAIL PROTECTED] wrote:

 On Mon, 2006-24-04 at 14:20 +0200, Marius Mauch wrote:
  On Sun, 23 Apr 2006 08:55:58 -0700
  Brian [EMAIL PROTECTED] wrote:
  
   I was thinking that /etc/portage/sets/glsa could be a symlink to
   set list in the current metadata/glsa directory of the portage
   tree.  That file should be relatively easy to auto-generate from
   the existing glsa*.xml files there already.  Perhaps a
   FEATURES=GLSA_SET would generate that file on completion of an
   emerge --sync  I could also then put a GLSA field into
   porthole's package Summary view as well as a GLSA notebook
   page(s) to display the appropriate glsa?.xml file(s).
  
  Too complicated. First you currently need gentoolkit for glsa.py,
  and portage shouldn't depend on gentoolkit.
 
 I did not mean portage should and I din't want to depend on gentoolkit
 either.

Not sure I understand your idea then, I was under the impression that
`FEATURE=GLSA_SET emerge --sync` would create that file, is that not
what you meant?

   Also you can't store
  system-specific files in the tree.
 
 Yeah, that was a bit of a thought evolution while I was typing. I
 realized after I hit send.  At first I thought it could be included
 in the sync.  Then thought it's only a duplication of the data already
 there, so why not generate it (save bandwidth), since the data will
 only change at sync time, just do it once then.

Ehm, you couldn't include it in the sync as it's system specific. It
would have to be generated locally, or you have to treat it special
again (only update packages that are installed, don't install new
packages).

   And finally using an intermediate
  file creates some additional issues (check for IO/FS problems,
  checking permissions, etc).
  Any reason you need a real file for this instead of just generating
  the list on the fly?
 
 I thought a smaller stripped down glsa.py module could generate the
 file at completion of the sync.  Then no special code is needed
 internal in porthole beyond checking for set inclusion, atom
 matching, just a glsa_flag=True to ignore members that are not
 already installed.
 
 Once portage was able to handle sets, it would almost automatically be
 able to handle a glsa set as well.  The only difference is not
 installing a set member that is not already installed.  

*Shrug*, generating the list dynamically shouldn't take more than 10 or
20 lines using glsa.py, basically it's

pkg_list = []
glsa_list = [Glsa(x) for x in get_glsa_list(glsadir, glsaconfig)]
for x in glsa_list:
if x.isVulnerable():
pkg_list.extend(x.getMergelist())

plus some error handling. Add some dep_getkey() calls if you don't want
the glsa resolver logic of minimal intrusion (might be problematic
though).
If you need a file interface wrap the list in a StringIO instance.
It's really better to keep interdependencies to a minimum here, and
when portage gets set support it will generate the glsa update list
dynamically anyway, so portage wouldn't benefit from a file at all.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] 2.1 release candidate soon?

2006-04-06 Thread Marius Mauch
On Thu, 06 Apr 2006 19:11:49 -0700
Zac Medico [EMAIL PROTECTED] wrote:

 The manifest code doesn't have very many use cases so I'd expect that
 we would have hit most major problems by now (even with a small
 sample).  Any necessary changes are likely to be small patches.  As
 an alternative, we can cut the 2.1 branch at the point before
 manifest2 was merged (2.1_pre7, essentially).

Releasing 2.1 without manifest2 is a no go, it would significantly
delay the deployment and transition. I'm not requesting to delay 2.1
for another few months, just one more pre release so people get a
chance to test it for one or two weeks.

  The remaining feature I'd like to get into 2.1 is the
  tree-format-check issue, but that could probably be slipped in in
  the rc phase (don't really like that idea, but it's an option).
 
 I don't want to rush the development of new features such as
 manifest2 or the tree-format-check.  We have a 2.1 branch that, in
 it's current state (2.1_pre7-r4, for example), provides significant
 benefits over the 2.0.x branch.  By delaying 2.1's release for the
 addition of _new_ features, we run the risk of the release being
 delayed indefinitely by just one more feature syndrome.
 Personally, I'd rather have shorter release periods so that just one
 more feature syndrome becomes less of an issue.

Ehm, this is not just one more feature, both manifest2 and
the tree-format-check are things to improve forward compability (or for
the latter even enable forward compability at all), so delaying them
will hinder future development, not only for us.
Also this isn't exactly news to you all as I sent my intentions already
a while ago, and last I asked you all agreed with them, so is there any
reason to rush this now?

Marius
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] tree dependency check

2006-03-29 Thread Marius Mauch

Marius Mauch schrieb:
So after manifest2 is in, I'll revive the other issue that IMO is a 
requirement for 2.1: enforcing dependencies needed to use the tree (see 
old threads or glep44 for reasoning). A patch for that is available at 
dev.gentoo.org/~genone/patches/treedeps.diff. Unless somebody objects 
I'll add that somewhere next week.


Ok, from a discussion with Zac and a few others it seems that we should 
add a layer of indirection, so instead of specifying atoms the tree has 
a freeform version identifier, and portage can use it to look up the 
required atoms by using a (remote) mapping file.
This new approach is implemented in 
dev.gentoo.org/~genone/pacthes/format-check.diff (it's basically a 
combination of strategies D and E in d.g.o/~genone/docs/treedeps.txt)


This new patch also has a few other improvements:
- checks overlays too
- at sync time first check the tree format of the remote tree, if we 
can't handle it the sync is aborted
- if no format specification is found warn the user about it (with the 
intention to fail in a future version)


New intended merge date is this weekend or beginning of next week, so if 
you have any objections, make them now.


Marius
--
gentoo-portage-dev@gentoo.org mailing list



[gentoo-portage-dev] tree dependency check

2006-03-25 Thread Marius Mauch
So after manifest2 is in, I'll revive the other issue that IMO is a 
requirement for 2.1: enforcing dependencies needed to use the tree (see 
old threads or glep44 for reasoning). A patch for that is available at 
dev.gentoo.org/~genone/patches/treedeps.diff. Unless somebody objects 
I'll add that somewhere next week.


Marius
--
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [PATCH] Manifest2 reloaded

2006-03-15 Thread Marius Mauch

Marius Mauch schrieb:

The first should be delayed until there is some consensus how the gpg
stuff should work in the future, the others I don't see the use for.
Also I only checked portage.py for changes, so emerge/repoman/... might
still have to be fixed.
Last but not least: I did some basic testing with this and the
important stuff seems to work, but I'm quite sure the code still has a
lot of bugs/issues, and this being a core functionality it needs a
*lot* of testing, so I'd really appreciate if you could all give it a
spin (but do not commit anything to the tree without manually checking
it first).


Does the lack of feedback (only got a reaction from Brian so far) mean 
that noone tried it or that it doesn't have any issues?


Marius
--
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Few things, which imho would make portage better

2006-03-14 Thread Marius Mauch

tvali wrote:

Ok, i send a lot of them, but hopefully they're interesting :)

I did research a bit about adding SQL support to portage -- as much as
i see, mysql is smallest sql server, which could be emerged with
python module.

In beginning, i think that SQL database structure should be created,
which supports basic fields from ebuilds (DESCRIPTION, HOMEPAGE,
SRC_URI, LICENSE, SLOT, KEYWORDS, IUSE, DEPEND, RDEPEND).

Then emerge --createsql command should be set up, which adds data from
/usr/portage/ filetree into this sql database.


You're talking about the cache, take a look at the cache subsystem and 
write a mysql module for it. This will never become a default though (we 
would get killed if portage starts to depend on mysql).



I have some questions now:
* Where i could find basic ebuild specification?


Developer handbook, ebuild(5), bash(1)


* Where i could find basic portage tree datastructure specification,
if there is any?


Not really.


* Does portage have some code documentation for faster learning?
(or should i just read the code?)


code documentation ... what's that? ;)

Marius
--
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Few things, which imho would make portage better

2006-03-14 Thread Marius Mauch

tvali wrote:

I did think about some priorities too, so that it could be perfect for me.

It should be possible to add package with a priority. I will give you an 
use case and explanation how i would use portage.


Heh, make the dep resolver even more complex ;)
Also don't really see a need for such a feature, pretty much no benefit 
with a lot of additional complexity.


Marius
--
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Hello! portage UI

2006-03-14 Thread Marius Mauch

Brian wrote:

On Tue, 2006-14-03 at 13:14 +0200, tvali wrote:


Ok, i was, yes, speaking about kde.

I will check out this Porthole :) I was actually thinking more about c
++, but nothing against python -- i was quite a fan of python when i
first found it.



I believe Kuroo is in C, maybe c++


For the record: Kuroo is known to be conceptually broken (uses the cache 
directly), but the author is looking to fix that in a future version.
Generally writing portage utils in languages other than python is 
difficult as you can't use the primary API directly.


Marius
--
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [rfc] variable naming for marking binaries as QA ignorable

2006-03-05 Thread Marius Mauch
On Sun, 5 Mar 2006 20:46:25 -0500
Mike Frysinger [EMAIL PROTECTED] wrote:

 On Sunday 05 March 2006 19:48, Kevin F. Quinn (Gentoo) wrote:
  Ned Ludd [EMAIL PROTECTED] wrote:
   On Fri, 2006-03-03 at 23:32 -0500, Mike Frysinger wrote:
so we've found some cases where a package installs objects that
either need to be ignored by some of the scanelf checks ...
   
...
   
what this e-mail is about is naming convention ... i'm thinking
that an ebuild sets up a variable with a list of relative paths
to $D of files that should be skipped for various checks ... so
with slmodem, we'd have like: QA_EXEC_STACK=usr/sbin/slmodemd
usr/sbin/slmodem_test
   
if, in the future, we need to add an ignore list for TEXTRELs,
we'd use QA_TEXTRELS=
  
   This becomes tricky when looking at tests across all CHOSTs.
   What holds true for one arch defiantly is not the case for others.
 
  This could be done via the profiles, perhaps - package.qa, something
  like package.mask/use/keywords:
 
 i hate such things ... imo this information should stay in the ebuild
 and nowhere else ...
 
 be trivial to expand the support like:
 QA_TEXTRELS=...   # for all arches
 QA_TEXTRELS_arch=...   # for just one arch
 
 so in the case of slmodem:
 QA_EXEC_STACK=usr/sbin/slmodemd
 in the case of some other package that only has issues on x86:
 QA_EXEC_STACK_x86=some/foo
 
 this thread was about the naming convention :P
 does QA_EXEC_STACK and QA_TEXTRELS work for people ?

Personally I'd call it QA_IGNORE_*, but that's just me.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.
-- 
gentoo-portage-dev@gentoo.org mailing list



[gentoo-portage-dev] [PATCH] Manifest2 reloaded

2006-03-03 Thread Marius Mauch
So while on my way to FOSDEM I decided to do something useful with the
time and wrote a new manifest2 implementation. This has nothing to do
with the original prototype I posted a while ago, it's been written
completely from scratch.
Basically all functionality (creation, parsing, validation) is
encapsulated in the new portage_manifest.Manifest class, including
compability code to read/write old style digests.
The changes to portage.py only change the digest*() functions to use
this new class instead of handling the task themselves (exception:
digestCheckFiles() which apparently was only used internally by other
digest* functions), they should more or less behave like with the old
code. Any new code however should use the Manifest() class directly
however.
While this patch implements the basic functionality some extra stuff
that was in the old code isn't included yet:
- gpg verification
- FEATURES=autoaddcvs
- FEATURES=cvs (probably obsolete anyway)
- emerge --digest / FEATURES=digest (may or may not work)

The first should be delayed until there is some consensus how the gpg
stuff should work in the future, the others I don't see the use for.
Also I only checked portage.py for changes, so emerge/repoman/... might
still have to be fixed.
Last but not least: I did some basic testing with this and the
important stuff seems to work, but I'm quite sure the code still has a
lot of bugs/issues, and this being a core functionality it needs a
*lot* of testing, so I'd really appreciate if you could all give it a
spin (but do not commit anything to the tree without manually checking
it first).

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.
diff -ru --exclude=CVS --exclude=.svn -N pym/portage.py.org pym/portage.py
--- pym/portage.py.org	2006-03-04 02:25:20.957635000 +
+++ pym/portage.py	2006-03-04 03:12:19.545785750 +
@@ -90,6 +90,7 @@
 
 	from portage_data import ostype, lchown, userland, secpass, uid, wheelgid, \
 	 portage_uid, portage_gid
+	from portage_manifest import Manifest
 
 	import portage_util
 	from portage_util import atomic_ofstream, dump_traceback, getconfig, grabdict, \
@@ -2049,181 +2050,67 @@
 			return 0
 	return 1
 
-
-def digestCreate(myfiles,basedir,oldDigest={}):
-	Takes a list of files and the directory they are in and returns the
-	dict of dict[filename][CHECKSUM_KEY] = hash
-	returns None on error.
-	mydigests={}
-	for x in myfiles:
-		print ,x
-		myfile=os.path.normpath(basedir+///+x)
-		if os.path.exists(myfile):
-			if not os.access(myfile, os.R_OK):
-print !!! Given file does not appear to be readable. Does it exist?
-print !!! File:,myfile
-return None
-			mydigests[x] = portage_checksum.perform_multiple_checksums(myfile, hashes=portage_const.MANIFEST1_HASH_FUNCTIONS)
-			mysize   = os.stat(myfile)[stat.ST_SIZE]
-		else:
-			if x in oldDigest:
-# DeepCopy because we might not have a unique reference.
-mydigests[x] = copy.deepcopy(oldDigest[x])
-mysize   = copy.deepcopy(oldDigest[x][size])
-			else:
-print !!! We have a source URI, but no file...
-print !!! File:,myfile
-return None
-
-		if mydigests[x].has_key(size) and (mydigests[x][size] != mysize):
-			raise portage_exception.DigestException, Size mismatch during checksums
-		mydigests[x][size] = copy.deepcopy(mysize)
-	return mydigests
-
-def digestCreateLines(filelist, mydict):
-	mylines = []
-	mydigests = copy.deepcopy(mydict)
-	for myarchive in filelist:
-		mysize = mydigests[myarchive][size]
-		if len(mydigests[myarchive]) == 0:
-			raise portage_exception.DigestException, No generate digest for '%(file)s' % {file:myarchive}
-		for sumName in mydigests[myarchive].keys():
-			if sumName not in portage_checksum.get_valid_checksum_keys():
-continue
-			mysum = mydigests[myarchive][sumName]
-
-			myline  = sumName[:]
-			myline +=  +mysum
-			myline +=  +myarchive
-			myline +=  +str(mysize)
-			mylines.append(myline)
-	return mylines
-
-def digestgen(myarchives,mysettings,overwrite=1,manifestonly=0):
+def digestgen(myarchives,mysettings,db=None,overwrite=1,manifestonly=0):
 	generates digest file if missing.  Assumes all files are available.	If
-	overwrite=0, the digest will only be created if it doesn't already exist.
-
-	# archive files
-	basedir=mysettings[DISTDIR]+/
-	digestfn=mysettings[FILESDIR]+/digest-+mysettings[PF]
-
-	# portage files -- p(ortagefiles)basedir
-	pbasedir=mysettings[O]+/
-	manifestfn=pbasedir+Manifest
-
-	if not manifestonly:
-		if not os.path.isdir(mysettings[FILESDIR]):
-			os.makedirs(mysettings[FILESDIR])
-		mycvstree=cvstree.getentries(pbasedir, recursive=1)
-
-		if (cvs in features) and os.path.exists(pbasedir+/CVS):
-			if not cvstree.isadded(mycvstree,files):
-if autoaddcvs in features:
-	print  Auto-adding files/ dir to CVS...
-	spawn(cd +pbasedir+; cvs 

Re: [gentoo-portage-dev] Switching LOCALE during build?

2006-02-28 Thread Marius Mauch

Patrick Lauer wrote:

Hi all,

this is an idea we had at FOSDEM to make bugreports easier to read:

During build portage should change the locale to en_us so that any error
messages are easy to read even if the user has set it differently. This
shouldn't affect users much and could get rid of the bugreports where it
is hard to understand what went wrong because everything is in french.

Portage changes should be limited to setting LC_ALL in all subshells it
spawns.

Does this cause any unexpected problems? (Programs changing their
language during build would be one possible side-effect). Is it
needed/useful?

Thanks for any feedback,


http://bugs.gentoo.org/show_bug.cgi?id=57973
--
gentoo-portage-dev@gentoo.org mailing list



[gentoo-portage-dev] [Patch] Restriction framework, new search code, metascan2

2006-02-18 Thread Marius Mauch
This is another attempt to prepare the way for including metascan, this
time based on a new restriction framework (which has absolutely nothing
to do with the one in savior/bcportage) included in the attached and
completely undocumented patch (once you get the concept it should be
self-explanatory however).
It basically consists of three class hierarchies:
- dataclasses responsibe for delivering the data to match against
- filterclasses implementing the match algorithms
- boolean connectors to combine multiple restrictions
There is a fourth type called prefilter that can adjust the match
algorithm of filter classes (e.g. a Split() class to break multi-value
metadata fields up for the string matchers).

The filterclasses and connectors implement the same interface for
performing the matches, so the most basic application would be:

for cpv in cpvlist:
if restriction.filter.PartialMatch(data.Metadata(key, db), 
value).match(cpv):
matches.append(cpv)

For a more complex application see the patched emerge or metascan2
(at d.g.o/~genone/scripts).

A word of warning: I've just written all this code in the past few hours, 
it is absolutely unoptimized, incomplete and certainly has bugs (none known 
though).
That also means I'm not interested in comments about the code or algorithm 
yet, as this needs a lot of work and so far I've only done the work to get 
it working, mainly sending this mail so people can play with it a bit.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.
diff -ruN vanilla/bin/emerge portage/bin/emerge
--- vanilla/bin/emerge	2006-02-16 09:38:57.0 +0100
+++ portage/bin/emerge	2006-02-18 12:19:52.0 +0100
@@ -189,7 +189,8 @@
 --tree,
 --update,   --upgradeonly,
 --usepkg,   --usepkgonly,
---verbose,  --version
+--verbose,  --version,
+--newsearch,--regex
 ]
 
 shortmapping={
@@ -693,7 +694,86 @@
 	if x in myparams:
 		myparams.remove(x)
 
+class dumb_formatter(object):
+	def __init__(self):
+		pass
+	
+	def generateReport(self, result):
+		rval = \n
+		for pkg in result.keys():
+			rval += str(result[pkg])+\n\n
+		return rval
+
 # search functionality
+class restriction_search(object):
+	DATA_NAME = 1
+	DATA_DESC = 2
+
+	FILTER_REGEX = 1
+	FILTER_PARTIAL = 2
+	FILTER_EXACT = 3
+
+	def __init__(self, formatter):
+		global myopts
+		if --searchdesc in myopts:
+			self.key = self.DATA_DESC
+		else:
+			self.key = self.DATA_NAME
+		if --regex in myopts:
+			self.type = self.FILTER_REGEX
+		else:
+			self.type = self.FILTER_PARTIAL
+		self.formatter = formatter
+	
+	def execute(self, searchkey):
+		from restrictions import data, filter
+		
+		search_category = False
+		if searchkey.startswith(@):
+			search_category = True
+			searchkey = searchkey[1:]
+		
+		if self.key == self.DATA_DESC:
+			primarydata = data.Metadata(DESCRIPTION, portage.portdb)
+		elif self.key == self.DATA_NAME:
+			primarydata = data.PackageName(with_category=search_category)
+		if self.type == self.FILTER_REGEX:
+			primaryfilter = filter.RegexMatch(primarydata, searchkey)
+			presearch_re = re.compile(searchkey)
+		elif self.type == self.FILTER_PARTIAL:
+			primaryfilter = filter.PartialMatch(primarydata, searchkey, ignoreCase=True)
+			presearch_re = re.compile(.*+re.escape(searchkey)+.*)
+		
+		self.result = {}
+		for package in portage.portdb.cp_all():
+			update_spinner()
+			# This is required to get similar performance as the old code, if we 
+			# don't then we have to do at least one xmatch() call per package
+			# slowing things down extremely
+			if self.key == self.DATA_NAME and not presearch_re.match(package):
+continue
+			masked = False
+			cpv = portage.portdb.xmatch(bestmatch-visible,package)
+			if len(cpv) == 0:
+cpv = portage.best(portage.portdb.xmatch(match-all,package))
+masked = True
+			if len(cpv) == 0:
+print package
+			if primaryfilter.match(cpv):
+update_spinner()
+self.result[package] = {}
+self.result[package][masked] = masked
+self.result[package][cpv_tree] = cpv
+self.result[package][cpv_installed] = portage.best(portage.db[/][vartree].dbapi.match(package))
+metadata_result = portage.portdb.aux_get(cpv, [DESCRIPTION, LICENSE, HOMEPAGE])
+self.result[package][desc] = metadata_result[0]
+self.result[package][license] = metadata_result[1]
+self.result[package][homepage] = metadata_result[2]
+self.result[package][size] = format_size(portage.portdb.getsize(cpv))
+		
+	def output(self):
+		print self.formatter.generateReport(self.result)
+
 class search:
 
 	#
@@ -3059,7 +3139,10 @@
 	if not myfiles:
 		print emerge: no search terms provided.
 	else:
-		searchinstance = search()
+		if --newsearch in myopts:
+			searchinstance = restriction_search(dumb_formatter())
+		else:
+			searchinstance = search()
 		for mysearch in myfiles:
 			try:

[gentoo-portage-dev] Deprecating 'emerge action' syntax

2006-02-16 Thread Marius Mauch
Right now 'emerge action' and 'emerge --action' are both supported. But
as we learned with the rsync case 'emerge action' has potential
namespace conflicts with 'emerge package' I'd propose to deprecate
'emerge action' before we hit another real conflict.
(The alternative would be to deprecate 'emerge package' in favor of a
to-be-written 'emerge install package', but that's even more
problematic)
Technically it's a no-brainer, only potential problem would be user
confusion.
Any objections against this for pre5?

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.
--- bin/emerge.org	2006-02-16 09:38:57.0 +0100
+++ bin/emerge	2006-02-16 12:21:26.0 +0100
@@ -272,7 +272,7 @@
 			sys.exit(1)
 	elif (not myaction) and (x in actions):
 		if x not in [system, world]:
-			#print red(*** Deprecated use of action '+x+')
+			print red(*** Deprecated use of action '%s', use '--%s' instead % (x, x))
 			if x==rsync:
 # emerge rsync
 print


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Deprecating 'emerge action' syntax

2006-02-16 Thread Marius Mauch
On Thu, 16 Feb 2006 13:04:20 +
Ciaran McCreesh [EMAIL PROTECTED] wrote:

 On Thu, 16 Feb 2006 12:31:25 +0100 Marius Mauch [EMAIL PROTECTED]
 wrote:
 | Right now 'emerge action' and 'emerge --action' are both supported.
 | But as we learned with the rsync case 'emerge action' has potential
 | namespace conflicts with 'emerge package' I'd propose to deprecate
 | 'emerge action' before we hit another real conflict.
 | (The alternative would be to deprecate 'emerge package' in favor of
 a | to-be-written 'emerge install package', but that's even more
 | problematic)
 | Technically it's a no-brainer, only potential problem would be user
 | confusion.
 | Any objections against this for pre5?
 
 Mmm, is this taking advantage of Daniel no longer being able to revert
 it? :)

Heh, almost forgot why the code has been commented for ages ;)

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Order of operations: buildpkg

2006-01-24 Thread Marius Mauch
On Tue, 24 Jan 2006 08:19:22 -0500
Mike Frysinger [EMAIL PROTECTED] wrote:

 On Tuesday 24 January 2006 03:43, Francesco Riosa wrote:
  Indeed, could someone shade a light on what happen to /var/db/pkg
  and world file when using ebuild this manner ?
  Could be rephrased as Does it act exactly the same way emerge
  does ?
 
 the 'qmerge' step would take care of updating /var/db/pkg

IIRC the thing it won't do is auto-clenaing the old entries though,
need to run a emerge --clean afterwards.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] confcache integration

2006-01-24 Thread Marius Mauch
On Tue, 24 Jan 2006 04:50:31 -0800
Brian Harring [EMAIL PROTECTED] wrote:

 Yo.
 
 Looking to integrate confcache support into trunk some time in the 
 near future- had users testing it for about 2 months (give or take), 
 so far it's behaved pretty decently.  A few packages eat themselves 
 when ran with --cache (bad autotooling), hence the addition of 
 restrict=confcache being added to the patch.
 
 Either way, patch is attached, looking for any complaints regarding
 it (or integrating confcache)- will be sending an email off to
 gentoo-dev in the next few days to get feedback from the general dev
 populace,, but sending this email to get feedback on the
 implementation in the meantime.

That directory stuff really needs a wrapper function, it's duplicated
way too often.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] making aux_get more usable for vardbapi

2006-01-24 Thread Marius Mauch
On Mon, 23 Jan 2006 14:08:00 -0800
Brian Harring [EMAIL PROTECTED] wrote:

 On Mon, Jan 23, 2006 at 07:44:44PM +0100, Marius Mauch wrote:
  On Mon, 23 Jan 2006 03:47:11 -0800
  Can't follow your thinking here. As said, the code won't corrupt any
  data, at worst it will tell the user that it couldn't extract some
  keys (and even that only if there would be such a stupid case which
  itself has a chance of like 10^-10 or so, or can you name a single
  case?).
 
 Yeah, any extension of your code to pick up EAPI is going to false 
 positive on somewhere around a quarter of my vdb nodes- I use a 
 debug function from my bashrc for prefix testing (holdover from eapi 
 testing days).  Why?  Because the eapi var didn't exist (thus no env 
 assignment), the only possible match is in the middle of a here op-
 
 debug_dump() {
   ...
   echo HERE 2
 EAPI=${EAPI-unset}
 PF=$PF
 CATEGORY=$CATEGORY
 HERE
   ...
 }
 
 Setting EAPI to unset strikes me as corruption here, because now 
 portage will *never* do anything with that vdb node- the eapi differs 
 from what it supports (it's been set to effectively a random value), 
 thus I have to fix the vdb entry myself if I ever want to get rid of 
 it.
 
 Guessing the response on that one is well, you're using the bashrc, 
 and yes, I am- the vdb code should be *robust*, not choking on a
 users debug code.

Not even trying to comment on this one.

  So what package contains filter-env then?
 
 Portage- been in cvs/svn for over a year now :)

Not in trunk or any release (except the alpha snapshot). And I'm not
going to blindly copy unknown code from completely different codebases
and later be blamed for doing it improperly.

   Aside from that, if the code is in debate (as this is), I really 
   don't think it should get slid into svn 2 weeks later effectively 
   unchanged- didn't write that original email just for the hell of
   it :)
  
  As said, I disagree with your assessment of the situation. If you
  can name a single case where the code breaks or filter-env hits
  the tree I might reconsider, but not before.
 
 Well, if you disagreed with the original response, continue the 
 conversation prior to commiting- otherwise we see a commit, then a 
 rebuttal a few hours later.  Not really how things should go for a 
 contested piece of code (at least when the only two to weigh in our 
 flat out opposed on it)- especially if the code's effect is
 nontrivial and it hasn't had any actual peer review (only comment was
 on your algo).

Interesting, you now count for two people? Or who is this second person
you're talking of? I still think you're making more out of this than it
really is. Also there really isn't a point in discussing a difference in
opinion IMO, such attempts lead nowhere. I considered your comment, but
couldn't come up with a reason why a theoretical issue should hold this
up.

 Issues with the code.

Now we're getting somewhere.

 1) Scans for metadata that didn't exist at the time of the ebuild's 
 installation, only possible match is unintended.  Not good.

Not really sure what that's supposed to mean, assuming you mean that
nodes will lack SRC_URI/HOMEPAGE/... from env.bz2 is about as likely as
here doc statements containing them (and indicates a broken package
anyway).

 2) your code is breaking upon *all* newlines, regardless if it's 
 in the middle of a variable assignment.  Yes, bash does use $'' for 
 assignment, but you're relying (hoping really) that the variable has 
 been filtered by python portage and the ebuild defined var is
 stomped. This is a bad idea- description comes to mind.  Your code
 seems to be aware of this possibility also, since it's doing rather
 loose quote filtering.

Have to think about this.

 3) Stop using regex all over the place.  split/join is a helluva lot 
 faster, and collapses that nasty regex that attempts to remove 
 whitespace down to two lines (a one time translate, and the
 split/join to kill whitespace).

Ok (cosmetics).

 4) Code specifically tries to find, and remove variable chars- 
 this is *really* the wrong thing to do.  If you're trying to work 
 around catching a var reference, either your parsing screwed up, or 
 you're mangling a DESCRIPTION value that you shouldn't be hosing.

define variable chars, otherwise doesn't make sense.

 5) hardcoded vdb.
 6) Non root aware VDB.

Ok (trivial)

 7) Only is aware of environment.bz2- older portage versions differed 
 in this afaik (I *do* know PORT_ENV_FILE overloading for ebd I had to 
 address this case already, so it's out there).

Well, it's present as long as I can remember, but an sanity check
wouldn't hurt there.

 8) perms on the new files?

Ok.

 9) general code comments; 'if s != ', use 'if s:'.  Catch the 
 exceptions for check mode, don't let vdbkeyhandler nuke world file 
 checking due to a potential exception.  Right now, your code will TB 
 if ran by non-root for a check run rather then stating got to be 
 root.

a) hate theses

Re: [gentoo-portage-dev] making aux_get more usable for vardbapi

2006-01-23 Thread Marius Mauch
On Wed, 11 Jan 2006 12:39:03 -0800
Brian Harring [EMAIL PROTECTED] wrote:

 Regex you've got there allows for pulling the wrong text- recall, ebd 
 originally was doing grep based filtering (regex).  Had to rewrite 
 that in a major hurry since bash syntax (specifically here ops)
 forces you to track state/constructs rather then just a regex...

Not really an issue in this case. First the code bails out if more than
one match is found, so unless the metadata assignment is NOT found by
it we don't get the wrong info. Also a mismatch in this special is so
extremely unlikely that honestly I don't really care about it,
especially as this is a one time conversion (might be different if I'd
have added the on the fly extraction). All together makes this pretty
much a hypothetical issue with no practical relevance.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Plausible idea for GLEP 19?

2006-01-22 Thread Marius Mauch
On Sun, 22 Jan 2006 10:25:37 -0600
Mikey [EMAIL PROTECTED] wrote:

 On Saturday 21 January 2006 22:39, Marius Mauch wrote:
 
  Check the archives for gentoo-dev and gentoo-server, several
  implementation plans have been presented in the not-so-distant past.
  However everyone seems to have a slightly different goal he wants to
  achieve, so maybe the best would be for people to make their goals
  very clear.
 
 I have checked the archives of gentoo-dev, gentoo-server, and have 
 researched everything I can find about glep 19.  I have come to the 
 conclusion that those projects are the /dev/null of gentoo
 projects. Post a request somewhere - hey, check out glep 19, wink
 wink.

Well, posting YAIP (yet another implementation plan) won't really help
either.

 Let me make my goal clear.  I would like to see some simple features
 added that does not require disruption of current usage, future
 plans, or require massive changes.  I am not interested in reviving
 dead projects or loft goals.

Don't see the goal here, just the constraints. Are you after a
non-moving tree, a tree with just security fixes, visibility filters in
portage, a new `emerge moo` graphics, ... ?

  No point, would rather add a RSYNC_OPTS var finally instead, which
  gives the same functionality (and much more).
 
 If that would work, great.  Not sure how rsync handles
 ordering/precedence of conflicting options, now or in the future.
 Also not sure how the environment may or may not be manipulated in
 the future by emerge itself. Right now the options are hard-coded
 into emerge, the simplest place to change it in my mind is right
 there where it happens...

Yeah, RSYNC_OPTIONS would remove all that hardcoded stuff and put it in
make.globals instead.

   2) Implement a new revision numbering scheme for ebuilds, -sX.
   Similar to -rX, but for glsa updates only.  It could be an
   abbreviation for sticky, security, or stable, whatever you want.
  
   For example if I am currently running mysql-4.0.25, the only
   candidate an emerge -u would consider would be mysql-4.0.25-s1,
   mysql-4.0.25-s2, etc In other words, emerge considers only
   -sX in its upgrade calculations, instead of -rX, and only
   considers the same version.
 
  No way. No visible benefit (you know about glep 14 / glsacheck,
  right?) and tons of issues (redundant ebuilds, ordering
  screwups, ...)
 
 Yes, I am aware of glsacheck.  How long has it been in testing?
 Every time I try it I get inconsistent results.  Something about
 WARNING: This tool is completely new and not very tested, so it
 should not be used on production systems.  kind of makes me hesitate
 to use it.

I removed the warning in gentoolkit-2.1 (wanted to do that for quite
some, just didn't get around to do it). What kind of inconsistent
results are you speaking of?

 As far as redundant ebuilds and ordering screwups.  If you change
 line 3173 of portage.py to:
 
   if len(myrev) and myrev[0] in [r,s]:
 
 Everything works quite well actually.  The s is sorted after the r,
 so -s7 would install after -r6 or instead of -r7.  It is actually a
 much simpler solution than glsa, could be introduced immediately to
 the portage tree, and use of it could be optional.  People could use
 them in their own overlays, backport their own security fixes.

Ok, so you just want to have another letter that does exactly the same
as -rX, just with a semantic difference.
Still has issues as this allows for multiple equal versions to exist
(as -rX == -sX). And no, it could not be used immideately in the tree
as unaware portage versions would fail with interesting errors (see
glep 45 for background info), same reason the versioning extensions in
2.1 can't be used yet (even if 2.1 would be stable). And I'm quite sure
that if this would be used in the tree we'd hit versioning screwups
sooner or later (-sX - -rX+1 - -rX+2 - -sX+1).

  No chance you get people to agree on something that will bloat the
  tree without any real benefit. Mind that we already revbump
  packages for security updates.
 
 Packages are not only revbumped for security updates.  I have not
 researched it much, but I would be willing to be the vast majority of
 revbumps are _rarely_ for security updates.  Most of the time glsas
 suggest to bump up to the next version of the package, not
 revision...  There is also no way to distinguish between a revbump
 that alters the package via a revbump that is only because a glibc
 has just been marked stable for another architecture, for example.

I didn't claim that all/the majority of revbumps are security related.
And there is a way to distinguish the different kind of revbumps: read
the changelog.

  No, this includes way too many changes to core functions (version
  comparison, resolver) with no visible benefit (from my POV). In
  case you haven't done so already take a good look at glep 14 and
  glsa-check (which implements the least-invasive update algorithm
  you seem to be after).
 
 I am happy

Re: [gentoo-portage-dev] Plausible idea for GLEP 19?

2006-01-22 Thread Marius Mauch
On Sun, 22 Jan 2006 13:02:37 -0600
Mikey [EMAIL PROTECTED] wrote:

 On Sunday 22 January 2006 13:02, Marius Mauch wrote:
 
  Well, posting YAIP (yet another implementation plan) won't really
  help either.
 
 Correct, plans never seem to go anywhere in regards to this...
 
  Don't see the goal here, just the constraints. Are you after a
  non-moving tree, a tree with just security fixes, visibility
  filters in portage, a new `emerge moo` graphics, ... ?
 
 The existing tree with additional functionality.  Implemented via a
 new revision type.
 
No point, would rather add a RSYNC_OPTS var finally instead,
which gives the same functionality (and much more).
 
  Yeah, RSYNC_OPTIONS would remove all that hardcoded stuff and put
  it in make.globals instead.
 
 From make.globals:
 
 #*
 #**  DO NOT EDIT THIS FILE  **
 # ***
 #  CHANGES TO make.conf *OVERRIDE* THIS FILE 
 # ***
 # ** Incremental Variables Accumulate Across Files **
 # **  USE, CONFIG_*, and FEATURES are incremental  **
 # ***

Your point being?

  I removed the warning in gentoolkit-2.1 (wanted to do that for quite
  some, just didn't get around to do it). What kind of inconsistent
  results are you speaking of?
 
 http://www.gentoo.org/proj/en/portage/glsa-integration.xml, section
 on known problems.

Which except for the SLOT issues are all fixed as far as I'm concernced.

 The plan is nice.  It does not, however, address the needs of some
 users to have a STABLE system as well.  Some users can't willy nilly
 upgrade to the next version of a package because they might have
 requirements to stay at the same version.  Through something as
 simple as adding a new revision specifier, a framework can be in
 place to backport security fixes or ONLY apply revisions that are
 security related...

I get see your point here.
`glsa-check -p affected` will shouw you exactly the *minimal* updates
necessary to fix all known vulnerabilities. Not that hard to isolate
the version bumps from the revbumps, is it?
Also your idea has another major problem that renders it more or less
useless:
- you have foo-1.0 installed
- foo-1.0-r1 with major changes is added to the tree
- a security update based on -r1 is added to the tree as foo-1.0-s2
Now what?
a) install -s2 including the changes from -r1
b) add another ebuild -s1 that is foo-1.0 with the security update but
without the -r1 updates

With a) there is no point at all in using the -sX system, with b) you
get into the equality issues (here -r1 and -s1 are not
functionally equal).

  Well, it only needs a way to feed a set of nodes into the dep
  resolver. But that in turn is quite a task as the dep resolver code
  is nasty.
 
 I have looked at the dep resolver and don't want to go near it with a
 10 foot pole.  I only want to do the functional equivalent of
 filtering out anything but *-sX$ (pseudo regex) during the final
 doemerge or when displaying in case of --pretend.

That's not really what you want.
-s updates might (will) be overlaid with version or revision bumps from
time to time, for this to be of any use it has to happen at the
resolver level (visiblity filter).

  What functionality does it actually add? The changes you described
  so far just allow multiple letters as revision prefixes. The main
  thing of your proposal was probably to limit updates to -s updates,
  and that's a tricky goal.
 
 It enables ebuild maintainers to backport security patches.  It
 allows them to fix security problems that are not glsa alerts.  It
 does not require the use of a new tool or integration of glsa into
 portage.  It allows users to distinguish between revbumps of a
 trivial nature and revbumps of a security nature.

Everything but the last one is already possible. For the last one you
currently have to read the changelog.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Re: Plausible idea for GLEP 19?

2006-01-22 Thread Marius Mauch
On Sun, 22 Jan 2006 14:55:48 -0600
Mikey [EMAIL PROTECTED] wrote:

 Users are discouraged from changing make.globals.  It would be better
 to implement in make.conf or as a command line option to emerge
 itself.  It is a configuration item for a user preference, not a
 (global) portage preference...

Ehm, did you read that comment you posted? make.conf overrides
make.globals, so you don't have a point here.

  Which except for the SLOT issues are all fixed as far as I'm
  concernced.
 
 That is not a small issue.

portage itself has similar issues with SLOTs

  That's not really what you want.
  -s updates might (will) be overlaid with version or revision bumps
  from time to time, for this to be of any use it has to happen at the
  resolver level (visiblity filter).
 
 Normal emerges would take -s2 over -r1 or -s1.  The change is
 transparent when not in glsa-only mode.

You didn't understand what I said. If you just play around with the
output info you'll miss updates.

  Everything but the last one is already possible. For the last one
  you currently have to read the changelog.
 
 It does not allow for backporting, it is not safe to use in an
 automated environment, it does not work in a production environment,
 and it does not scale well.

Sorry? What prevents anyone from doing backports? It's as safe as your
idea for both automatic and manual (unless you assume that all security
updates will only ever be a -sX bump), and the last two are just random
comments that don't mean anything.

Anyway, showed you why it won't work and why it won't get implemented,
so rather pointless to continue this.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Plausible idea for GLEP 19?

2006-01-21 Thread Marius Mauch

Mikey wrote:
I have been emailing the published addresses for GLEP 19 for 2 months now 
with no success.  I am very interested in any ideas or projects that might 
help gentoo be more server friendly, in an enterprise environment, for 
lack of a better term.  

I have an idea towards stabilizing portage for production environments, 
but I am not knowledgeable enough about portage to know if it would even be 
plausible.  If this is the wrong place to ask this, please feel free to 
point me in a better direction.


Check the archives for gentoo-dev and gentoo-server, several 
implementation plans have been presented in the not-so-distant past. 
However everyone seems to have a slightly different goal he wants to 
achieve, so maybe the best would be for people to make their goals very 
clear.


Basically, add a new value for FEATURES.  For lack of a better name, call 
it sticky.


FEATURES=sticky

If this flag is present in make.conf:

1) emerge --sync does only updates, not deletes (don't ditch old ebuilds).


No point, would rather add a RSYNC_OPTS var finally instead, which gives 
the same functionality (and much more).


2) Implement a new revision numbering scheme for ebuilds, -sX.  Similar to 
-rX, but for glsa updates only.  It could be an abbreviation for sticky, 
security, or stable, whatever you want.  

For example if I am currently running mysql-4.0.25, the only candidate an 
emerge -u would consider would be mysql-4.0.25-s1, mysql-4.0.25-s2, etc  
In other words, emerge considers only -sX in its upgrade calculations, 
instead of -rX, and only considers the same version.


No way. No visible benefit (you know about glep 14 / glsacheck, right?) 
and tons of issues (redundant ebuilds, ordering screwups, ...)


3) Package maintainers could create duplicate ebuilds for security-only 
related revisions to packages, some other team could maintain them, this be 
somehow automated, or this could be left up to the users to maintain 
through their own overlays.  My idea is fuzzy here...


No chance you get people to agree on something that will bloat the tree 
without any real benefit. Mind that we already revbump packages for 
security updates.



Plausible?


No, this includes way too many changes to core functions (version 
comparison, resolver) with no visible benefit (from my POV). In case you 
haven't done so already take a good look at glep 14 and glsa-check 
(which implements the least-invasive update algorithm you seem to be after).


Marius
--
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Fwd: /etc/portage/profile/{pmask,arch.list, categories}

2006-01-10 Thread Marius Mauch
On Mon, 9 Jan 2006 10:21:49 -0500
Alec Warner [EMAIL PROTECTED] wrote:

 
 My Apologies, KMail died, and when I restarted it it had the mail but
 stripped the attachment ;)
 --  Forwarded Message  --
 
 Subject: /etc/portage/profile/{pmask,arch.list, categories}
 Date: Monday 09 January 2006 09:59
 From: Alec Warner [EMAIL PROTECTED]
 To: gentoo-portage-dev@gentoo.org
 
 Apparently these weren't working ( not being read in ) due to a lack
 of /etc/portage/profile being in Locations in config in portage.py
 
 I split a patch for hte issue by simply adding the extra location and
 fixing the odd grabfile_package calls.  It seems grabfile package was
 stripping the
 - so it wouldn't nuke existing entries in the pmask file.
 
 Comments welcome as usual.

The profile stacking doesn't work that way ...
$PORTDIR/profiles/ and /etc/portage aren't part of the profile
stack, /etc/portage/profile is and actually works as intended. So this
patch would actually create a bug instead of fixing one
And if you think those dirs should be part of the stacking the patch is
still wrong as it has to be done the other way round.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


[gentoo-portage-dev] making aux_get more usable for vardbapi

2006-01-10 Thread Marius Mauch
Currently vardbapi.aux_get only works for a subset of all auxdbkeys, as
some like KEYWORDS or DESCRIPTIOn aren't stored in vdb directly.
They are however stored in environment.bz2, but not accessible
there.
This is unintuitive and limits tools like equery or my own auxget
and metascan tools in their usability.

There are two solutions to this problem:
a) enhance vardbapi.aux_get so it can use environment.bz2
b) store more keys in vdb

Now there is a tradeoff to made: a) doesn't need space but is slow
while b) is fast but needs space, both in non-trivial amounts (runtime
increased from 1s to 9s for a metascan -i run and from 0.5s to 0.8s
for auxget -i, haven't actually checked the size increase, expect
somewhere between 1 and 10 megabytes on a typical install).

I'm attaching a patch that implements both (each in it's own hunk) as
well as a new emaint option to create the missing entries offline.

A not so obvious issue with a) is that due to the recent
storage optimizations (empty entries not being stored) it's worse than
I originally expected, as any entry missing a file will be looked up in
env.bz2 instead. Only way to avoid that would be to add special casing
in aux_get which I really dislike.

Opinions?

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.
--- /home/gentoo/svn/portage/main/trunk/bin/ebuild.sh	2006-01-08 04:30:16.0 +0100
+++ /usr/lib/portage/bin/ebuild.sh	2006-01-10 18:27:15.0 +0100
@@ -914,7 +914,8 @@
 	for f in ASFLAGS CATEGORY CBUILD CC CFLAGS CHOST CTARGET CXX \
 		CXXFLAGS DEPEND EXTRA_ECONF EXTRA_EINSTALL EXTRA_MAKE \
 		FEATURES INHERITED IUSE LDFLAGS LIBCFLAGS LIBCXXFLAGS \
-		LICENSE PDEPEND PF PKGUSE PROVIDE RDEPEND RESTRICT SLOT; do
+		LICENSE PDEPEND PF PKGUSE PROVIDE RDEPEND RESTRICT SLOT \
+		KEYWORDS HOMEPAGE SRC_URI DESCRIPTION; do
 		[ -n ${!f} ]  echo $(echo ${!f} | tr '\n,\r,\t' ' , , ' | sed s/'  \+'/' '/g)  ${f}
 	done
 	echo ${USE}		 USE
--- /home/gentoo/svn/portage/main/trunk/pym/portage.py	2006-01-08 04:30:11.0 +0100
+++ /usr/lib/portage/pym/portage.py	2006-01-10 18:28:00.0 +0100
@@ -4501,17 +4477,32 @@
 	def aux_get(self, mycpv, wants):
 		global auxdbkeys
 		results = []
+		envlines = None
 		for x in wants:
-			myfn = self.root+VDB_PATH+/+str(mycpv)+/+str(x)
-			if os.access(myfn,os.R_OK):
-myf = open(myfn, r)
+			mydir = self.root+VDB_PATH+/+str(mycpv)+/
+			if os.access(mydir+str(x),os.R_OK):
+myf = open(mydir+str(x), r)
 myd = myf.read()
 myf.close()
-myd = re.sub([\n\r\t]+, ,myd)
-myd = re.sub( +, ,myd)
-myd = string.strip(myd)
+			# Fallback to searching environment.bz2 if no key-specific file exists
+			elif os.access(mydir+environment.bz2, os.R_OK):
+if not envlines:
+	env = os.popen(bzip2 -dcq +mydir+environment.bz2, r)
+	envlines = env.read().split(\n)
+	env.close()
+myd = [l for l in envlines if l.strip(\\').startswith(x+=)]
+if len(myd)  1:
+	raise portage_exception.CorruptionError(multiple matches for +x+ found in +mydir+environment.bz2)
+elif len(myd) == 0:
+	myd = 
+else:
+	myd = myd[0].split(=,1)[1]
+	myd = myd.lstrip($).strip(\'\)
+	myd = re.sub(([nrt])+,  , myd)
 			else:
 myd = 
+			myd = re.sub([\n\r\t]+, ,myd)
+			myd = re.sub( +, ,myd)
+			myd = string.strip(myd)
 			results.append(myd)
 		if EAPI in wants:
 			idx = wants.index(EAPI)
--- /home/gentoo/svn/portage/main/trunk/bin/emaint	2006-01-08 04:30:16.0 +0100
+++ /usr/lib/portage/bin/emaint	2006-01-10 19:29:20.0 +0100
@@ -4,7 +4,7 @@
 from copy import copy
 from optparse import OptionParser, OptionValueError
 
-
+import re
 
 import os, portage, portage_const
 class WorldHandler(object):
@@ -44,8 +44,65 @@
 			errors.append(portage_const.WORLD_FILE +  could not be opened for writing)
 		return errors
 
+class VdbKeyHandler(object):
+	def name():
+		return vdbkeys
+	name = staticmethod(name)
+
+	def __init__(self):
+		self.list = portage.db[/][vartree].dbapi.cpv_all()
+		self.missing = []
+		self.keys = [HOMEPAGE, SRC_URI, KEYWORDS, DESCRIPTION]
+		
+		for p in self.list:
+			mydir = /var/db/pkg/+p
+			ismissing = True
+			for k in self.keys:
+if os.path.exists(mydir+/+k):
+	ismissing = False
+	break
+			if ismissing:
+self.missing.append(p)
+		
+	def check(self):
+		return [%s has missing keys % x for x in self.missing]
+	
+	def fix(self):
+	
+		errors = []
+	
+		for p in self.missing:
+			mydir = /var/db/pkg/+p
+			if not os.access(mydir+/environment.bz2, os.R_OK):
+errors.append(Can't access %s % (mydir+/environment.bz2))
+			elif not os.access(mydir, os.W_OK):
+errors.append(Can't create files in %s % mydir)
+			else:
+env = os.popen(bzip2 -dcq +mydir+/environment.bz2, r)
+envlines = env.read().split(\n)
+env.close()
+for k in self.keys:
+	s = [l for l in 

Re: [gentoo-portage-dev] Package moves in overlays

2006-01-09 Thread Marius Mauch

Mike Auty wrote:

Yeah,
I agree that the updates should only affect that individual tree. 
Whilst I can imagine cases where a move in the main tree should affect 
overlay trees (such as recategorizing some net-misc ebuilds into a 
net-proxy category), that could at least be emulated with a per-tree 
move directive fairly easily.
It sounds as though the answer is that there currently no way to do 
these kinds of moves, so what's the next step?  Should I try and hack up 
a version to deal with updates on a per tree basis?  Is it actively 
being developed and I'm just unaware of it?  What can I do to help get 
this feature into a future version of portage?  It would be very useful 
to have, because at the moment network administrator's can't quite use 
overlays as a software management mechanism...

Thanks for you time,


As currently vdb/pkgdir don't track the repo where a package comes from
it's not possible to do it in a proper way (those are affected by moves, 
not the repositories themselves).


Marius
--
gentoo-portage-dev@gentoo.org mailing list



[gentoo-portage-dev] [RFC] making the tree depend on portage

2005-12-19 Thread Marius Mauch
Ok, the subject might be confusing, so let me explain this a bit:

Whenever we want/need to make structural changes to the tree that are
going to break backwards compability we have a serious problem (see
GLEP 44 in case you don't know about it). To reduce the impact of that
problem I've got the idea to make the tree itself (so not any
particular ebuild or profile) DEPEND on a minimal portage version,
which the users would be forced to upgrade to (maybe with an override)
before they can do anything else (with the exception of --sync).
Manifest2 is one example for such a situation, another one is the
request to not create manifest entries for ChangeLog and metadata.xml
anymore (needs =2.0.51.20 on user side).
Don't really like this idea myself, but somthing needs to be done to at
least reduce the problem, having to wait years for old portage versions
to (almost) vanish can't be a permanent solution.

Also not talking about implementation details yet, just after comments
about the general idea of forced portage updates.

And just in case anybody wonders: this cannot be fixed with EAPI or
adding a portage dep on packages as those only take effect when the
ebuild is already parsed while the mentioned problems occur much
earlier.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] chunking up portage

2005-12-16 Thread Marius Mauch

Brian Harring wrote:

On Thu, Dec 15, 2005 at 01:54:06PM +0200, Marius Mauch wrote:


Brian Harring wrote:

So... thoughts?  I'm not much for making portage depend on tarsync 
just for emerge-webrsync improvements, would rather chunk the bugger 
out.


How about runtime detection?


runtime detection is questionable from my standpoint, since while 
coding for it is good, without hard dep pulling it in the only folks 
who will ever have a faster emerge-webrsync are those who happen to 
know the hidden trick to merge tarsync.


Well, how do you plan to inform users of the splitted package for 
webrsync? Could just do the same for tarsync instead, or for example if 
no tarsync is found print a nice little message about falling back to 
slow default, emerge tarsync to avoid.


Marius
--
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] chunking up portage

2005-12-15 Thread Marius Mauch

Brian Harring wrote:
So... thoughts?  I'm not much for making portage depend on tarsync 
just for emerge-webrsync improvements, would rather chunk the bugger 
out.


How about runtime detection?

Marius
--
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] ACCEPT_KEYWORDS depreciation

2005-12-15 Thread Marius Mauch

Carsten Lohrke wrote:

On Thursday 15 December 2005 13:05, Marius Mauch wrote:


package.keywords isn't better or worse than ACCEPT_KEYWORDS, it's just
different in its behavior.



I disagree. ACCEPT_KEYWORDS is problematic for (new) users, because of its
behaviour.


problematic != worse

*shrug*, if there is a strong desire for it I'd suggest to check for 
existance of package.keywords and print a warning only if it doesn't 
exist and ACCEPT_KEYWORDS=~arch is used (as people who have a 
package.keywords file/dir clearly know the difference, right?)


Marius
--
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-12-07 Thread Marius Mauch
On Wed, 7 Dec 2005 21:33:00 +0900
Jason Stubbs [EMAIL PROTECTED] wrote:

  It isn't about expectations.
 
 Ok, I misunderstood your previous posts on this topic then.
 
  I just think it's bad engineering to use the same version prefix for
  two rather different codebases. ... After all, wasn't engineering
  the reason why we're going to increase the minor?
 
 I don't understand where the conflict comes in between the two.
 Internally, the old 2.1 has been known as HEAD, trunk and now
 2.1-experimental. Externally, it's been known as 2.1.0_alpha20050718.
 The set of new features available in 2.1.0_alpha20050718 are pretty
 much all available in current trunk as far as I know... You'll need
 to explain the issue in a little bit more detail.

I guess it's a mindset thing. You say that HEAD and trunk are
names, for me they are just locations that tell me _where_ I can find
them, not _what_ they are. When savior goes into trunk at some
point in the future, where will trunk go? branches/2.1-production?
Then we have two 2.1 branches which codebases aren't really related,
but for someone who hasn't worked on both it will look like one is
based on the other.
This is also the reason why I didn't just put the savior code into a
branches/3.0 but gave it a symbolic name instead. *If* trunk and the
existing 2.1 branch would have similar symbolic names I probably
wouldn't have objected in the first place, but they don't (and
retroactively adding one for 2.1 just for this seems like a bad idea).
I guess one can say it's about identification. Yes, it plays with the
usability aspects of version numbers, but in this case I prefer
technical aspects over usability aspects.

I hope this makes it a bit more understandable.

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-12-06 Thread Marius Mauch
On Tue, 6 Dec 2005 23:19:38 +0900
Jason Stubbs [EMAIL PROTECTED] wrote:

 So I'm going to make a decision and offer until Friday (Saturday in
 my time) for opposers to solidify and state any opposition. If
 there's no solid opposition, Saturday I will put current trunk into
 ~arch as 2.1_beta20051210. I will also post on the 2.0.53 bug that
 fixes are available for the ldconfig bug and the tee bug stating that
 we'd like to also add trunk's cache subsystem to 2.0.54 and that
 dependening on the next council meeting(?) the SHA1 enabling as well.
 Doing it this way will make the ~arch users happy straight away. If
 we look at it as our responsibility to get fixes and new
 functionality into ~arch then our jobs done and we can get back to
 business.

Well, I've already stated several times that IMO using a 2.1 branch is
wrong (use 2.2 instead), but if I'm overvoted, so it shall be.
As for the cache rewrite in 2.0.54, I don't really prefer one way or the
other, from an engineering POV it is 2.2 material, but if it is a major
improvement and well tested it can also be in .54 (just in case my
previous mail was misunderstood).

 As for stable users? If arch teams are willing to selectively choose
 what fixes they want backported to stable (when they're not prepared
 to move the ~arch version into stable) things will go much smoother.
 Of course, it would still be our responsibility to get those things
 backported and assert some confidence that it is working. However,
 once the requested fixes are backported all that needs to be done is
 put out the patched stable version with ~arch keywords and then leave
 it up to the arch teams again. Except for the slight extra burden on
 (which I believe many would actually find to be a blessing), it
 should be a win-win situation.

Just in case you forgot and also for general reference, when I asked
the arch teams about the portage keywording policy a few months ago
(wether we should stable even without testing on all archs or to
delegate that to arch teams) everyone seemed to be happy with the old
policy, at least nobody voted for a change. As portage doesn't really
have any arch specific code and a rather short dep list IMO it also
doesn't yield any real benefit other than more people testing it (which
is of course always a good thing).

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-12-06 Thread Marius Mauch
On Wed, 7 Dec 2005 08:41:27 +0900
Jason Stubbs [EMAIL PROTECTED] wrote:

 On Wednesday 07 December 2005 01:01, Marius Mauch wrote:
  On Tue, 6 Dec 2005 23:19:38 +0900
  Jason Stubbs [EMAIL PROTECTED] wrote:
   If there's no solid opposition, Saturday I will put current trunk
   into ~arch as 2.1_beta20051210.
 
  Well, I've already stated several times that IMO using a 2.1 branch
  is wrong (use 2.2 instead), but if I'm overvoted, so it shall be.
 
 As Brian stated, 2.2 being a version higher than 2.1 will have all
 the same expectations placed on it. From what I can see, 1% of users
 know anything about 2.1 so 99% would be wondering why there was a
 jump from 2.0 to 2.2. Do you have anything against 2.1 other than
 fearing that people will expect more from it than it will turn out to
 be?

It isn't about expectations.
I just think it's bad engineering to use the same version prefix for
two rather different codebases. As for your concerns about confused
users, I don't share them. 90% will simply ignore it, 5% will do a
quick search and find something like my 2.1 thread in the forums, and
the remaining 5% either know already the situation or will draw their
own conclusions.
After all, wasn't engineering the reason why we're going to increase
the minor?

 Really, the bottom line is that regardless of what the response was
 when you asked about portage keywording, if all the arch teams had
 confidence in what we thought 2.0.53 would have been stable a long
 time ago. On the surface the only benefit is extra testing (which has
 already payed off) but it also allows others to take an active hand
 in the quality of portage as well as strengthens the communication
 channels. It's these auxillary benefits as well as the benefit of
 being able to focus on trunk more (which will yield faster rollout of
 features) that make me think it is the best way to go.

Ok, but it still doesn't really have anything to do with arch teams,
just general QA. Also I didn't mean to criticize you, just stating
that this option exists.

 I can't tell if you followed what I said in my last email so I'll
 reiterate. Trunk will go into ~arch on Saturday. 2.0.54 will go out
 (also in ~arch) two weeks after that with the two fixes and include
 the cache rewrite based on the opinion of a broad range of users
 (rather than just the noise makers). SHA1 will of course also go in
 based on how it is voted.

Ehm, what's the point of having .54 in ~arch after trunk is in
~arch? You won't get much testing that way as ~arch users would
already use trunk and stable users likely won't know about .54 ...
(typical visibility problem)

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Latest version vs specific version

2005-12-02 Thread Marius Mauch

Andrea Carpani wrote:
Hi all. Here's my problem. 


I'm using a lot binary packages of in portage and custom created
ebuilds. I have a virtual ebuild I use that contains only dependencies
and I use this one to merge given versions of packages all in one shot:
sort of a shanpshot of a given moment in time (sort of tag).

Eg:

virtual-1.0.ebuild contains

RDEPEND=
a-2.0.ebuild
b-3.0.ebuild
c-4.0.ebuild
d-5.0.ebuild


So that emerge -K virtual-1.0 emerges all a,b,c,d on given versions and:

virtual-2.0.ebuild contains

RDEPEND=
a-2.1.ebuild
b-3.1.ebuild
c-4.1.ebuild
d-5.1.ebuild


My problem is that if a-2.0.ebuild and a-2.1.ebuild contain

RDEPEND=


c-1.0




and I have binary packages for both c-4.0 and c-4.1 when I emerge
virtual-2.0.ebuild i get c-4.1.ebuild but if I emerge virtual-1.0 I get
both c-4.0 (required as a dependency for virtual-1.0) and c-4.1 (as a
dep for a-2.0 who gets the latest c even though c-4.0 would have
satisfied the dependency.

Any info on how to avoid this?


It might be helpful if you'd post a real example and not some invalid 
pseudo-ebuild syntax.


Marius
--
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-11-26 Thread Marius Mauch

Jason Stubbs wrote:

On Saturday 26 November 2005 11:07, Marius Mauch wrote:


On Sat, 26 Nov 2005 00:01:15 +0900
Jason Stubbs [EMAIL PROTECTED] wrote:


The only other new thing in trunk that I know of is logging but
there's still a question mark over the ordering of messages... Can
that be resolved soon? Anything else missing? Any reasons against any
of the above?


Resolved how? I'm not really sure I understood the original problem
(other than listdir being underdeterministic in theory).



TGL suggested that all the messages go into a single file with some sort of 
prefix that would then be parsed on the python side. The original order of 
messages could then be maintained. However, seeing as there needs to be no 
compatibility with the temporary files it could wait for later anyway.


a) haven't seen a patch for it, so no clue about how complex it is code 
wise and b) I generally dislike any markup/parsing in the temporary 
files. I'd rather get it out as-is now and incorporate any feedback 
later. As you said this interface doesn't have to be compatible, also 
I intended to add a general might change-note for elog in the release 
notes.



- the pycrypto hash additions (for .54)


This is only useful if the vote goes in favour of adding further hash types to 
Manfiest, right? If the vote goes that way I've got no issues with it, but if 
it doesn't it would essentially be dead code.


Well, the vote was more for the SHA1 change actually as that's the one 
triggering the size increase. The pycrypto stuff itself doesn't do 
anything really, it would just make the size increase more apparent.



- Killing off auto-use+USE_ORDER?


Yep, I'd really like to see this one gone too. We should probably state the 
intention on -dev first though as there might be a lot of people against it.


Well, Spanky liked the suggestion ;)


- the recursive grab* functions I just posted (for .54)


Needs a small amount of work (/etc/portage/package.mask/foo/bar would break 
it) but I like the general idea.


How would it break?


- integration of set modules, either as emerge targets (requires
serious gutting of emerge) or a first-class atoms (semantically
tricky, no clue about implementation yet)


I'm working on this with my refactoring. Defininately a post-.54 thing unless 
you want to quickly hack it into getlist()?


Don't think so given that offhand I don't even know what getlist() does ...

Oh, btw, two things that are in trunk but weren't listed in your 
original mail:
- the rewritten versioning code (including the cvs and mult-suffix 
enhancements)

- finally killing of the stupid masked by -* message

Marius
--
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Help with KDE Arts

2005-11-25 Thread Marius Mauch

Robert wrote:

Hey, for some reason I cannot seem to install arts (KDE).


Try asking that on the gentooo-user list, it has nothing to do with 
portage development.


Marius
--
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] Re: Bugzilla Bug 112779: New and Improved Way to Handle /etc/portage

2005-11-25 Thread Marius Mauch
On Thu, 17 Nov 2005 09:30:15 +0200
Marius Mauch [EMAIL PROTECTED] wrote:

 Anthony Gorecki wrote:
  On Wednesday, November 16, 2005 23:12, Zac Medico wrote:
  
 I wouldn't mind having a feature like this.  I would provide a way
 for 
  
  automatic unmasking tools to keep their changes separate and easily 
  reversible.
  
  This seems to be borderlining on being unnecessary, in my opinion.
  A commented section in package.unmask could work just as easily,
  and it would likely save time for the user. kde-base/kdelibs is
  just as easy to find in a sorted, sectioned file as it is in
  multiple files:
  
  # GNOME Packages:
  [...]
  
  # KDE Packages:
  [...]
 
 I think the point is more with a) temporary enabling/disabling of 
 sections and b) sharing sections. Having multiple files those
 situations are a bit easier to handle.
 Shouldn't be too hard offhand, basically
 if isdir(foo):
   for x in listdir(foo):
   mylines.extend(grabfile(x))
 else:
   mylines = grabfile(foo)
 

Ok, patch is now available at
dev.gentoo.org/~genone/patches/portage-recursive-grab+config.diff

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] .53, .54 and beyond...

2005-11-25 Thread Marius Mauch
On Sat, 26 Nov 2005 00:01:15 +0900
Jason Stubbs [EMAIL PROTECTED] wrote:

 Hi all,
 
 I don't think there's really anything else that can be done for
 2.0.53 so am thinking that we should probably push _rc7 + docs out
 and let the arch teams mark it stable when they're ready (or stick
 with 2.0.51.22-r3 if it pleaseth them).
 
 We should put out a 2.0.54_pre1 out soon after that. What I'm
 wondering in is what will be going in? So far I'm thinking along the
 lines of:
 
 * cache rewrite
 * exec cleanup
 * ldconfig fix
 * sha1 enabling
 
 The only other new thing in trunk that I know of is logging but
 there's still a question mark over the ordering of messages... Can
 that be resolved soon? Anything else missing? Any reasons against any
 of the above?

Resolved how? I'm not really sure I understood the original problem
(other than listdir being underdeterministic in theory).

 What's on the map for after that? There's a few things listed on the
 new (still unreleased?) project index and I'm looking to get the
 dependency stuff refactored and moved out of emerge.. What are the
 shortterm goals?

- the pycrypto hash additions (for .54)
- Manifest2 verification support (need the GLEP first so the format is
sanctioned). Technically it's complete, just generation is still
unfinished. (so a maybe for .54 depending on the timeframe)
- Killing off auto-use+USE_ORDER?
- the recursive grab* functions I just posted (for .54)
- addition of auxget/metascan tools (could be for .54)
- integration of set modules, either as emerge targets (requires
serious gutting of emerge) or a first-class atoms (semantically
tricky, no clue about implementation yet)

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] multiple hash functions

2005-11-23 Thread Marius Mauch
On Wed, 23 Nov 2005 14:12:22 -0600
Brian Harring [EMAIL PROTECTED] wrote:

 Add the keys only if there is a func that can be used- list of 
 required chksums is a config thing (and repoman thing during 
 commiting), so I'm not seeing any reason to have None as a value in 
 your hashfunc mapping...

Yeah, makes a bit more sense. Good that nobody saw the nasty trick I had
before ;)

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Manifest signing

2005-11-22 Thread Marius Mauch
On Sat, 19 Nov 2005 20:59:07 +0900
Jason Stubbs [EMAIL PROTECTED] wrote:

 On Saturday 19 November 2005 20:41, Mike Auty wrote:
  If portage can already handle multiple hash formats,
 
 Portage can't handle multiple hash formats at the moment. It is only
 smart enough to not throw a fit when other hash formats appear.

Actually all current version (=2.0.51 IIRC) handle SHA1 digests
already (others aren't implemented in portage_checksum.py, but will be
silently ignored). We just don't generate SHA1 yet due to compability
issues, but those aren't relevant anymore.

 The user will be able to configure what algorithm(s) are used.

You mean for creation or verification (or both)?
Generation must be done for all available algorithms IMO.
Not so sure what to do about verification until the tree is mostly
covered with SHA1.

  Having to maintain backwards compatibility with old versions of
  portage is a good idea, however just how far back must be supported?
 
 A year is a good guide.

I'd say 9 months after the ebuild for that version vanished from
the tree as a minimum, but a year is also fine.
There are two usergroups creating problems:
- the ones that only update packages they're interested in (so no
complete system or world update ever)
- the ones who only ever update once per year or so (for whatever
reason)
Maybe we should request an addition to the handbook You must update
sys-apps/portage at least every 6 months?

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Display of keyword in emerge : code proposal

2005-11-22 Thread Marius Mauch
On Fri, 18 Nov 2005 19:13:13 +0100
jb benoit [EMAIL PROTECTED] wrote:

 $ diff -Naur portage.py.sav portage.py
 --- portage.py.sav  2005-11-17 15:32:20.0 +0100
 +++ portage.py  2005-11-17 15:15:24.0 +0100
 @@ -3866,6 +3866,42 @@
 pmaskfile.close()
 return None
 
 +def getprintablemaskingstatus(mycpv):
 +
 +   #checking
 +   global portdb
 +   mysplit = catpkgsplit(mycpv)
 +   if not mysplit:
 +   raise ValueError(invalid CPV: %s % mycpv)
 +   if not portdb.cpv_exists(mycpv):
 +   raise KeyError(CPV %s does not exist % mycpv)
 +   mycp=mysplit[0]+/+mysplit[1]
 +
 +   #gathering data
 +   mygroups = portdb.aux_get(mycpv, [KEYWORDS])[0].split()
 +   myarch = settings[ARCH]
 +   maskdict=settings.pmaskdict
 +
 +   rValue=[]
 +   test=0
 +
 +   # keyword chercking
 +   for gp in mygroups:
 +   if gp==~+myarch:
 +   kmask=~+myarch
 +   rValue.append(kmask)
 +   test = 1
 +
 +   # package.mask checking
 +if test == 1:
 +   if maskdict.has_key(mycp):
 +   for x in maskdict[mycp]:
 +   if mycpv in
 portdb.xmatch(match-all, x):
 +   rValue.append(~M)
 +
 +   return rValue
 +
 +
  def getmaskingstatus(mycpv):
 global portdb
 mysplit = catpkgsplit(mycpv)

- Doesn't work with binpkgs (though that's probably also a problem in
getmaskingstatus() itself)
- there is more than keyword and p.mask for masking (profiles)
- the function name is misleading (you're not checking the actual
masking status)
- you don't check for non-~arch and package.mask'ed packages
- you don't check for non-$ARCH ACCEPT_KEYWORDS/package.keywords entries
- other semantic issues I' not going to repeat
- completely useless without docs.

We didn't say it's complicated for no reason ;)

Marius

PS: But thanks anyway, looking at getmaskingstatus() reminds me of
removing the stupid -* message.

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] inital Manifest2 support

2005-11-22 Thread Marius Mauch
On Fri, 18 Nov 2005 18:40:14 -0600
Brian Harring [EMAIL PROTECTED] wrote:

 On Wed, Nov 16, 2005 at 04:33:02AM +0100, Marius Mauch wrote:
  -   digestfn  =
  mysettings[FILESDIR]+/digest-+mysettings[PF]
  +   digestfn  =
  portdb.finddigest(mysettings[CATEGORY]+/+mysettings[PF])
 Like this mod personally, but don't like the '/'.  You're passing in 
 two chunks of data, pass them in seperate imo, rather then collapsing 
 it for every call.

Not sure I understand this, you mean changing finddigest() to use two
params? Doesn't really make sense to me, should just work for any CPV
entry.

 Kind of disliking the 'calledFor' being thrown around here, plus the 
 v2 check, then resorting to a seperate method of checking.
 Integrate 'em potentially?
 
 You're basically treating it as two seperate funcs (from how I look
 at it), should split it up (even if the seperated funcs call each
 other, or a common func).

Yeah, was just going the least intrusive route for now, so keep the API
compatible. This wasn't intended for inclusion yet anyway, just some
sneak preview ;)

  @@ -2218,7 +2229,7 @@
  return 1
   
   
  -def digestParseFile(myfilename):
  +def digestParseFile(myfilename, calledFor=manifest):
  (filename) -- Parses a given file for entries matching:
  MD5 MD5_STRING_OF_HEX_CHARS FILE_NAME FILE_SIZE
  Ignores lines that do not begin with 'MD5' and returns a
  @@ -2230,6 +2241,10 @@
   
  mydigests={}
  for x in mylines:
  +   tmp = portage_manifest.convertManifest2ToDigest(x,
  calledAs=calledFor)
  +   if tmp[0] == True:
  +   mydigests[tmp[1]] = tmp[2]
  +   continue
  myline=string.split(x)
  if len(myline)  4:
  #invalid line
  @@ -2258,7 +2273,10 @@
  the basedir given. Returns 1 only if all files exist and
  match the md5s. 
  for x in myfiles:
  -   if not mydigests.has_key(x):
  +   if
  portage_manfest.getManifestVersion(basedir+/Manifest) == 2 and
  x.startswith(files/digest-):
 hardcoding sucks, kthnx
 If possible, try to shove it into the func so modification (down the
 line) occurs in one spot, rather then having to resort to
 search/replace :)

Shove what into which func? I'm bad at guessing today.

 
  Index: pym/portage_checksum.py
  ===
  --- pym/portage_checksum.py (revision 2312)
  +++ pym/portage_checksum.py (working copy)
  @@ -126,3 +126,12 @@
  portage_locks.unlockfile(mylock)
   
  return (myhash,mysize)
  +
  +
  +hashfunc_map = {MD5: md5hash, SHA1: sha1hash}
  +
  +def perform_multiple_checksums(filename, hashes=[MD5],
  calc_prelink=0):
  +   rVal = {}
  +   for x in hashes:
  +   rVal[x] = perform_checksum(filename,
  hashfunc_map[x], calc_prelink)[0]
  +   return rVal
 Not sure if I like the [0] tbh.
 Still think the size test should be broken off as it's own pseudo 
 hashfunc, and implemented in that way.  Course, would need to weight 
 it somehow, so that size test is done first, then the more expensive 
 chf tests...

Yep, see above about least intrusive.

  --- pym/portage_manifest.py (revision 0)
  +++ pym/portage_manifest.py (revision 0)
  @@ -0,0 +1,163 @@
  +import os
  +
  +import portage, portage_exception, portage_checksum,
  portage_versions +
  +class Manifest2Entry:
  +   def __init__(self, entrytype, name, config,
  methods=portage_checksum.get_valid_checksum_keys()):
 The methods trick you're doing is a one time evaluation... not sure
 if that's the best approach tbh.
 
  +   if entrytype not in [EBUILD, AUXFILE,
  MISCFILE, SRCURI]:
  +   raise
  portage_exception.PortageException(unknown Manifest entry type:
  %s % entrytype)
 break that out into a class constant, and please kill off the capps 
 (they kill kittens, or something).

Ok on the constant, but I like the caps here ;)

  +   self.type = entrytype
  +   self.name = name
  +   self.methods = methods
  +   self.config = config
 I realize stable does this all over, but I really am not much for 
 storing a config instance on this attribute.
 
 Thoughts on yanking the settings you need up front?

No problem, just took the easy route for now.

 
  +   def generate(self, pkgdir, genOldStyle=True):
  +   if self.type == EBUILD or self.type ==
  MISCFILE:
  +   filename = pkgdir+/+self.name
  +   elif self.type == AUXFILE:
  +   filename = pkgdir+/files/+self.name
  +   elif self.type == SRCURI:
  +   filename =
  self.config[DISTDIR]+/+self.name
  +   if not os.path.exists(filename):
  +   raise
  portage_exception.PortageException(file for Manifest entry doesn't
  exist: %s % filename)
  +   
  +   self.size = os.stat(filename).st_size
  +   self.checksums =
  portage_checksum.perform_multiple_checksums(filename

Re: [gentoo-portage-dev] Manifest signing

2005-11-22 Thread Marius Mauch
On Fri, 18 Nov 2005 22:01:27 -0800
Robin H. Johnson [EMAIL PROTECTED] wrote:

 Ergo, instead of a Manifest being re-generated each time, it needs to
 act like a FIFO queue.

Or in other words: transactional manifests.

 Each queue element consists of:
 - checksum/existing Manifest element of items changed in that action
 ONLY (however it should be possible to forcibly include files).
 - Signature around the above checksum.
 
 So now the new Manifest structure looks roughly like this
 (abbreviated): -- PGP
 MD5 ...
 MD5 ...
 -- SIG
 -- SIG
 -- PGP
 MD5 ...
 -- SIG
 -- SIG
 etc.
 
 This has one important implication for backwards compatibility in
 checking of Manifests.
 In the case that a filename appears more than once in the file, only
 the last instance of it should be used, as that is the one that
 relates to the current version of the file. It's 4 lines of code in
 the current portage that need to be removed for this to work (see my
 -core post for where exactly).

 Generation of the above is reasonably simple, just make the checksums
 in a string, clear-sign via gpg separately/tmpfile, append to
 Manifest.

And get an ever growing Manifest? Surely you're going to kill old
entries somehow (you only explained the when below).
Also, I expect a lot more problems. Easy one:
- dev A commits a file with MD5, SHA1 and SHA512 checksum
- dev B later commits the same file with just MD5 and SHA1
- user wants to use all available hashes
This posts two questions: 
a) should all the manifest entries from dev A be removed (assuming
those are the only in the signature block)?
b) if the old SHA512 entry is still present, how can we find out that
it should be ignored?
Contrary to my previous post on -dev this shows that even verification
isn't so easy.
So my previous objections to this still stand. I'm not going to
massively complicate the Manifest code for IMO very little gain.

 Even still, re-checking every digest in the tree should not happen on
 every CVS-rsync window. It's computationally pointless. Just check
 the changes.

And you get the changes how?
Hmm, thinking about it a simple `find $PORTDIR -mmin -x` with some
additional parsing should be sufficient and not be too bad on
performance (took about 2 minutes on my rather slow server with
$PORTDIR not being cached, about a second with it being cached).
That requires proper timestamps though (which I'm not sure we have).

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.


signature.asc
Description: PGP signature


  1   2   >