Re: [gentoo-portage-dev] equery displays warnings about masked deps, even when those deps are deeper than --depth specification

2010-01-13 Thread Douglas Anderson
On Wed, Jan 13, 2010 at 2:50 AM, Amit Dor-Shifer wrote:

 Douglas Anderson wrote:

 The newest version of gentoolkit has slightly changed the way depgraph
 prints output. Could you try checking with the latest unstable version of
 gentoolkit and submit a bug if you get the same results?

  No change w/0.3.0_rc7:

rc8(-r1) is newest in tree and depgraph is slightly changed in it. Same
deal, please test with your setup and let me know. I'll have time this
weekend to take a closer look.

Appreciate the testing!

Re: [gentoo-portage-dev] equery displays warnings about masked deps, even when those deps are deeper than --depth specification

2010-01-12 Thread Douglas Anderson
On Tue, Jan 12, 2010 at 1:50 AM, Amit Dor-Shifer wrote:

 amit0 ~ # qfile -v $(which equery)
 app-portage/gentoolkit- (/usr/bin/equery)

The newest version of gentoolkit has slightly changed the way depgraph
prints output. Could you try checking with the latest unstable version of
gentoolkit and submit a bug if you get the same results?


P.S. Replying below the quote is the convention on most mailing lists. It's
more readable if everyone does it... thanks :)

Re: [gentoo-dev] New metadata fields

2009-06-03 Thread Douglas Anderson
The equery 'meta' module in gentoolkit-0.3.0* can show all the
upstream fields, though not many packages currently make use of it.

Oh yeah, and there's a traceback in the upstream code that's been
fixed but won't be in the tree until rc7, so you may want to wait to
check it out.

$ equery m --upstream lince
 * net-p2p/lince
Remote ID:  sourceforge: lincetorrent

Re: [gentoo-portage-dev] Symlinks with distutils

2009-05-05 Thread Douglas Anderson

On May 5, 2009, at 10:37 PM, Michael A. Smith  

In theory, doing symlinks with distutils isn't a big deal, but
distutils.core.setup doesn't have the capability built in.  
does, but it's not clear how to trigger that intuitively within  
setup().) So
for the changes to gentoolkit for 0.3, I vote we keep the ebuild  
dosyms, and
leave distutils out of the business of symlinks, to be revisited at  
a later

date. Agree?

Best wishes,
Michael (a.k.a. kojiro)

Sounds good to me.

[gentoo-portage-dev] Deprecate portage.abssymlink

2009-03-10 Thread Douglas Anderson
I can't make out the different between portage.abssymlink and
os.path.realpath, besides that realpath is used a LOT more in portage
and abssympath dies if it gets passed something that's not a symlink.

Let's get rid of it!

1st patch is for trunk/pym/
2nd patch is for trunk/pym/dbapi/ (only actual instance of
abssymlink left)

---	2009-03-10 21:07:06.0 +0900
+++	2009-03-10 21:20:34.0 +0900
@@ -180,12 +180,10 @@ def getcwd():
 def abssymlink(symlink):
-	This reads symlinks, resolving the relative symlinks, and returning the absolute.
-	mylink=os.readlink(symlink)
-	if mylink[0] != '/':
-		mydir=os.path.dirname(symlink)
-		mylink=mydir+/+mylink
-	return os.path.normpath(mylink)
+	Not currently used by Portage (DEPRECATED 03/2009)
+	writemsg(DEPRECATED: portage.abssymlink: use os.path.realpath instead\n,
+		noiselevel=-1)
+	return os.path.realpath(symlink)
 dircache = {}
---	2009-03-10 21:14:12.0 +0900
+++	2009-03-10 21:16:07.0 +0900
@@ -3,8 +3,18 @@
 # $Id$
 __all__ = [PreservedLibsRegistry, LinkageMap,
-	vardbapi, vartree, dblink] + \
-	[write_contents, tar_contents]
+	vardbapi, vartree, dblink, write_contents, tar_contents]
+import os, re, shutil, stat, errno, copy, subprocess
+import logging
+import shlex
+import sys
+from itertools import izip
+	import cPickle as pickle
+except ImportError:
+	import pickle
 import portage
@@ -32,20 +42,10 @@ from portage.exception import CommandNot
 from portage import listdir, dep_expand, digraph, flatten, key_expand, \
 	doebuild_environment, doebuild, env_update, prepare_build_dirs, \
-	abssymlink, movefile, _movefile, bsd_chflags, cpv_getkey
+	movefile, _movefile, bsd_chflags, cpv_getkey
 from portage.cache.mappings import slot_dict_class
-import os, re, shutil, stat, errno, copy, subprocess
-import logging
-import shlex
-import sys
-from itertools import izip
-	import cPickle as pickle
-except ImportError:
-	import pickle
 class PreservedLibsRegistry(object):
 	 This class handles the tracking of preserved library objects 
@@ -3649,7 +3649,7 @@ class dblink(object):
 			if stat.S_ISLNK(mymode):
 # we are merging a symbolic link
-myabsto = abssymlink(mysrc)
+myabsto = os.path.realpath(mysrc)
 if myabsto.startswith(srcroot):
 	myabsto = myabsto[len(srcroot):]
 myabsto = myabsto.lstrip(sep)

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

2009-02-23 Thread Douglas Anderson
On Mon, Feb 23, 2009 at 7:02 PM, Tiziano Müller wrote:
 Am Montag, den 23.02.2009, 22:25 +1300 schrieb Alistair Bush:

 Tiziano Müller wrote:
  What is proposed in glep-55 seems to aim to solve both issues at the
  same time (it isn't stated) by switching file extension every time the
  eapi is changed. This is slightly against the principle of the least
  surprise and apparently is disliked by enough people to lead the
  situation to be discussed in the council.
  Instead of switching file extension every time the eapi is changed you
  could also increment it only when a new EAPI breaks sourcing the ebuild
  compared to the requirements of the prior EAPI.
  (This way you'd in fact split EAPI into a major- and a minor-version.)

 Doesn't that just add extra complexity for no gain.
 Yes, sure. I was just looking for a solution for the we have countless 
 .eapi-X after 10 years problem.

No one wants to be working with ebuild-29 or something like that in a
few years and trying to figure out which feature came in which EAPI.
Instead of bumping EAPI for each little change, save them up and bump
no more than once a year or less, each bump bringing in some major new
feature. With a little common sense and planning, we could make this a
non-issue and give ebuild authors and PM devs alike a little time to
get used to each change.

Re: [gentoo-portage-dev] equery: RFC and code review

2009-02-12 Thread Douglas Anderson
On Thu, Feb 12, 2009 at 4:10 PM, Alec Warner wrote:
 I think one of the major things that annoyed me when reading the code
 is that you did add a new option parser...and you added it about a
 dozen times...(regexp used to basically find the start of the same
 code block over and over).

 anta...@kyoto ~/code/checkouts/genscripts-read-only/equery $ grep
 opts = .x *.py | wc -l

 Now the option parsing is quite complicated so I'm not sure if a
 complicated data structure to pass to a generic function is really a
 good use of ones time.  But it stands out.

Actually, I think you'll find that the amount of duplicated opt
parsing code in each module is really minimal, essentially only that
one line that your regex matched. Each module has its own set of
options and handles those options differently, so putting it all in
__init__ would be a royal mess.

I can see how at a glance it would appear like duplicated code, but
check out the different between normal getopt and gnu_getopt and how I
have those working between __init__ and the modules. There's nothing
being parsed twice. If you can come up with a simpler, cleaner way,
write back with a little more detail. But I just reexamined it and
even tried moving some stuff around, and I think it's about as good as
it gets right now.

 None of your files has a copyright notice.

There's one in The rest of equery will reside in
python/site-packages. Does each one need the same copyright notice? (I
honestly don't know the policy)

 I pray pp.print_error prints to stderr and not stdout; otherwise tends
 to be a big nitpick of mine when people write errors to stdout.

$ grep -A 2 'def print_error' /usr/lib/gentoolkit/pym/gentoolkit/
def print_error(s):
Prints an error string to stderr.
sys.stderr.write(!!! ) + s + \n)

Safe :) has this gem:
 q.append(('^' if query[0] == '/' else '/') + re.escape(query) + '$')

 I realize it works; but honestly, not readable.

You're right, that's pretty nasty. But look what it came from!

q = map(lambda x: ((len(x) and x[0] == /) and ^ or /) +
re.escape(x) + $, query)

But I agree that it can be further improved. How's this:

if query.startswith('/'):
query = ^%s$ % re.escape(query)
query = /%s$ % re.escape(query)

 You use # FOO comments often; sometimes too often.

Yep, good point. I comment a little excessively, and believe it or not
emeta (now the meta module) was my first ever python script, so it had
lots of obvious comments. I'll try to get rid of the silly stuff as
I run into them again.

 in your find_matches function is too long (180 lines or
 4.5 of my screen heights).  You've already split it up into convenient
 sections by comments (which are fine by the way) but you can probably
 make subroutines out of some of the subsections.

Good idea. I'll do that.

 in you have InvalidRegex with a TODO to move it to another
 file.  It seems like might be a worthy place.

Yep, not 100% done yet and modifying more blocks to use exceptions,
and then moving the exceptions out to is on my list of

 I think there are a number of cases where I'd prefer to see string
 interpolation instead of explicit addition:

 '%s/%s' % (cat, package)

 But thats just me being picky (and you use string interpolation often,
 which is good).

Explicit string catting is also one of my pet peeves, but py-2.6 is
getting a brand new string formatting syntax and % style
interpolation will be deprecated in 3k, so I only rewrote strings
where readability was suffering. I'm biding my time... but I
definitely agree with you here.

 Thats what I found in about 20 minutes of looking.


Muchly appreciated!

[gentoo-portage-dev] equery: deprecate --category filtering in belongs

2009-02-07 Thread Douglas Anderson
Hi, does anyone use --category filtering in equery belongs? I want to
get rid of it, or at least deprecate it. My reasoning:

* We use 'equery belongs' when don't know to what package a file
belongs. Even if we have a suspicion, most users would have to look up
the category of the package before typing it in.
* Even if you happen to know the exact category of the package that
installed the file (why are you using belongs?), typing
--category=app-portage takes more time than is saved by filtering by
category (about 5 seconds more by my unscientific test).

Even in a script setting, I see no use for this. The time saved is minuscule:

$ time equery belongs /usr/bin/equery --category app-portage
[ Searching for file(s) /usr/bin/equery in app-portage... ]
app-portage/gentoolkit- (/usr/bin/equery)

sys 0m0.076s
$ time equery belongs /usr/bin/equery
[ Searching for file(s) /usr/bin/equery in *... ]
app-portage/gentoolkit- (/usr/bin/equery)

sys 0m0.102s

* Lastly, it's confusing. belongs takes a filename as input, but you
can filter by category? All other modules that take can filter by
category take a pkgspec.


Re: [gentoo-dev] Re: Last Rites: app-portage/udept

2008-12-15 Thread Douglas Anderson
On Tue, Dec 16, 2008 at 9:06 AM, Daniel Pielmeier wrote:
 Duncan schrieb am 16.12.2008 00:47:
 While I'm at it, is there anything useful to display metadata.xml?  In
 particular, the long descriptions and use flags can be useful.  With
 use.desc and especially the local version thereof going deprecated, and
 with additional info about global flags sometimes in the metadata...

 Regarding metadata.xml there is (besides querying Willikins on IRC)
 emeta. Take a look at bug 248278 [1].


emeta will likely become `equery meta' in the next release of
gentoolkit, but feel free to use it from the overlay for now.

Regarding equery, there's a changes option that is just waiting to
be written. Considering udept's changleog function is about 30 lines,
it should be trivial to do. If people are interested, I can handle
that. It's something I would like to have, also.

I hope to have the upgraded gentoolkit available for testing within a few weeks.


[gentoo-dev] Re: Digest of issue 653 (33679-33728)

2008-12-09 Thread Douglas Anderson

[gentoo-dev] Re: Digest of issue 653 (33679-33728)

2008-12-09 Thread Douglas Anderson
blah (switching emails, sorry for that)

On Tue, Dec 9, 2008 at 7:14 PM, Douglas Anderson [EMAIL PROTECTED]


Re: [gentoo-portage-dev] Re: equery refactorization

2008-12-06 Thread Douglas Anderson
On Sun, Dec 7, 2008 at 12:02 PM, Michael A. Smith [EMAIL PROTECTED] wrote:
 Hash: SHA1

 Regarding gentoolkit/trunk/src/equery/tests

 I discovered all the test kit that's in equery, and have been refactoring
 They're written in bash, not python, so they're a candidate for some kind of
 python unit testing. Right now, however, that's not a priority for me, so
 just making the bash cleaner and hopefully faster and more maintainable. I
 think it'll be helpful as we refactor.

 The question is, how maintainable are the help tests? These are tests that
 try to confirm that the --help output of each module is correct. I think it
 might be more work than it's worth to try to maintain those...


I know some people like to write the tests and then write the code to
match, but I don't think it's a good idea for you to refactor the
tests as I'm refactoring the codebase :)

Especially since I'm chopping and moving things, renaming functions,
etc, as long as I think it'll help in the long term.

I even changed the format of the help output ;) Why? Because we have
two user-oriented tools with a similar modular design, equery and
eselect, and yet they have a totally different naming scheme and
behave quite differently. It's unnecessarily confusing so I tried to
make them more uniform (I'll upload some code shortly). I always
though equery's --help was cluttered and confusing. A complete
overview is what `man equery' is for, IMHO.

I also changed the way equery handles input slightly. For example this
I think is unnecessarily lenient and in the end confusing, because it
goes against what most other tools do (raise an exception):

$ equery -q -i list mozilla-firefox
!!! unknown global option -i, reusing as local option

So, I don't think we should be working on the tests until we have most
of the code refactored, but I re-extend my invitation for help on that
because there's quite a bit to do!


Re: [gentoo-portage-dev] Re: equery refactorization

2008-12-06 Thread Douglas Anderson
On Sun, Dec 7, 2008 at 12:34 PM, Michael A. Smith [EMAIL PROTECTED] wrote:
 Hash: SHA1

 Let me rephrase:

 The tests as they are written are not all that helpful or functional.
 I'm refactoring them so that they are. If I don't, they won't be any good at

 These are not sophisticated tests that comprehensively review your code.
 simply do some sanity checks on the output of equery.

 - -Michael

OK, I just wanted you to understand that --help and some error
messages are bound to change as I attempt to clarify them.

I also thought about renaming the list(l) option as search,
because if you look at the help output, almost every module lists
something. equery's list is actually a search, I don't see why we
shouldn't name it that. I think maybe list was used because there were
already two s options, stats and size. Stats is not implemented so
I'm taking it out of help for now. Size can use the short z, becaues
that's quite unique. That would free up s for search and it would be
a whole lot clearer.

Yes? No?


Re: [gentoo-portage-dev] Python style (Was: equery refactorization)

2008-12-06 Thread Douglas Anderson
On Sun, Dec 7, 2008 at 9:50 AM, Zac Medico [EMAIL PROTECTED] wrote:
 Hash: SHA1

 Michael A. Smith wrote:
 Actually I don't like spaces for indentation at all. A tab character and a
 space character take the same number of bytes, so it takes two-to-eight
 as much space per indentation to store the same indentation as a single tab
 character. On the average project that can add up to several kilobytes of
 difference. Furthermore, it completely _prevents_ collaborators from
 enjoying a
 bit of customization as to the amount of indentation. If I like my
 to be two spaces, and you like yours to be four, then we can each set
 our text
 editors that way and continue to happily share code that uses tabs for

 I disagree with both PEP8 and Gentoo SOC styles on the matter. I would
 love it
 if someone could explain to me the actual benefit of using spaces over

 FWIW, I feel the same way.

 - --
 Version: GnuPG v2.0.9 (GNU/Linux)


Again, not starting a holy war here, I'm happy using tabs as that's
what people want, but you asked why most Python people prefer spaces,
and one of the reasons is that it's considered Pythonic to match the
indentation level of an opening parenthesis, for example:

With tabs you can't do that faithfully. And yes tabs do use less
space, but at the expense of readability. Stripping doc string out
would also use less space. As PEP 20 says, readability counts. As
Python is largely reliant on indentation for meaning in code, spaces
work well in python where they're not necessary in, say, bash. That's
just MHO.


Re: [gentoo-portage-dev] Re: equery refactorization

2008-12-05 Thread Douglas Anderson
On Thu, Dec 4, 2008 at 2:43 PM, Alec Warner [EMAIL PROTECTED] wrote:
 nitpick feel free to ignore me
 Don't put stuff in

 Make a file called equery (no .py) and do all the work in the modules
 you import; eg.

 from equery import driver

 if __name__ == __main__:

 Then put all this code in (option parsing, signal handling,
 etc...).  Don't try to hide the code in; it confuses
 people who are trying to figure out what the module is for (since has very specific duties in declaring what is in the
 module when you inspect the query module).  Putting the code in a file
 named '' or similarly makes it pretty obvious (to me anyway)
 what the code in that file is for (to drive a program).

 Does that make sense or am I full of crap?

I see what you're saying, but using a second file,, just to
do's job seems even more confusing. is the
standard python constructor, and it's required to be in every module
directory, if I understand correctly. Since you have to have an file in the directory, which gets sourced anyway, it might
as well be used for what it's meant for, which is handling all the
initial setup of the package. If I'm misunderstanding the purpose of
__init__, please let me know.

So two best ways I can think to set it up are:
1) /usr/bin/equery only import equery from /usr/lib/gentoolkit/equery. in that dir runs all the setup work and handles input args
(this is quite common), and imports and runs the requested module.
This is a similar setup used by something like iotop.


2) /usr/bin/equery contains all the init stuff and opt handling, then
imports the separate modules as needed. This style is used by
something like pybugz, although you still have to have an
in the module folder, it can be a lot sparser.

I was leaning toward #1 because it keeps all the code in the same directory.

 A little RFC:
 1) Spaces or tabs? Python standard is spaces, Gentoo seems to be
 predominantly tabs. I personally like to use spaces when I'm writing
 Python, but if that would annoy everyone later on, I'll stick to tabs.

 Gentoo has no official coding standards.  I'd personally prefer spaces
 (along with basically everything else in the Google Python Style
 Guide[1]), but I'm probably not going to nitpick.  It is my opinion
 that tabs are used because that is what the tools were written in and
 it is annoying to change from tabs to spaces ;)


I'm with you there, I really like that style guide as well. We should
adopt it :)


Re: [gentoo-portage-dev] search functionality in emerge

2008-11-23 Thread Douglas Anderson

It would be great it you could speed search up a bit!

As these other guys have pointed out, we do have some indexing tools in
Gentoo already. Most users don't understand why that kind of functionality
isn't built directly into Portage, but IIRC it has something to do with the
fact that these fast search indexes aren't able to be written to by more
than one process at the same time, so for example if you had two emerges
finishing at the same time, Portage's current flat hash file can handle
that, but the faster db-based indexes can't.

Anyways, that's the way I, as a curious user, understood the problem.

You might be interested in reading this, very old forum thread about a
previous attempt:

On Sun, Nov 23, 2008 at 11:33 PM, Pacho Ramos 

 El dom, 23-11-2008 a las 16:01 +0200, tvali escribió:
  Try esearch.
  emerge esearch
  esearch ...
  2008/11/23 Emma Strubell [EMAIL PROTECTED]
  Hi everyone. My name is Emma, and I am completely new to this
  list. I've been using Gentoo since 2004, including Portage of
  course, and before I say anything else I'd like to say thanks
  to everyone for such a kickass package management system!!
  Anyway, for my final project in my Data Structures 
  Algorithms class this semester, I would like to modify the
  search functionality in emerge. Something I've always noticed
  about 'emerge -s' or '-S' is that, in general, it takes a very
  long time to perform the searches. (Although, lately it does
  seem to be running faster, specifically on my laptop as
  opposed to my desktop. Strangely, though, it seems that when I
  do a simple 'emerge -av whatever' on my laptop it takes a very
  long time for emerge to find the package and/or determine the
  dependecies -  whatever it's doing behind that spinner. I can
  definitely go into more detail about this if anyone's
  interested. It's really been puzzling me!) So, as my final
  project I've proposed to improve the time it takes to perform
  a search using emerge. My professor suggested that I look into
  implementing indexing.
  However, I've started looking at the code, and I must admit
  I'm pretty overwhelmed! I don't know where to start. I was
  wondering if anyone on here could give me a quick overview of
  how the search function currently works, an idea as to what
  could be modified or implemented in order to improve the
  running time of this code, or any tip really as to where I
  should start or what I should start looking at. I'd really
  appreciate any help or advice!!
  Thanks a lot, and keep on making my Debian-using professor
  jealous :]
  Kuskilt foorumist: - kui inglise keelt oskad.
  Muide, üle 120 oled väga tark, üle 140 oled geenius, mingi 170 oled ju
  mingi täica pea nagu prügikast...

 I use eix:
 emerge eix