Re: [gentoo-portage-dev] Hello! portage UI
On Tuesday 14 March 2006 17:48, tvali wrote: 2006/3/14, Marius Mauch [EMAIL PROTECTED]: Brian wrote: On Tue, 2006-14-03 at 13:14 +0200, tvali wrote: Ok, i was, yes, speaking about kde. I will check out this Porthole :) I was actually thinking more about c ++, but nothing against python -- i was quite a fan of python when i first found it. I believe Kuroo is in C, maybe c++ For the record: Kuroo is known to be conceptually broken (uses the cache directly), but the author is looking to fix that in a future version. Generally writing portage utils in languages other than python is difficult as you can't use the primary API directly. There seems not to be c++ bintings to python, anyway there is dev-python/sip. I havent done such thing, but i suppose that it would not be impossible to start portage inside Kuroo as a thread, then interact with this thread, having sip between them. If it's used for creation of PyQt, such two-way interaction should be possible. Actually, C++ being strongly related to C, can just use the C python api's. As such it could directly interface with python, and use the python portage api. Paul -- Paul de Vrieze Gentoo Developer Mail: [EMAIL PROTECTED] Homepage: http://www.devrieze.net pgpyvIdbNhH88.pgp Description: PGP signature
Re: sync suggestions [was Re: [gentoo-portage-dev] Few things, which imho would make portage better]
But if any user-side changes are assumed to be separated? I mean if there is a boolean field user, which is triggered for user-changed tables. Or, to be simpler, i use 2 tables in my example. Lets assume that user wants to change description of dev-lang/php -- so that user has to change dev-lang/php in table user-tree, but leave the same in table portage-tree unchanged. Of course, current example would make queries into portage-db more complex than they should be, so more optimized version should be found -- but, anyway, there are ways to make things work. Imagine that (i take only ebuild files into consideration here): * Portage tree is kept in SQL base, which contains the following fields: ** Id ** LongName (dev-db/mysql-4.1.14) ** Name (dev-db/mysql) ** ShortName (mysql) ** Slot ** Server -- which server or server group contains that ebuild (if same ebuild is in several server, it should be repeated in SQL). If empty, then this ebuild is created by user ** ServerStatus -- false if deleted from server (only used if UserInfo is not NULL) ** Status -- if false, this row will not be used ** Current -- if this ebuild is what should be installed if emerge mysql is written on this system ** Description ** /.../ -- other fields parsed from ebuild ** ServerInfo -- ebuild file from server ** UserInfo -- user additions to that file * Dependency tree ** Will be updated from prev. table Now, updates in server should be in the following form: * Id * ServerInfo * Action -- add/delete/update All other fields will be parsed out from Action in user's computer. Any changes to portage tree will be then done via portage commands, not directly to SQL. 2006/3/15, Brian Harring [EMAIL PROTECTED]: On Tue, Mar 14, 2006 at 03:50:18PM +0200, tvali wrote: Another question now is about sync. I did read somewhere, that this is not good user behavior to sync more than once per day. I understand that as if this is a huge download even if there is nothing changed. Isnt it nice idea to have this database just optimized? I mean (assuming portage using SQL now) -- that would be really simple to log every change in portage tree as series of SQL queries, which would reproduce this change. Pushing the delta (what you're suggesting) is only usable if it can be guranteed the user hasn't modified their tree at all (thus resulting in cache db differing from upstreams). That right there is the brass tacks of it; You wouldn't be able to push just the changes, you would have to regenerate the _whole_ db (slow, 20k inserts assuming only one table). Sidenote... please post seperate threads for seperate ideas/discussions, else it's damn hard to look back and pull the specific thread were something was discussed. ~harring -- tvali (e-mail: [EMAIL PROTECTED]; msn: [EMAIL PROTECTED]; icq: 317-492-912) Ühe eesti internetifirma lehel kohtasin tsitaati: If you don't do it excellently, dont do it at all. Because if it's not excellent, it won't be profitable or fun, and if you're not in business for fun or profit, what the hell are you doing here? Robert Townsend -- gentoo-portage-dev@gentoo.org mailing list
Re: esearch integration [was Re: [gentoo-portage-dev] Few things, which imho would make portage better]
Yes SQL tables are better for that as it's simpler to update them :) 2006/3/15, Brian Harring [EMAIL PROTECTED]: On Tue, Mar 14, 2006 at 04:33:06PM +0200, tvali wrote: I did think about it now and it seems to me that probably it would be much faster if esearch is not just another package, but part of portage. I mean -- functions of portage, which query db, should use esearch index wherever they need information, which exists in that index. As much as i can understand, /var/cache/edb/ contains esearch database in many files and esearchdb.py is search index as python script. No... esearch is a static db- only useful for 'frozen' trees, eg rsync distributed trees with no eclasses in overlays. All cvs users (devs) run unfrozen trees (readonly/readwrite is better terminology), thus portage updates the cache db on the fly as needed. If esearch was integrated into portage the result would be stale metadata for cvs users, and stale metadata for rsync users when overlays with eclasses are involved- no go. That and esearch last I looked just generates a giant dict (thus the cache is in memory), which kind of blows the 25mb mem usage 2.1 now sports :) ~harring -- tvali (e-mail: [EMAIL PROTECTED]; msn: [EMAIL PROTECTED]; icq: 317-492-912) Ühe eesti internetifirma lehel kohtasin tsitaati: If you don't do it excellently, dont do it at all. Because if it's not excellent, it won't be profitable or fun, and if you're not in business for fun or profit, what the hell are you doing here? Robert Townsend -- gentoo-portage-dev@gentoo.org mailing list
[gentoo-portage-dev] DB and binary dependency
Hello, There is any provision for binary dependency on Gentoo/Portage? The way it works now is quite messy with things like revdep-rebuild. I have an idea to solve this problem: after software is build, you check which files it links (ldd binaries libraries) and check the used against installed packages. If a library is not provided by an installed package we could have a policy to inform user or just abort installation. Also, if we implement these dependencies in rpm-generator, we could just generate RPM packages and install it in the RPM-DB and let it handle these kind of things. With these in place emerge would handle the build stage (where it excels), but rpm would handle the binary installation and dependencies (where it excels). -- Gustavo Sverzut Barbieri -- Mobile: +55 (81) 9927 0010 Phone: +1 (347) 624 6296; [EMAIL PROTECTED] Jabber: [EMAIL PROTECTED] ICQ#: 17249123 MSN: [EMAIL PROTECTED] GPG: 0xB640E1A2 @ wwwkeys.pgp.net -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] kudoos to all
On Thursday 16 March 2006 00:24, Brian wrote: I just want to say that all the speedup work everyone has been doing is really noticeable. Porthole now can reload the entire database 10763 packages (includes some in PORTDIR_OVERLAY) in about 3 seconds on my system. Athlon-xp 2000 based. the cache update after a sync no longer takes as long as compiling the mozilla suite .) good work! :) currently @ portage-2.1_pre6 Most of that work can be attributed to Brian Harring. For a real kicker though, throw portdbapi.auxdbmodule = cache.metadata_overlay.database into /etc/portage/modules and FEATURES=-metadata-transfer into make.conf. Then rm -rf /var/cache/edb/dep. This one goes to Zac Medico, the current release coordinator, and builds on Brian's work to cut out cache updating altogether. New in pre6 (pre5?) and in need of testing - especially when local modifications are made - but very promising. -- Jason Stubbs -- gentoo-portage-dev@gentoo.org mailing list
Re: portage from C (was Re: [gentoo-portage-dev] Hello! portage UI)
sorry lines char tt; for (tt=0; tt40; tt++) fputc(tt, inputf); /.../ fputc(EOF, inputf); are not needed at all and do nothing -- i was just testing a thought before i catched how those pipes and forks work in linux :) i have mostly done things in windows and c++ and WinApi so i dont catch everything immedietly yet :) 2006/3/16, tvali [EMAIL PROTECTED]: So I was the last one :) My example is using python interpreter -- it's maybe simpler to use in many cases, but theoretically slower. It is, anyway, not so advanced to import portage, but you can add that line ;) 2006/3/15, solar [EMAIL PROTECTED]: On Wed, 2006-03-15 at 21:42 +0100, Kevin F. Quinn (Gentoo) wrote: On Wed, 15 Mar 2006 09:52:13 -0500 solar [EMAIL PROTECTED] wrote: On Wed, 2006-03-15 at 10:51 +0100, Paul de Vrieze wrote: [snip] Actually, C++ being strongly related to C, can just use the C python api's. As such it could directly interface with python, and use the python portage api. If you could demonstrate an 'import portage' and a simple printf(PORTDIR=%s\n, PORTDIR); in C I'd be highly interested. Try attached :) Had a go 'coz I was curious. Compile with: gcc -o getportdir getportdir.c -lpython2.4 Does the equivalent of: #!/usr/bin/portage import portage; print portage.settings[PORTDIR]; (more or less). Docs on the API itself (which comes with Python) are at http://docs.python.org/api/api.html thanks Kevin, marienz also put 3 examples together to demonstrate it. /me feels like a kid in a candy store. thanks guys. -- solar [EMAIL PROTECTED] Gentoo Linux -- gentoo-portage-dev@gentoo.org mailing list -- tvali (e-mail: [EMAIL PROTECTED]; msn: [EMAIL PROTECTED]; icq: 317-492-912) Ühe eesti internetifirma lehel kohtasin tsitaati: If you don't do it excellently, dont do it at all. Because if it's not excellent, it won't be profitable or fun, and if you're not in business for fun or profit, what the hell are you doing here? Robert Townsend -- tvali (e-mail: [EMAIL PROTECTED]; msn: [EMAIL PROTECTED]; icq: 317-492-912) Ühe eesti internetifirma lehel kohtasin tsitaati: If you don't do it excellently, dont do it at all. Because if it's not excellent, it won't be profitable or fun, and if you're not in business for fun or profit, what the hell are you doing here? Robert Townsend -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] [PATCH] Manifest2 reloaded
Marius Mauch schrieb: The first should be delayed until there is some consensus how the gpg stuff should work in the future, the others I don't see the use for. Also I only checked portage.py for changes, so emerge/repoman/... might still have to be fixed. Last but not least: I did some basic testing with this and the important stuff seems to work, but I'm quite sure the code still has a lot of bugs/issues, and this being a core functionality it needs a *lot* of testing, so I'd really appreciate if you could all give it a spin (but do not commit anything to the tree without manually checking it first). Does the lack of feedback (only got a reaction from Brian so far) mean that noone tried it or that it doesn't have any issues? Marius -- gentoo-portage-dev@gentoo.org mailing list
[gentoo-portage-dev] Wildcards in package.keywords
Hi, I just recently finished an installation KDE 3.5.2. The process, of course, required the use of a great deal of entries in the package.keywords file to get around the fact that KDE is not in stable yet. I was able to write a simple Python script to automate adding all of the KDE packages, but it was still a hassle. Would it be possible to add in a feature that just needs this in package.keywords: kde-base/* ~x86 or, to apply it to a single version, this: =kde-base/*-3.5.1 ~x86 Regular expressions would, of course, work too. They might be a little bit of overkill, though. Chow, Aaron DeVoreg How To Tick People Off #20. TYPE IN UPPERCASE. -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] [PATCH] Manifest2 reloaded
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Marius Mauch wrote:
Marius Mauch schrieb:
The first should be delayed until there is some consensus how the gpg
stuff should work in the future, the others I don't see the use for.
Also I only checked portage.py for changes, so emerge/repoman/... might
still have to be fixed.
Last but not least: I did some basic testing with this and the
important stuff seems to work, but I'm quite sure the code still has a
lot of bugs/issues, and this being a core functionality it needs a
*lot* of testing, so I'd really appreciate if you could all give it a
spin (but do not commit anything to the tree without manually checking
it first).
Does the lack of feedback (only got a reaction from Brian so far) mean
that noone tried it or that it doesn't have any issues?
The patch applies and seems to work well. At a quick glance the code looks
pretty clean and it's nice to migrate more code out of portage.py to a separate
module. I've attached a refreshed version of the patch that applies cleanly
against current svn (I've made no changes).
Zac
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2.2 (GNU/Linux)
iD8DBQFEGP1S/ejvha5XGaMRAl/7AJ9cZbjhWtjCz+ac2/tjQNUoivj0twCg7xAG
cYvDbMiqU5HtpNrVk7fs6RM=
=Eqlo
-END PGP SIGNATURE-
=== added file 'pym/portage_manifest.py'
--- /dev/null
+++ pym/portage_manifest.py
@@ -0,0 +1,314 @@
+import os, sets
+
+import portage, portage_exception, portage_versions, portage_const
+from portage_checksum import *
+from portage_exception import *
+
+class FileNotInManifestException(PortageException):
+ pass
+
+def manifest2AuxfileFilter(filename):
+ filename = filename.strip(/)
+ return not (filename in [CVS, .svn] or filename[:len(digest-)] == digest-)
+
+def manifest2MiscfileFilter(filename):
+ filename = filename.strip(/)
+ return not (filename in [CVS, .svn, files, Manifest] or filename[-7:] == .ebuild)
+
+class Manifest(object):
+ def __init__(self, pkgdir, db, mysettings, hashes=portage_const.MANIFEST2_HASH_FUNCTIONS, manifest1_compat=True, fromScratch=False):
+ self.pkgdir = pkgdir+os.sep
+ self.fhashdict = {}
+ self.hashes = hashes
+ self.hashes.append(size)
+ if manifest1_compat:
+ self.hashes.extend(portage_const.MANIFEST1_HASH_FUNCTIONS)
+ self.hashes = sets.Set(self.hashes)
+ for t in portage_const.MANIFEST2_IDENTIFIERS:
+ self.fhashdict[t] = {}
+ self._read()
+ self.compat = manifest1_compat
+ self.db = db
+ self.mysettings = mysettings
+ if mysettings.has_key(PORTAGE_ACTUAL_DISTDIR):
+ self.distdir = mysettings[PORTAGE_ACTUAL_DISTDIR]
+ else:
+ self.distdir = mysettings[DISTDIR]
+
+ def guessType(self, filename):
+ if filename.startswith(files/digest-):
+ return None
+ if filename.startswith(files/):
+ return AUX
+ elif filename.endswith(.ebuild):
+ return EBUILD
+ elif filename in [ChangeLog, metadata.xml]:
+ return MISC
+ else:
+ return DIST
+
+ def getFullname(self):
+ return self.pkgdir+Manifest
+
+ def getDigests(self):
+ rval = {}
+ for t in portage_const.MANIFEST2_IDENTIFIERS:
+ rval.update(self.fhashdict[t])
+ return rval
+
+ def _readDigests(self):
+ mycontent =
+ for d in portage.listdir(self.pkgdir+files, filesonly=True, recursive=False):
+ if d.startswith(digest-):
+mycontent += open(self.pkgdir+files+os.sep+d, r).read()
+ return mycontent
+
+ def _read(self):
+ if not os.path.exists(self.getFullname()):
+ return
+ fd = open(self.getFullname(), r)
+ mylines = fd.readlines()
+ fd.close()
+ mylines.extend(self._readDigests().split(\n))
+ for l in mylines:
+ myname =
+ mysplit = l.split()
+ if len(mysplit) == 4 and mysplit[0] in portage_const.MANIFEST1_HASH_FUNCTIONS:
+myname = mysplit[2]
+mytype = self.guessType(myname)
+if mytype == AUX and myname.startswith(files/):
+ myname = myname[6:]
+if mytype == None:
+ continue
+mysize = int(mysplit[3])
+myhashes = {mysplit[0]: mysplit[1]}
+ if len(mysplit) 4 and mysplit[0] in portage_const.MANIFEST2_IDENTIFIERS:
+mytype = mysplit[0]
+myname = mysplit[1]
+mysize = int(mysplit[2])
+myhashes = dict(zip(mysplit[3::2], mysplit[4::2]))
+ if len(myname) == 0:
+continue
+ if not self.fhashdict[mytype].has_key(myname):
+self.fhashdict[mytype][myname] = {}
+ self.fhashdict[mytype][myname].update(myhashes)
+ self.fhashdict[mytype][myname][size] = mysize
+
+ def _writeDigests(self):
+ cpvlist = [self.pkgdir.rstrip(/).split(/)[-2]+/+x[:-7] for x in portage.listdir(self.pkgdir) if x.endswith(.ebuild)]
+ rval = []
+ for cpv in cpvlist:
+ dname = self.pkgdir+files+os.sep+digest-+portage.catsplit(cpv)[1]
+ mylines = []
+ distlist = self._getCpvDistfiles(cpv)
+ for f in self.fhashdict[DIST].keys():
+if f in distlist:
+ for h in self.fhashdict[DIST][f].keys():
+ if h not in portage_const.MANIFEST1_HASH_FUNCTIONS:
+ continue
+ myline = .join([h, str(self.fhashdict[DIST][f][h]), f,
Re: [gentoo-portage-dev] [PATCH] Manifest2 reloaded
On Wed, Mar 15, 2006 at 09:53:24PM -0800, Zac Medico wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Marius Mauch wrote:
Marius Mauch schrieb:
The first should be delayed until there is some consensus how the gpg
stuff should work in the future, the others I don't see the use for.
Also I only checked portage.py for changes, so emerge/repoman/... might
still have to be fixed.
Last but not least: I did some basic testing with this and the
important stuff seems to work, but I'm quite sure the code still has a
lot of bugs/issues, and this being a core functionality it needs a
*lot* of testing, so I'd really appreciate if you could all give it a
spin (but do not commit anything to the tree without manually checking
it first).
Does the lack of feedback (only got a reaction from Brian so far) mean
that noone tried it or that it doesn't have any issues?
The patch applies and seems to work well. At a quick glance the code looks
pretty clean and it's nice to migrate more code out of portage.py to a
separate module. I've attached a refreshed version of the patch that applies
cleanly against current svn (I've made no changes).
Zac
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2.2 (GNU/Linux)
iD8DBQFEGP1S/ejvha5XGaMRAl/7AJ9cZbjhWtjCz+ac2/tjQNUoivj0twCg7xAG
cYvDbMiqU5HtpNrVk7fs6RM=
=Eqlo
-END PGP SIGNATURE-
=== added file 'pym/portage_manifest.py'
--- /dev/null
+++ pym/portage_manifest.py
@@ -0,0 +1,314 @@
+import os, sets
+
+import portage, portage_exception, portage_versions, portage_const
+from portage_checksum import *
+from portage_exception import *
+
+class FileNotInManifestException(PortageException):
+ pass
+
+def manifest2AuxfileFilter(filename):
+ filename = filename.strip(/)
+ return not (filename in [CVS, .svn] or filename[:len(digest-)] ==
digest-)
+
+def manifest2MiscfileFilter(filename):
+ filename = filename.strip(/)
+ return not (filename in [CVS, .svn, files, Manifest] or
filename[-7:] == .ebuild)
python -m timeit -s 's=asdf*400;s+=fdsa.ebuild' 's.endswith(.ebuild)'
100 loops, best of 3: 0.88 usec per loop
python -m timeit -s 's=asdf*400;s+=fdsa.ebuild' 's[-7:] == .ebuild'
100 loops, best of 3: 0.564 usec per loop
Use endswith
oddly, worth noting that startswith differs in this behaviour...
python -m timeit -s 's=asdf*400;s+=fdsa.ebuild' 's[:7] == .ebuild'
100 loops, best of 3: 0.592 usec per loop
python -m timeit -s 's=asdf*400;s+=fdsa.ebuild' 's.startswith(.ebuild)'
100 loops, best of 3: 0.842 usec per loop
+class Manifest(object):
+ def __init__(self, pkgdir, db, mysettings,
hashes=portage_const.MANIFEST2_HASH_FUNCTIONS, manifest1_compat=True,
fromScratch=False):
+ self.pkgdir = pkgdir+os.sep
rstrip os.sep prior to adding it
+ self.fhashdict = {}
+ self.hashes = hashes
+ self.hashes.append(size)
+ if manifest1_compat:
+
self.hashes.extend(portage_const.MANIFEST1_HASH_FUNCTIONS)
+ self.hashes = sets.Set(self.hashes)
+ for t in portage_const.MANIFEST2_IDENTIFIERS:
+ self.fhashdict[t] = {}
+ self._read()
+ self.compat = manifest1_compat
+ self.db = db
+ self.mysettings = mysettings
+ if mysettings.has_key(PORTAGE_ACTUAL_DISTDIR):
+ self.distdir = mysettings[PORTAGE_ACTUAL_DISTDIR]
+ else:
+ self.distdir = mysettings[DISTDIR]
Why pass in mysettings?
Have the code push it in, manifest shouldn't know about the DISTDIR
key nor PORTAGE_ACTUAL_DISTDIR, should just have a directory to look
in.
+ def guessType(self, filename):
+ if filename.startswith(files/digest-):
+ return None
+ if filename.startswith(files/):
if you're intent on using os.sep, might want to correct the two '/'
uses above to use os.path.join/os.path.sep
If concerned about cost, just calculate it once in the class namespace
as a constant.
related, might I suggest converting away from internal strings to a
class level enumeration?
int comparison is faster then string, plus it unbinds the internal
code from the on disk symbols used (eg, just cause on disk is AUX
doesn't mean internally it should be throwing around AUX).
+ return AUX
+ elif filename.endswith(.ebuild):
+ return EBUILD
+ elif filename in [ChangeLog, metadata.xml]:
+ return MISC
+ else:
+ return DIST
+
+ def getFullname(self):
+ return self.pkgdir+Manifest
Err... move that into the initializer.
If you're concerned folks will screw up the var, use a property to
make it immutable.
Either way, func calls aren't cheap, and that's not really needed :)
+
+
Re: [gentoo-portage-dev] [PATCH] Manifest2 reloaded
Brian Harring wrote: python -m timeit -s 's=asdf*400;s+=fdsa.ebuild' 's.endswith(.ebuild)' 100 loops, best of 3: 0.88 usec per loop python -m timeit -s 's=asdf*400;s+=fdsa.ebuild' 's[-7:] == .ebuild' 100 loops, best of 3: 0.564 usec per loop Use endswith oddly, worth noting that startswith differs in this behaviour... python -m timeit -s 's=asdf*400;s+=fdsa.ebuild' 's[:7] == .ebuild' 100 loops, best of 3: 0.592 usec per loop python -m timeit -s 's=asdf*400;s+=fdsa.ebuild' 's.startswith(.ebuild)' 100 loops, best of 3: 0.842 usec per loop Um, those both read the same way to me. You just switched the ordering around, so the (starts|ends)with is on the bottom instead of the top, but both times (starts|ends)with is longer. Thanks, Donnie signature.asc Description: OpenPGP digital signature
