Re: [gentoo-portage-dev] [PATCH] _get_lock_fn: support multiprocessing spawn start method (bug 758230)
Thanks Zac!
On 04-12-2020 14:58:22 -0800, Zac Medico wrote:
> Ensure that _get_lock_fn arguments to multiprocessing.Process will
> successfully pickle, as required by the spawn start method, which
> is the default for macOS since Python 3.8.
>
> Since file descriptors are not inherited unless the fork start
> method is used, the subprocess should only try to close an
> inherited file descriptor for the fork start method.
>
> Bug: https://bugs.gentoo.org/758230
> Signed-off-by: Zac Medico
> ---
> lib/portage/locks.py | 36 +++-
> 1 file changed, 23 insertions(+), 13 deletions(-)
>
> diff --git a/lib/portage/locks.py b/lib/portage/locks.py
> index 1073343be..193045c03 100644
> --- a/lib/portage/locks.py
> +++ b/lib/portage/locks.py
> @@ -44,18 +44,7 @@ def _get_lock_fn():
> if _lock_fn is not None:
> return _lock_fn
>
> - def _test_lock(fd, lock_path):
> - os.close(fd)
> - try:
> - with open(lock_path, 'a') as f:
> - fcntl.lockf(f.fileno(),
> fcntl.LOCK_EX|fcntl.LOCK_NB)
> - except EnvironmentError as e:
> - if e.errno == errno.EAGAIN:
> - # Parent process holds lock, as expected.
> - sys.exit(0)
>
> - # Something went wrong.
> - sys.exit(1)
>
> fd, lock_path = tempfile.mkstemp()
> try:
> @@ -64,8 +53,16 @@ def _get_lock_fn():
> except EnvironmentError:
> pass
> else:
> - proc = multiprocessing.Process(target=_test_lock,
> - args=(fd, lock_path))
> + proc = multiprocessing.Process(
> + target=_subprocess_test_lock,
> + args=(
> + # Since file descriptors are not
> inherited unless the fork start
> + # method is used, the subprocess should
> only try to close an
> + # inherited file descriptor for the
> fork start method.
> + fd if
> multiprocessing.get_start_method() == "fork" else None,
> + lock_path,
> + ),
> + )
> proc.start()
> proc.join()
> if proc.exitcode == os.EX_OK:
> @@ -80,6 +77,19 @@ def _get_lock_fn():
> _lock_fn = fcntl.flock
> return _lock_fn
>
> +def _subprocess_test_lock(fd, lock_path):
> + if fd is not None:
> + os.close(fd)
> + try:
> + with open(lock_path, 'a') as f:
> + fcntl.lockf(f.fileno(), fcntl.LOCK_EX|fcntl.LOCK_NB)
> + except EnvironmentError as e:
> + if e.errno == errno.EAGAIN:
> + # Parent process holds lock, as expected.
> + sys.exit(0)
> +
> + # Something went wrong.
> + sys.exit(1)
>
> _open_fds = {}
> _open_inodes = {}
> --
> 2.26.2
>
>
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] Speeding up Tree Verification
On 30-06-2020 13:13:29 -0500, Sid Spry wrote: > On Tue, Jun 30, 2020, at 1:20 AM, Fabian Groffen wrote: > > Hi, > > > > On 29-06-2020 21:13:43 -0500, Sid Spry wrote: > > > Hello, > > > > > > I have some runnable pseudocode outlining a faster tree verification > > > algorithm. > > > Before I create patches I'd like to see if there is any guidance on > > > making the > > > changes as unobtrusive as possible. If the radical change in algorithm is > > > acceptable I can work on adding the changes. > > > > > > Instead of composing any kind of structured data out of the portage tree > > > my > > > algorithm just lists all files and then optionally batches them out to > > > threads. > > > There is a noticeable speedup by eliding the tree traversal operations > > > which > > > can be seen when running the algorithm with a single thread and comparing > > > it to > > > the current algorithm in gemato (which should still be discussed here?). > > > > I remember something that gemato used to use multiple threads, but > > because it totally saturated disk-IO, it was brought back to a single > > thread. People were complaining about unusable systems. > > > > I think this is an argument for cgroups limits support on the portage process > or > account as opposed to an argument against picking a better algorithm. That is > something I have been working towards, but I am only one man. But this requires a) cgroups support, and b) the privileges to use it. Shouldn't be a problem in the normal case, but just saying. > > In any case, can you share your performance results? What speedup did > > you see, on warm and hot FS caches? Which type of disk do you use? > > > > I ran all tests multiple times to make them warm off of a Samsung SSD, but > nothing very precise yet. > > % gemato verify --openpgp-key signkey.asc /var/db/repos/gentoo > [...] > INFO:root:Verifying /var/db/repos/gentoo... > INFO:root:/var/db/repos/gentoo verified in 16.45 seconds > > sometimes going higher, closer to 18s, vs. > > % ./veriftree.py > 4.763171965983929 > > So roughly an order of magnitude speedup without batching to threads. That is kind of a change. Makes one wonder if you really did the same work. > > You could compare against qmanifest, which uses OpenMP-based > > paralllelism while verifying the tree. On SSDs this does help. > > > > I lost my notes -- how do I specify to either gemato or qmanifest the GnuPG > directory? My code is partially structured as it is because I had problems > doing > this. I rediscovered -K/--openpgp-key in gemato but am unsure for qmanifest. qmanifest doesn't do much magic out of the standard gnupg practices. (It is using gpgme.) If you want it to use a different gnupg dir, you may change HOME, or GNUPGHOME. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] Speeding up Tree Verification
: str) -> bool:
> response = dns.resolver.query(domain, dns.rdatatype.NS)
> nsname = response.rrset[0]
> response = dns.resolver.query(str(nsname), dns.rdatatype.A)
> nsaddr = response.rrset[0].to_text()
>
> # DNSKEY
> request = dns.message.make_query(domain,
> dns.rdatatype.DNSKEY, want_dnssec=True)
> response = dns.query.udp(request, nsaddr)
> if response.rcode() != 0:
> raise Exception('Unable to request dnskey.')
>
> answer = response.answer
> if len(answer) != 2:
> raise Exception('Malformed answer to dnskey query.')
>
> name = dns.name.from_text(domain)
> try:
> dns.dnssec.validate(answer[0], answer[1], {name: answer[0]})
> except dns.dnssec.ValidationFailure as exc:
> # Validation failed. The raise causes python to abort with status 1.
> #raise exc
> return False
> except AttributeError as exc:
> # Validation may have failed; DNSKEY missing signer attribute. dig
> may report
> # domain as valid.
> #
> # TODO: Additional state where subdomain of valid domain may fail
> with 3 nested
> # KeyErrors. Avoid temptation to wildcard catch. Safer to put in
> process?
> #raise exc
> return False
> else:
> return True
>
> def hash_localpart(incoming: bytes) -> str:
> '''Z-base32 the localpart of an e-mail address
>
>
> https://tools.ietf.org/html/draft-koch-openpgp-webkey-service-08#section-3.1
> describes why this is needed.
>
> See https://tools.ietf.org/html/rfc6189#section-5.1.6 for a
> description of the z-base32 scheme.
> '''
> zb32 = "ybndrfg8ejkmcpqxot1uwisza345h769"
>
> b = hashlib.sha1(incoming).digest()
> ret = ""
> assert(len(b) * 8 == 160)
> for i in range(0, 160, 5):
> byte = i // 8
> offset = i - byte * 8
> # offset | bits remaining in k+1 | right-shift k+1
> # 3 | 0 | x
> # 4 | 1 | 7
> # 5 | 2 | 6
> # 6 | 3 | 5
> # 7 | 4 | 4
> if offset < 4:
> n = (b[byte] >> (3 - offset))
> else:
> n = (b[byte] << (offset - 3)) + (b[byte + 1] >> (11 - offset))
>
> ret += zb32[n & 0b1]
> return ret
>
> def build_web_key_uri(address: str) -> str:
> local, remote = address.split('@')
> local = hash_localpart(local.encode('utf-8'))
> return 'https://' + remote + '/.well-known/openpgpkey/hu/' + \
> local
>
> def stream_to_file(uri: str, fname: str) -> None:
> with requests.get(uri, verify=True, stream=True) as r:
> from pprint import pprint
> pprint(r.headers)
> with open(fname, 'wb') as f:
> shutil.copyfileobj(r.raw, f)
> ```
>
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] Add caching to a few commonly used functions
Hi Chun-Yu, On 26-06-2020 23:34:12 -0700, Chun-Yu Shei wrote: > Hi, > > I was recently interested in whether portage could be speed up, since > dependency resolution can sometimes take a while on slower machines. > After generating some flame graphs with cProfile and vmprof, I found 3 > functions which seem to be called extremely frequently with the same > arguments: catpkgsplit, use_reduce, and match_from_list. In the first > two cases, it was simple to cache the results in dicts, while > match_from_list was a bit trickier, since it seems to be a requirement > that it return actual entries from the input "candidate_list". I also > ran into some test failures if I did the caching after the > mydep.unevaluated_atom.use and mydep.repo checks towards the end of the > function, so the caching is only done up to just before that point. > > The catpkgsplit change seems to definitely be safe, and I'm pretty sure > the use_reduce one is too, since anything that could possibly change the > result is hashed. I'm a bit less certain about the match_from_list one, > although all tests are passing. > > With all 3 patches together, "emerge -uDvpU --with-bdeps=y @world" > speeds up from 43.53 seconds to 30.96 sec -- a 40.6% speedup. "emerge > -ep @world" is just a tiny bit faster, going from 18.69 to 18.22 sec > (2.5% improvement). Since the upgrade case is far more common, this > would really help in daily use, and it shaves about 30 seconds off > the time you have to wait to get to the [Yes/No] prompt (from ~90s to > 60s) on my old Sandy Bridge laptop when performing normal upgrades. > > Hopefully, at least some of these patches can be incorporated, and please > let me know if any changes are necessary. This sounds like a good job to me! Do you have any idea what the added memory pressure for these changes are? Thanks, Fabian > > Thanks, > Chun-Yu > > > -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] Use env to find python
On 16-06-2020 23:10:28 -0500, Sid Spry wrote: > On Tue, Jun 16, 2020, at 3:57 PM, Michał Górny wrote: > > '/usr/bin/env python' (with no extra options) is the portable shebang. > > > > I added `-S` to preserve the options passed via the shebang line. It seems > they can be left off, does anyone know otherwise? -S is not supported on some platforms. In any case, I think this patch is wrong. In the Prefix case, the shebangs are set during install, in the non-Prefix case I think there is machinery in place to replace the shebangs during install. It seems the problem is already solved, why do you need these shebangs changed? Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] Change BINPKG_COMPRESS default from bzip2 to xz
On 26-04-2020 21:29:42 +0200, Michał Górny wrote: > On Sun, 2020-04-26 at 09:55 -0700, Matt Turner wrote: > > Bug: https://bugs.gentoo.org/715108 > > Signed-off-by: Matt Turner > > --- > > Strawman patch. Bikeshed away. > > > > xz is generally slow and doesn't do parallel good. If we want to change > this, we should go for something cool like zstd that scales better. I'd go for zstd too. It seems to be the best of both worlds, good compression at a good speed. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.
On 13-12-2019 14:24:33 -0500, Michael Orlitzky wrote: > On 12/13/19 9:28 AM, Fabian Groffen wrote: > > > > We are providing those patches, maybe. In reality very often the > > patches originate from somewhere else though. And you don't want to > > have to respin all of those just because. At least that's what I feel. > > > > Just because... the context changed? A new "!" in a line of context can > be the difference between letting someone log in with the right password > and letting them log in with the wrong password. You should at least > have to stop and verify that the patch does what you think it does when > it "gains" fuzz. And at that point, git diff will give you a clean > version of it. Counter argument is that we've been doing this for decades, and always relied on maintainers to check the contents of their patches, without problems. We didn't introduce a predictable random number generator or something. Your very specific example just illustrates the niche this proposal is targetting. As with many of the proposals lately, they just seem to aim at more work for individual maintainers, with a very low gain ratio. Better, allow this to be a FEATURE, or whatever, that devs should enable, but don't spit this in the user's face by default. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.
On 13-12-2019 15:24:40 +0100, Michał Górny wrote: > > > ...and why do we consider it correct to apply patches when the context > > > doesn't match? If our only goal is to make things 'easier' for > > > 'everyone', then we could just pass -F and ignore all the context. > > > > > > Though I don't understand why include any context in the first place if > > > you don't care about it matching. Sounds like a waste of space to me! > > > > The patch command defaults to -F2. If that makes no sense, why is it > > the upstream default? > > > > You should ask upstream, not me. But if I were to guess, the answer > would be because patch(1) is used by random people trying to apply > random patches they've found somewhere. We on the other hand are > applying patches that *we* are supposed to provide. We are providing those patches, maybe. In reality very often the patches originate from somewhere else though. And you don't want to have to respin all of those just because. At least that's what I feel. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [RFC] Adding extra vars to md5-cache, for QA purposes
Hi, On 25-07-2019 14:20:50 +0200, Michał Górny wrote: > Hi, > > TL;DR: I'd like to make it possible for ebuilds to define additional > variables that will be stored in md5-cache. This would be useful for CI > and other tooling that right now has to parse ebuilds for other data. Only downside I can think of, is a diskspace increase for the md5-cache. Not sure if this is going to be substantial, but given things like PYTHON_COMPAT, perhaps a quick calculation of extra "cost" can be made. Should diskspace become a problem, one could consider to use a separate file/dir, that users could rsync-exclude, since Portage won't need it to operate properly. Thanks, Fabian > > > The idea is to add a new incremental ebuild/eclass variable (technical > name: QA_EXTRA_CACHE_VARS) that would define additional data to be > stored in cache. For example, python*-r1 eclasses would define > 'PYTHON_COMPAT', acct-user would define 'ACCT_USER_ID', etc. > > When regenerating cache, the PM would read this variable, and store > the values of all defined variables into md5-cache. As a result, > programs needing those variables can get them straight from cache > without having to attempt to run or parse ebuilds (which is both slow > and prone to bugs). > > This would benefit e.g. gpyutils that right now need to attempt to parse > PYTHON_COMPAT from ebuilds. It would also benefit writing future > pkgcheck checks for user/group ID collisions. > > > Notes: > > - since md5-cache uses key-value format and allows for future > extensions, the new values can be added without breaking anything; > > - md5-cache is not specified in the PMS, and the whole thing can be > implemented without need for EAPI bump, > > - I would like to have this implemented consistently both in Portage > and pkgcore, > > - we will need to clearly define how to dump arrays. > > > What do you think? > > -- > Best regards, > Michał Górny > -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH v5] collision_protect: use dynamic report interval
Pushed, thanks
On 10-01-2019 20:37:16 -0800, Zac Medico wrote:
> On 1/10/19 7:30 AM, Fabian Groffen wrote:
> > The reporting of files remaining can look somewhat odd since the report
> > interval is hardcoded to be per 1000 objects. Adjust this interval to
> > be time based. This means that modern (fast) machines likely will never
> > see the countdown messages at all. On slow setups the message will be
> > informative that there is progress, albeit rather slowly. While at it,
> > report percentage done.
> >
> > Output before this patch:
> >
> > * checking 6158 files for package collisions
> > 5158 files remaining ...
> > 4158 files remaining ...
> > 3158 files remaining ...
> > 2158 files remaining ...
> > 1158 files remaining ...
> > 158 files remaining ...
> >
> > Possible output after this patch on a slower machine:
> >
> > * checking 6158 files for package collisions
> > 48% done, 3145 files remaining ...
> > 96% done, 192 files remaining ...
> > 100% done
> >
> > Signed-off-by: Fabian Groffen
> > ---
> > lib/portage/dbapi/vartree.py | 15 +--
> > 1 file changed, 13 insertions(+), 2 deletions(-)
> >
> > diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
> > index 4b91caea8..909c0e473 100644
> > --- a/lib/portage/dbapi/vartree.py
> > +++ b/lib/portage/dbapi/vartree.py
> > @@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
> > 'portage.util.install_mask:install_mask_dir,InstallMask',
> > 'portage.util.listdir:dircache,listdir',
> > 'portage.util.movefile:movefile',
> > + 'portage.util.monotonic:monotonic',
> > 'portage.util.path:first_existing,iter_parents',
> > 'portage.util.writeable_check:get_ro_checker',
> > 'portage.util._xattr:xattr',
> > @@ -3475,13 +3476,21 @@ class dblink(object):
> > symlink_collisions = []
> > destroot = self.settings['ROOT']
> > totfiles = len(file_list) + len(symlink_list)
> > + previous = monotonic()
> > + progress_shown = False
> > + report_interval = 1.7 # seconds
> > + falign = len("%d" % totfiles)
> > showMessage(_(" %s checking %d files for package
> > collisions\n") % \
> > (colorize("GOOD", "*"), totfiles))
> > for i, (f, f_type) in enumerate(chain(
> > ((f, "reg") for f in file_list),
> > ((f, "sym") for f in symlink_list))):
> > - if i % 1000 == 0 and i != 0:
> > - showMessage(_("%d files remaining
> > ...\n") % (totfiles - i))
> > + current = monotonic()
> > + if current - previous > report_interval:
> > + showMessage(_("%3d%% done, %*d files
> > remaining ...\n") %
> > + (i * 100 / totfiles,
> > falign, totfiles - i))
> > + previous = current
> > + progress_shown = True
> >
> > dest_path = normalize_path(
> > os.path.join(destroot,
> > f.lstrip(os.path.sep)))
> > @@ -3570,6 +3579,8 @@ class dblink(object):
> > break
> > if stopmerge:
> > collisions.append(f)
> > + if progress_shown:
> > + showMessage(_("100% done\n"))
> > return collisions, dirs_ro, symlink_collisions,
> > plib_collisions
> >
> > def _lstat_inode_map(self, path_iter):
> >
>
> Looks good!
> --
> Thanks,
> Zac
>
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH v5] collision_protect: use dynamic report interval
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects. Adjust this interval to
be time based. This means that modern (fast) machines likely will never
see the countdown messages at all. On slow setups the message will be
informative that there is progress, albeit rather slowly. While at it,
report percentage done.
Output before this patch:
* checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...
Possible output after this patch on a slower machine:
* checking 6158 files for package collisions
48% done, 3145 files remaining ...
96% done, 192 files remaining ...
100% done
Signed-off-by: Fabian Groffen
---
lib/portage/dbapi/vartree.py | 15 +--
1 file changed, 13 insertions(+), 2 deletions(-)
diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..909c0e473 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.install_mask:install_mask_dir,InstallMask',
'portage.util.listdir:dircache,listdir',
'portage.util.movefile:movefile',
+ 'portage.util.monotonic:monotonic',
'portage.util.path:first_existing,iter_parents',
'portage.util.writeable_check:get_ro_checker',
'portage.util._xattr:xattr',
@@ -3475,13 +3476,21 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+ previous = monotonic()
+ progress_shown = False
+ report_interval = 1.7 # seconds
+ falign = len("%d" % totfiles)
showMessage(_(" %s checking %d files for package
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
- if i % 1000 == 0 and i != 0:
- showMessage(_("%d files remaining
...\n") % (totfiles - i))
+ current = monotonic()
+ if current - previous > report_interval:
+ showMessage(_("%3d%% done, %*d files
remaining ...\n") %
+ (i * 100 / totfiles,
falign, totfiles - i))
+ previous = current
+ progress_shown = True
dest_path = normalize_path(
os.path.join(destroot,
f.lstrip(os.path.sep)))
@@ -3570,6 +3579,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+ if progress_shown:
+ showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions,
plib_collisions
def _lstat_inode_map(self, path_iter):
--
2.20.1
Re: [gentoo-portage-dev] [PATCH v4] collision_protect: use dynamic report interval
On 09-01-2019 21:22:26 -0800, Zac Medico wrote:
> > + if tnow + tinterv < ninterv:
> > + showMessage(_("100% done\n"))
>
> It took me a moment to understand the calculation here. How about if we
> do something like this instead:
>
>
> previous = monotonic()
> progress_shown = False
You're right. I should've gone for readability in the first place.
There is no point in trying to save anything here.
Thanks,
Fabian
> for f in files:
> current = monotonic()
> if current - previous > tinterv:
> showMessage(...)
> previous = current
> progress_shown = True
>
> if progress_shown:
> showMessage(_("100% done\n"))
>
>
>
> > return collisions, dirs_ro, symlink_collisions,
> > plib_collisions
> >
> > def _lstat_inode_map(self, path_iter):
> >
>
>
> --
> Thanks,
> Zac
>
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH v4] collision_protect: use dynamic report interval
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects. Adjust this interval to
be time based. This means that modern (fast) machines likely will never
see the countdown messages at all. On slow setups the message will be
informative that there is progress, albeit rather slowly. While at it,
report percentage done.
Output before this patch:
* checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...
Possible output after this patch on a slower machine:
* checking 6158 files for package collisions
48% done, 3145 files remaining ...
96% done, 192 files remaining ...
100% done
Signed-off-by: Fabian Groffen
---
lib/portage/dbapi/vartree.py | 13 +++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..df192e6fc 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.install_mask:install_mask_dir,InstallMask',
'portage.util.listdir:dircache,listdir',
'portage.util.movefile:movefile',
+ 'portage.util.monotonic:monotonic',
'portage.util.path:first_existing,iter_parents',
'portage.util.writeable_check:get_ro_checker',
'portage.util._xattr:xattr',
@@ -3475,13 +3476,19 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+ tnow = monotonic()
+ tinterv = 2 # seconds
+ ninterv = tnow + tinterv
+ falign = len("%d" % totfiles)
showMessage(_(" %s checking %d files for package
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
- if i % 1000 == 0 and i != 0:
- showMessage(_("%d files remaining
...\n") % (totfiles - i))
+ if monotonic() > ninterv:
+ showMessage(_("%3d%% done, %*d files
remaining ...\n") %
+ (falign, i * 100 /
totfiles, totfiles - i))
+ ninterv = monotonic() + tinterv
dest_path = normalize_path(
os.path.join(destroot,
f.lstrip(os.path.sep)))
@@ -3570,6 +3577,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+ if tnow + tinterv < ninterv:
+ showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions,
plib_collisions
def _lstat_inode_map(self, path_iter):
--
2.20.1
Re: [gentoo-portage-dev] [PATCH v3] collision_protect: use dynamic report interval
On 08-01-2019 20:59:34 +, M. J. Everitt wrote:
> On 08/01/19 19:15, Zac Medico wrote:
> > On 1/8/19 5:42 AM, Fabian Groffen wrote:
> >> The reporting of files remaining can look somewhat odd since the report
> >> interval is hardcoded to be per 1000 objects. Adjust this interval to
> >> be time based. This means that modern (fast) machines likely will never
> >> see the countdown messages at all. On slow setups the message will be
> >> informative that there is progress, albeit rather slowly. While at it,
> >> report percentage done.
> >>
> >> Output before this patch:
> >>
> >> * checking 6158 files for package collisions
> >> 5158 files remaining ...
> >> 4158 files remaining ...
> >> 3158 files remaining ...
> >> 2158 files remaining ...
> >> 1158 files remaining ...
> >> 158 files remaining ...
> >>
> >> Possible output after this patch on a slower machine:
> >>
> >> * checking 6158 files for package collisions
> >> 48% done, 3145 files remaining ...
> >> 96% done, 192 files remaining ...
> >> 100% done
> >>
> >> Signed-off-by: Fabian Groffen
> >> ---
> >> lib/portage/dbapi/vartree.py | 11 +--
> >> 1 file changed, 9 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
> >> index 4b91caea8..244195fad 100644
> >> --- a/lib/portage/dbapi/vartree.py
> >> +++ b/lib/portage/dbapi/vartree.py
> >> @@ -3475,13 +3475,18 @@ class dblink(object):
> >>symlink_collisions = []
> >>destroot = self.settings['ROOT']
> >>totfiles = len(file_list) + len(symlink_list)
> >> + tnow = time.time()
> >> + tinterv = 2 # seconds
> >> + ninterv = tnow + tinterv
> >>showMessage(_(" %s checking %d files for package
> >> collisions\n") % \
> >>(colorize("GOOD", "*"), totfiles))
> >>for i, (f, f_type) in enumerate(chain(
> >>((f, "reg") for f in file_list),
> >>((f, "sym") for f in symlink_list))):
> >> - if i % 1000 == 0 and i != 0:
> >> - showMessage(_("%d files remaining
> >> ...\n") % (totfiles - i))
> >> + if time.time() > ninterv:
> >> + showMessage(_("%3d%% done, %d files
> >> remaining ...\n") %
> >> + (i * 100 / totfiles,
> >> totfiles - i))
> >> + ninterv = time.time() + tinterv
> >>
> >>dest_path = normalize_path(
> >>os.path.join(destroot,
> >> f.lstrip(os.path.sep)))
> >> @@ -3570,6 +3575,8 @@ class dblink(object):
> >>break
> >>if stopmerge:
> >>collisions.append(f)
> >> + if tnow + tinterv < ninterv:
> >> + showMessage(_("100% done\n"))
> >>return collisions, dirs_ro, symlink_collisions,
> >> plib_collisions
> >>
> >>def _lstat_inode_map(self, path_iter):
> >>
> > Please replace time.time() with portage.util.monotonic.monotonic().
Will do.
> It's a bit of a nit-pick, granted, but can we ensure that the count-down's
> remain padded/justified such that the numbers line up for easy at-a-glance
> inspection ? The optimal standard looks somewhat like the pre-merge Sizes:
>
> * Final size of build directory: 2696 KiB (2.6 MiB)
> * Final size of installed tree: 5372 KiB (5.2 MiB)
>
> Otherwise, I think this will be quite helpful. Thanks.
I think I understand what you mean, let me see if I can do something
without too much complexity.
Thanks,
Fabian
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH v3] collision_protect: use dynamic report interval
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects. Adjust this interval to
be time based. This means that modern (fast) machines likely will never
see the countdown messages at all. On slow setups the message will be
informative that there is progress, albeit rather slowly. While at it,
report percentage done.
Output before this patch:
* checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...
Possible output after this patch on a slower machine:
* checking 6158 files for package collisions
48% done, 3145 files remaining ...
96% done, 192 files remaining ...
100% done
Signed-off-by: Fabian Groffen
---
lib/portage/dbapi/vartree.py | 11 +--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..244195fad 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,18 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+ tnow = time.time()
+ tinterv = 2 # seconds
+ ninterv = tnow + tinterv
showMessage(_(" %s checking %d files for package
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
- if i % 1000 == 0 and i != 0:
- showMessage(_("%d files remaining
...\n") % (totfiles - i))
+ if time.time() > ninterv:
+ showMessage(_("%3d%% done, %d files
remaining ...\n") %
+ (i * 100 / totfiles,
totfiles - i))
+ ninterv = time.time() + tinterv
dest_path = normalize_path(
os.path.join(destroot,
f.lstrip(os.path.sep)))
@@ -3570,6 +3575,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+ if tnow + tinterv < ninterv:
+ showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions,
plib_collisions
def _lstat_inode_map(self, path_iter):
--
2.20.1
Re: [gentoo-portage-dev] [PATCH] collision_protect: use dynamic report interval
On 08-01-2019 09:17:04 +0100, Ulrich Mueller wrote: > >>>>> On Tue, 08 Jan 2019, Fabian Groffen wrote: > > > Output before this patch: > > > * checking 6111 files for package collisions > > [...] > > > After: > > > * checking 6158 files for package collisions > > [...] > > Why does the total number of files change? First was python2, second python3. The result was indicative only, the number doesn't change of course. Sorry for the confusion. Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] collision_protect: use dynamic report interval
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects. Adjust this interval to
be regular towards the end. While at it, report percentage done.
Output before this patch:
* checking 6111 files for package collisions
5111 files remaining ...
4111 files remaining ...
3111 files remaining ...
2111 files remaining ...
files remaining ...
111 files remaining ...
After:
* checking 6158 files for package collisions
16% done, 5131 files remaining ...
33% done, 4104 files remaining ...
50% done, 3077 files remaining ...
66% done, 2050 files remaining ...
83% done, 1023 files remaining ...
100% done
Signed-off-by: Fabian Groffen
---
lib/portage/dbapi/vartree.py | 14 +++---
1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..78f2b37f2 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,19 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+ bucksize = 1000
+ buckcnt = int(totfiles / bucksize)
+ if buckcnt == 0 or totfiles % bucksize > int(bucksize /
2):
+ buckcnt = buckcnt + 1
+ bucksize = int(totfiles / buckcnt) + 1
showMessage(_(" %s checking %d files for package
collisions\n") % \
- (colorize("GOOD", "*"), totfiles))
+ (colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
- if i % 1000 == 0 and i != 0:
- showMessage(_("%d files remaining
...\n") % (totfiles - i))
+ if i % bucksize == 0 and i != 0:
+ showMessage(_("%3d%% done, %d files
remaining ...\n") %
+ (i * 100 / totfiles,
totfiles - i))
dest_path = normalize_path(
os.path.join(destroot,
f.lstrip(os.path.sep)))
@@ -3570,6 +3576,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+ if bucksize < totfiles:
+ showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions,
plib_collisions
def _lstat_inode_map(self, path_iter):
--
2.20.1
[gentoo-portage-dev] [PATCH] lib/portage/dbapi/vartree: use dynamic report interval in _collision_protect
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects. Adjust this interval to
be regular towards the end. While at it, report percentage done.
Output before this patch:
* checking 6111 files for package collisions
5111 files remaining ...
4111 files remaining ...
3111 files remaining ...
2111 files remaining ...
files remaining ...
111 files remaining ...
After:
* checking 6158 files for package collisions
16% done, 5131 files remaining ...
33% done, 4104 files remaining ...
50% done, 3077 files remaining ...
66% done, 2050 files remaining ...
83% done, 1023 files remaining ...
Signed-off-by: Fabian Groffen
---
lib/portage/dbapi/vartree.py | 10 --
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..4d0bf2789 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,19 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+ bucksize = 1000
+ buckcnt = int(totfiles / bucksize)
+ if buckcnt == 0 or totfiles % bucksize > int(bucksize /
2):
+ buckcnt = buckcnt + 1
+ bucksize = int(totfiles / buckcnt) + 1
showMessage(_(" %s checking %d files for package
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
- if i % 1000 == 0 and i != 0:
- showMessage(_("%d files remaining
...\n") % (totfiles - i))
+ if i % bucksize == 0 and i != 0:
+ showMessage(_("%2d%% done, %d files
remaining ...\n") %
+ (i * 100 / totfiles,
totfiles - i))
dest_path = normalize_path(
os.path.join(destroot,
f.lstrip(os.path.sep)))
--
2.20.1
Re: [RFC] gpkg format proposal v2 (was: Re: [gentoo-portage-dev] [RFC] Improving Gentoo package format)
On 11-11-2018 21:53:33 +0100, Michał Górny wrote:
> Hi,
>
> Ok, here's the second version integrating the feedback received.
> The format is much simpler, based on nested tarballs inspired by Debian.
>
> The outer tarball is uncompressed and uses '.gpkg.tar' suffix. It
> contains (preferably in order but PM should also handle packages with
> mismatched order):
>
> 1. Optional (but recommended) "gpkg: ${PF}" package label that can be
> used to quickly distinguish Gentoo binpkgs from regular tarballs
> (for file(1)).
>
> 2. "metadata.tar${comp}" tarball containing binary package metadata
> as files.
>
> 3. Optional "metadata.tar${comp}.sig" containing detached signature
> for the metadata archive.
>
> 4. "contents.tar${comp}" tarball containing files to be installed.
>
> 5. Optional "contents.tar${comp}.sig" containing detached signature for
> the contents archive.
>
> Notes:
>
> a. ${comp} can be any compression format supported by binary packages.
> Technically, metadata and content archives may use different
> compression. Either or both may be uncompressed as well.
I'm wondering here, how much sense does it make to compress 2., 3.
and/or 4. if you compress the whole gpkg? I have the impression
compression on compression isn't beneficial here. Shouldn't just
compressing of the gpkg tar be sufficient?
As to allowing different compressors for a single gpkg, I think it would
be better to require all compressors to be the same, such that a PM or
tool can quickly see if it can "read" the file from the gpkg filename,
instead of having to fetch and open it first. Obviously, if you drop
compression of the inner tars, this point goes away.
Thanks,
Fabian
> b. While signatures are optional, the PM should have a switch
> controlling whether to expect them, and fail hard if they're not present
> when expected.
>
>
> Advantages
> --
> Guaranteed:
>
> + The binary package is still one file, so can be fetched easily.
>
> + File format is trivial and can be extracted using tar(1) + compressor.
>
> + The metadata and contents are compressed independently, and so can be
> easily extracted or modified independently.
>
> + The package format provides for separate metadata and content
> signatures, so they can be verified independently.
>
> + Metadata can be compressed now.
>
> Achieved by regular archives (but might be broken if modified by user):
>
> + Easy recognition by magic(1).
>
> + The metadata archive (and its signature) is packed first, so it may be
> read without fetching the whole binpkg.
>
>
> Why not .ar format?
> ---
> The use of .ar format has been proposed, akin to Debian. While
> the option is mostly feasible, and the simplicity of .ar format would
> reduce the outer size of binary packages, I think the format is simply
> too obscure. It lives mostly as static library format, and the tooling
> for it is part of binutils. LSB considers it deprecated. While I don't
> see it going away anytime soon, I'd rather not rely on it in order to
> save a few KiB.
>
>
> Is there anything left to address?
>
> --
> Best regards,
> Michał Górny
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [PATCH v2] rsync: quarantine data prior to verification (bug 660410)
lg = False
> is_synced = False
> if timestamp != 0 and "--quiet" not in opts:
> @@ -686,6 +752,12 @@ class RsyncSync(NewBase):
> elif (servertimestamp == 0) or (servertimestamp >
> timestamp):
> # actual sync
> command = rsynccommand[:]
> +
> + if self.repo.location != download_dir:
> + # Use shared hardlinks for files that
> are identical
> + # in the previous snapshot of the
> repository.
> + command.append('--link-dest=%s' %
> self.repo.location)
> +
> submodule_paths = self._get_submodule_paths()
> if submodule_paths:
> # The only way to select multiple
> directories to
> @@ -696,9 +768,10 @@ class RsyncSync(NewBase):
> # /./ is special syntax
> supported with the
> # rsync --relative option.
> command.append(syncuri + "/./"
> + path)
> - command.append(self.repo.location)
> else:
> - command.extend([syncuri + "/",
> self.repo.location])
> + command.append(syncuri + "/")
> +
> + command.append(download_dir)
>
> exitcode = None
> try:
> --
> 2.13.6
>
>
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)
On 01-04-2018 20:29:59 +0200, Michał Górny wrote: > > > This essentially looks like ~700 lines of code to try to workaround > > > broken networking. I would rather try to do that using 5 lines of code > > > but that's just me, and my programs aren't enterprise quality. I just > > > hope it actually solves as many problems as it's going to introduce. > > > > The vast majority of this code is generic and reusable, and I do intend > > to reuse it. For example, the executor support will be an essential > > piece for the asyncio.AbstractEventLoop for bug 649588. > > Sure it is and sure you will. But tell me: who is going to maintain it > all? Because as far as I can see, we're still dealing with a bus factor > of one and all you're doing is making it worse. More code, more > complexity, more creeping featurism and hacks. Well, well, well. This could be said about a lot of code, including your own. > Last time you went away, you left us with a horribly unmaintainable > package manager full of complexity, hacks and creeping featurism > and a Portage team whose members had barely any knowledge of the code. > Just when things started moving again, you came back and we're back to > square one. I don't see why this has to be made personal. In the olde days people just pushed whatever they needed, now it's reviewed and acked, so how can it be back to square one? > Today Portage once again is a one-developer project, full of more > complexity, more hacks and more creeping featurism. And we once again > have a bus factor of one -- one developer who apparently knows > everything, does everything and tries to be nice to everyone, except he > really ignores others, makes a lot of empty promises and consistently > makes the health of the project go from bad to worse. Errr, didn't you recently start your own fork with creeping featurism of its own because mainline was/is too slow? > So, please tell me: what happens when you leave again? How have you used > your time in the project? What have you done to make sure that > the project stays alive without you? Because as far as I can see, adding > few thousand of lines of practically unreviewed code every month does > not help with that. Why is this Zac's problem specifically? Isn't this a general problem of Gentoo? And isn't your attitude contributing in a large factor to the bus-factor getting down from 1 to 0? > I forked Portage because I didn't want to fight with you. When I forked > it, I declared that I will merge mainline changes regularly for > the benefit of my users. But after a week, I really start feeling like > that's going to be a really bad idea. Like it's time to forget about > mainline Portage as a completely dead end, and go our separate ways. So you fork. Like many forks, that is because people feel that it isn't going "their way". Yay. That doesn't make you "right". > I'm seriously worried about the future of Gentoo. I'd really appreciate > if you started focusing on that as well. I get that all this stuff looks > cool on paper but few months or years from now, someone will curse > 'whoever wrote that code' while trying to fix some nasty bug. Or get > things moving forward. Or implement EAPI 8. Perhaps it's time to look into the mirror. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH v2] __dyn_install: improve reporting of build and image sizes
On 27-08-2017 12:53:23 -0700, Zac Medico wrote: > On 08/27/2017 08:06 AM, Fabian Groffen wrote: > > Prior to this commit, the reported sizes would look like: > > > > * Final size of build directory: 34942 KiB > > * Final size of installed tree: 5627 KiB > > > > Because the sizes aren't aligned, it is hard to (visually) compare them. > > On top of this, because the numbers are sometimes bigger, print a human > > friendly size after the KiB size if applicable, like so: > > > > * Final size of build directory: 1906 KiB (1.8 MiB) > > * Final size of installed tree: 7 KiB > > > > It should be noted that in case both sizes have a human-readable > > variant, they are also aligned. > > > > The helper functions are defined and used in a subshell to avoid > > pollution of the caller's environment. > > --- > > bin/phase-functions.sh | 53 > > ++++++---- > > 1 file changed, 49 insertions(+), 4 deletions(-) > > Looks good. Please merge. Pushed, thanks! Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
[gentoo-portage-dev] [PATCH v2] __dyn_install: improve reporting of build and image sizes
Prior to this commit, the reported sizes would look like:
* Final size of build directory: 34942 KiB
* Final size of installed tree: 5627 KiB
Because the sizes aren't aligned, it is hard to (visually) compare them.
On top of this, because the numbers are sometimes bigger, print a human
friendly size after the KiB size if applicable, like so:
* Final size of build directory: 1906 KiB (1.8 MiB)
* Final size of installed tree: 7 KiB
It should be noted that in case both sizes have a human-readable
variant, they are also aligned.
The helper functions are defined and used in a subshell to avoid
pollution of the caller's environment.
---
bin/phase-functions.sh | 53 ++
1 file changed, 49 insertions(+), 4 deletions(-)
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index dfd8733c8..ce174ba91 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -598,10 +598,55 @@ __dyn_install() {
# record build & installed size in build log
if type -P du &>/dev/null; then
- local sz=( $(du -ks "${WORKDIR}") )
- einfo "Final size of build directory: ${sz[0]} KiB"
- sz=( $(du -ks "${D}") )
- einfo "Final size of installed tree: ${sz[0]} KiB"
+ local nsz=( $(du -ks "${WORKDIR}") )
+ local isz=( $(du -ks "${D}") )
+
+ # subshell to avoid polluting the caller env with the helper
+ # functions below
+ (
+ # align $1 to the right to the width of the widest of
$1 and $2
+ padl() {
+ local s1=$1
+ local s2=$2
+ local width=${#s1}
+ [[ ${#s2} -gt ${width} ]] && width=${#s2}
+ printf "%*s" ${width} "${s1}"
+ }
+
+ # transform number in KiB into MiB, GiB or TiB based on
size
+ human() {
+ local s1=$1
+ local units=( KiB MiB GiB TiB )
+
+ s1=$((s1 * 10))
+ while [[ ${s1} -gt 10240 && ${#units[@]} -gt 1
]] ; do
+ s1=$((s1 / 1024 ))
+ units=( ${units[@]:1} )
+ done
+
+ local r=${s1: -1}
+ s1=$((s1 / 10))
+ printf "%s.%s %s" "${s1}" "${r}" "${units[0]}"
+ }
+
+ size() {
+ local s1=$1
+ local s2=$2
+ local out="$(padl "${s1}" "${s2}") KiB"
+
+ if [[ ${s1} -gt 1024 ]] ; then
+ s1=$(human ${s1})
+ if [[ ${s2} -gt 1024 ]] ; then
+ s2=$(human ${s2})
+ s1=$(padl ${s1} ${s2})
+ fi
+ out+=" (${s1})"
+ fi
+ echo "${out}"
+ }
+ einfo "Final size of build directory: $(size ${nsz[0]}
${isz[0]})"
+ einfo "Final size of installed tree: $(size ${isz[0]}
${nsz[0]})"
+ )
__vecho
fi
--
2.14.1
Re: [gentoo-portage-dev] [PATCH] _collision_protect: report progress in work todo
On 26-08-2017 14:46:30 -0700, Zac Medico wrote:
> On 08/24/2017 06:28 AM, Fabian Groffen wrote:
> > Currently Portage reports its progress in checking collisions forward
> > every 1000th file like so:
> >
> > * checking 4149 files for package collisions
> > 1000 files checked ...
> > 2000 files checked ...
> > 3000 files checked ...
> > 4000 files checked ...
> >>>> Merging sys-apps/portage-2.3.8 to /
> >
> > Change it to countdown style so it is easier to anticipate what the
> > next action will be:
> >
> > * checking 4149 files for package collisions
> > 3149 files remaining ...
> > 2149 files remaining ...
> > 1149 files remaining ...
> > 149 files remaining ...
> >>>> Merging sys-apps/portage-2.3.8 to /
> > ---
> > pym/portage/dbapi/vartree.py | 5 +++--
> > 1 file changed, 3 insertions(+), 2 deletions(-)
> >
> > diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
> > index 7c8f150bb..04a40b732 100644
> > --- a/pym/portage/dbapi/vartree.py
> > +++ b/pym/portage/dbapi/vartree.py
> > @@ -3420,13 +3420,14 @@ class dblink(object):
> > dirs_ro = set()
> > symlink_collisions = []
> > destroot = self.settings['ROOT']
> > + totfiles = len(file_list) + len(symlink_list)
> > showMessage(_(" %s checking %d files for package
> > collisions\n") % \
> > - (colorize("GOOD", "*"), len(file_list) +
> > len(symlink_list)))
> > + (colorize("GOOD", "*"), totfiles))
> > for i, (f, f_type) in enumerate(chain(
> > ((f, "reg") for f in file_list),
> > ((f, "sym") for f in symlink_list))):
> > if i % 1000 == 0 and i != 0:
> > - showMessage(_("%d files checked ...\n")
> > % i)
> > + showMessage(_("%d files remaining
> > ...\n") % (totfiles - i))
> >
> > dest_path = normalize_path(
> > os.path.join(destroot,
> > f.lstrip(os.path.sep)))
> >
>
> Looks good. Please merge.
Pushed, thanks
Fabian
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: Digital signature
[gentoo-portage-dev] [PATCH] _collision_protect: report progress in work todo
Currently Portage reports its progress in checking collisions forward
every 1000th file like so:
* checking 4149 files for package collisions
1000 files checked ...
2000 files checked ...
3000 files checked ...
4000 files checked ...
>>> Merging sys-apps/portage-2.3.8 to /
Change it to countdown style so it is easier to anticipate what the
next action will be:
* checking 4149 files for package collisions
3149 files remaining ...
2149 files remaining ...
1149 files remaining ...
149 files remaining ...
>>> Merging sys-apps/portage-2.3.8 to /
---
pym/portage/dbapi/vartree.py | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
index 7c8f150bb..04a40b732 100644
--- a/pym/portage/dbapi/vartree.py
+++ b/pym/portage/dbapi/vartree.py
@@ -3420,13 +3420,14 @@ class dblink(object):
dirs_ro = set()
symlink_collisions = []
destroot = self.settings['ROOT']
+ totfiles = len(file_list) + len(symlink_list)
showMessage(_(" %s checking %d files for package
collisions\n") % \
- (colorize("GOOD", "*"), len(file_list) +
len(symlink_list)))
+ (colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
if i % 1000 == 0 and i != 0:
- showMessage(_("%d files checked ...\n")
% i)
+ showMessage(_("%d files remaining
...\n") % (totfiles - i))
dest_path = normalize_path(
os.path.join(destroot,
f.lstrip(os.path.sep)))
--
2.14.1
[gentoo-portage-dev] [PATCH] __dyn_install: improve reporting of build and image sizes
Prior to this commit, the reported sizes would look like:
* Final size of build directory: 34942 KiB
* Final size of installed tree: 5627 KiB
Because the sizes aren't aligned, it is hard to (visually) compare them.
On top of this, because the numbers are sometimes bigger, print a human
friendly size after the KiB size if applicable, like so:
* Final size of build directory: 1906 KiB (1.8 MiB)
* Final size of installed tree: 7 KiB
It should be noted that in case both sizes have a human-readable
variant, they are also aligned.
---
bin/phase-functions.sh | 49 +
1 file changed, 45 insertions(+), 4 deletions(-)
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index dfd8733c8..af45a0d49 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -598,10 +598,51 @@ __dyn_install() {
# record build & installed size in build log
if type -P du &>/dev/null; then
- local sz=( $(du -ks "${WORKDIR}") )
- einfo "Final size of build directory: ${sz[0]} KiB"
- sz=( $(du -ks "${D}") )
- einfo "Final size of installed tree: ${sz[0]} KiB"
+ local nsz=( $(du -ks "${WORKDIR}") )
+ local isz=( $(du -ks "${D}") )
+
+ # align $1 to the right to the width of the widest of $1 and $2
+ padl() {
+ local s1=$1
+ local s2=$2
+ local width=${#s1}
+ [[ ${#s2} -gt ${width} ]] && width=${#s2}
+ printf "%*s" ${width} "${s1}"
+ }
+
+ # transform number in KiB into MiB, GiB or TiB based on size
+ human() {
+ local s1=$1
+ local units=( KiB MiB GiB TiB )
+
+ s1=$((s1 * 10))
+ while [[ ${s1} -gt 10240 && ${#units[@]} -gt 1 ]] ; do
+ s1=$((s1 / 1024 ))
+ units=( ${units[@]:1} )
+ done
+
+ local r=${s1: -1}
+ s1=$((s1 / 10))
+ printf "%s.%s %s" "${s1}" "${r}" "${units[0]}"
+ }
+
+ size() {
+ local s1=$1
+ local s2=$2
+ local out="$(padl "${s1}" "${s2}") KiB"
+
+ if [[ ${s1} -gt 1024 ]] ; then
+ s1=$(human ${s1})
+ if [[ ${s2} -gt 1024 ]] ; then
+ s2=$(human ${s2})
+ s1=$(padl ${s1} ${s2})
+ fi
+ out+=" (${s1})"
+ fi
+ echo "${out}"
+ }
+ einfo "Final size of build directory: $(size ${nsz[0]}
${isz[0]})"
+ einfo "Final size of installed tree: $(size ${isz[0]}
${nsz[0]})"
__vecho
fi
--
2.14.1
[gentoo-portage-dev] [PATCH] etc-update: fix hang when using_editor is set, bug #544440
Don't try to use an interactive editor to obtain the difference between
files. Fall back to plain `diff -q` in this case.
---
bin/etc-update | 19 +++
1 file changed, 15 insertions(+), 4 deletions(-)
diff --git a/bin/etc-update b/bin/etc-update
index e0f7224dc..ea69f1478 100755
--- a/bin/etc-update
+++ b/bin/etc-update
@@ -649,10 +649,21 @@ do_distconf() {
elif [[ -L ${efile} || -L ${file} ]] ; then
# not the same file types
continue
- elif diff_command "${file}" "${efile}" &> /dev/null; then
- # replace identical copy
- mv "${file}" "${efile}"
- break
+ else
+ local ret=
+ if [[ ${using_editor} == 0 ]] ; then
+ diff_command "${file}" "${efile}" &> /dev/null
+ ret=$?
+ else
+ # fall back to plain diff
+ diff -q "${file}" "${efile}" &> /dev/null
+ ret=$?
+ fi
+ if [[ ${ret} == 0 ]] ; then
+ # replace identical copy
+ mv "${file}" "${efile}"
+ break
+ fi
fi
done
}
--
2.11.1
[gentoo-portage-dev] [PATCH] getgccversion: check for clang if gcc is not found
In Gentoo Prefix on Mac OS X systems, LLVM/clang is used as the
toolchain, so no gcc/gcc-config around. Probe for clang after we failed
to locate gcc, so we can show clang version instead.
---
pym/_emerge/actions.py | 39 ++-
1 file changed, 38 insertions(+), 1 deletion(-)
diff --git a/pym/_emerge/actions.py b/pym/_emerge/actions.py
index 59626ad..1d324aa 100644
--- a/pym/_emerge/actions.py
+++ b/pym/_emerge/actions.py
@@ -39,6 +39,7 @@ from portage import os
from portage import shutil
from portage import eapi_is_supported, _encodings, _unicode_decode
from portage.cache.cache_errors import CacheError
+from portage.const import EPREFIX
from portage.const import GLOBAL_CONFIG_PATH, VCS_DIRS,
_DEPCLEAN_LIB_CHECK_DEFAULT
from portage.const import SUPPORTED_BINPKG_FORMATS, TIMESTAMP_FORMAT
from portage.dbapi.dep_expand import dep_expand
@@ -2406,12 +2407,23 @@ def getgccversion(chost=None):
gcc_ver_command = ['gcc', '-dumpversion']
gcc_ver_prefix = 'gcc-'
+ clang_ver_command = ['clang', '--version']
+ clang_ver_prefix = 'clang-'
+
+ ubinpath = os.path.join('/', portage.const.EPREFIX, 'usr', 'bin')
+
gcc_not_found_error = red(
"!!! No gcc found. You probably need to 'source /etc/profile'\n" +
"!!! to update the environment of this terminal and possibly\n" +
"!!! other terminals also.\n"
)
+ def getclangversion(output):
+ version = re.search('clang version ([0-9.]+) ', output)
+ if version:
+ return version.group(1)
+ return "unknown"
+
if chost:
try:
proc = subprocess.Popen(["gcc-config", "-c"],
@@ -2439,7 +2451,7 @@ def getgccversion(chost=None):
return gcc_ver_prefix + myoutput
try:
- proc = subprocess.Popen(gcc_ver_command,
+ proc = subprocess.Popen([ubinpath + "/" + gcc_ver_command[0]] +
gcc_ver_command[1:],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
except OSError:
myoutput = None
@@ -2450,6 +2462,31 @@ def getgccversion(chost=None):
if mystatus == os.EX_OK:
return gcc_ver_prefix + myoutput
+ try:
+ proc = subprocess.Popen(
+ [ubinpath + "/" + chost + "-" + clang_ver_command[0]] +
clang_ver_command[1:],
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+ except OSError:
+ myoutput = None
+ mystatus = 1
+ else:
+ myoutput = _unicode_decode(proc.communicate()[0]).rstrip("\n")
+ mystatus = proc.wait()
+ if mystatus == os.EX_OK:
+ return clang_ver_prefix + getclangversion(myoutput)
+
+ try:
+ proc = subprocess.Popen([ubinpath + "/" + clang_ver_command[0]]
+ clang_ver_command[1:],
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+ except OSError:
+ myoutput = None
+ mystatus = 1
+ else:
+ myoutput = _unicode_decode(proc.communicate()[0]).rstrip("\n")
+ mystatus = proc.wait()
+ if mystatus == os.EX_OK:
+ return clang_ver_prefix + getclangversion(myoutput)
+
portage.writemsg(gcc_not_found_error, noiselevel=-1)
return "[unavailable]"
--
2.7.0
[gentoo-portage-dev] [PATCH] consider clang when looking up compiler version
This is a patch from prefix branch to make the compiler detection consider clang. Fabian Groffen (1): getgccversion: check for clang if gcc is not found pym/_emerge/actions.py | 39 ++- 1 file changed, 38 insertions(+), 1 deletion(-) -- 2.7.0
Re: [gentoo-portage-dev] [PATCH 3/3] Add CPU model name to output of getportageversion as fifth element
On 14-08-2013 22:10:40 +0200, Mark Kubacki wrote: From: W-Mark Kubacki wm...@hurrikane.de It will read like this: Portage 2.1.13.7 (default/linux/amd64/13.0, gcc-4.6.2, glibc-2.18, 3.9.0-rc8-mark-signed+ x86_64, Intel(R) Core(TM) i7-3770T CPU @ 2.50GHz) That new fifth element will be the CPU model name of the host running Portage. It is not the target architecture! FYI: % python3 Python 3.3.2 (default, Jul 24 2013, 11:14:02) [GCC 4.2.1 (Gentoo 4.2.1_p5666-r1, Apple Inc. build 5666) (dot 3)] on darwin Type help, copyright, credits or license for more information. import platform platform.release() '12.4.0' platform.machine() 'x86_64' platform.processor() 'i386' % python Python 3.2.3 (default, May 6 2013, 21:19:05) [GCC 4.7.2] on sunos5 Type help, copyright, credits or license for more information. import platform platform.release() '5.11' platform.machine() 'i86pc' platform.processor() 'i386' % python Python 3.3.2 (default, Jul 15 2013, 13:51:24) [GCC 4.7.2] on sunos5 Type help, copyright, credits or license for more information. import platform platform.release() '5.10' platform.machine() 'sun4u' platform.processor() 'sparc' % python Python 3.2.5 (default, Jul 15 2013, 11:37:08) [GCC 4.6.3] on linux2 Type help, copyright, credits or license for more information. import platform platform.release() '3.8.13-gentoo' platform.machine() 'x86_64' platform.processor() 'AMD Athlon(tm) 64 X2 Dual Core Processor 3800+' e.g. it seems to me only on Linux it gives fancy model output. Note that the first and second system were running a 64-bit Python as well as kernel. Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH 0/3] Add cross-prefix support
Hi Ruud, On 17-06-2013 09:21:41 +0200, Ruud Koolen wrote: This patch series adds support for using a portage installed in one prefix to build packages with a different prefix. Thanks so much for your work. I hope to have a look at this soon, and merge this. From your work I assume your patches don't break normal installs/bootstraps, right? (You tested that?) Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
[gentoo-portage-dev] Re: proj/portage:master commit in: man/, pym/portage/dbapi/, pym/portage/, cnf/
On 25-10-2012 08:45:36 +, Zac Medico wrote: URL: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=10b6d012 Add FEATURES=merge-sync, for bug #439584. + if not self._device_path_map or \ + merge-sync not in self.settings.features: + return + + syncfs = self._get_syncfs() + if syncfs is None: + try: + proc = subprocess.Popen([sync]) + except EnvironmentError: + pass + else: + proc.wait() + else: + for path in self._device_path_map.values(): + try: + fd = os.open(path, os.O_RDONLY) + except OSError: + pass + else: + try: + syncfs(fd) can't you use os.fdatasync() or os.fsync() instead? -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-portage-dev] Re: proj/portage:master commit in: man/, pym/portage/dbapi/, pym/portage/, cnf/
On 25-10-2012 02:45:41 -0700, Zac Medico wrote: can't you use os.fdatasync() or os.fsync() instead? os.fdatasync() or os.fsync() only work for single files, right? We don't want to sync them individually, since that wastes disk bandwidth. It's much more efficient to sync the whole filesystem at once, like syncfs does. You use syncfs, which according to the man-page requires a filedescriptor. Hence my suggestion. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-portage-dev] About adding a repoman check to test for missing files in FILESDIR
On 08-10-2012 20:47:41 +0200, Pacho Ramos wrote:
El lun, 08-10-2012 a las 11:32 -0700, Zac Medico escribió:
Given the expressiveness of the bash language, it's really a non-trivial
thing to check without actually executing bash. See discussion here:
https://bugs.gentoo.org/show_bug.cgi?id=431196
Then, maybe I could have a custom repoman script that would check for
them in my files/ directory (either using bash to check for them or
simply running ls) :/
Yes, but still it is very hard to make it 100% reliable. If you stick
to simple ${FILESDIR}/${PN}-x.y.patch patterns, it's very well
feasible, though. See for inspiration the fetch_files() function from [1].
[1]
http://overlays.gentoo.org/proj/alt/browser/trunk/prefix-overlay/scripts/eupdate.in
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: Digital signature
[gentoo-portage-dev] Re: proj/portage:master commit in: pym/portage/dbapi/
On 26-11-2011 16:56:41 +0530, Nirbheek Chauhan wrote: On Sat, Nov 26, 2011 at 4:28 PM, Fabian Groffen grob...@gentoo.org wrote: On 26-11-2011 01:54:35 +, Arfrever Frehtes Taifersar Arahesis wrote: commit: 1d4ac47c28706094230cb2c4e6ee1c1c71629aa0 T Org AuthorDate: Sat Nov 26 01:52:49 2011 + Commit: Arfrever Frehtes Taifersar Arahesis arfrever AT gentoo DOT org CommitDate: Sat Nov 26 01:52:49 2011 + URL: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=1d4ac47c dblink.mergeme(): Merge files in alphabetic order. What's the advantage of this? I don't really like to pay for sorting a potentially huge list just for some eye-candy. (That's omitted by default these days anyway...) Any other opinions on this one? If it should be sorted[1], it should really be sorted in the reverse order of distfile-download size. That would be extremely useful on systems with slow internet connections. Too many times have I sat waiting for libreoffice-bin to download while a webkit-gtk recompile waits in the queue. We already have the information during dependency resolution with --verbose, and it costs very little. Besides, sorting even 30,000 entries (if you're merging every ebuild in portage) should not take more than a few secs. A linux kernel has around that much of files, and I really wonder if it's worth waiting a couple of seconds (probably more on sparc and arm systems) just because then the files are in sorted order. 1. I'm obviously assuming that dep nodes that do not depend on each other would be sorted I think this is per package. I didn't watch closely enough the reply-to headers, the gentoo-portage-dev list was my original target, which obviously makes more sense for this context. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
[gentoo-portage-dev] Re: proj/portage:master commit in: pym/repoman/, pym/portage/tests/repoman/
On 21-10-2011 06:09:57 +, Zac Medico wrote: UpdateChangeLog: split out/test copyright regex This also fixes a case where something like Copyright 2011 would be replaced with Copyright 2011-2011 . I like this, but it should only happen when one would live in 1999 (that's why I added it unconditionally), see below. I'm affraid we need the 1999 to be configured, for legal reasons (though I don't claim to be an expert on this case, just judging from the comments from echangelog). + def testCopyrightUpdate(self): + test_cases = ( + ( + '2011', + '# Copyright 1999-2008 Gentoo Foundation; Distributed under the GPL v2', + '# Copyright 1999-2011 Gentoo Foundation; Distributed under the GPL v2', we should add a case like # Copyright 2010-2011 Gentoo Foundation; Distributed ... echangelog would have rewritten it to 1999-2011, because of legal reasons IIRC. +def _update_copyright_year(year, line): + + These two regexes are taken from echangelog + update_copyright(), except that we don't hardcode + 1999 here (in order to be more generic). This comment makes little to no more sense, since the patterns taken from echangelog have changed considerably now :) -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-portage-dev] Re: proj/portage:master commit in: pym/repoman/, pym/portage/tests/repoman/
On 21-10-2011 00:30:05 -0700, Zac Medico wrote: On 10/21/2011 12:09 AM, Fabian Groffen wrote: On 21-10-2011 06:09:57 +, Zac Medico wrote: UpdateChangeLog: split out/test copyright regex This also fixes a case where something like Copyright 2011 would be replaced with Copyright 2011-2011 . I like this, but it should only happen when one would live in 1999 (that's why I added it unconditionally), see below. I'm affraid we need the 1999 to be configured, for legal reasons (though I don't claim to be an expert on this case, just judging from the comments from echangelog). I don't see anything like that in echangelog. What would be the point of re-writing copyright start dates, when those should already be correctly inherited from skel.ChangeLog anyway? Sure, but this is mainly intended for ebuilds and files (like init scripts?), I guess. I considered using skel.ebuild for .ebuild files, perhaps that's just the best to avoid hardcoding any years. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH 2 of 3] repoman: get ChangeLog header from skel.ChangeLog
On 19-10-2011 14:58:39 -0700, Zac Medico wrote: On 10/19/2011 12:55 PM, Fabian Groffen wrote: + if clold_lines[-1].strip(): + f.write(clold_lines[-1]) If the old ChangeLog happens to be an empty file, then clold_lines[-1] will raise IndexError. So, this is safer: if clold_lines and clold_lines[-1].strip(): I applied your fix, and moved it from the other patch in this one, where it belongs. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
[gentoo-portage-dev] [PATCH] repoman: update copyright on modified files
To retain the behaviour of echangelog, update the copyrights on modified
files (mostly ebuilds) when necessary. Also update the ChangeLog's
copyright.
diff --git a/pym/repoman/utilities.py b/pym/repoman/utilities.py
--- a/pym/repoman/utilities.py
+++ b/pym/repoman/utilities.py
@@ -523,9 +523,77 @@
return outvcs
+def update_copyright(fn_path, year, pretend):
+
+ Check file for a Copyright statement, and update its year. The
+ patterns used for replacing copyrights are taken from echangelog.
+ Only the first lines of each file that start with a hash ('#') are
+ considered, until a line is found that doesn't start with a hash.
+
+
+ try:
+ fn_hdl = io.open(_unicode_encode(fn_path,
+ encoding=_encodings['fs'], errors='strict'),
+ mode='r', encoding=_encodings['repo.content'],
errors='replace')
+ except EnvironmentError:
+ return
+
+ orig_header = []
+ new_header = []
+
+ for line in fn_hdl:
+ line_strip = line.strip()
+ orig_header.append(line)
+ if not line_strip or line_strip[:1] != '#':
+ new_header.append(line)
+ break
+
+ # these two regexes are taken from
+ # echangelog update_copyright()
+ line = re.sub(r'^(# Copyright \d+) ',
+ r'\1-%s ' % year, line)
+ line = re.sub(r'^(# Copyright) \d\d\d\d-\d\d\d\d',
+ r'\1 1999-%s' % year, line)
+ new_header.append(line)
+
+ difflines = 0
+ for line in difflib.unified_diff(orig_header, new_header,
+ fromfile=fn_path, tofile=fn_path, n=0):
+ util.writemsg_stdout(line, noiselevel=-1)
+ difflines += 1
+ util.writemsg_stdout(\n, noiselevel=-1)
+
+ # unified diff has three lines to start with
+ if difflines 3 and not pretend:
+ # write new file with changed header
+ f, fnnew_path = mkstemp()
+ f = io.open(f, mode='w', encoding=_encodings['repo.content'],
+ errors='backslashreplace')
+ for line in new_header:
+ f.write(line);
+ for line in fn_hdl:
+ f.write(line)
+ f.close()
+ try:
+ fn_stat = os.stat(fn_path)
+ except OSError:
+ fn_stat = None
+
+ shutil.move(fnnew_path, fn_path)
+
+ if fn_stat is None:
+ util.apply_permissions(fn_path, mode=0o644)
+ else:
+ util.apply_stat_permissions(fn_path, fn_stat)
+ fn_hdl.close()
+
def UpdateChangeLog(pkgdir, category, package, new, removed, changed, \
msg, pretend, repodir):
-Write an entry to an existing ChangeLog, or create a new one.
+
+ Write an entry to an existing ChangeLog, or create a new one.
+ Updates copyright year on changed files, and updates the header of
+ ChangeLog with the contents of skel.ChangeLog.
+
# figure out who to write as
if 'GENTOO_COMMITTER_NAME' in os.environ and \
@@ -548,6 +616,16 @@
logging.critical(err)
return None
+ year = time.strftime('%Y')
+ date = time.strftime('%d %b %Y')
+
+ # check modified files and the ChangeLog for copyright updates
+ # patches and diffs (identified by .patch and .diff) are excluded
+ for fn in new + changed:
+ if fn.endswith('.diff') or fn.endswith('.patch'):
+ continue
+ update_copyright(os.path.join(pkgdir, fn), year, pretend)
+
cl_path = os.path.join(pkgdir, 'ChangeLog')
clold_lines = []
clnew_lines = []
@@ -584,8 +662,9 @@
clold_lines.append(line)
if line_strip[:1] != '#':
break
- if clskel_file is None:
- clnew_lines.append(line)
+ line = re.sub(r'^(# Copyright)
\d\d\d\d-\d\d\d\d',
+ r'\1 1999-%s' % year, line)
+ clnew_lines.append(line)
if not line_strip:
break
elif clskel_file is not None:
@@ -597,7 +676,7 @@
line = line.replace('CATEGORY', category)
line = line.replace('PACKAGE_NAME', package)
line = re.sub(r'^(# Copyright
\d\d\d\d)-\d\d\d\d ',
- r'\1-%s ' % time.strftime('%Y'), line)
+
[gentoo-portage-dev] [PATCH 3 of 3] repoman: update copyright on modified files
To retain the behaviour of echangelog, update the copyrights on modified
files (mostly ebuilds) when necessary. We also use this to update the
ChangeLog's copyright.
diff --git a/pym/repoman/utilities.py b/pym/repoman/utilities.py
--- a/pym/repoman/utilities.py
+++ b/pym/repoman/utilities.py
@@ -523,9 +523,84 @@
return outvcs
+def update_copyrights(pkgdir, files, year, pretend):
+
+ Check list of files for Copyright statements, and update them to
+ year. The patterns used for replacing copyrights are taken from
+ echangelog. Only the first lines of each file that start with a
+ hash ('#') are considered, until a line is found that doesn't start
+ with a hash.
+ patches and diffs (identified by .patch and .diff) are excluded
+
+
+ for fn in files:
+ if fn.endswith('.diff') or fn.endswith('.patch'):
+ continue
+
+ fn_path = os.path.join(pkgdir, fn)
+ try:
+ fn_hdl = io.open(_unicode_encode(fn_path,
+ encoding=_encodings['fs'], errors='strict'),
+ mode='r', encoding=_encodings['repo.content'],
errors='replace')
+ except EnvironmentError:
+ continue
+
+ orig_header = []
+ new_header = []
+
+ for line in fn_hdl:
+ line_strip = line.strip()
+ orig_header.append(line)
+ if not line_strip or line_strip[:1] != '#':
+ new_header.append(line)
+ break
+
+ # these two regexes are taken from
+ # echangelog update_copyright()
+ line = re.sub(r'^(# Copyright \d+) ',
+ r'\1-%s ' % year, line)
+ line = re.sub(r'^(# Copyright) \d\d\d\d-\d\d\d\d',
+ r'\1 1999-%s' % year, line)
+ new_header.append(line)
+
+ difflines = 0
+ for line in difflib.unified_diff(orig_header, new_header,
+ fromfile=fn_path, tofile=fn_path, n=0):
+ util.writemsg_stdout(line, noiselevel=-1)
+ difflines += 1
+ util.writemsg_stdout(\n, noiselevel=-1)
+
+ # unified diff has three lines to start with
+ if difflines 3 and not pretend:
+ # write new file with changed header
+ f, fnnew_path = mkstemp()
+ f = io.open(f, mode='w',
encoding=_encodings['repo.content'],
+ errors='backslashreplace')
+ for line in new_header:
+ f.write(line);
+ for line in fn_hdl:
+ f.write(line)
+ f.close()
+ try:
+ fn_stat = os.stat(fn_path)
+ except OSError:
+ fn_stat = None
+
+ shutil.move(fnnew_path, fn_path)
+
+ if fn_stat is None:
+ util.apply_permissions(fn_path, mode=0o644)
+ else:
+ util.apply_stat_permissions(fn_path, fn_stat)
+ fn_hdl.close()
+
def UpdateChangeLog(pkgdir, category, package, new, removed, changed, \
msg, pretend, repodir):
-Write an entry to an existing ChangeLog, or create a new one.
+
+ Write an entry to an existing ChangeLog, or create a new one.
+ Updates copyright year on changed files, and updates the header of
+ ChangeLog with the contents of skel.ChangeLog.
+
# figure out who to write as
if 'GENTOO_COMMITTER_NAME' in os.environ and \
@@ -548,6 +623,12 @@
logging.critical(err)
return None
+ year = time.strftime('%Y')
+ date = time.strftime('%d %b %Y')
+
+ # check modified files and the ChangeLog for copyright updates
+ update_copyrights(pkgdir, new + changed + ['ChangeLog'], year, pretend)
+
cl_path = os.path.join(pkgdir, 'ChangeLog')
clold_lines = []
clnew_lines = []
@@ -578,7 +659,8 @@
# construct correct header first
try:
if clold_file is not None:
- # retain header from old ChangeLog
+ # retain header from old ChangeLog, its copyright has
+ # already been updated by update_copyrights
for line in clold_file:
line_strip = line.strip()
clold_lines.append(line)
@@ -597,7 +679,7 @@
line =
[gentoo-portage-dev] [PATCH 0 of 3] repoman changelog patches
The following three patches change repoman's echangelog feature slightly: 1) default for updating the changelog is retrieved from layout.conf 2) skel.ChangeLog is used for constructing the header of new ChangeLogs 3) modified files are checked for copyright years to be updated Please review before I push them.
Re: [gentoo-portage-dev] [PATCH 2/2] prepstrip: add support for elfutils strip
On 11-10-2011 10:18:16 -0400, Mike Frysinger wrote: For the sake of it, can we try to detect GNU binutils here? Our profiles restrict strip, so no immediate problem here if not. adding detection here will fix most, but i don't think all, of your troubles the code still unconditionally runs `strip -g` which i'm guessing won't work for you ... also, we only run strip on ELF files. so if you aren't an ELF system, even fixing the -g won't help. Actually, on Solaris (ELF) the way Portage uses strip will damage the files in such a way that they become unusable. And we're using GNU binutils there. So I guess it's ok to assume here. In the profiles stripping is just disabled everywhere it breaks. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-portage-dev] EbuildProcess logs poll-error to already removed $T (on AIX)
() File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/PackageMerge.py, line 43, in _start self.returncode = self.merge.merge() File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/MergeListItem.py, line 147, in merge retval = self._install_task.install() File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/EbuildBuild.py, line 348, in install rval = merge.execute() File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/EbuildMerge.py, line 26, in execute blockers=self.find_blockers) File /big5tk/local/gprefix/usr/lib/portage/pym/portage/dbapi/vartree.py, line 4064, in merge mydbapi=mydbapi, prev_mtimes=prev_mtimes) File /big5tk/local/gprefix/usr/lib/portage/pym/portage/dbapi/vartree.py, line 3915, in merge mydbapi=mydbapi, prev_mtimes=prev_mtimes) File /big5tk/local/gprefix/usr/lib/portage/pym/portage/dbapi/vartree.py, line 3936, in _merge cleanup=cleanup, mydbapi=mydbapi, prev_mtimes=prev_mtimes) File /big5tk/local/gprefix/usr/lib/portage/pym/portage/dbapi/vartree.py, line 3341, in treewalk others_in_slot=others_in_slot) File /big5tk/local/gprefix/usr/lib/portage/pym/portage/dbapi/vartree.py, line 1700, in unmerge myebuildpath, cleanrm) File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/Scheduler.py, line 737, in _dblink_ebuild_phase ebuild_phase.wait() File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AsynchronousTask.py, line 40, in wait self._wait() File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/CompositeTask.py, line 72, in _wait task.wait() File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AsynchronousTask.py, line 40, in wait self._wait() File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/SubProcess.py, line 76, in _wait self.scheduler.schedule(self._reg_id) File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/PollScheduler.py, line 286, in _schedule_wait handler(f, event) File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/SpawnProcess.py, line 224, in _dummy_handler self._unregister_if_appropriate(event) File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AbstractPollTask.py, line 56, in _unregister_if_appropriate self._log_poll_exception(event) File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AbstractEbuildProcess.py, line 234, in _log_poll_exception (self.__class__.__name__, event,)]) File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AbstractEbuildProcess.py, line 229, in _elog log_path=self.settings.get(PORTAGE_LOG_FILE)) File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/PollScheduler.py, line 327, in _task_output mode='ab') IOError: [Errno 2] No such file or directory: '/big5tk/tmp/gprefix/portage/._unmerge_/app-misc/mime-types-8/temp/build.log' -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] Portage on AIX
Hi, On 24-11-2010 12:17:12 -0600, Perry Smith wrote: I don't use Gentoo but I bumped into Portage on a recent quest. Long story short, I'd like to try and port it to AIX. I'm writing this list to see if you think that is possible or not. We use Portage Prefix on AIX, so I think we can say it works. Do you really need Portage (the package manager) or the entire set of applications provided by it? -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] Portage on AIX
On 24-11-2010 12:29:41 -0600, Perry Smith wrote: On Nov 24, 2010, at 12:22 PM, Fabian Groffen wrote: Hi, On 24-11-2010 12:17:12 -0600, Perry Smith wrote: I don't use Gentoo but I bumped into Portage on a recent quest. Long story short, I'd like to try and port it to AIX. I'm writing this list to see if you think that is possible or not. We use Portage Prefix on AIX, so I think we can say it works. Do you really need Portage (the package manager) or the entire set of applications provided by it? I'm assuming the latter does not exist. I've love to just say install ruby right now and have it work but that isn't what I'm expecting. I'm hoping, over time, to provide some of those packages over time. I hope I'm making sense here... Gentoo Prefix[1] on AIX is sort of that. Not sure if ruby is ported yet, but at least the base system (the core you need to get Portage sanely running, and being able to emerge other stuff + a handful of indispensible tools) compiles. You'll find people on the gentoo-alt list[2] that maintain the AIX port. [1] http://prefix.gentoo.org/ [2] http://archives.gentoo.org/gentoo-alt/ -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] [API] First steps for creating an API for portage
On 18-06-2010 02:08:04 +0200, René 'Necoro' Neumann wrote: In parallel (or thereafter), we build the C-bindings. The API for these bindings probably look different -- but I guess they should be implemented in terms of the library created above. By example: - Operation: get the list of categories - Python-API: portage.api.categories() : Category list - Implementation: def categories(): return - C-API: category * categories() - C-Implementation: some wrapper around portage.api.categories If you want to deliver a C implementation, I'd wonder why you wouldn't just make a full C implementation and create Python wrappers for them instead of the other way around. Might accidentially speed up Portage, and make tools like portage-utils happy. -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] [API] First steps for creating an API for portage
On 18-06-2010 01:27:57 -0700, Brian Dolbec wrote: Unless I'm way off the mark, your suggesting to re-code portage into C then create a python interface to it. Others have started such a project, but none have completed aside from paludis (C++). It's actually on my wishlist for a long time, just because Python is an extreme pain in my environments, and given its current maintainer it is going to be an extreme pain for a long time as well. If there are other apps that would like to get info from portage via this API, that is fine with me. I have intentions of making the C interface available to all apps that desire to. If there is data that they want to get from portage via the API, then they should send their wish list in and we'll do our best to get it in there for them. Currently the API will be based on the packagekit portage backend code that was produced last summer and other code from porthole, portato, kportagetray. We are also going to put together a layman API for consumer apps (guis frontends) to use to operate layman without the need to run it in and parse terminal output. It too will hopefully be available via a C interface for non-python apps. I don't want to push you into the C corner, but tools like qfile and qlist are in orders of magnitude faster than their equery variants for a good reason. Maybe this simple command gives you an idea why I think python and portage are slow. % time portageq envvar CHOST powerpc-apple-darwin8 0.488u 0.704s 0:03.09 38.1% 0+0k 25+70io 0pf+0w (three full seconds to return a very simple var) That said, if you design C APIs, please design them from a C point of view, initially implemented by your Python functionality doing the necessary wrapping to get a sane C structure. Then they can be replaced by native C code as RSI and time permits in the future. -- Fabian Groffen Gentoo on a different level
[gentoo-portage-dev] [PATCH] config_root variables in pym/portage/const.py
In Prefix, there is a distinction between variables used with
config_root and target_root. Quoting from prefix' const.py:
# We have a most confusing situation here, which is most of all pretty
# weak for protecting us from making mistakes.
# First there is a config_root (PORTAGE_CONFIGROOT) which can be a path
# somewhere, from which all paths need to be relative (e.g.
# etc/portage), hence those constants do NOT have EPREFIX, because
# config_root contains EPREFIX by default -- overriding it loses the
# EPREFIX as one would expect.
# Second there is target_root (ROOT) which is to install somewhere
# completely else, in Prefix of limited use. Because this is an offset
# always given, the EPREFIX should always be applied in it. Those
# constants (like VDB_PATH) DO have EPREFIX.
# Unfortunately this file is ordered quite horrible in this respect.
The attached patch makes all variables against config_root relative (by
removing the leading '/'), with the result that all lstrip(os.sep) calls
in the code can be removed. Please note that all but two occurences did
use config_root. For one (pym/portage/__init__.py; sandbox stuff) I've
added it, for the other (in pym/portage/output.py) I don't know how to
get config_root, so substituted '/', which breaks Prefix by design.
How about this patch, and if it is a good idea, how about grouping all
of these config_root variables such that it's easier to see what is
what? I actually ran into this because Sebastian was hitting it with
his smolt-gentoo tool.
--
Fabian Groffen
Gentoo on a different level
Index: pym/portage/__init__.py
===
--- pym/portage/__init__.py (revision 13819)
+++ pym/portage/__init__.py (working copy)
@@ -1309,7 +1309,7 @@
if not config_profile_path:
config_profile_path = \
- os.path.join(config_root,
PROFILE_PATH.lstrip(os.path.sep))
+ os.path.join(config_root, PROFILE_PATH)
if os.path.isdir(config_profile_path):
self.profile_path = config_profile_path
else:
@@ -1325,7 +1325,7 @@
self.module_priority= [user,default]
self.modules= {}
self.modules[user] = getconfig(
- os.path.join(config_root,
MODULES_FILE_PATH.lstrip(os.path.sep)))
+ os.path.join(config_root, MODULES_FILE_PATH))
if self.modules[user] is None:
self.modules[user] = {}
self.modules[default] = {
@@ -1389,7 +1389,7 @@
self.profiles = []
if local_config and self.profiles:
custom_prof = os.path.join(
- config_root,
CUSTOM_PROFILE_PATH.lstrip(os.path.sep))
+ config_root, CUSTOM_PROFILE_PATH)
if os.path.exists(custom_prof):
self.user_profile_dir = custom_prof
self.profiles.append(custom_prof)
@@ -1465,7 +1465,7 @@
del rawpuseforce
make_conf = getconfig(
- os.path.join(config_root,
MAKE_CONF_FILE.lstrip(os.path.sep)),
+ os.path.join(config_root, MAKE_CONF_FILE),
tolerant=tolerant, allow_sourcing=True)
if make_conf is None:
make_conf = {}
@@ -1561,7 +1561,7 @@
self.configdict[defaults]=self.configlist[-1]
self.mygcfg = getconfig(
- os.path.join(config_root,
MAKE_CONF_FILE.lstrip(os.path.sep)),
+ os.path.join(config_root, MAKE_CONF_FILE),
tolerant=tolerant, allow_sourcing=True,
expand=expand_map)
if self.mygcfg is None:
self.mygcfg = {}
@@ -1610,8 +1610,7 @@
self.pkeywordsdict = {}
self._plicensedict = {}
self.punmaskdict = {}
- abs_user_config = os.path.join(config_root,
- USER_CONFIG_PATH.lstrip(os.path.sep))
+ abs_user_config = os.path.join(config_root,
USER_CONFIG_PATH)
# locations for categories and arch.list files
locations = [os.path.join(self[PORTDIR], profiles)]
@@ -1838,7 +1837,7 @@
(sandbox in self.features or usersandbox in
self.features
Re: [gentoo-portage-dev] prefix-portage-2.2.00.13734 with --nodeps merges in reverse order
On 02-07-2009 15:31:16 +0200, Michael Haubenwallner wrote: Hi, seems prefix-portage as of r13734, when used with '--nodeps', does merge packages in reverse order than given on cmdline, but I don't believe this is a prefix issue... prefix-portage r13683 does merge in correct order. 13734 is after the _emerge split -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] prefix portage chaining
On 20-03-2009 11:35:09 +0100, Markus Duft wrote: i'll try and explain what i want in the first place: i'm porting things to native windows. since windows isn't too cooperative, i'm unable to merge most things (and with other things, i simply don't want to), and thus i need to take those things from somewhere else (more or less the complete @system). I _am_ able to build all those things for interix (which is the host system in the windows case). So what i want is a setup of two prefix instances with a certain relation to each other: the native windows prefix should be able to recognize installed packages from the other instance, and resolve dependencies by eventually using the other .../var/db/... Since Windows and Interix seem to be two different things to me, can you explain why in this case Portage can resolve the dependencies from the Interix system to use for the Windows system? What dependencies are we talking about? Build tools? Libraries? Runtime dependencies? This could be (and is) quite usefull for all other platforms too. For exmaple i could use prefix chaining on a linux box. I could create a prefix containing a base system, and then for testing of i-don't-know-whatever, i could create another small prefix inheriting all installed packages from the other one. this way new prefixes can stay very slim, but still the parent prefix is not altered on merges. That potentially is useful, in the case where you want to upgrade a critical system package and test it out or something. Can't think of an example other than Portage itself for the moment, though. (And that one can be hairy, for instance if the vdb format changes NEEDED - NEEDED.ELF.2) one issue not handled by the current patch is, that prefixes can have different CHOST/ARCH/... (which is the case with x86-interix and x86-winnt for example). Then how does it work for you? -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] How to extract the version/revision of an installed package?
On 25-11-2008 16:15:09 -0800, Ned Ludd wrote:
On Tue, 2008-11-25 at 18:05 +0200, Amit Dor-Shifer wrote:
Given the following:
# qlist -Iv sys-apps/portage
sys-apps/portage-2.1.4.5
How do I safely extract the 2.1.4.5?
(I don't necessarily need to use qlist. Just want to get the version of
an installed package within a bash script)
qatom $(qlist -ICv sys-apps/portage) | awk '{print $3}'
I happen to do it in bash for my update script, it hasn't broke yet:
get_ebuildversion() {
if [[ -z $1 ]] ; then
return
fi
# strip extension
t=${1%.ebuild}
# abort if this is not an ebuild
if [[ $t == $1 ]] ; then
return
fi
# strip package name
t=${t#*-}
# sometimes there are dashes in the package name
while [[ ${t:0:1} 0 || ${t:0:1} 9 ]] ; do
[[ $t != ${t#*-} ]] \
t=${t#*-} \
|| break
done
echo ${t}
}
you might want to remove the .ebuild check and strip the category ($1#*/})
--
Fabian Groffen
Gentoo on a different level
Re: [gentoo-portage-dev] search functionality in emerge
On 24-11-2008 10:34:28 +0100, René 'Necoro' Neumann wrote: tvali schrieb: There is daemon, which notices about filesystem changes - http://pyinotify.sourceforge.net/ would be a good choice. Disadvantage: Has to run all the time (I see already some people crying: oh noez. not yet another daemon...). ... and it is Linux only, which spoils the fun. -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] Re: portage-2.2-rc3 parallel merges quit being parallel
On 28-07-2008 08:10:47 +, Duncan wrote: René 'Necoro' Neumann [EMAIL PROTECTED] posted [EMAIL PROTECTED], A very basic thingy: watch qlop -cC qlop is part of portage-utils. And it seems to only work correctly here, when run as root :) Thanks! =8^) and when sandbox is enabled, as it looks for [sandbox] in the process list -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] [PATCH] Repoman subversion support
On 24-07-2008 17:38:51 +0200, Arfrever Frehtes Taifersar Arahesis wrote: `svn list` returns only files which actually exist in the repository. It doesn't check working copy. You should use `svn st`. Ok, you're right. But svn status simply sucks: [usr/portage/eclass] % touch x11.eclass.rej [usr/portage/eclass] % svn status ? files [usr/portage/eclass] % In other words, it auto-ignores stuff that really causes the messed up manifests if someone isn't paying attention. IIRC this ignorance isn't limited to .rej, but also other extentions, such as the trailing ~, which some editors leave behind. I guess we need to use --no-ignore, and then also recognise the I entries. Bah. -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] [PATCH] Repoman subversion support
On 14-05-2008 17:56:38 +0200, Fabian Groffen wrote: On 14-05-2008 16:27:28 +0200, Arfrever Frehtes Taifersar Arahesis wrote: 2008-05-14 00:32 Marius Mauch [EMAIL PROTECTED] napisał(a): Merged in r10325 with some minor changes (removed the 'svn update' bit until someone remembers why exactly it's there During committing, only files, which are being committed, are being updated, so `svn up` is certainly a good idea. Please add it. Details here are that the code did this: if vcs == cvs: myf=open(checkdir+/CVS/Entries,r) if vcs == svn: myf=os.popen(svn update /dev/null; svn list) genone's remark here refers to this `svn update`, of which I don't recall any more why I added it. All I know is that I think I needed it for some reason to get a correct output of list. However, I have no proof here, so he removed the svn update, which gives a non-announced possible modification of the local repository. I've seen numerous examples like bug 232825[1] now, and I'm almost confident the svn update resolved that problem. Problem is that the file (a .rej in the case of the bug) exists in the local dir, but repoman doesn't notice it, since svn list doesn't return it (as '?'). Note that this sometimes happens, and sometimes not. I haven't found the rule here yet. [1] http://bugs.gentoo.org/show_bug.cgi?id=232825 -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] [PATCH] show binhosts as repository
On 14-05-2008 00:45:10 +0200, Marius Mauch wrote: On Mon, 12 May 2008 20:53:35 +0200 Fabian Groffen [EMAIL PROTECTED] wrote: The following patch shows the url to the binhost in an emerge -av as repository name, instead of unknown. Unfortunately the patch doesn't store the binhost url, such that portage can't show where the package comes from when unmerged. Not sure if this is the right thing to do, or if the reponame in those cases should be the one from the source repo that was used to generate those binary packages. Not saying that the binhost name here is useless, quite the opposite, just a bit concerned about mixing different things here. The idea behind it is that the repo_name of the binhost is not necessarily useful. In many cases it will contain the default repo_name value, since people just create binpkgs with a standard Portage install. I agree it is a hack. Perhaps a combination of the repo_name and its url would be the best. -- Fabian Groffen Gentoo on a different level -- gentoo-portage-dev@lists.gentoo.org mailing list
[gentoo-portage-dev] [PATCH] unpack .deb files with deb2tgz
Attached patch is necessary for some extreme platforms, as can be read
in the comments.
--
Fabian Groffen
Gentoo on a different level
--- ../../trunk/bin/ebuild.sh 2008-04-27 17:36:18 +0200
+++ ./bin/ebuild.sh 2008-04-13 11:41:55 +0200
@@ -372,9 +372,22 @@
LHa|LHA|lha|lzh)
lha xfq ${srcdir}${x} || die $myfail
;;
- a|deb)
+ a)
ar x ${srcdir}${x} || die $myfail
;;
+ deb)
+ # Unpacking .deb archives can not always be
done with
+ # `ar`. For instance on AIX this doesn't work
out. If
+ # we have `deb2targz` installed, prefer it over
`ar` for
+ # that reason. We just make sure on AIX
`deb2targz` is
+ # installed.
+ if type -P deb2targz /dev/null; then
+ deb2targz ${srcdir}/${x} || die
$myfail
+ mv ${srcdir}/${x/.deb/.tar.gz}
data.tar.gz
+ else
+ ar x ${srcdir}/${x} || die $myfail
+ fi
+ ;;
lzma)
if [ ${y} == tar ]; then
lzma -dc ${srcdir}${x} | tar xof -
${tar_opts}
[gentoo-portage-dev] [PATCH] remove eselect compiler usage
eselect compiler has been removed from the tree, hence its usage can be removed from portage. -- Fabian Groffen Gentoo on a different level --- ../../trunk/pym/_emerge/__init__.py 2008-05-12 19:25:21 +0200 +++ ./pym/_emerge/__init__.py 2008-05-12 17:16:49 +0200 @@ -303,12 +318,6 @@ !!! other terminals also.\n ) - mystatus, myoutput = commands.getstatusoutput(eselect compiler show) - if mystatus == os.EX_OK and len(myoutput.split(/)) == 2: - part1, part2 = myoutput.split(/) - if part1.startswith(chost + -): - return myoutput.replace(chost + -, gcc_ver_prefix, 1) - mystatus, myoutput = commands.getstatusoutput(gcc-config -c) if mystatus == os.EX_OK and myoutput.startswith(chost + -): return myoutput.replace(chost + -, gcc_ver_prefix, 1)
[gentoo-portage-dev] [PATCH] show binhosts as repository
The following patch shows the url to the binhost in an emerge -av as repository name, instead of unknown. Unfortunately the patch doesn't store the binhost url, such that portage can't show where the package comes from when unmerged. -- Fabian Groffen Gentoo on a different level --- ../../trunk/pym/_emerge/__init__.py 2008-05-12 19:25:21 +0200 +++ ./pym/_emerge/__init__.py 2008-05-12 17:16:49 +0200 @@ -1117,6 +1126,6 @@ pkg_chost = pkg.metadata.get(CHOST) if pkg_chost and pkg_chost != pkgsettings[CHOST]: return False if not portage.eapi_is_supported(pkg.metadata[EAPI]): return False if not pkg.installed and \ @@ -4447,13 +4471,18 @@ pkg = x metadata = pkg.metadata ebuild_path = None - repo_name = metadata[repository] + if pkg_type == binary: + repo_name = self.roots[myroot].settings.get(PORTAGE_BINHOST) + else: + repo_name = metadata[repository] if pkg_type == ebuild: ebuild_path = portdb.findname(pkg_key) if not ebuild_path: # shouldn't happen raise portage.exception.PackageNotFound(pkg_key) repo_path_real = os.path.dirname(os.path.dirname( os.path.dirname(ebuild_path))) + elif pkg_type == binary: + repo_path_real = repo_name else: repo_path_real = portdb.getRepositoryPath(repo_name) pkg_use = metadata[USE].split() @@ -5422,6 +5451,11 @@ self._repo_paths_real = [ os.path.realpath(repo_path) \ for repo_path in repo_paths ] + if root_config.settings.get(PORTAGE_BINHOST): + binhost = root_config.settings.get(PORTAGE_BINHOST) + self._repo_paths.append(binhost) + self._repo_paths_real.append(binhost) + # pre-allocate index for PORTDIR so that it always has index 0. for root_config in roots.itervalues(): portdb = root_config.trees[porttree].dbapi
[gentoo-portage-dev] [PATCH] include status support for xterm-color and interix
Attached patch adds statusbar support for xterm-color and interix terminals. -- Fabian Groffen Gentoo on a different level --- ../../trunk/pym/portage/output.py 2008-05-12 19:25:13 +0200 +++ ./pym/portage/output.py 2008-05-08 21:17:50 +0200 @@ -246,7 +246,7 @@ if len(mystr) max_len: mystr = mystr[:max_len] myt=os.environ[TERM] - legal_terms = [xterm,Eterm,aterm,rxvt,screen,kterm,rxvt-unicode,gnome] + legal_terms = [xterm,xterm-color,Eterm,aterm,rxvt,screen,kterm,rxvt-unicode,gnome,interix] for term in legal_terms: if myt.startswith(term): if not raw:
[gentoo-portage-dev] IUSE and userland_, elibc_, kernel_, etc.
Just to have it clear and to be sure: On 04-11-2007 03:33:41 +, [EMAIL PROTECTED] wrote: Modified: diffutils-2.8.7-r2.ebuild Log: do *not* include userland_GNU in IUSE (Portage version: 2.1.3.16) Index: diffutils-2.8.7-r2.ebuild -IUSE=nls static userland_GNU +IUSE=nls static On 04-11-2007 08:10:29 +, Zac Medico wrote: Author: zmedico Date: 2007-11-04 08:10:29 + (Sun, 04 Nov 2007) New Revision: 8420 Modified: main/trunk/pym/portage/dbapi/bintree.py Log: When evaluating *DEPEND conditionals for the Packages metadata index, do not use IUSE to filter USE since there is currently no guarantee that IUSE properly defines all of the necessary flags. These two changes now mean that without having userland_GNU in IUSE DEPEND=!userland_GNU? ( some/package ) will correctly end up in the Packages file, such that Portage will properly calculate dependencies when reading from a binhost, right? -- Fabian Groffen Gentoo on a different level -- [EMAIL PROTECTED] mailing list
Re: [gentoo-portage-dev] pty/termios problems on alternative platforms
On 25-05-2007 15:04:24 +0200, Fabian Groffen wrote: ... raise an exception on the pty code, but only bails out on the termios.tcgetattr function because it gets an invalid argument. from the code pym/portage/__init__.py around line 2288 from pty import openpty master_fd, slave_fd = openpty() # Disable the ECHO attribute so the terminal behaves properly # if the subprocess needs to read input from stdin. import termios term_attr = termios.tcgetattr(master_fd) term_attr[3] = ~termios.ECHO termios.tcsetattr(master_fd, termios.TCSAFLUSH, term_attr) if I comment out the complete termios (last four lines) it appears to work, though I don't know what repercussions this has on the rest of the code. Judging from the comment, it feels not so right. -- Fabian Groffen Gentoo on a different level -- [EMAIL PROTECTED] mailing list
Re: [gentoo-portage-dev] pty/termios problems on alternative platforms
On 25-05-2007 12:44:10 -0700, Zac Medico wrote: if I comment out the complete termios (last four lines) it appears to work, though I don't know what repercussions this has on the rest of the code. Judging from the comment, it feels not so right. Try replacing master_fd with slave_fd in both the tcgetattr and tcsetattr calls. I'm not sure if that will solve it but it's worth a try. It lets it continue without a backtrace... so I guess that sort of works? -- Fabian Groffen Gentoo on a different level -- [EMAIL PROTECTED] mailing list
Re: [gentoo-portage-dev] [PATCH] Repoman SVN support
On 24-11-2006 23:29:53 -0800, Zac Medico wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Fabian Groffen wrote: Please find attached the patch of the prefix repoman sources against the current trunk. I like the idea of having svn support, but I think it will be much nicer if we separate the cvs and svn support into plugins. That way, it should make the code cleaner and we'll more easily be able to add support for additional revision control systems if we want. I would prefer that, as I'm an OO-freak, but like I said I only revamped this patch into a working one. Besides that, I'm not a Python guru, so more than a few if-s is getting a challenge already. All in all, if there is an interface somewhere, then the code parts can be copied in there for sure, quite easily. -- Fabian Groffen Gentoo on a different level -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] [PATCH] Repoman SVN support
On 25-11-2006 03:00:10 -0500, Alec Warner wrote: Fabian Groffen wrote: Hi all, Please find attached the patch of the prefix repoman sources against the current trunk. I hope I removed all hunks that have prefix-only changes. You realize I wrote generic SCM support into repoman over the summer right? :P It was never merged (afaik I slacked on it, but also because E no... probably because at that time I wasn't making my hands dirty on portage code. it uses Popen and you pesky OSX folks are stuck on python2.3, so I need to convert to portage.spawn). You can drop that, as I already require python 2.4, since portage needs at least 2.3.1 currently and I don't feel like checking with an older version to see if it breaks more. During bootstrapping I first install python 2.4. On a slightly related side note: you can also drop all your bash-2 code, as we end up running bash-3 too. Would only be nice if the basic functionally would work with bash-2, otherwise I need to bootstrap bash too. (Worth the time if it simplifies portage.) However the support was generic, so for pretty much any system that 'is kinda like cvs/svn' should work fine with a few tweaks to the exec mapping. biasThis of course excludes GIT, since it's a huge non-standard piece of crap./bias Great! I feel that SVN won't be the only thing folks use, so I'm looking forward to your interface from a (hopefully) functional view point. I may work on fixing that code up and putting it into HEAD before I retire. E... what? The first is nice, the latter for sure isn't. -- Fabian Groffen Gentoo on a different level -- gentoo-portage-dev@gentoo.org mailing list
[gentoo-portage-dev] [PATCH] Repoman SVN support
Hi all,
Please find attached the patch of the prefix repoman sources against the
current trunk. I hope I removed all hunks that have prefix-only
changes.
The current diff is mainly SVN support for Repoman. I extracted the SVN
diffs only. Most of the original work was done by Kito, and recently I
wiped the dust off of it, and finalised it into some working version.
So far the good news, the bad news now regarding this patch: I had to
somehow hardcode the svn repo being used, as I didn't find a way to let
repoman figure out dynamically what the offset in the SVN repository
is where the portage tree starts. This is kind of similar to how CVS
support is done, where this offset is also hardcoded.
While I think repoman's CVS support has always been used as-is --
nowadays people prefer to start non-CVS repos, the SVN support as in
this patch would, considering the many overlays in SVN around, surely
benefit from some sort of solution that allows to set the repository
offset or something, e.g. through a variable. Thoughts?
--
Fabian Groffen
Gentoo on a different level
--- ../../trunk/bin/repoman 2006-11-23 18:15:36 +0100
+++ bin/repoman 2006-11-20 21:52:30 +0100
@@ -407,9 +409,80 @@
os.environ[FEATURES]=repoman_settings[FEATURES]+ cvs
isCvs = True
-if not --pretend in myoptions and not isCvs:
+ try:
+ isCvs=True
+ myrepofile=open(CVS/Repository)
+ myreporoot=myrepofile.readline()[:-1]
+ myrepofile.close()
+ myrepofile=open(CVS/Root)
+ myreporootpath=string.split(myrepofile.readline()[:-1], :)[-1]
+ myrepofile.close()
+ if myreporootpath == myreporoot[:len(myreporootpath)]:
+ # goofy os x cvs co.
+ myreporoot = myreporoot[len(myreporootpath):]
+ while myreporoot and myreporoot[0] == '/':
+ myreporoot = myreporoot[1:]
+ except SystemExit, e:
+ raise # Need to propogate this
+ except:
+ err(Error grabbing repository information; exiting.)
+ myreporootpath=os.path.normpath(myreporootpath)
+ if myreporootpath==/space/cvsroot:
+ print
+ print
+ print red(You're using the wrong cvsroot. For Manifests to be
correct, you must)
+ print red(use the same base cvsroot path that the servers use.
Please try the)
+ print red(following script to remedy this:)
+ print
+ print cd my_cvs_tree
+ print
+ print rm -Rf [a-z]*
+ print cvs up
+ print
+ if portage.userland==BSD or portage.userland==Darwin:
+ print find ./ -type f -regex '.*/CVS/Root$' -print0 |
xargs -0 sed \\
+ else:
+ print find ./ -type f -regex '.*/CVS/Root$' -print0 |
xargs -0r sed \\
+ fi
+ print -i 's:/space/cvsroot$:/home/cvsroot:'
+ print
+ print red(You must clear and re-update your tree as all header
tags will cause)
+ print red(problems in manifests for all rsync and
/home/cvsroot users.)
+ print
+ print You should do this to any other gentoo trees your have
as well,
+ print excluding the deletions. 'gentoo-src' should be
/home/cvsroot.
+ print
+ sys.exit(123)
+
+isSvn=False
+myreporoot=None
+if os.path.isdir(.svn):
+ if svn not in repoman_settings.features:
+ print
+ print
+ print red('!!! You do not have ')+bold('FEATURES=svn
')+red(enabled...)
+ print red(!!! )+bold(Adding \svn\ to FEATURES)
+ print
+ os.environ[FEATURES]=repoman_settings[FEATURES]+ svn
+
+ try:
+ isSvn=True
+ myrepoinfo = os.popen(svn info).readlines()
+ myurl = myrepoinfo[1][5:].rstrip()
+ myreproot = myrepoinfo[2][17:].rstrip()
+ # This one is EVIL, but I don't know how to solve the problem of
+ # getting rid of /trunk/prefix-overlay. This only works on the
+ # prefix overlay.
+ myreporoot = /.join(myurl[len(myreproot):].split(/)[2:])
+ myreporootpath = / + /.join(myreproot.split(/)[3:])
+ except SystemExit, e:
+ raise # Need to propogate this
+ except:
+ err(Error grabbing repository information; exiting.)
+
+if not --pretend in myoptions and not isCvs and not isSvn:
print
- print darkgreen(Not in a CVS repository; enabling pretend mode.)
+ print darkgreen(Not in a CVS or SVN repository; enabling pretend
mode.)
myoptions.append(--pretend);
@@ -490,14 +563,17 @@
myreporoot = os.path.basename(portdir_overlay
