Re: [gentoo-portage-dev] [PATCH] _get_lock_fn: support multiprocessing spawn start method (bug 758230)

2020-12-05 Thread Fabian Groffen
Thanks Zac!

On 04-12-2020 14:58:22 -0800, Zac Medico wrote:
> Ensure that _get_lock_fn arguments to multiprocessing.Process will
> successfully pickle, as required by the spawn start method, which
> is the default for macOS since Python 3.8.
> 
> Since file descriptors are not inherited unless the fork start
> method is used, the subprocess should only try to close an
> inherited file descriptor for the fork start method.
> 
> Bug: https://bugs.gentoo.org/758230
> Signed-off-by: Zac Medico 
> ---
>  lib/portage/locks.py | 36 +++-
>  1 file changed, 23 insertions(+), 13 deletions(-)
> 
> diff --git a/lib/portage/locks.py b/lib/portage/locks.py
> index 1073343be..193045c03 100644
> --- a/lib/portage/locks.py
> +++ b/lib/portage/locks.py
> @@ -44,18 +44,7 @@ def _get_lock_fn():
>   if _lock_fn is not None:
>   return _lock_fn
>  
> - def _test_lock(fd, lock_path):
> - os.close(fd)
> - try:
> - with open(lock_path, 'a') as f:
> - fcntl.lockf(f.fileno(), 
> fcntl.LOCK_EX|fcntl.LOCK_NB)
> - except EnvironmentError as e:
> - if e.errno == errno.EAGAIN:
> - # Parent process holds lock, as expected.
> - sys.exit(0)
>  
> - # Something went wrong.
> - sys.exit(1)
>  
>   fd, lock_path = tempfile.mkstemp()
>   try:
> @@ -64,8 +53,16 @@ def _get_lock_fn():
>   except EnvironmentError:
>   pass
>   else:
> - proc = multiprocessing.Process(target=_test_lock,
> - args=(fd, lock_path))
> + proc = multiprocessing.Process(
> + target=_subprocess_test_lock,
> + args=(
> + # Since file descriptors are not 
> inherited unless the fork start
> + # method is used, the subprocess should 
> only try to close an
> + # inherited file descriptor for the 
> fork start method.
> + fd if 
> multiprocessing.get_start_method() == "fork" else None,
> + lock_path,
> + ),
> + )
>   proc.start()
>   proc.join()
>   if proc.exitcode == os.EX_OK:
> @@ -80,6 +77,19 @@ def _get_lock_fn():
>   _lock_fn = fcntl.flock
>   return _lock_fn
>  
> +def _subprocess_test_lock(fd, lock_path):
> + if fd is not None:
> + os.close(fd)
> + try:
> + with open(lock_path, 'a') as f:
> + fcntl.lockf(f.fileno(), fcntl.LOCK_EX|fcntl.LOCK_NB)
> + except EnvironmentError as e:
> + if e.errno == errno.EAGAIN:
> +     # Parent process holds lock, as expected.
> + sys.exit(0)
> +
> + # Something went wrong.
> + sys.exit(1)
>  
>  _open_fds = {}
>  _open_inodes = {}
> -- 
> 2.26.2
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Speeding up Tree Verification

2020-07-01 Thread Fabian Groffen
On 30-06-2020 13:13:29 -0500, Sid Spry wrote:
> On Tue, Jun 30, 2020, at 1:20 AM, Fabian Groffen wrote:
> > Hi,
> > 
> > On 29-06-2020 21:13:43 -0500, Sid Spry wrote:
> > > Hello,
> > > 
> > > I have some runnable pseudocode outlining a faster tree verification 
> > > algorithm.
> > > Before I create patches I'd like to see if there is any guidance on 
> > > making the
> > > changes as unobtrusive as possible. If the radical change in algorithm is
> > > acceptable I can work on adding the changes.
> > > 
> > > Instead of composing any kind of structured data out of the portage tree 
> > > my
> > > algorithm just lists all files and then optionally batches them out to 
> > > threads.
> > > There is a noticeable speedup by eliding the tree traversal operations 
> > > which
> > > can be seen when running the algorithm with a single thread and comparing 
> > > it to
> > > the current algorithm in gemato (which should still be discussed here?).
> > 
> > I remember something that gemato used to use multiple threads, but
> > because it totally saturated disk-IO, it was brought back to a single
> > thread.  People were complaining about unusable systems.
> > 
> 
> I think this is an argument for cgroups limits support on the portage process 
> or
> account as opposed to an argument against picking a better algorithm. That is
> something I have been working towards, but I am only one man.

But this requires a) cgroups support, and b) the privileges to use it.
Shouldn't be a problem in the normal case, but just saying.

> > In any case, can you share your performance results?  What speedup did
> > you see, on warm and hot FS caches?  Which type of disk do you use?
> > 
> 
> I ran all tests multiple times to make them warm off of a Samsung SSD, but
> nothing very precise yet.
> 
> % gemato verify --openpgp-key signkey.asc /var/db/repos/gentoo
> [...]
> INFO:root:Verifying /var/db/repos/gentoo...
> INFO:root:/var/db/repos/gentoo verified in 16.45 seconds
> 
> sometimes going higher, closer to 18s, vs.
> 
> % ./veriftree.py
> 4.763171965983929
> 
> So roughly an order of magnitude speedup without batching to threads.

That is kind of a change.  Makes one wonder if you really did the same
work.

> > You could compare against qmanifest, which uses OpenMP-based
> > paralllelism while verifying the tree.  On SSDs this does help.
> > 
> 
> I lost my notes -- how do I specify to either gemato or qmanifest the GnuPG
> directory? My code is partially structured as it is because I had problems 
> doing
> this. I rediscovered -K/--openpgp-key in gemato but am unsure for qmanifest.

qmanifest doesn't do much magic out of the standard gnupg practices.
(It is using gpgme.)  If you want it to use a different gnupg dir, you
may change HOME, or GNUPGHOME.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Speeding up Tree Verification

2020-06-30 Thread Fabian Groffen
: str) -> bool:
> response = dns.resolver.query(domain, dns.rdatatype.NS)
> nsname = response.rrset[0]
> response = dns.resolver.query(str(nsname), dns.rdatatype.A)
> nsaddr = response.rrset[0].to_text()
> 
> # DNSKEY
> request = dns.message.make_query(domain,
> dns.rdatatype.DNSKEY, want_dnssec=True)
> response = dns.query.udp(request, nsaddr)
> if response.rcode() != 0:
> raise Exception('Unable to request dnskey.')
> 
> answer = response.answer
> if len(answer) != 2:
> raise Exception('Malformed answer to dnskey query.')
> 
> name = dns.name.from_text(domain)
> try:
> dns.dnssec.validate(answer[0], answer[1], {name: answer[0]})
> except dns.dnssec.ValidationFailure as exc:
> # Validation failed. The raise causes python to abort with status 1.
> #raise exc
> return False
> except AttributeError as exc:
> # Validation may have failed; DNSKEY missing signer attribute. dig 
> may report
> # domain as valid.
> #
> # TODO: Additional state where subdomain of valid domain may fail 
> with 3 nested
> # KeyErrors. Avoid temptation to wildcard catch. Safer to put in 
> process?
> #raise exc
> return False
> else:
> return True
> 
> def hash_localpart(incoming: bytes) -> str:
> '''Z-base32 the localpart of an e-mail address
> 
> 
> https://tools.ietf.org/html/draft-koch-openpgp-webkey-service-08#section-3.1
> describes why this is needed.
> 
> See https://tools.ietf.org/html/rfc6189#section-5.1.6 for a
> description of the z-base32 scheme.
> '''
> zb32 = "ybndrfg8ejkmcpqxot1uwisza345h769"
> 
> b = hashlib.sha1(incoming).digest()
> ret = ""
> assert(len(b) * 8 == 160)
> for i in range(0, 160, 5):
> byte = i // 8
> offset = i - byte * 8
> # offset | bits remaining in k+1 | right-shift k+1
> # 3 | 0 | x
> # 4 | 1 | 7
> # 5 | 2 | 6
> # 6 | 3 | 5
> # 7 | 4 | 4
> if offset < 4:
> n = (b[byte] >> (3 - offset))
> else:
> n = (b[byte] << (offset - 3)) + (b[byte + 1] >> (11 - offset))
> 
> ret += zb32[n & 0b1]
> return ret
> 
> def build_web_key_uri(address: str) -> str:
> local, remote = address.split('@')
> local = hash_localpart(local.encode('utf-8'))
> return 'https://' + remote + '/.well-known/openpgpkey/hu/' + \
> local
> 
> def stream_to_file(uri: str, fname: str) -> None:
> with requests.get(uri, verify=True, stream=True) as r:
> from pprint import pprint
> pprint(r.headers)
> with open(fname, 'wb') as f:
> shutil.copyfileobj(r.raw, f)
> ```
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Add caching to a few commonly used functions

2020-06-27 Thread Fabian Groffen
Hi Chun-Yu,

On 26-06-2020 23:34:12 -0700, Chun-Yu Shei wrote:
> Hi,
> 
> I was recently interested in whether portage could be speed up, since
> dependency resolution can sometimes take a while on slower machines.
> After generating some flame graphs with cProfile and vmprof, I found 3
> functions which seem to be called extremely frequently with the same
> arguments: catpkgsplit, use_reduce, and match_from_list.  In the first
> two cases, it was simple to cache the results in dicts, while
> match_from_list was a bit trickier, since it seems to be a requirement
> that it return actual entries from the input "candidate_list".  I also
> ran into some test failures if I did the caching after the
> mydep.unevaluated_atom.use and mydep.repo checks towards the end of the
> function, so the caching is only done up to just before that point.
> 
> The catpkgsplit change seems to definitely be safe, and I'm pretty sure
> the use_reduce one is too, since anything that could possibly change the
> result is hashed.  I'm a bit less certain about the match_from_list one,
> although all tests are passing.
> 
> With all 3 patches together, "emerge -uDvpU --with-bdeps=y @world"
> speeds up from 43.53 seconds to 30.96 sec -- a 40.6% speedup.  "emerge
> -ep @world" is just a tiny bit faster, going from 18.69 to 18.22 sec
> (2.5% improvement).  Since the upgrade case is far more common, this
> would really help in daily use, and it shaves about 30 seconds off
> the time you have to wait to get to the [Yes/No] prompt (from ~90s to
> 60s) on my old Sandy Bridge laptop when performing normal upgrades.
> 
> Hopefully, at least some of these patches can be incorporated, and please
> let me know if any changes are necessary.

This sounds like a good job to me!  Do you have any idea what the added
memory pressure for these changes are?

Thanks,
Fabian
> 
> Thanks,
> Chun-Yu
> 
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] Use env to find python

2020-06-17 Thread Fabian Groffen
On 16-06-2020 23:10:28 -0500, Sid Spry wrote:
> On Tue, Jun 16, 2020, at 3:57 PM, Michał Górny wrote:
> > '/usr/bin/env python' (with no extra options) is the portable shebang.
> > 
> 
> I added `-S` to preserve the options passed via the shebang line. It seems 
> they can be left off, does anyone know otherwise?

-S is not supported on some platforms.

In any case, I think this patch is wrong.  In the Prefix case, the
shebangs are set during install, in the non-Prefix case I think there is
machinery in place to replace the shebangs during install.

It seems the problem is already solved, why do you need these shebangs
changed?

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] Change BINPKG_COMPRESS default from bzip2 to xz

2020-04-26 Thread Fabian Groffen
On 26-04-2020 21:29:42 +0200, Michał Górny wrote:
> On Sun, 2020-04-26 at 09:55 -0700, Matt Turner wrote:
> > Bug: https://bugs.gentoo.org/715108
> > Signed-off-by: Matt Turner 
> > ---
> > Strawman patch. Bikeshed away.
> > 
> 
> xz is generally slow and doesn't do parallel good.  If we want to change
> this, we should go for something cool like zstd that scales better.

I'd go for zstd too.  It seems to be the best of both worlds, good
compression at a good speed.

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.

2019-12-13 Thread Fabian Groffen
On 13-12-2019 14:24:33 -0500, Michael Orlitzky wrote:
> On 12/13/19 9:28 AM, Fabian Groffen wrote:
> > 
> > We are providing those patches, maybe.  In reality very often the
> > patches originate from somewhere else though.  And you don't want to
> > have to respin all of those just because.  At least that's what I feel.
> > 
> 
> Just because... the context changed? A new "!" in a line of context can
> be the difference between letting someone log in with the right password
> and letting them log in with the wrong password. You should at least
> have to stop and verify that the patch does what you think it does when
> it "gains" fuzz. And at that point, git diff will give you a clean
> version of it.

Counter argument is that we've been doing this for decades, and always
relied on maintainers to check the contents of their patches, without
problems.  We didn't introduce a predictable random number generator or
something.
Your very specific example just illustrates the niche this proposal is
targetting.

As with many of the proposals lately, they just seem to aim at more work
for individual maintainers, with a very low gain ratio.
Better, allow this to be a FEATURE, or whatever, that devs should enable,
but don't spit this in the user's face by default.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.

2019-12-13 Thread Fabian Groffen
On 13-12-2019 15:24:40 +0100, Michał Górny wrote:
> > > ...and why do we consider it correct to apply patches when the context
> > > doesn't match?  If our only goal is to make things 'easier' for
> > > 'everyone', then we could just pass -F and ignore all the context.
> > > 
> > > Though I don't understand why include any context in the first place if
> > > you don't care about it matching.  Sounds like a waste of space to me!
> > 
> > The patch command defaults to -F2. If that makes no sense, why is it
> > the upstream default?
> > 
> 
> You should ask upstream, not me.  But if I were to guess, the answer
> would be because patch(1) is used by random people trying to apply
> random patches they've found somewhere.  We on the other hand are
> applying patches that *we* are supposed to provide.

We are providing those patches, maybe.  In reality very often the
patches originate from somewhere else though.  And you don't want to
have to respin all of those just because.  At least that's what I feel.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [RFC] Adding extra vars to md5-cache, for QA purposes

2019-07-26 Thread Fabian Groffen
Hi,

On 25-07-2019 14:20:50 +0200, Michał Górny wrote:
> Hi,
> 
> TL;DR: I'd like to make it possible for ebuilds to define additional
> variables that will be stored in md5-cache.  This would be useful for CI
> and other tooling that right now has to parse ebuilds for other data.

Only downside I can think of, is a diskspace increase for the md5-cache.
Not sure if this is going to be substantial, but given things like
PYTHON_COMPAT, perhaps a quick calculation of extra "cost" can be made.
Should diskspace become a problem, one could consider to use a separate
file/dir, that users could rsync-exclude, since Portage won't need it to
operate properly.

Thanks,
Fabian

> 
> 
> The idea is to add a new incremental ebuild/eclass variable (technical
> name: QA_EXTRA_CACHE_VARS) that would define additional data to be
> stored in cache.  For example, python*-r1 eclasses would define
> 'PYTHON_COMPAT', acct-user would define 'ACCT_USER_ID', etc.
> 
> When regenerating cache, the PM would read this variable, and store
> the values of all defined variables into md5-cache.  As a result,
> programs needing those variables can get them straight from cache
> without having to attempt to run or parse ebuilds (which is both slow
> and prone to bugs).
> 
> This would benefit e.g. gpyutils that right now need to attempt to parse
> PYTHON_COMPAT from ebuilds.  It would also benefit writing future
> pkgcheck checks for user/group ID collisions.
> 
> 
> Notes:
> 
> - since md5-cache uses key-value format and allows for future
> extensions, the new values can be added without breaking anything;
> 
> - md5-cache is not specified in the PMS, and the whole thing can be
> implemented without need for EAPI bump,
> 
> - I would like to have this implemented consistently both in Portage
> and pkgcore,
> 
> - we will need to clearly define how to dump arrays.
> 
> 
> What do you think?
> 
> -- 
> Best regards,
> Michał Górny
> 



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH v5] collision_protect: use dynamic report interval

2019-01-11 Thread Fabian Groffen
Pushed, thanks

On 10-01-2019 20:37:16 -0800, Zac Medico wrote:
> On 1/10/19 7:30 AM, Fabian Groffen wrote:
> > The reporting of files remaining can look somewhat odd since the report
> > interval is hardcoded to be per 1000 objects.  Adjust this interval to
> > be time based.  This means that modern (fast) machines likely will never
> > see the countdown messages at all.  On slow setups the message will be
> > informative that there is progress, albeit rather slowly.  While at it,
> > report percentage done.
> > 
> > Output before this patch:
> > 
> >  * checking 6158 files for package collisions
> > 5158 files remaining ...
> > 4158 files remaining ...
> > 3158 files remaining ...
> > 2158 files remaining ...
> > 1158 files remaining ...
> > 158 files remaining ...
> > 
> > Possible output after this patch on a slower machine:
> > 
> >  * checking 6158 files for package collisions
> >  48% done,  3145 files remaining ...
> >  96% done,   192 files remaining ...
> > 100% done
> > 
> > Signed-off-by: Fabian Groffen 
> > ---
> >  lib/portage/dbapi/vartree.py | 15 +--
> >  1 file changed, 13 insertions(+), 2 deletions(-)
> > 
> > diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
> > index 4b91caea8..909c0e473 100644
> > --- a/lib/portage/dbapi/vartree.py
> > +++ b/lib/portage/dbapi/vartree.py
> > @@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
> > 'portage.util.install_mask:install_mask_dir,InstallMask',
> > 'portage.util.listdir:dircache,listdir',
> > 'portage.util.movefile:movefile',
> > +   'portage.util.monotonic:monotonic',
> > 'portage.util.path:first_existing,iter_parents',
> > 'portage.util.writeable_check:get_ro_checker',
> > 'portage.util._xattr:xattr',
> > @@ -3475,13 +3476,21 @@ class dblink(object):
> > symlink_collisions = []
> > destroot = self.settings['ROOT']
> > totfiles = len(file_list) + len(symlink_list)
> > +   previous = monotonic()
> > +   progress_shown = False
> > +   report_interval = 1.7  # seconds
> > +   falign = len("%d" % totfiles)
> > showMessage(_(" %s checking %d files for package 
> > collisions\n") % \
> > (colorize("GOOD", "*"), totfiles))
> > for i, (f, f_type) in enumerate(chain(
> > ((f, "reg") for f in file_list),
> > ((f, "sym") for f in symlink_list))):
> > -   if i % 1000 == 0 and i != 0:
> > -   showMessage(_("%d files remaining 
> > ...\n") % (totfiles - i))
> > +   current = monotonic()
> > +   if current - previous > report_interval:
> > +   showMessage(_("%3d%% done,  %*d files 
> > remaining ...\n") %
> > +   (i * 100 / totfiles, 
> > falign, totfiles - i))
> > +   previous = current
> > +   progress_shown = True
> >  
> > dest_path = normalize_path(
> > os.path.join(destroot, 
> > f.lstrip(os.path.sep)))
> > @@ -3570,6 +3579,8 @@ class dblink(object):
> > break
> > if stopmerge:
> > collisions.append(f)
> > +   if progress_shown:
> > +   showMessage(_("100% done\n"))
> > return collisions, dirs_ro, symlink_collisions, 
> > plib_collisions
> >  
> > def _lstat_inode_map(self, path_iter):
> > 
> 
> Looks good!
> -- 
> Thanks,
> Zac
> 




-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH v5] collision_protect: use dynamic report interval

2019-01-10 Thread Fabian Groffen
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be time based.  This means that modern (fast) machines likely will never
see the countdown messages at all.  On slow setups the message will be
informative that there is progress, albeit rather slowly.  While at it,
report percentage done.

Output before this patch:

 * checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...

Possible output after this patch on a slower machine:

 * checking 6158 files for package collisions
 48% done,  3145 files remaining ...
 96% done,   192 files remaining ...
100% done

Signed-off-by: Fabian Groffen 
---
 lib/portage/dbapi/vartree.py | 15 +--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..909c0e473 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.install_mask:install_mask_dir,InstallMask',
'portage.util.listdir:dircache,listdir',
'portage.util.movefile:movefile',
+   'portage.util.monotonic:monotonic',
'portage.util.path:first_existing,iter_parents',
'portage.util.writeable_check:get_ro_checker',
'portage.util._xattr:xattr',
@@ -3475,13 +3476,21 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+   previous = monotonic()
+   progress_shown = False
+   report_interval = 1.7  # seconds
+   falign = len("%d" % totfiles)
showMessage(_(" %s checking %d files for package 
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
-   if i % 1000 == 0 and i != 0:
-   showMessage(_("%d files remaining 
...\n") % (totfiles - i))
+   current = monotonic()
+   if current - previous > report_interval:
+   showMessage(_("%3d%% done,  %*d files 
remaining ...\n") %
+   (i * 100 / totfiles, 
falign, totfiles - i))
+   previous = current
+   progress_shown = True
 
dest_path = normalize_path(
os.path.join(destroot, 
f.lstrip(os.path.sep)))
@@ -3570,6 +3579,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+   if progress_shown:
+   showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions, 
plib_collisions
 
def _lstat_inode_map(self, path_iter):
-- 
2.20.1




Re: [gentoo-portage-dev] [PATCH v4] collision_protect: use dynamic report interval

2019-01-10 Thread Fabian Groffen
On 09-01-2019 21:22:26 -0800, Zac Medico wrote:
> > +   if tnow + tinterv < ninterv:
> > +   showMessage(_("100% done\n"))
> 
> It took me a moment to understand the calculation here. How about if we
> do something like this instead:
> 
> 
> previous = monotonic()
> progress_shown = False

You're right.  I should've gone for readability in the first place.
There is no point in trying to save anything here.

Thanks,
Fabian

> for f in files:
>   current = monotonic()
>   if current - previous > tinterv:
>   showMessage(...)
>   previous = current
>   progress_shown = True
> 
> if progress_shown:
>   showMessage(_("100% done\n"))
> 
> 
> 
> > return collisions, dirs_ro, symlink_collisions, 
> > plib_collisions
> >  
> > def _lstat_inode_map(self, path_iter):
> > 
> 
> 
> -- 
> Thanks,
> Zac
> 




-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH v4] collision_protect: use dynamic report interval

2019-01-09 Thread Fabian Groffen
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be time based.  This means that modern (fast) machines likely will never
see the countdown messages at all.  On slow setups the message will be
informative that there is progress, albeit rather slowly.  While at it,
report percentage done.

Output before this patch:

 * checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...

Possible output after this patch on a slower machine:

 * checking 6158 files for package collisions
 48% done,  3145 files remaining ...
 96% done,   192 files remaining ...
100% done

Signed-off-by: Fabian Groffen 
---
 lib/portage/dbapi/vartree.py | 13 +++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..df192e6fc 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.install_mask:install_mask_dir,InstallMask',
'portage.util.listdir:dircache,listdir',
'portage.util.movefile:movefile',
+   'portage.util.monotonic:monotonic',
'portage.util.path:first_existing,iter_parents',
'portage.util.writeable_check:get_ro_checker',
'portage.util._xattr:xattr',
@@ -3475,13 +3476,19 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+   tnow = monotonic()
+   tinterv = 2  # seconds
+   ninterv = tnow + tinterv
+   falign = len("%d" % totfiles)
showMessage(_(" %s checking %d files for package 
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
-   if i % 1000 == 0 and i != 0:
-   showMessage(_("%d files remaining 
...\n") % (totfiles - i))
+   if monotonic() > ninterv:
+   showMessage(_("%3d%% done,  %*d files 
remaining ...\n") %
+   (falign, i * 100 / 
totfiles, totfiles - i))
+   ninterv = monotonic() + tinterv
 
dest_path = normalize_path(
os.path.join(destroot, 
f.lstrip(os.path.sep)))
@@ -3570,6 +3577,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+   if tnow + tinterv < ninterv:
+   showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions, 
plib_collisions
 
def _lstat_inode_map(self, path_iter):
-- 
2.20.1




Re: [gentoo-portage-dev] [PATCH v3] collision_protect: use dynamic report interval

2019-01-09 Thread Fabian Groffen
On 08-01-2019 20:59:34 +, M. J. Everitt wrote:
> On 08/01/19 19:15, Zac Medico wrote:
> > On 1/8/19 5:42 AM, Fabian Groffen wrote:
> >> The reporting of files remaining can look somewhat odd since the report
> >> interval is hardcoded to be per 1000 objects.  Adjust this interval to
> >> be time based.  This means that modern (fast) machines likely will never
> >> see the countdown messages at all.  On slow setups the message will be
> >> informative that there is progress, albeit rather slowly.  While at it,
> >> report percentage done.
> >>
> >> Output before this patch:
> >>
> >>  * checking 6158 files for package collisions
> >> 5158 files remaining ...
> >> 4158 files remaining ...
> >> 3158 files remaining ...
> >> 2158 files remaining ...
> >> 1158 files remaining ...
> >> 158 files remaining ...
> >>
> >> Possible output after this patch on a slower machine:
> >>
> >>  * checking 6158 files for package collisions
> >>  48% done, 3145 files remaining ...
> >>  96% done, 192 files remaining ...
> >> 100% done
> >>
> >> Signed-off-by: Fabian Groffen 
> >> ---
> >>  lib/portage/dbapi/vartree.py | 11 +--
> >>  1 file changed, 9 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
> >> index 4b91caea8..244195fad 100644
> >> --- a/lib/portage/dbapi/vartree.py
> >> +++ b/lib/portage/dbapi/vartree.py
> >> @@ -3475,13 +3475,18 @@ class dblink(object):
> >>symlink_collisions = []
> >>destroot = self.settings['ROOT']
> >>totfiles = len(file_list) + len(symlink_list)
> >> +  tnow = time.time()
> >> +  tinterv = 2  # seconds
> >> +  ninterv = tnow + tinterv
> >>showMessage(_(" %s checking %d files for package 
> >> collisions\n") % \
> >>(colorize("GOOD", "*"), totfiles))
> >>for i, (f, f_type) in enumerate(chain(
> >>((f, "reg") for f in file_list),
> >>((f, "sym") for f in symlink_list))):
> >> -  if i % 1000 == 0 and i != 0:
> >> -  showMessage(_("%d files remaining 
> >> ...\n") % (totfiles - i))
> >> +  if time.time() > ninterv:
> >> +  showMessage(_("%3d%% done, %d files 
> >> remaining ...\n") %
> >> +  (i * 100 / totfiles, 
> >> totfiles - i))
> >> +  ninterv = time.time() + tinterv
> >>  
> >>dest_path = normalize_path(
> >>os.path.join(destroot, 
> >> f.lstrip(os.path.sep)))
> >> @@ -3570,6 +3575,8 @@ class dblink(object):
> >>break
> >>if stopmerge:
> >>collisions.append(f)
> >> +  if tnow + tinterv < ninterv:
> >> +  showMessage(_("100% done\n"))
> >>return collisions, dirs_ro, symlink_collisions, 
> >> plib_collisions
> >>  
> >>def _lstat_inode_map(self, path_iter):
> >>
> > Please replace time.time() with portage.util.monotonic.monotonic().

Will do.

> It's a bit of a nit-pick, granted, but can we ensure that the count-down's
> remain padded/justified such that the numbers line up for easy at-a-glance
> inspection ? The optimal standard looks somewhat like the pre-merge Sizes:
> 
>  * Final size of build directory: 2696 KiB (2.6 MiB)
>  * Final size of installed tree:  5372 KiB (5.2 MiB)
> 
> Otherwise, I think this will be quite helpful. Thanks.

I think I understand what you mean, let me see if I can do something
without too much complexity.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH v3] collision_protect: use dynamic report interval

2019-01-08 Thread Fabian Groffen
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be time based.  This means that modern (fast) machines likely will never
see the countdown messages at all.  On slow setups the message will be
informative that there is progress, albeit rather slowly.  While at it,
report percentage done.

Output before this patch:

 * checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...

Possible output after this patch on a slower machine:

 * checking 6158 files for package collisions
 48% done, 3145 files remaining ...
 96% done, 192 files remaining ...
100% done

Signed-off-by: Fabian Groffen 
---
 lib/portage/dbapi/vartree.py | 11 +--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..244195fad 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,18 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+   tnow = time.time()
+   tinterv = 2  # seconds
+   ninterv = tnow + tinterv
showMessage(_(" %s checking %d files for package 
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
-   if i % 1000 == 0 and i != 0:
-   showMessage(_("%d files remaining 
...\n") % (totfiles - i))
+   if time.time() > ninterv:
+   showMessage(_("%3d%% done, %d files 
remaining ...\n") %
+   (i * 100 / totfiles, 
totfiles - i))
+   ninterv = time.time() + tinterv
 
dest_path = normalize_path(
os.path.join(destroot, 
f.lstrip(os.path.sep)))
@@ -3570,6 +3575,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+   if tnow + tinterv < ninterv:
+   showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions, 
plib_collisions
 
def _lstat_inode_map(self, path_iter):
-- 
2.20.1




Re: [gentoo-portage-dev] [PATCH] collision_protect: use dynamic report interval

2019-01-08 Thread Fabian Groffen
On 08-01-2019 09:17:04 +0100, Ulrich Mueller wrote:
> >>>>> On Tue, 08 Jan 2019, Fabian Groffen wrote:
> 
> > Output before this patch:
> 
> >  * checking 6111 files for package collisions
> >  [...]
> 
> > After:
> 
> >  * checking 6158 files for package collisions
> >  [...]
> 
> Why does the total number of files change?

First was python2, second python3.  The result was indicative only, the
number doesn't change of course.

Sorry for the confusion.

Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH] collision_protect: use dynamic report interval

2019-01-08 Thread Fabian Groffen
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be regular towards the end.  While at it, report percentage done.

Output before this patch:

 * checking 6111 files for package collisions
5111 files remaining ...
4111 files remaining ...
3111 files remaining ...
2111 files remaining ...
 files remaining ...
111 files remaining ...

After:

 * checking 6158 files for package collisions
 16% done, 5131 files remaining ...
 33% done, 4104 files remaining ...
 50% done, 3077 files remaining ...
 66% done, 2050 files remaining ...
 83% done, 1023 files remaining ...
100% done

Signed-off-by: Fabian Groffen 
---
 lib/portage/dbapi/vartree.py | 14 +++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..78f2b37f2 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,19 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+   bucksize = 1000
+   buckcnt = int(totfiles / bucksize)
+   if buckcnt == 0 or totfiles % bucksize > int(bucksize / 
2):
+   buckcnt = buckcnt + 1
+   bucksize = int(totfiles / buckcnt) + 1
showMessage(_(" %s checking %d files for package 
collisions\n") % \
-   (colorize("GOOD", "*"), totfiles))
+   (colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
-   if i % 1000 == 0 and i != 0:
-   showMessage(_("%d files remaining 
...\n") % (totfiles - i))
+   if i % bucksize == 0 and i != 0:
+   showMessage(_("%3d%% done, %d files 
remaining ...\n") %
+   (i * 100 / totfiles, 
totfiles - i))
 
dest_path = normalize_path(
os.path.join(destroot, 
f.lstrip(os.path.sep)))
@@ -3570,6 +3576,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+   if bucksize < totfiles:
+   showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions, 
plib_collisions
 
def _lstat_inode_map(self, path_iter):
-- 
2.20.1




[gentoo-portage-dev] [PATCH] lib/portage/dbapi/vartree: use dynamic report interval in _collision_protect

2019-01-07 Thread Fabian Groffen
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be regular towards the end.  While at it, report percentage done.

Output before this patch:

 * checking 6111 files for package collisions
5111 files remaining ...
4111 files remaining ...
3111 files remaining ...
2111 files remaining ...
 files remaining ...
111 files remaining ...

After:

 * checking 6158 files for package collisions
16% done, 5131 files remaining ...
33% done, 4104 files remaining ...
50% done, 3077 files remaining ...
66% done, 2050 files remaining ...
83% done, 1023 files remaining ...

Signed-off-by: Fabian Groffen 
---
 lib/portage/dbapi/vartree.py | 10 --
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..4d0bf2789 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,19 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+   bucksize = 1000
+   buckcnt = int(totfiles / bucksize)
+   if buckcnt == 0 or totfiles % bucksize > int(bucksize / 
2):
+   buckcnt = buckcnt + 1
+   bucksize = int(totfiles / buckcnt) + 1
showMessage(_(" %s checking %d files for package 
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
-   if i % 1000 == 0 and i != 0:
-   showMessage(_("%d files remaining 
...\n") % (totfiles - i))
+   if i % bucksize == 0 and i != 0:
+   showMessage(_("%2d%% done, %d files 
remaining ...\n") %
+   (i * 100 / totfiles, 
totfiles - i))
 
dest_path = normalize_path(
os.path.join(destroot, 
f.lstrip(os.path.sep)))
-- 
2.20.1




Re: [RFC] gpkg format proposal v2 (was: Re: [gentoo-portage-dev] [RFC] Improving Gentoo package format)

2018-11-12 Thread Fabian Groffen
On 11-11-2018 21:53:33 +0100, Michał Górny wrote:
> Hi,
> 
> Ok, here's the second version integrating the feedback received.
> The format is much simpler, based on nested tarballs inspired by Debian.
> 
> The outer tarball is uncompressed and uses '.gpkg.tar' suffix.  It
> contains (preferably in order but PM should also handle packages with
> mismatched order):
> 
> 1. Optional (but recommended) "gpkg: ${PF}" package label that can be
> used to quickly distinguish Gentoo binpkgs from regular tarballs
> (for file(1)).
> 
> 2. "metadata.tar${comp}" tarball containing binary package metadata
> as files.
> 
> 3. Optional "metadata.tar${comp}.sig" containing detached signature
> for the metadata archive.
> 
> 4. "contents.tar${comp}" tarball containing files to be installed.
> 
> 5. Optional "contents.tar${comp}.sig" containing detached signature for
> the contents archive.
> 
> Notes:
> 
> a. ${comp} can be any compression format supported by binary packages. 
> Technically, metadata and content archives may use different
> compression.  Either or both may be uncompressed as well.

I'm wondering here, how much sense does it make to compress 2., 3.
and/or 4. if you compress the whole gpkg?  I have the impression
compression on compression isn't beneficial here.  Shouldn't just
compressing of the gpkg tar be sufficient?

As to allowing different compressors for a single gpkg, I think it would
be better to require all compressors to be the same, such that a PM or
tool can quickly see if it can "read" the file from the gpkg filename,
instead of having to fetch and open it first.  Obviously, if you drop
compression of the inner tars, this point goes away.

Thanks,
Fabian

> b. While signatures are optional, the PM should have a switch
> controlling whether to expect them, and fail hard if they're not present
> when expected.
> 
> 
> Advantages
> --
> Guaranteed:
> 
> + The binary package is still one file, so can be fetched easily.
> 
> + File format is trivial and can be extracted using tar(1) + compressor.
> 
> + The metadata and contents are compressed independently, and so can be
> easily extracted or modified independently.
> 
> + The package format provides for separate metadata and content
> signatures, so they can be verified independently.
> 
> + Metadata can be compressed now.
> 
> Achieved by regular archives (but might be broken if modified by user):
> 
> + Easy recognition by magic(1).
> 
> + The metadata archive (and its signature) is packed first, so it may be
> read without fetching the whole binpkg.
> 
> 
> Why not .ar format?
> ---
> The use of .ar format has been proposed, akin to Debian.  While
> the option is mostly feasible, and the simplicity of .ar format would
> reduce the outer size of binary packages, I think the format is simply
> too obscure.  It lives mostly as static library format, and the tooling
> for it is part of binutils.  LSB considers it deprecated.  While I don't
> see it going away anytime soon, I'd rather not rely on it in order to
> save a few KiB.
> 
> 
> Is there anything left to address?
> 
> -- 
> Best regards,
> Michał Górny



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH v2] rsync: quarantine data prior to verification (bug 660410)

2018-07-08 Thread Fabian Groffen
lg = False
>   is_synced = False
>   if timestamp != 0 and "--quiet" not in opts:
> @@ -686,6 +752,12 @@ class RsyncSync(NewBase):
>   elif (servertimestamp == 0) or (servertimestamp > 
> timestamp):
>   # actual sync
>   command = rsynccommand[:]
> +
> + if self.repo.location != download_dir:
> + # Use shared hardlinks for files that 
> are identical
> + # in the previous snapshot of the 
> repository.
> + command.append('--link-dest=%s' % 
> self.repo.location)
> +
>   submodule_paths = self._get_submodule_paths()
>   if submodule_paths:
>   # The only way to select multiple 
> directories to
> @@ -696,9 +768,10 @@ class RsyncSync(NewBase):
>   # /./ is special syntax 
> supported with the
>   # rsync --relative option.
>   command.append(syncuri + "/./" 
> + path)
> - command.append(self.repo.location)
>   else:
> - command.extend([syncuri + "/", 
> self.repo.location])
> + command.append(syncuri + "/")
> +
> + command.append(download_dir)
>  
>   exitcode = None
>   try:
> -- 
> 2.13.6
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)

2018-04-01 Thread Fabian Groffen
On 01-04-2018 20:29:59 +0200, Michał Górny wrote:
> > > This essentially looks like ~700 lines of code to try to workaround
> > > broken networking. I would rather try to do that using 5 lines of code
> > > but that's just me, and my programs aren't enterprise quality. I just
> > > hope it actually solves as many problems as it's going to introduce.
> > 
> > The vast majority of this code is generic and reusable, and I do intend
> > to reuse it. For example, the executor support will be an essential
> > piece for the asyncio.AbstractEventLoop for bug 649588.
> 
> Sure it is and sure you will. But tell me: who is going to maintain it
> all? Because as far as I can see, we're still dealing with a bus factor
> of one and all you're doing is making it worse. More code, more
> complexity, more creeping featurism and hacks.

Well, well, well.  This could be said about a lot of code, including
your own.

> Last time you went away, you left us with a horribly unmaintainable
> package manager full of complexity, hacks and creeping featurism
> and a Portage team whose members had barely any knowledge of the code.
> Just when things started moving again, you came back and we're back to
> square one.

I don't see why this has to be made personal.  In the olde days people
just pushed whatever they needed, now it's reviewed and acked, so how
can it be back to square one?

> Today Portage once again is a one-developer project, full of more
> complexity, more hacks and more creeping featurism. And we once again
> have a bus factor of one -- one developer who apparently knows
> everything, does everything and tries to be nice to everyone, except he
> really ignores others, makes a lot of empty promises and consistently
> makes the health of the project go from bad to worse.

Errr, didn't you recently start your own fork with creeping featurism of
its own because mainline was/is too slow?

> So, please tell me: what happens when you leave again? How have you used
> your time in the project? What have you done to make sure that
> the project stays alive without you? Because as far as I can see, adding
> few thousand of lines of practically unreviewed code every month does
> not help with that.

Why is this Zac's problem specifically?  Isn't this a general problem of
Gentoo?  And isn't your attitude contributing in a large factor to
the bus-factor getting down from 1 to 0?

> I forked Portage because I didn't want to fight with you. When I forked
> it, I declared that I will merge mainline changes regularly for
> the benefit of my users. But after a week, I really start feeling like
> that's going to be a really bad idea. Like it's time to forget about
> mainline Portage as a completely dead end, and go our separate ways.

So you fork.  Like many forks, that is because people feel that it isn't
going "their way".  Yay.  That doesn't make you "right".

> I'm seriously worried about the future of Gentoo. I'd really appreciate
> if you started focusing on that as well. I get that all this stuff looks
> cool on paper but few months or years from now, someone will curse
> 'whoever wrote that code' while trying to fix some nasty bug. Or get
> things moving forward. Or implement EAPI 8.

Perhaps it's time to look into the mirror.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH v2] __dyn_install: improve reporting of build and image sizes

2017-08-28 Thread Fabian Groffen
On 27-08-2017 12:53:23 -0700, Zac Medico wrote:
> On 08/27/2017 08:06 AM, Fabian Groffen wrote:
> > Prior to this commit, the reported sizes would look like:
> > 
> >  * Final size of build directory: 34942 KiB
> >  * Final size of installed tree: 5627 KiB
> > 
> > Because the sizes aren't aligned, it is hard to (visually) compare them.
> > On top of this, because the numbers are sometimes bigger, print a human
> > friendly size after the KiB size if applicable, like so:
> > 
> >  * Final size of build directory: 1906 KiB (1.8 MiB)
> >  * Final size of installed tree: 7 KiB
> > 
> > It should be noted that in case both sizes have a human-readable
> > variant, they are also aligned.
> > 
> > The helper functions are defined and used in a subshell to avoid
> > pollution of the caller's environment.
> > ---
> >  bin/phase-functions.sh | 53 
> > ++++++----
> >  1 file changed, 49 insertions(+), 4 deletions(-)
> 
> Looks good. Please merge.

Pushed, thanks!

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


[gentoo-portage-dev] [PATCH v2] __dyn_install: improve reporting of build and image sizes

2017-08-27 Thread Fabian Groffen
Prior to this commit, the reported sizes would look like:

 * Final size of build directory: 34942 KiB
 * Final size of installed tree: 5627 KiB

Because the sizes aren't aligned, it is hard to (visually) compare them.
On top of this, because the numbers are sometimes bigger, print a human
friendly size after the KiB size if applicable, like so:

 * Final size of build directory: 1906 KiB (1.8 MiB)
 * Final size of installed tree: 7 KiB

It should be noted that in case both sizes have a human-readable
variant, they are also aligned.

The helper functions are defined and used in a subshell to avoid
pollution of the caller's environment.
---
 bin/phase-functions.sh | 53 ++
 1 file changed, 49 insertions(+), 4 deletions(-)

diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index dfd8733c8..ce174ba91 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -598,10 +598,55 @@ __dyn_install() {
 
# record build & installed size in build log
if type -P du &>/dev/null; then
-   local sz=( $(du -ks "${WORKDIR}") )
-   einfo "Final size of build directory: ${sz[0]} KiB"
-   sz=( $(du -ks "${D}") )
-   einfo "Final size of installed tree: ${sz[0]} KiB"
+   local nsz=( $(du -ks "${WORKDIR}") )
+   local isz=( $(du -ks "${D}") )
+
+   # subshell to avoid polluting the caller env with the helper
+   # functions below
+   (
+   # align $1 to the right to the width of the widest of 
$1 and $2
+   padl() {
+   local s1=$1
+   local s2=$2
+   local width=${#s1}
+   [[ ${#s2} -gt ${width} ]] && width=${#s2}
+   printf "%*s" ${width} "${s1}"
+   }
+
+   # transform number in KiB into MiB, GiB or TiB based on 
size
+   human() {
+   local s1=$1
+   local units=( KiB MiB GiB TiB )
+
+   s1=$((s1 * 10))
+   while [[ ${s1} -gt 10240 && ${#units[@]} -gt 1 
]] ; do
+   s1=$((s1 / 1024 ))
+   units=( ${units[@]:1} )
+   done
+
+   local r=${s1: -1}
+   s1=$((s1 / 10))
+   printf "%s.%s %s" "${s1}" "${r}" "${units[0]}"
+   }
+
+   size() {
+   local s1=$1
+   local s2=$2
+   local out="$(padl "${s1}" "${s2}") KiB"
+
+   if [[ ${s1} -gt 1024 ]] ; then
+   s1=$(human ${s1})
+   if [[ ${s2} -gt 1024 ]] ; then
+   s2=$(human ${s2})
+   s1=$(padl ${s1} ${s2})
+   fi
+   out+=" (${s1})"
+   fi
+   echo "${out}"
+   }
+   einfo "Final size of build directory: $(size ${nsz[0]} 
${isz[0]})"
+   einfo "Final size of installed tree:  $(size ${isz[0]} 
${nsz[0]})"
+   )
__vecho
fi
 
-- 
2.14.1




Re: [gentoo-portage-dev] [PATCH] _collision_protect: report progress in work todo

2017-08-27 Thread Fabian Groffen
On 26-08-2017 14:46:30 -0700, Zac Medico wrote:
> On 08/24/2017 06:28 AM, Fabian Groffen wrote:
> > Currently Portage reports its progress in checking collisions forward
> > every 1000th file like so:
> > 
> >  * checking 4149 files for package collisions
> > 1000 files checked ...
> > 2000 files checked ...
> > 3000 files checked ...
> > 4000 files checked ...
> >>>> Merging sys-apps/portage-2.3.8 to /
> > 
> > Change it to countdown style so it is easier to anticipate what the
> > next action will be:
> > 
> >  * checking 4149 files for package collisions
> > 3149 files remaining ...
> > 2149 files remaining ...
> > 1149 files remaining ...
> > 149 files remaining ...
> >>>> Merging sys-apps/portage-2.3.8 to /
> > ---
> >  pym/portage/dbapi/vartree.py | 5 +++--
> >  1 file changed, 3 insertions(+), 2 deletions(-)
> > 
> > diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
> > index 7c8f150bb..04a40b732 100644
> > --- a/pym/portage/dbapi/vartree.py
> > +++ b/pym/portage/dbapi/vartree.py
> > @@ -3420,13 +3420,14 @@ class dblink(object):
> > dirs_ro = set()
> > symlink_collisions = []
> > destroot = self.settings['ROOT']
> > +   totfiles = len(file_list) + len(symlink_list)
> > showMessage(_(" %s checking %d files for package 
> > collisions\n") % \
> > -   (colorize("GOOD", "*"), len(file_list) + 
> > len(symlink_list)))
> > +   (colorize("GOOD", "*"), totfiles))
> > for i, (f, f_type) in enumerate(chain(
> > ((f, "reg") for f in file_list),
> > ((f, "sym") for f in symlink_list))):
> > if i % 1000 == 0 and i != 0:
> > -   showMessage(_("%d files checked ...\n") 
> > % i)
> > +   showMessage(_("%d files remaining 
> > ...\n") % (totfiles - i))
> >  
> > dest_path = normalize_path(
> > os.path.join(destroot, 
> > f.lstrip(os.path.sep)))
> > 
> 
> Looks good. Please merge.

Pushed, thanks

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


[gentoo-portage-dev] [PATCH] _collision_protect: report progress in work todo

2017-08-24 Thread Fabian Groffen
Currently Portage reports its progress in checking collisions forward
every 1000th file like so:

 * checking 4149 files for package collisions
1000 files checked ...
2000 files checked ...
3000 files checked ...
4000 files checked ...
>>> Merging sys-apps/portage-2.3.8 to /

Change it to countdown style so it is easier to anticipate what the
next action will be:

 * checking 4149 files for package collisions
3149 files remaining ...
2149 files remaining ...
1149 files remaining ...
149 files remaining ...
>>> Merging sys-apps/portage-2.3.8 to /
---
 pym/portage/dbapi/vartree.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
index 7c8f150bb..04a40b732 100644
--- a/pym/portage/dbapi/vartree.py
+++ b/pym/portage/dbapi/vartree.py
@@ -3420,13 +3420,14 @@ class dblink(object):
dirs_ro = set()
symlink_collisions = []
destroot = self.settings['ROOT']
+   totfiles = len(file_list) + len(symlink_list)
showMessage(_(" %s checking %d files for package 
collisions\n") % \
-   (colorize("GOOD", "*"), len(file_list) + 
len(symlink_list)))
+   (colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
if i % 1000 == 0 and i != 0:
-   showMessage(_("%d files checked ...\n") 
% i)
+   showMessage(_("%d files remaining 
...\n") % (totfiles - i))
 
dest_path = normalize_path(
os.path.join(destroot, 
f.lstrip(os.path.sep)))
-- 
2.14.1




[gentoo-portage-dev] [PATCH] __dyn_install: improve reporting of build and image sizes

2017-08-24 Thread Fabian Groffen
Prior to this commit, the reported sizes would look like:

 * Final size of build directory: 34942 KiB
 * Final size of installed tree: 5627 KiB

Because the sizes aren't aligned, it is hard to (visually) compare them.
On top of this, because the numbers are sometimes bigger, print a human
friendly size after the KiB size if applicable, like so:

 * Final size of build directory: 1906 KiB (1.8 MiB)
 * Final size of installed tree: 7 KiB

It should be noted that in case both sizes have a human-readable
variant, they are also aligned.
---
 bin/phase-functions.sh | 49 +
 1 file changed, 45 insertions(+), 4 deletions(-)

diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index dfd8733c8..af45a0d49 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -598,10 +598,51 @@ __dyn_install() {
 
# record build & installed size in build log
if type -P du &>/dev/null; then
-   local sz=( $(du -ks "${WORKDIR}") )
-   einfo "Final size of build directory: ${sz[0]} KiB"
-   sz=( $(du -ks "${D}") )
-   einfo "Final size of installed tree: ${sz[0]} KiB"
+   local nsz=( $(du -ks "${WORKDIR}") )
+   local isz=( $(du -ks "${D}") )
+
+   # align $1 to the right to the width of the widest of $1 and $2
+   padl() {
+   local s1=$1
+   local s2=$2
+   local width=${#s1}
+   [[ ${#s2} -gt ${width} ]] && width=${#s2}
+   printf "%*s" ${width} "${s1}"
+   }
+
+   # transform number in KiB into MiB, GiB or TiB based on size
+   human() {
+   local s1=$1
+   local units=( KiB MiB GiB TiB )
+
+   s1=$((s1 * 10))
+   while [[ ${s1} -gt 10240 && ${#units[@]} -gt 1 ]] ; do
+   s1=$((s1 / 1024 ))
+   units=( ${units[@]:1} )
+   done
+
+   local r=${s1: -1}
+   s1=$((s1 / 10))
+   printf "%s.%s %s" "${s1}" "${r}" "${units[0]}"
+   }
+
+   size() {
+   local s1=$1
+   local s2=$2
+   local out="$(padl "${s1}" "${s2}") KiB"
+
+   if [[ ${s1} -gt 1024 ]] ; then
+   s1=$(human ${s1})
+   if [[ ${s2} -gt 1024 ]] ; then
+   s2=$(human ${s2})
+   s1=$(padl ${s1} ${s2})
+   fi
+   out+=" (${s1})"
+   fi
+   echo "${out}"
+   }
+   einfo "Final size of build directory: $(size ${nsz[0]} 
${isz[0]})"
+   einfo "Final size of installed tree:  $(size ${isz[0]} 
${nsz[0]})"
__vecho
fi
 
-- 
2.14.1




[gentoo-portage-dev] [PATCH] etc-update: fix hang when using_editor is set, bug #544440

2017-02-22 Thread Fabian Groffen
Don't try to use an interactive editor to obtain the difference between
files.  Fall back to plain `diff -q` in this case.
---
 bin/etc-update | 19 +++
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/bin/etc-update b/bin/etc-update
index e0f7224dc..ea69f1478 100755
--- a/bin/etc-update
+++ b/bin/etc-update
@@ -649,10 +649,21 @@ do_distconf() {
elif [[ -L ${efile} || -L ${file} ]] ; then
# not the same file types
continue
-   elif diff_command "${file}" "${efile}" &> /dev/null; then
-   # replace identical copy
-   mv "${file}" "${efile}"
-   break
+   else
+   local ret=
+   if [[ ${using_editor} == 0 ]] ; then
+   diff_command "${file}" "${efile}" &> /dev/null
+   ret=$?
+   else
+   # fall back to plain diff
+   diff -q "${file}" "${efile}" &> /dev/null
+   ret=$?
+   fi
+   if [[ ${ret} == 0 ]] ; then
+   # replace identical copy
+   mv "${file}" "${efile}"
+   break
+   fi
fi
done
 }
-- 
2.11.1




[gentoo-portage-dev] [PATCH] getgccversion: check for clang if gcc is not found

2016-02-18 Thread Fabian Groffen
In Gentoo Prefix on Mac OS X systems, LLVM/clang is used as the
toolchain, so no gcc/gcc-config around.  Probe for clang after we failed
to locate gcc, so we can show clang version instead.
---
 pym/_emerge/actions.py | 39 ++-
 1 file changed, 38 insertions(+), 1 deletion(-)

diff --git a/pym/_emerge/actions.py b/pym/_emerge/actions.py
index 59626ad..1d324aa 100644
--- a/pym/_emerge/actions.py
+++ b/pym/_emerge/actions.py
@@ -39,6 +39,7 @@ from portage import os
 from portage import shutil
 from portage import eapi_is_supported, _encodings, _unicode_decode
 from portage.cache.cache_errors import CacheError
+from portage.const import EPREFIX
 from portage.const import GLOBAL_CONFIG_PATH, VCS_DIRS, 
_DEPCLEAN_LIB_CHECK_DEFAULT
 from portage.const import SUPPORTED_BINPKG_FORMATS, TIMESTAMP_FORMAT
 from portage.dbapi.dep_expand import dep_expand
@@ -2406,12 +2407,23 @@ def getgccversion(chost=None):
gcc_ver_command = ['gcc', '-dumpversion']
gcc_ver_prefix = 'gcc-'
 
+   clang_ver_command = ['clang', '--version']
+   clang_ver_prefix = 'clang-'
+
+   ubinpath = os.path.join('/', portage.const.EPREFIX, 'usr', 'bin')
+
gcc_not_found_error = red(
"!!! No gcc found. You probably need to 'source /etc/profile'\n" +
"!!! to update the environment of this terminal and possibly\n" +
"!!! other terminals also.\n"
)
 
+   def getclangversion(output):
+   version = re.search('clang version ([0-9.]+) ', output)
+   if version:
+   return version.group(1)
+   return "unknown"
+
if chost:
try:
proc = subprocess.Popen(["gcc-config", "-c"],
@@ -2439,7 +2451,7 @@ def getgccversion(chost=None):
return gcc_ver_prefix + myoutput
 
try:
-   proc = subprocess.Popen(gcc_ver_command,
+   proc = subprocess.Popen([ubinpath + "/" + gcc_ver_command[0]] + 
gcc_ver_command[1:],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
except OSError:
myoutput = None
@@ -2450,6 +2462,31 @@ def getgccversion(chost=None):
if mystatus == os.EX_OK:
return gcc_ver_prefix + myoutput
 
+   try:
+   proc = subprocess.Popen(
+   [ubinpath + "/" + chost + "-" + clang_ver_command[0]] + 
clang_ver_command[1:],
+   stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+   except OSError:
+   myoutput = None
+   mystatus = 1
+   else:
+   myoutput = _unicode_decode(proc.communicate()[0]).rstrip("\n")
+   mystatus = proc.wait()
+   if mystatus == os.EX_OK:
+   return clang_ver_prefix + getclangversion(myoutput)
+
+   try:
+   proc = subprocess.Popen([ubinpath + "/" + clang_ver_command[0]] 
+ clang_ver_command[1:],
+   stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+   except OSError:
+   myoutput = None
+   mystatus = 1
+   else:
+   myoutput = _unicode_decode(proc.communicate()[0]).rstrip("\n")
+   mystatus = proc.wait()
+   if mystatus == os.EX_OK:
+   return clang_ver_prefix + getclangversion(myoutput)
+
portage.writemsg(gcc_not_found_error, noiselevel=-1)
return "[unavailable]"
 
-- 
2.7.0




[gentoo-portage-dev] [PATCH] consider clang when looking up compiler version

2016-02-18 Thread Fabian Groffen
This is a patch from prefix branch to make the compiler detection
consider clang.

Fabian Groffen (1):
  getgccversion: check for clang if gcc is not found

 pym/_emerge/actions.py | 39 ++-
 1 file changed, 38 insertions(+), 1 deletion(-)

-- 
2.7.0




Re: [gentoo-portage-dev] [PATCH 3/3] Add CPU model name to output of getportageversion as fifth element

2013-08-15 Thread Fabian Groffen
On 14-08-2013 22:10:40 +0200, Mark Kubacki wrote:
 From: W-Mark Kubacki wm...@hurrikane.de
 
 It will read like this:
  Portage 2.1.13.7 (default/linux/amd64/13.0, gcc-4.6.2, glibc-2.18, 
  3.9.0-rc8-mark-signed+ x86_64, Intel(R) Core(TM) i7-3770T CPU @ 2.50GHz)
 
 That new fifth element will be the CPU model name of the host
 running Portage. It is not the target architecture!

FYI:

% python3
Python 3.3.2 (default, Jul 24 2013, 11:14:02) 
[GCC 4.2.1 (Gentoo 4.2.1_p5666-r1, Apple Inc. build 5666) (dot 3)] on darwin
Type help, copyright, credits or license for more information.
 import platform
 platform.release()
'12.4.0'
 platform.machine()
'x86_64'
 platform.processor()
'i386'
 

% python
Python 3.2.3 (default, May  6 2013, 21:19:05) 
[GCC 4.7.2] on sunos5
Type help, copyright, credits or license for more information.
 import platform
 platform.release()
'5.11'
 platform.machine()
'i86pc'
 platform.processor()
'i386'
 

% python
Python 3.3.2 (default, Jul 15 2013, 13:51:24) 
[GCC 4.7.2] on sunos5
Type help, copyright, credits or license for more information.
 import platform
 platform.release()
'5.10'
 platform.machine()
'sun4u'
 platform.processor()
'sparc'
 

% python
Python 3.2.5 (default, Jul 15 2013, 11:37:08) 
[GCC 4.6.3] on linux2
Type help, copyright, credits or license for more information.
 import platform
 platform.release()
'3.8.13-gentoo'
 platform.machine()
'x86_64'
 platform.processor()
'AMD Athlon(tm) 64 X2 Dual Core Processor 3800+'
 

e.g. it seems to me only on Linux it gives fancy model output.  Note
that the first and second system were running a 64-bit Python as well as
kernel.

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


Re: [gentoo-portage-dev] [PATCH 0/3] Add cross-prefix support

2013-06-17 Thread Fabian Groffen
Hi Ruud,

On 17-06-2013 09:21:41 +0200, Ruud Koolen wrote:
 This patch series adds support for using a portage installed in one prefix
 to build packages with a different prefix.

Thanks so much for your work.  I hope to have a look at this soon, and
merge this.  From your work I assume your patches don't break normal
installs/bootstraps, right?  (You tested that?)

Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


[gentoo-portage-dev] Re: proj/portage:master commit in: man/, pym/portage/dbapi/, pym/portage/, cnf/

2012-10-25 Thread Fabian Groffen
On 25-10-2012 08:45:36 +, Zac Medico wrote:
 URL:
 http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=10b6d012
 
 Add FEATURES=merge-sync, for bug #439584.

 + if not self._device_path_map or \
 + merge-sync not in self.settings.features:
 + return
 +
 + syncfs = self._get_syncfs()
 + if syncfs is None:
 + try:
 + proc = subprocess.Popen([sync])
 + except EnvironmentError:
 + pass
 + else:
 + proc.wait()
 + else:
 + for path in self._device_path_map.values():
 + try:
 + fd = os.open(path, os.O_RDONLY)
 + except OSError:
 + pass
 + else:
 + try:
 + syncfs(fd)

can't you use os.fdatasync() or os.fsync() instead?


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


Re: [gentoo-portage-dev] Re: proj/portage:master commit in: man/, pym/portage/dbapi/, pym/portage/, cnf/

2012-10-25 Thread Fabian Groffen
On 25-10-2012 02:45:41 -0700, Zac Medico wrote:
  can't you use os.fdatasync() or os.fsync() instead?
 
 os.fdatasync() or os.fsync() only work for single files, right? We don't
 want to sync them individually, since that wastes disk bandwidth. It's
 much more efficient to sync the whole filesystem at once, like syncfs does.

You use syncfs, which according to the man-page requires a
filedescriptor.  Hence my suggestion.


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


Re: [gentoo-portage-dev] About adding a repoman check to test for missing files in FILESDIR

2012-10-09 Thread Fabian Groffen
On 08-10-2012 20:47:41 +0200, Pacho Ramos wrote:
 El lun, 08-10-2012 a las 11:32 -0700, Zac Medico escribió:
  Given the expressiveness of the bash language, it's really a non-trivial
  thing to check without actually executing bash. See discussion here:
  
https://bugs.gentoo.org/show_bug.cgi?id=431196
 
 Then, maybe I could have a custom repoman script that would check for
 them in my files/ directory (either using bash to check for them or
 simply running ls)  :/

Yes, but still it is very hard to make it 100% reliable.  If you stick
to simple ${FILESDIR}/${PN}-x.y.patch patterns, it's very well
feasible, though.  See for inspiration the fetch_files() function from [1].


[1] 
http://overlays.gentoo.org/proj/alt/browser/trunk/prefix-overlay/scripts/eupdate.in


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


[gentoo-portage-dev] Re: proj/portage:master commit in: pym/portage/dbapi/

2011-11-26 Thread Fabian Groffen
On 26-11-2011 16:56:41 +0530, Nirbheek Chauhan wrote:
 On Sat, Nov 26, 2011 at 4:28 PM, Fabian Groffen grob...@gentoo.org wrote:
  On 26-11-2011 01:54:35 +, Arfrever Frehtes Taifersar Arahesis wrote:
  commit:     1d4ac47c28706094230cb2c4e6ee1c1c71629aa0
  T Org
  AuthorDate: Sat Nov 26 01:52:49 2011 +
  Commit:     Arfrever Frehtes Taifersar Arahesis arfrever AT gentoo 
  DOT org
  CommitDate: Sat Nov 26 01:52:49 2011 +
  URL:        
  http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=1d4ac47c
 
  dblink.mergeme(): Merge files in alphabetic order.
 
  What's the advantage of this?  I don't really like to pay for sorting a
  potentially huge list just for some eye-candy.  (That's omitted by
  default these days anyway...)
  Any other opinions on this one?
 
 
 If it should be sorted[1], it should really be sorted in the reverse
 order of distfile-download size. That would be extremely useful on
 systems with slow internet connections. Too many times have I sat
 waiting for libreoffice-bin to download while a webkit-gtk recompile
 waits in the queue.
 
 We already have the information during dependency resolution with
 --verbose, and it costs very little. Besides, sorting even 30,000
 entries (if you're merging every ebuild in portage) should not take
 more than a few secs.

A linux kernel has around that much of files, and I really wonder if
it's worth waiting a couple of seconds (probably more on sparc and arm
systems) just because then the files are in sorted order.

 1. I'm obviously assuming that dep nodes that do not depend on each
 other would be sorted

I think this is per package.

I didn't watch closely enough the reply-to headers, the
gentoo-portage-dev list was my original target, which obviously makes
more sense for this context.


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


[gentoo-portage-dev] Re: proj/portage:master commit in: pym/repoman/, pym/portage/tests/repoman/

2011-10-21 Thread Fabian Groffen
On 21-10-2011 06:09:57 +, Zac Medico wrote:
 UpdateChangeLog: split out/test copyright regex
 
 This also fixes a case where something like Copyright 2011  would be
 replaced with Copyright 2011-2011 .

I like this, but it should only happen when one would live in 1999
(that's why I added it unconditionally), see below.  I'm affraid we need
the 1999 to be configured, for legal reasons (though I don't claim to be
an expert on this case, just judging from the comments from echangelog).

 + def testCopyrightUpdate(self):
 + test_cases = (
 + (
 + '2011',
 + '# Copyright 1999-2008 Gentoo Foundation; 
 Distributed under the GPL v2',
 + '# Copyright 1999-2011 Gentoo Foundation; 
 Distributed under the GPL v2',

we should add a case like

# Copyright 2010-2011 Gentoo Foundation; Distributed ...

echangelog would have rewritten it to 1999-2011, because of legal
reasons IIRC.

 +def _update_copyright_year(year, line):
 + 
 + These two regexes are taken from echangelog
 + update_copyright(), except that we don't hardcode
 + 1999 here (in order to be more generic).

This comment makes little to no more sense, since the patterns taken
from echangelog have changed considerably now :)


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


Re: [gentoo-portage-dev] Re: proj/portage:master commit in: pym/repoman/, pym/portage/tests/repoman/

2011-10-21 Thread Fabian Groffen
On 21-10-2011 00:30:05 -0700, Zac Medico wrote:
 On 10/21/2011 12:09 AM, Fabian Groffen wrote:
  On 21-10-2011 06:09:57 +, Zac Medico wrote:
  UpdateChangeLog: split out/test copyright regex
 
  This also fixes a case where something like Copyright 2011  would be
  replaced with Copyright 2011-2011 .
  
  I like this, but it should only happen when one would live in 1999
  (that's why I added it unconditionally), see below.  I'm affraid we need
  the 1999 to be configured, for legal reasons (though I don't claim to be
  an expert on this case, just judging from the comments from echangelog).
 
 I don't see anything like that in echangelog. What would be the point of
 re-writing copyright start dates, when those should already be correctly
 inherited from skel.ChangeLog anyway?

Sure, but this is mainly intended for ebuilds and files (like init
scripts?), I guess.  I considered using skel.ebuild for .ebuild files,
perhaps that's just the best to avoid hardcoding any years.


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


Re: [gentoo-portage-dev] [PATCH 2 of 3] repoman: get ChangeLog header from skel.ChangeLog

2011-10-20 Thread Fabian Groffen
On 19-10-2011 14:58:39 -0700, Zac Medico wrote:
 On 10/19/2011 12:55 PM, Fabian Groffen wrote:
  +   if clold_lines[-1].strip():
  +   f.write(clold_lines[-1])
 
 If the old ChangeLog happens to be an empty file, then clold_lines[-1]
 will raise IndexError. So, this is safer:
 
if clold_lines and clold_lines[-1].strip():

I applied your fix, and moved it from the other patch in this one, where
it belongs.


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


[gentoo-portage-dev] [PATCH] repoman: update copyright on modified files

2011-10-20 Thread Fabian Groffen
To retain the behaviour of echangelog, update the copyrights on modified
files (mostly ebuilds) when necessary.  Also update the ChangeLog's
copyright.

diff --git a/pym/repoman/utilities.py b/pym/repoman/utilities.py
--- a/pym/repoman/utilities.py
+++ b/pym/repoman/utilities.py
@@ -523,9 +523,77 @@
 
return outvcs
 
+def update_copyright(fn_path, year, pretend):
+   
+   Check file for a Copyright statement, and update its year.  The
+   patterns used for replacing copyrights are taken from echangelog.
+   Only the first lines of each file that start with a hash ('#') are
+   considered, until a line is found that doesn't start with a hash.
+   
+
+   try:
+   fn_hdl = io.open(_unicode_encode(fn_path,
+   encoding=_encodings['fs'], errors='strict'),
+   mode='r', encoding=_encodings['repo.content'], 
errors='replace')
+   except EnvironmentError:
+   return
+
+   orig_header = []
+   new_header = []
+
+   for line in fn_hdl:
+   line_strip = line.strip()
+   orig_header.append(line)
+   if not line_strip or line_strip[:1] != '#':
+   new_header.append(line)
+   break
+
+   # these two regexes are taken from
+   # echangelog update_copyright()
+   line = re.sub(r'^(# Copyright \d+) ',
+   r'\1-%s ' % year, line)
+   line = re.sub(r'^(# Copyright) \d\d\d\d-\d\d\d\d',
+   r'\1 1999-%s' % year, line)
+   new_header.append(line)
+
+   difflines = 0
+   for line in difflib.unified_diff(orig_header, new_header,
+   fromfile=fn_path, tofile=fn_path, n=0):
+   util.writemsg_stdout(line, noiselevel=-1)
+   difflines += 1
+   util.writemsg_stdout(\n, noiselevel=-1)
+
+   # unified diff has three lines to start with
+   if difflines  3 and not pretend:
+   # write new file with changed header
+   f, fnnew_path = mkstemp()
+   f = io.open(f, mode='w', encoding=_encodings['repo.content'],
+   errors='backslashreplace')
+   for line in new_header:
+   f.write(line);
+   for line in fn_hdl:
+   f.write(line)
+   f.close()
+   try:
+   fn_stat = os.stat(fn_path)
+   except OSError:
+   fn_stat = None
+
+   shutil.move(fnnew_path, fn_path)
+
+   if fn_stat is None:
+   util.apply_permissions(fn_path, mode=0o644)
+   else:
+   util.apply_stat_permissions(fn_path, fn_stat)
+   fn_hdl.close()
+
 def UpdateChangeLog(pkgdir, category, package, new, removed, changed, \
msg, pretend, repodir):
-Write an entry to an existing ChangeLog, or create a new one. 
+   
+   Write an entry to an existing ChangeLog, or create a new one.
+   Updates copyright year on changed files, and updates the header of
+   ChangeLog with the contents of skel.ChangeLog.
+   
 
# figure out who to write as
if 'GENTOO_COMMITTER_NAME' in os.environ and \
@@ -548,6 +616,16 @@
logging.critical(err)
return None
 
+   year = time.strftime('%Y')
+   date = time.strftime('%d %b %Y')
+
+   # check modified files and the ChangeLog for copyright updates
+   # patches and diffs (identified by .patch and .diff) are excluded
+   for fn in new + changed:
+   if fn.endswith('.diff') or fn.endswith('.patch'):
+   continue
+   update_copyright(os.path.join(pkgdir, fn), year, pretend)
+
cl_path = os.path.join(pkgdir, 'ChangeLog')
clold_lines = []
clnew_lines = []
@@ -584,8 +662,9 @@
clold_lines.append(line)
if line_strip[:1] != '#':
break
-   if clskel_file is None:
-   clnew_lines.append(line)
+   line = re.sub(r'^(# Copyright) 
\d\d\d\d-\d\d\d\d',
+   r'\1 1999-%s' % year, line)
+   clnew_lines.append(line)
if not line_strip:
break
elif clskel_file is not None:
@@ -597,7 +676,7 @@
line = line.replace('CATEGORY', category)
line = line.replace('PACKAGE_NAME', package)
line = re.sub(r'^(# Copyright 
\d\d\d\d)-\d\d\d\d ',
-   r'\1-%s ' % time.strftime('%Y'), line)
+   

[gentoo-portage-dev] [PATCH 3 of 3] repoman: update copyright on modified files

2011-10-19 Thread Fabian Groffen
To retain the behaviour of echangelog, update the copyrights on modified
files (mostly ebuilds) when necessary.  We also use this to update the
ChangeLog's copyright.

diff --git a/pym/repoman/utilities.py b/pym/repoman/utilities.py
--- a/pym/repoman/utilities.py
+++ b/pym/repoman/utilities.py
@@ -523,9 +523,84 @@
 
return outvcs
 
+def update_copyrights(pkgdir, files, year, pretend):
+   
+   Check list of files for Copyright statements, and update them to
+   year.  The patterns used for replacing copyrights are taken from
+   echangelog.  Only the first lines of each file that start with a
+   hash ('#') are considered, until a line is found that doesn't start
+   with a hash.
+   patches and diffs (identified by .patch and .diff) are excluded
+   
+
+   for fn in files:
+   if fn.endswith('.diff') or fn.endswith('.patch'):
+   continue
+
+   fn_path = os.path.join(pkgdir, fn)
+   try:
+   fn_hdl = io.open(_unicode_encode(fn_path,
+   encoding=_encodings['fs'], errors='strict'),
+   mode='r', encoding=_encodings['repo.content'], 
errors='replace')
+   except EnvironmentError:
+   continue
+
+   orig_header = []
+   new_header = []
+
+   for line in fn_hdl:
+   line_strip = line.strip()
+   orig_header.append(line)
+   if not line_strip or line_strip[:1] != '#':
+   new_header.append(line)
+   break
+
+   # these two regexes are taken from
+   # echangelog update_copyright()
+   line = re.sub(r'^(# Copyright \d+) ',
+   r'\1-%s ' % year, line)
+   line = re.sub(r'^(# Copyright) \d\d\d\d-\d\d\d\d',
+   r'\1 1999-%s' % year, line)
+   new_header.append(line)
+
+   difflines = 0
+   for line in difflib.unified_diff(orig_header, new_header,
+   fromfile=fn_path, tofile=fn_path, n=0):
+   util.writemsg_stdout(line, noiselevel=-1)
+   difflines += 1
+   util.writemsg_stdout(\n, noiselevel=-1)
+
+   # unified diff has three lines to start with
+   if difflines  3 and not pretend:
+   # write new file with changed header
+   f, fnnew_path = mkstemp()
+   f = io.open(f, mode='w', 
encoding=_encodings['repo.content'],
+   errors='backslashreplace')
+   for line in new_header:
+   f.write(line);
+   for line in fn_hdl:
+   f.write(line)
+   f.close()
+   try:
+   fn_stat = os.stat(fn_path)
+   except OSError:
+   fn_stat = None
+
+   shutil.move(fnnew_path, fn_path)
+
+   if fn_stat is None:
+   util.apply_permissions(fn_path, mode=0o644)
+   else:
+   util.apply_stat_permissions(fn_path, fn_stat)
+   fn_hdl.close()
+
 def UpdateChangeLog(pkgdir, category, package, new, removed, changed, \
msg, pretend, repodir):
-Write an entry to an existing ChangeLog, or create a new one. 
+   
+   Write an entry to an existing ChangeLog, or create a new one.
+   Updates copyright year on changed files, and updates the header of
+   ChangeLog with the contents of skel.ChangeLog.
+   
 
# figure out who to write as
if 'GENTOO_COMMITTER_NAME' in os.environ and \
@@ -548,6 +623,12 @@
logging.critical(err)
return None
 
+   year = time.strftime('%Y')
+   date = time.strftime('%d %b %Y')
+
+   # check modified files and the ChangeLog for copyright updates
+   update_copyrights(pkgdir, new + changed + ['ChangeLog'], year, pretend)
+
cl_path = os.path.join(pkgdir, 'ChangeLog')
clold_lines = []
clnew_lines = []
@@ -578,7 +659,8 @@
# construct correct header first
try:
if clold_file is not None:
-   # retain header from old ChangeLog
+   # retain header from old ChangeLog, its copyright has
+   # already been updated by update_copyrights
for line in clold_file:
line_strip = line.strip()
clold_lines.append(line)
@@ -597,7 +679,7 @@
line = 

[gentoo-portage-dev] [PATCH 0 of 3] repoman changelog patches

2011-10-19 Thread Fabian Groffen
The following three patches change repoman's echangelog feature
slightly:

1) default for updating the changelog is retrieved from layout.conf
2) skel.ChangeLog is used for constructing the header of new ChangeLogs
3) modified files are checked for copyright years to be updated

Please review before I push them.



Re: [gentoo-portage-dev] [PATCH 2/2] prepstrip: add support for elfutils strip

2011-10-11 Thread Fabian Groffen
On 11-10-2011 10:18:16 -0400, Mike Frysinger wrote:
  For the sake of it, can we try to detect GNU binutils here?  Our
  profiles restrict strip, so no immediate problem here if not.
  
 adding detection here will fix most, but i don't think all, of your troubles
 
 the code still unconditionally runs `strip -g` which i'm guessing won't work 
 for you ...
 
 also, we only run strip on ELF files.  so if you aren't an ELF system, even 
 fixing the -g won't help.

Actually, on Solaris (ELF) the way Portage uses strip will damage the
files in such a way that they become unusable.  And we're using GNU
binutils there.  So I guess it's ok to assume here.  In the profiles
stripping is just disabled everywhere it breaks.


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


Re: [gentoo-portage-dev] EbuildProcess logs poll-error to already removed $T (on AIX)

2011-03-25 Thread Fabian Groffen
()
   File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/PackageMerge.py, 
 line 43, in _start
 self.returncode = self.merge.merge()
   File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/MergeListItem.py, 
 line 147, in merge
 retval = self._install_task.install()
   File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/EbuildBuild.py, 
 line 348, in install
 rval = merge.execute()
   File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/EbuildMerge.py, 
 line 26, in execute
 blockers=self.find_blockers)
   File /big5tk/local/gprefix/usr/lib/portage/pym/portage/dbapi/vartree.py, 
 line 4064, in merge
 mydbapi=mydbapi, prev_mtimes=prev_mtimes)
   File /big5tk/local/gprefix/usr/lib/portage/pym/portage/dbapi/vartree.py, 
 line 3915, in merge
 mydbapi=mydbapi, prev_mtimes=prev_mtimes)
   File /big5tk/local/gprefix/usr/lib/portage/pym/portage/dbapi/vartree.py, 
 line 3936, in _merge
 cleanup=cleanup, mydbapi=mydbapi, prev_mtimes=prev_mtimes)
   File /big5tk/local/gprefix/usr/lib/portage/pym/portage/dbapi/vartree.py, 
 line 3341, in treewalk
 others_in_slot=others_in_slot)
   File /big5tk/local/gprefix/usr/lib/portage/pym/portage/dbapi/vartree.py, 
 line 1700, in unmerge
 myebuildpath, cleanrm)
   File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/Scheduler.py, line 
 737, in _dblink_ebuild_phase
 ebuild_phase.wait()
   File 
 /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AsynchronousTask.py, line 
 40, in wait
 self._wait()
   File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/CompositeTask.py, 
 line 72, in _wait
 task.wait()
   File 
 /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AsynchronousTask.py, line 
 40, in wait
 self._wait()
   File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/SubProcess.py, 
 line 76, in _wait
 self.scheduler.schedule(self._reg_id)
   File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/PollScheduler.py, 
 line 286, in _schedule_wait
 handler(f, event)
   File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/SpawnProcess.py, 
 line 224, in _dummy_handler
 self._unregister_if_appropriate(event)
   File 
 /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AbstractPollTask.py, line 
 56, in _unregister_if_appropriate
 self._log_poll_exception(event)
   File 
 /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AbstractEbuildProcess.py, 
 line 234, in _log_poll_exception
 (self.__class__.__name__, event,)])
   File 
 /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AbstractEbuildProcess.py, 
 line 229, in _elog
 log_path=self.settings.get(PORTAGE_LOG_FILE))
   File /big5tk/local/gprefix/usr/lib/portage/pym/_emerge/PollScheduler.py, 
 line 327, in _task_output
 mode='ab')
 IOError: [Errno 2] No such file or directory: 
 '/big5tk/tmp/gprefix/portage/._unmerge_/app-misc/mime-types-8/temp/build.log'
 


-- 
Fabian Groffen
Gentoo on a different level



Re: [gentoo-portage-dev] Portage on AIX

2010-11-24 Thread Fabian Groffen
Hi,

On 24-11-2010 12:17:12 -0600, Perry Smith wrote:
 I don't use Gentoo but I bumped into Portage on a recent quest.  Long story 
 short, I'd like to try and port it to AIX.  I'm writing this list to see if 
 you think that is possible or not.

We use Portage Prefix on AIX, so I think we can say it works.  Do you
really need Portage (the package manager) or the entire set of
applications provided by it?


-- 
Fabian Groffen
Gentoo on a different level



Re: [gentoo-portage-dev] Portage on AIX

2010-11-24 Thread Fabian Groffen
On 24-11-2010 12:29:41 -0600, Perry Smith wrote:
 
 On Nov 24, 2010, at 12:22 PM, Fabian Groffen wrote:
 
  Hi,
  
  On 24-11-2010 12:17:12 -0600, Perry Smith wrote:
  I don't use Gentoo but I bumped into Portage on a recent quest.  Long 
  story short, I'd like to try and port it to AIX.  I'm writing this list to 
  see if you think that is possible or not.
  
  We use Portage Prefix on AIX, so I think we can say it works.  Do you
  really need Portage (the package manager) or the entire set of
  applications provided by it?
 
 I'm assuming the latter does not exist.  I've love to just say install ruby 
 right now and have it work but that isn't what I'm expecting.  I'm hoping, 
 over time, to provide some of those packages over time.  I hope I'm making 
 sense here...

Gentoo Prefix[1] on AIX is sort of that.  Not sure if ruby is ported
yet, but at least the base system (the core you need to get Portage
sanely running, and being able to emerge other stuff + a handful of
indispensible tools) compiles.  You'll find people on the gentoo-alt
list[2] that maintain the AIX port.

[1] http://prefix.gentoo.org/
[2] http://archives.gentoo.org/gentoo-alt/


-- 
Fabian Groffen
Gentoo on a different level




Re: [gentoo-portage-dev] [API] First steps for creating an API for portage

2010-06-18 Thread Fabian Groffen
On 18-06-2010 02:08:04 +0200, René 'Necoro' Neumann wrote:
 In parallel (or thereafter), we build the C-bindings. The API for these
 bindings probably look different -- but I guess they should be
 implemented in terms of the library created above.
 
 By example:
 
 - Operation: get the list of categories
 - Python-API: portage.api.categories() : Category list
 - Implementation: def categories(): return 
 - C-API: category * categories()
 - C-Implementation: some wrapper around portage.api.categories

If you want to deliver a C implementation, I'd wonder why you wouldn't
just make a full C implementation and create Python wrappers for them
instead of the other way around.  Might accidentially speed up Portage,
and make tools like portage-utils happy.


-- 
Fabian Groffen
Gentoo on a different level



Re: [gentoo-portage-dev] [API] First steps for creating an API for portage

2010-06-18 Thread Fabian Groffen
On 18-06-2010 01:27:57 -0700, Brian Dolbec wrote:
 Unless I'm way off the mark, your suggesting to re-code portage into C
 then create a python interface to it.  Others have started such a
 project, but none have completed aside from paludis (C++).

It's actually on my wishlist for a long time, just because Python is an
extreme pain in my environments, and given its current maintainer it is
going to be an extreme pain for a long time as well.

 If there are other apps that would like to get info from portage via
 this API, that is fine with me.  I have intentions of making the C
 interface available to all apps that desire to.  If there is data that
 they want to get from portage via the API, then they should send their
 wish list in and we'll do our best to get it in there for them.
 
 Currently the API will be based on the packagekit portage backend code
 that was produced last summer and other code from porthole, portato,
 kportagetray.  We are also going to put together a layman API for
 consumer apps (guis frontends) to use to operate layman without the need
 to run it in and parse terminal output.  It too will hopefully be
 available via a C interface for non-python apps.

I don't want to push you into the C corner, but tools like qfile and
qlist are in orders of magnitude faster than their equery variants for a
good reason.  Maybe this simple command gives you an idea why I think
python and portage are slow.

% time portageq envvar CHOST
powerpc-apple-darwin8
0.488u 0.704s 0:03.09 38.1% 0+0k 25+70io 0pf+0w

(three full seconds to return a very simple var)

That said, if you design C APIs, please design them from a C point of
view, initially implemented by your Python functionality doing the
necessary wrapping to get a sane C structure.  Then they can be replaced
by native C code as RSI and time permits in the future.


-- 
Fabian Groffen
Gentoo on a different level



[gentoo-portage-dev] [PATCH] config_root variables in pym/portage/const.py

2009-07-12 Thread Fabian Groffen
In Prefix, there is a distinction between variables used with
config_root and target_root.  Quoting from prefix' const.py:

# We have a most confusing situation here, which is most of all pretty
# weak for protecting us from making mistakes.
# First there is a config_root (PORTAGE_CONFIGROOT) which can be a path
# somewhere, from which all paths need to be relative (e.g.
# etc/portage), hence those constants do NOT have EPREFIX, because
# config_root contains EPREFIX by default -- overriding it loses the
# EPREFIX as one would expect.
# Second there is target_root (ROOT) which is to install somewhere
# completely else, in Prefix of limited use.  Because this is an offset
# always given, the EPREFIX should always be applied in it.  Those
# constants (like VDB_PATH) DO have EPREFIX.
# Unfortunately this file is ordered quite horrible in this respect.

The attached patch makes all variables against config_root relative (by
removing the leading '/'), with the result that all lstrip(os.sep) calls
in the code can be removed.  Please note that all but two occurences did
use config_root.  For one (pym/portage/__init__.py; sandbox stuff) I've
added it, for the other (in pym/portage/output.py) I don't know how to
get config_root, so substituted '/', which breaks Prefix by design.

How about this patch, and if it is a good idea, how about grouping all
of these config_root variables such that it's easier to see what is
what?  I actually ran into this because Sebastian was hitting it with
his smolt-gentoo tool.


-- 
Fabian Groffen
Gentoo on a different level
Index: pym/portage/__init__.py
===
--- pym/portage/__init__.py (revision 13819)
+++ pym/portage/__init__.py (working copy)
@@ -1309,7 +1309,7 @@
 
if not config_profile_path:
config_profile_path = \
-   os.path.join(config_root, 
PROFILE_PATH.lstrip(os.path.sep))
+   os.path.join(config_root, PROFILE_PATH)
if os.path.isdir(config_profile_path):
self.profile_path = config_profile_path
else:
@@ -1325,7 +1325,7 @@
self.module_priority= [user,default]
self.modules= {}
self.modules[user] = getconfig(
-   os.path.join(config_root, 
MODULES_FILE_PATH.lstrip(os.path.sep)))
+   os.path.join(config_root, MODULES_FILE_PATH))
if self.modules[user] is None:
self.modules[user] = {}
self.modules[default] = {
@@ -1389,7 +1389,7 @@
self.profiles = []
if local_config and self.profiles:
custom_prof = os.path.join(
-   config_root, 
CUSTOM_PROFILE_PATH.lstrip(os.path.sep))
+   config_root, CUSTOM_PROFILE_PATH)
if os.path.exists(custom_prof):
self.user_profile_dir = custom_prof
self.profiles.append(custom_prof)
@@ -1465,7 +1465,7 @@
del rawpuseforce
 
make_conf = getconfig(
-   os.path.join(config_root, 
MAKE_CONF_FILE.lstrip(os.path.sep)),
+   os.path.join(config_root, MAKE_CONF_FILE),
tolerant=tolerant, allow_sourcing=True)
if make_conf is None:
make_conf = {}
@@ -1561,7 +1561,7 @@
self.configdict[defaults]=self.configlist[-1]
 
self.mygcfg = getconfig(
-   os.path.join(config_root, 
MAKE_CONF_FILE.lstrip(os.path.sep)),
+   os.path.join(config_root, MAKE_CONF_FILE),
tolerant=tolerant, allow_sourcing=True, 
expand=expand_map)
if self.mygcfg is None:
self.mygcfg = {}
@@ -1610,8 +1610,7 @@
self.pkeywordsdict = {}
self._plicensedict = {}
self.punmaskdict = {}
-   abs_user_config = os.path.join(config_root,
-   USER_CONFIG_PATH.lstrip(os.path.sep))
+   abs_user_config = os.path.join(config_root, 
USER_CONFIG_PATH)
 
# locations for categories and arch.list files
locations = [os.path.join(self[PORTDIR], profiles)]
@@ -1838,7 +1837,7 @@
(sandbox in self.features or usersandbox in 
self.features

Re: [gentoo-portage-dev] prefix-portage-2.2.00.13734 with --nodeps merges in reverse order

2009-07-02 Thread Fabian Groffen
On 02-07-2009 15:31:16 +0200, Michael Haubenwallner wrote:
 Hi,
 
 seems prefix-portage as of r13734, when used with '--nodeps', does merge
 packages in reverse order than given on cmdline, but I don't believe
 this is a prefix issue...
 
 prefix-portage r13683 does merge in correct order.

13734 is after the _emerge split


-- 
Fabian Groffen
Gentoo on a different level



Re: [gentoo-portage-dev] prefix portage chaining

2009-03-25 Thread Fabian Groffen
On 20-03-2009 11:35:09 +0100, Markus Duft wrote:
 i'll try and explain what i want in the first place: i'm porting things
 to native windows. since windows isn't too cooperative, i'm unable to
 merge most things (and with other things, i simply don't want to), and
 thus i need to take those things from somewhere else (more or less the
 complete @system). I _am_ able to build all those things for interix
 (which is the host system in the windows case). So what i want is a
 setup of two prefix instances with a certain relation to each other: the
 native windows prefix should be able to recognize installed packages
 from the other instance, and resolve dependencies by eventually using
 the other .../var/db/...

Since Windows and Interix seem to be two different things to me, can you
explain why in this case Portage can resolve the dependencies from the
Interix system to use for the Windows system?  What dependencies are we
talking about?  Build tools?  Libraries?  Runtime dependencies?

 This could be (and is) quite usefull for all other platforms too. For
 exmaple i could use prefix chaining on a linux box. I could create a
 prefix containing a base system, and then for testing of
 i-don't-know-whatever, i could create another small prefix inheriting
 all installed packages from the other one. this way new prefixes can
 stay very slim, but still the parent prefix is not altered on merges.

That potentially is useful, in the case where you want to upgrade a
critical system package and test it out or something.  Can't think of an
example other than Portage itself for the moment, though.  (And that one
can be hairy, for instance if the vdb format changes NEEDED -
NEEDED.ELF.2)

 one issue not handled by the current patch is, that prefixes can have
 different CHOST/ARCH/... (which is the case with x86-interix and
 x86-winnt for example).

Then how does it work for you?


-- 
Fabian Groffen
Gentoo on a different level



Re: [gentoo-portage-dev] How to extract the version/revision of an installed package?

2008-11-26 Thread Fabian Groffen
On 25-11-2008 16:15:09 -0800, Ned Ludd wrote:
 On Tue, 2008-11-25 at 18:05 +0200, Amit Dor-Shifer wrote:
  Given the following:
  # qlist -Iv sys-apps/portage
  sys-apps/portage-2.1.4.5
  
  How do I safely extract the 2.1.4.5?
  
  (I don't necessarily need to use qlist. Just want to get the version of 
  an installed package within a bash script)
 
 qatom $(qlist -ICv sys-apps/portage) | awk '{print $3}'

I happen to do it in bash for my update script, it hasn't broke yet:

get_ebuildversion() {
if [[ -z $1 ]] ; then
return
fi

# strip extension
t=${1%.ebuild}
# abort if this is not an ebuild
if [[ $t == $1 ]] ; then
return
fi
# strip package name
t=${t#*-}
# sometimes there are dashes in the package name
while [[ ${t:0:1}  0 || ${t:0:1}  9 ]] ; do
[[ $t != ${t#*-} ]] \
 t=${t#*-} \
|| break
done

echo ${t}
}

you might want to remove the .ebuild check and strip the category ($1#*/})


-- 
Fabian Groffen
Gentoo on a different level



Re: [gentoo-portage-dev] search functionality in emerge

2008-11-24 Thread Fabian Groffen
On 24-11-2008 10:34:28 +0100, René 'Necoro' Neumann wrote:
 tvali schrieb:
  There is daemon, which notices about filesystem changes -
  http://pyinotify.sourceforge.net/ would be a good choice.
 
 Disadvantage: Has to run all the time (I see already some people crying:
 oh noez. not yet another daemon...).

... and it is Linux only, which spoils the fun.


-- 
Fabian Groffen
Gentoo on a different level



Re: [gentoo-portage-dev] Re: portage-2.2-rc3 parallel merges quit being parallel

2008-07-28 Thread Fabian Groffen
On 28-07-2008 08:10:47 +, Duncan wrote:
 René 'Necoro' Neumann [EMAIL PROTECTED] posted [EMAIL PROTECTED],
  A very basic thingy:
  watch qlop -cC
  
  qlop is part of portage-utils. And it seems to only work correctly here,
  when run as root :)
 
 Thanks! =8^)

and when sandbox is enabled, as it looks for [sandbox] in the process
list


-- 
Fabian Groffen
Gentoo on a different level



Re: [gentoo-portage-dev] [PATCH] Repoman subversion support

2008-07-27 Thread Fabian Groffen
On 24-07-2008 17:38:51 +0200, Arfrever Frehtes Taifersar Arahesis wrote:
 `svn list` returns only files which actually exist in the repository. It
 doesn't check working copy. You should use `svn st`.

Ok, you're right.  But svn status simply sucks:

[usr/portage/eclass] % touch x11.eclass.rej
[usr/portage/eclass] % svn status
?  files
[usr/portage/eclass] %

In other words, it auto-ignores stuff that really causes the messed up
manifests if someone isn't paying attention.  IIRC this ignorance isn't
limited to .rej, but also other extentions, such as the trailing ~,
which some editors leave behind.  I guess we need to use --no-ignore,
and then also recognise the I entries.  Bah.


-- 
Fabian Groffen
Gentoo on a different level



Re: [gentoo-portage-dev] [PATCH] Repoman subversion support

2008-07-24 Thread Fabian Groffen
On 14-05-2008 17:56:38 +0200, Fabian Groffen wrote:
 On 14-05-2008 16:27:28 +0200, Arfrever Frehtes Taifersar Arahesis wrote:
  2008-05-14 00:32 Marius Mauch [EMAIL PROTECTED] napisał(a):
   Merged in r10325 with some minor changes (removed the 'svn update' bit
   until someone remembers why exactly it's there
  
  During committing, only files, which are being committed, are being
  updated, so `svn up` is certainly a good idea. Please add it.
 
 Details here are that the code did this:
 
 if vcs == cvs:
 myf=open(checkdir+/CVS/Entries,r)
 if vcs == svn:
 myf=os.popen(svn update  /dev/null; svn list)
 
 genone's remark here refers to this `svn update`, of which I don't
 recall any more why I added it.  All I know is that I think I needed it
 for some reason to get a correct output of list.  However, I have no
 proof here, so he removed the svn update, which gives a non-announced
 possible modification of the local repository.

I've seen numerous examples like bug 232825[1] now, and I'm almost
confident the svn update resolved that problem.  Problem is that the
file (a .rej in the case of the bug) exists in the local dir, but
repoman doesn't notice it, since svn list doesn't return it (as '?').
Note that this sometimes happens, and sometimes not.  I haven't found
the rule here yet.

[1] http://bugs.gentoo.org/show_bug.cgi?id=232825

-- 
Fabian Groffen
Gentoo on a different level



Re: [gentoo-portage-dev] [PATCH] show binhosts as repository

2008-05-14 Thread Fabian Groffen
On 14-05-2008 00:45:10 +0200, Marius Mauch wrote:
 On Mon, 12 May 2008 20:53:35 +0200
 Fabian Groffen [EMAIL PROTECTED] wrote:
 
  The following patch shows the url to the binhost in an emerge -av as
  repository name, instead of unknown.  Unfortunately the patch
  doesn't store the binhost url, such that portage can't show where the
  package comes from when unmerged.
 
 Not sure if this is the right thing to do, or if the reponame in those
 cases should be the one from the source repo that was used to generate
 those binary packages. Not saying that the binhost name here is
 useless, quite the opposite, just a bit concerned about mixing
 different things here.

The idea behind it is that the repo_name of the binhost is not
necessarily useful.  In many cases it will contain the default repo_name
value, since people just create binpkgs with a standard Portage install.

I agree it is a hack.  Perhaps a combination of the repo_name and its
url would be the best.


-- 
Fabian Groffen
Gentoo on a different level
-- 
gentoo-portage-dev@lists.gentoo.org mailing list



[gentoo-portage-dev] [PATCH] unpack .deb files with deb2tgz

2008-05-12 Thread Fabian Groffen
Attached patch is necessary for some extreme platforms, as can be read
in the comments.

-- 
Fabian Groffen
Gentoo on a different level
--- ../../trunk/bin/ebuild.sh   2008-04-27 17:36:18 +0200
+++ ./bin/ebuild.sh 2008-04-13 11:41:55 +0200
@@ -372,9 +372,22 @@
LHa|LHA|lha|lzh)
lha xfq ${srcdir}${x} || die $myfail
;;
-   a|deb)
+   a)
ar x ${srcdir}${x} || die $myfail
;;
+   deb)
+   # Unpacking .deb archives can not always be 
done with
+   # `ar`.  For instance on AIX this doesn't work 
out.  If
+   # we have `deb2targz` installed, prefer it over 
`ar` for
+   # that reason.  We just make sure on AIX 
`deb2targz` is
+   # installed.
+   if type -P deb2targz  /dev/null; then
+   deb2targz ${srcdir}/${x} || die 
$myfail
+   mv ${srcdir}/${x/.deb/.tar.gz} 
data.tar.gz
+   else
+   ar x ${srcdir}/${x} || die $myfail
+   fi
+   ;;
lzma)
if [ ${y} == tar ]; then
lzma -dc ${srcdir}${x} | tar xof - 
${tar_opts}


[gentoo-portage-dev] [PATCH] remove eselect compiler usage

2008-05-12 Thread Fabian Groffen
eselect compiler has been removed from the tree, hence its usage can be
removed from portage.

-- 
Fabian Groffen
Gentoo on a different level
--- ../../trunk/pym/_emerge/__init__.py 2008-05-12 19:25:21 +0200
+++ ./pym/_emerge/__init__.py   2008-05-12 17:16:49 +0200
@@ -303,12 +318,6 @@
!!! other terminals also.\n
)
 
-   mystatus, myoutput = commands.getstatusoutput(eselect compiler show)
-   if mystatus == os.EX_OK and len(myoutput.split(/)) == 2:
-   part1, part2 = myoutput.split(/)
-   if part1.startswith(chost + -):
-   return myoutput.replace(chost + -, gcc_ver_prefix, 1)
-
mystatus, myoutput = commands.getstatusoutput(gcc-config -c)
if mystatus == os.EX_OK and myoutput.startswith(chost + -):
return myoutput.replace(chost + -, gcc_ver_prefix, 1)


[gentoo-portage-dev] [PATCH] show binhosts as repository

2008-05-12 Thread Fabian Groffen
The following patch shows the url to the binhost in an emerge -av as
repository name, instead of unknown.  Unfortunately the patch doesn't
store the binhost url, such that portage can't show where the package
comes from when unmerged.

-- 
Fabian Groffen
Gentoo on a different level
--- ../../trunk/pym/_emerge/__init__.py 2008-05-12 19:25:21 +0200
+++ ./pym/_emerge/__init__.py   2008-05-12 17:16:49 +0200
@@ -1117,6 +1126,6 @@
pkg_chost = pkg.metadata.get(CHOST)
if pkg_chost and pkg_chost != pkgsettings[CHOST]:
return False
if not portage.eapi_is_supported(pkg.metadata[EAPI]):
return False
if not pkg.installed and \
@@ -4447,13 +4471,18 @@
pkg = x
metadata = pkg.metadata
ebuild_path = None
-   repo_name = metadata[repository]
+   if pkg_type == binary:
+   repo_name = 
self.roots[myroot].settings.get(PORTAGE_BINHOST)
+   else:
+   repo_name = metadata[repository]
if pkg_type == ebuild:
ebuild_path = portdb.findname(pkg_key)
if not ebuild_path: # shouldn't happen
raise 
portage.exception.PackageNotFound(pkg_key)
repo_path_real = 
os.path.dirname(os.path.dirname(
os.path.dirname(ebuild_path)))
+   elif pkg_type == binary:
+   repo_path_real = repo_name
else:
repo_path_real = 
portdb.getRepositoryPath(repo_name)
pkg_use = metadata[USE].split()
@@ -5422,6 +5451,11 @@
self._repo_paths_real = [ os.path.realpath(repo_path) \
for repo_path in repo_paths ]
 
+   if root_config.settings.get(PORTAGE_BINHOST):
+   binhost = root_config.settings.get(PORTAGE_BINHOST)
+   self._repo_paths.append(binhost)
+   self._repo_paths_real.append(binhost)
+
# pre-allocate index for PORTDIR so that it always has index 0.
for root_config in roots.itervalues():
portdb = root_config.trees[porttree].dbapi


[gentoo-portage-dev] [PATCH] include status support for xterm-color and interix

2008-05-12 Thread Fabian Groffen
Attached patch adds statusbar support for xterm-color and interix
terminals.

-- 
Fabian Groffen
Gentoo on a different level
--- ../../trunk/pym/portage/output.py   2008-05-12 19:25:13 +0200
+++ ./pym/portage/output.py 2008-05-08 21:17:50 +0200
@@ -246,7 +246,7 @@
if len(mystr)  max_len:
mystr = mystr[:max_len]
myt=os.environ[TERM]
-   legal_terms = 
[xterm,Eterm,aterm,rxvt,screen,kterm,rxvt-unicode,gnome]
+   legal_terms = 
[xterm,xterm-color,Eterm,aterm,rxvt,screen,kterm,rxvt-unicode,gnome,interix]
for term in legal_terms:
if myt.startswith(term):
if not raw:


[gentoo-portage-dev] IUSE and userland_, elibc_, kernel_, etc.

2007-11-04 Thread Fabian Groffen
Just to have it clear and to be sure:

On 04-11-2007 03:33:41 +, [EMAIL PROTECTED] wrote:
 Modified: diffutils-2.8.7-r2.ebuild
 Log:
   do *not* include userland_GNU in IUSE
   (Portage version: 2.1.3.16)
 
 
 Index: diffutils-2.8.7-r2.ebuild

 -IUSE=nls static userland_GNU
 +IUSE=nls static


On 04-11-2007 08:10:29 +, Zac Medico wrote:
 Author: zmedico
 Date: 2007-11-04 08:10:29 + (Sun, 04 Nov 2007)
 New Revision: 8420
 
 Modified:
main/trunk/pym/portage/dbapi/bintree.py
 Log:
 When evaluating *DEPEND conditionals for the Packages metadata
 index, do not use IUSE to filter USE since there is currently
 no guarantee that IUSE properly defines all of the necessary
 flags.

These two changes now mean that without having userland_GNU in IUSE

  DEPEND=!userland_GNU? ( some/package )

will correctly end up in the Packages file, such that Portage will
properly calculate dependencies when reading from a binhost, right?


-- 
Fabian Groffen
Gentoo on a different level
-- 
[EMAIL PROTECTED] mailing list



Re: [gentoo-portage-dev] pty/termios problems on alternative platforms

2007-05-25 Thread Fabian Groffen
On 25-05-2007 15:04:24 +0200, Fabian Groffen wrote:
...
 raise an exception on the pty code, but only bails out on the
 termios.tcgetattr function because it gets an invalid argument.

from the code pym/portage/__init__.py around line 2288

from pty import openpty
master_fd, slave_fd = openpty()
# Disable the ECHO attribute so the terminal behaves properly
# if the subprocess needs to read input from stdin.
import termios
term_attr = termios.tcgetattr(master_fd)
term_attr[3] = ~termios.ECHO
termios.tcsetattr(master_fd, termios.TCSAFLUSH, term_attr)

if I comment out the complete termios (last four lines) it appears to
work, though I don't know what repercussions this has on the rest of the
code.  Judging from the comment, it feels not so right.

-- 
Fabian Groffen
Gentoo on a different level
-- 
[EMAIL PROTECTED] mailing list



Re: [gentoo-portage-dev] pty/termios problems on alternative platforms

2007-05-25 Thread Fabian Groffen
On 25-05-2007 12:44:10 -0700, Zac Medico wrote:
  if I comment out the complete termios (last four lines) it appears to
  work, though I don't know what repercussions this has on the rest of the
  code.  Judging from the comment, it feels not so right.
 
 Try replacing master_fd with slave_fd in both the tcgetattr and
 tcsetattr calls.  I'm not sure if that will solve it but it's worth
 a try.

It lets it continue without a backtrace... so I guess that sort of
works?


-- 
Fabian Groffen
Gentoo on a different level

-- 
[EMAIL PROTECTED] mailing list



Re: [gentoo-portage-dev] [PATCH] Repoman SVN support

2006-11-25 Thread Fabian Groffen
On 24-11-2006 23:29:53 -0800, Zac Medico wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 Fabian Groffen wrote:
  Please find attached the patch of the prefix repoman sources against the
  current trunk.
 
 I like the idea of having svn support, but I think it will be much
 nicer if we separate the cvs and svn support into plugins.  That
 way, it should make the code cleaner and we'll more easily be able
 to add support for additional revision control systems if we want.

I would prefer that, as I'm an OO-freak, but like I said I only revamped
this patch into a working one.  Besides that, I'm not a Python guru,
so more than a few if-s is getting a challenge already.

All in all, if there is an interface somewhere, then the code parts can
be copied in there for sure, quite easily.

-- 
Fabian Groffen
Gentoo on a different level
-- 
gentoo-portage-dev@gentoo.org mailing list



Re: [gentoo-portage-dev] [PATCH] Repoman SVN support

2006-11-25 Thread Fabian Groffen
On 25-11-2006 03:00:10 -0500, Alec Warner wrote:
 Fabian Groffen wrote:
 Hi all,
 
 Please find attached the patch of the prefix repoman sources against the
 current trunk.  I hope I removed all hunks that have prefix-only
 changes.
 
 You realize I wrote generic SCM support into repoman over the summer 
 right? :P It was never merged (afaik I slacked on it, but also because 

E no... probably because at that time I wasn't making my hands dirty
on portage code.

 it uses Popen and you pesky OSX folks are stuck on python2.3, so I need 
 to convert to portage.spawn).

You can drop that, as I already require python 2.4, since portage needs
at least 2.3.1 currently and I don't feel like checking with an older
version to see if it breaks more.  During bootstrapping I first install
python 2.4.  On a slightly related side note: you can also drop all your
bash-2 code, as we end up running bash-3 too.  Would only be nice if the
basic functionally would work with bash-2, otherwise I need to bootstrap
bash too.  (Worth the time if it simplifies portage.)

 However the support was generic, so for pretty much any system that 'is 
 kinda like cvs/svn' should work fine with a few tweaks to the exec 
 mapping.  biasThis of course excludes GIT, since it's a huge 
 non-standard piece of crap./bias

Great!  I feel that SVN won't be the only thing folks use, so I'm
looking forward to your interface from a (hopefully) functional view
point.

 I may work on fixing that code up and putting it into HEAD before I retire.

E... what?  The first is nice, the latter for sure isn't.


-- 
Fabian Groffen
Gentoo on a different level
-- 
gentoo-portage-dev@gentoo.org mailing list



[gentoo-portage-dev] [PATCH] Repoman SVN support

2006-11-23 Thread Fabian Groffen
Hi all,

Please find attached the patch of the prefix repoman sources against the
current trunk.  I hope I removed all hunks that have prefix-only
changes.

The current diff is mainly SVN support for Repoman.  I extracted the SVN
diffs only.  Most of the original work was done by Kito, and recently I
wiped the dust off of it, and finalised it into some working version.
So far the good news, the bad news now regarding this patch: I had to
somehow hardcode the svn repo being used, as I didn't find a way to let
repoman figure out dynamically what the offset in the SVN repository
is where the portage tree starts.  This is kind of similar to how CVS
support is done, where this offset is also hardcoded.

While I think repoman's CVS support has always been used as-is --
nowadays people prefer to start non-CVS repos, the SVN support as in
this patch would, considering the many overlays in SVN around, surely
benefit from some sort of solution that allows to set the repository
offset or something, e.g. through a variable.  Thoughts?


-- 
Fabian Groffen
Gentoo on a different level
--- ../../trunk/bin/repoman 2006-11-23 18:15:36 +0100
+++ bin/repoman 2006-11-20 21:52:30 +0100
@@ -407,9 +409,80 @@
os.environ[FEATURES]=repoman_settings[FEATURES]+ cvs
isCvs = True
 
-if not --pretend in myoptions and not isCvs:
+   try:
+   isCvs=True
+   myrepofile=open(CVS/Repository)
+   myreporoot=myrepofile.readline()[:-1]
+   myrepofile.close()
+   myrepofile=open(CVS/Root)
+   myreporootpath=string.split(myrepofile.readline()[:-1], :)[-1]
+   myrepofile.close()
+   if myreporootpath == myreporoot[:len(myreporootpath)]:
+   # goofy os x cvs co.
+   myreporoot = myreporoot[len(myreporootpath):]
+   while myreporoot and myreporoot[0] == '/':
+   myreporoot = myreporoot[1:]
+   except SystemExit, e:
+   raise  # Need to propogate this
+   except:
+   err(Error grabbing repository information; exiting.)
+   myreporootpath=os.path.normpath(myreporootpath)
+   if myreporootpath==/space/cvsroot:
+   print
+   print
+   print red(You're using the wrong cvsroot. For Manifests to be 
correct, you must)
+   print red(use the same base cvsroot path that the servers use. 
Please try the)
+   print red(following script to remedy this:)
+   print
+   print cd my_cvs_tree
+   print
+   print rm -Rf [a-z]*
+   print cvs up
+   print
+   if portage.userland==BSD or portage.userland==Darwin:
+   print find ./ -type f -regex '.*/CVS/Root$' -print0 | 
xargs -0 sed \\
+   else:
+   print find ./ -type f -regex '.*/CVS/Root$' -print0 | 
xargs -0r sed \\
+   fi
+   print   -i 's:/space/cvsroot$:/home/cvsroot:'
+   print
+   print red(You must clear and re-update your tree as all header 
tags will cause)
+   print red(problems in manifests for all rsync and 
/home/cvsroot users.)
+   print
+   print You should do this to any other gentoo trees your have 
as well,
+   print excluding the deletions. 'gentoo-src' should be 
/home/cvsroot.
+   print
+   sys.exit(123)
+
+isSvn=False
+myreporoot=None
+if os.path.isdir(.svn):
+   if svn not in repoman_settings.features:
+   print
+   print
+   print red('!!! You do not have ')+bold('FEATURES=svn 
')+red(enabled...)
+   print red(!!! )+bold(Adding \svn\ to FEATURES)
+   print
+   os.environ[FEATURES]=repoman_settings[FEATURES]+ svn
+
+   try:
+   isSvn=True
+   myrepoinfo = os.popen(svn info).readlines()
+   myurl = myrepoinfo[1][5:].rstrip()
+   myreproot = myrepoinfo[2][17:].rstrip()
+   # This one is EVIL, but I don't know how to solve the problem of
+   # getting rid of /trunk/prefix-overlay.  This only works on the
+   # prefix overlay.
+   myreporoot = /.join(myurl[len(myreproot):].split(/)[2:])
+   myreporootpath = / + /.join(myreproot.split(/)[3:])
+   except SystemExit, e:
+   raise  # Need to propogate this
+   except:
+   err(Error grabbing repository information; exiting.)
+
+if not --pretend in myoptions and not isCvs and not isSvn:
print
-   print darkgreen(Not in a CVS repository; enabling pretend mode.)
+   print darkgreen(Not in a CVS or SVN repository; enabling pretend 
mode.)
myoptions.append(--pretend);
 
 
@@ -490,14 +563,17 @@
myreporoot = os.path.basename(portdir_overlay