Re: [gentoo-dev] [RFC] New Manifest hashes and how to enable them

2017-04-03 Thread Ulrich Mueller
> On Mon, 3 Apr 2017, Dirkjan Ochtman wrote:

> This seems pretty hasty.

> First of all, SHA-256 should be safe for all intents and purposes,
> and for the foreseeable future. This is nothing like Git's usage of
> SHA-1, which was known to be on the way to brokenville for a long
> time. I don't think there is a solid reason for deprecating it now.

Hasty? The plan has been outlined already 7 years ago, in GLEP 59 [1].

IIUC, it is enough to keep the strongest checksum of a given set,
namely SHA512 for SHA-2. The increase in security by keeping a second
weaker checksum of the same family (i.e. SHA256) is negligible.

So the only reason for keeping SHA256 was that old Portage versions
didn't support SHA512. However, by now SHA512 is supported since more
than 5 years, namely since Portage version 2.1.10.44 which went stable
on the last arch in February 2012.

I don't have a strong opinion if we should add SHA-3, but I don't
think that it is urgently needed.

Ulrich

[1] https://wiki.gentoo.org/wiki/GLEP:59#Checksum_depreciation_timing


pgpS0JtE20Ogj.pgp
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH 2/2] EbuildBuild: async spawn_nofetch in _fetchonly_exit (bug 614116)

2017-04-03 Thread Zac Medico
On Mon, Apr 3, 2017 at 1:21 PM, Brian Dolbec  wrote:
> On Sun,  2 Apr 2017 19:36:54 -0700
> Zac Medico  wrote:
>
>> Replace a synchronous spawn_nofetch call with an asynchronous one,
>> in order to avoid event loop recursion which is not compatible with
>> asyncio. This involves refactoring of spawn_nofetch to provide an
>> asynchronous interface.
>>
>> X-Gentoo-bug: 614116
>> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=614116
>> ---
>>  pym/_emerge/EbuildBuild.py   |  18 -
>>  pym/portage/package/ebuild/_spawn_nofetch.py | 106
>> +-- 2 files changed, 85 insertions(+), 39
>> deletions(-)
>>
>
> looks good
>

Thanks, pushed:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=285d5d038d8bb8a17d853816e156147c8c59f248
-- 
Thanks,
Zac



Re: [gentoo-portage-dev] [PATCH 1/2] EbuildBuild: eliminate call to digestgen (bug 614116)

2017-04-03 Thread Zac Medico
On Mon, Apr 3, 2017 at 1:21 PM, Brian Dolbec  wrote:
> On Sun,  2 Apr 2017 19:36:53 -0700
> Zac Medico  wrote:
>
>> Eliminate the call to digestgen in EbuildBuild._fetchonly_exit,
>> and make Scheduler._generate_digests call it earlier when
>> --fetchonly is enabled. This avoids event loop recursion which
>> is not compatible with asyncio (digestgen makes many calls that
>> can trigger event loop recursion).
>>
>> X-Gentoo-bug: 614116
>> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=614116
>
> looks good
>

Thanks, pushed:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=f05d0864653e082bd60db67b52132c4ba6515339

-- 
Thanks,
Zac



Re: [gentoo-dev] Review: xemacs-packages-r1.eclass

2017-04-03 Thread Michael Orlitzky

On 04/02/2017 05:05 AM, David Seifert wrote:


[[ ${XEMACS_PKG_CAT} ]] || die "XEMACS_PKG_CAT was not defined before inheriting 
xemacs-packages-r1.eclass"
case ${XEMACS_PKG_CAT} in
standard|mule|contrib)
;;
*)
die "Unsupported package category in XEMACS_PKG_CAT"
;;
esac



I think that the first check is redundant, but it isn't important if you 
don't feel like changing it.


Elsewhere: quoting the variables when they're used would produce better 
error messages if somebody puts something stupid in the variables.





Re: [gentoo-portage-dev] [PATCH 2/2] EbuildBuild: async spawn_nofetch in _fetchonly_exit (bug 614116)

2017-04-03 Thread Brian Dolbec
On Sun,  2 Apr 2017 19:36:54 -0700
Zac Medico  wrote:

> Replace a synchronous spawn_nofetch call with an asynchronous one,
> in order to avoid event loop recursion which is not compatible with
> asyncio. This involves refactoring of spawn_nofetch to provide an
> asynchronous interface.
> 
> X-Gentoo-bug: 614116
> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=614116
> ---
>  pym/_emerge/EbuildBuild.py   |  18 -
>  pym/portage/package/ebuild/_spawn_nofetch.py | 106
> +-- 2 files changed, 85 insertions(+), 39
> deletions(-)
> 
> diff --git a/pym/_emerge/EbuildBuild.py b/pym/_emerge/EbuildBuild.py
> index 11eb1c9..48f4704 100644
> --- a/pym/_emerge/EbuildBuild.py
> +++ b/pym/_emerge/EbuildBuild.py
> @@ -22,7 +22,7 @@ import portage
>  from portage import _encodings, _unicode_decode, _unicode_encode, os
>  from portage.package.ebuild.digestcheck import digestcheck
>  from portage.package.ebuild.doebuild import _check_temp_dir
> -from portage.package.ebuild._spawn_nofetch import spawn_nofetch
> +from portage.package.ebuild._spawn_nofetch import
> SpawnNofetchWithoutBuilddir 
>  class EbuildBuild(CompositeTask):
>  
> @@ -165,8 +165,22 @@ class EbuildBuild(CompositeTask):
>   def _fetchonly_exit(self, fetcher):
>   self._final_exit(fetcher)
>   if self.returncode != os.EX_OK:
> + self.returncode = None
>   portdb =
> self.pkg.root_config.trees[self._tree].dbapi
> - spawn_nofetch(portdb, self._ebuild_path,
> settings=self.settings)
> + self._start_task(SpawnNofetchWithoutBuilddir(
> + background=self.background,
> + portdb=portdb,
> + ebuild_path=self._ebuild_path,
> + scheduler=self.scheduler,
> + settings=self.settings),
> + self._nofetch_without_builddir_exit)
> + return
> +
> + self.wait()
> +
> + def _nofetch_without_builddir_exit(self, nofetch):
> + self._final_exit(nofetch)
> + self.returncode = 1
>   self.wait()
>  
>   def _pre_clean_exit(self, pre_clean_phase):
> diff --git a/pym/portage/package/ebuild/_spawn_nofetch.py
> b/pym/portage/package/ebuild/_spawn_nofetch.py index 0fc53c8..bbfd5b7
> 100644 --- a/pym/portage/package/ebuild/_spawn_nofetch.py
> +++ b/pym/portage/package/ebuild/_spawn_nofetch.py
> @@ -14,11 +14,14 @@ from portage.package.ebuild.prepare_build_dirs
> import prepare_build_dirs from portage.util._async.SchedulerInterface
> import SchedulerInterface from portage.util._eventloop.EventLoop
> import EventLoop from portage.util._eventloop.global_event_loop
> import global_event_loop +from _emerge.CompositeTask import
> CompositeTask from _emerge.EbuildPhase import EbuildPhase
>  
> -def spawn_nofetch(portdb, ebuild_path, settings=None, fd_pipes=None):
> +
> +class SpawnNofetchWithoutBuilddir(CompositeTask):
>   """
> - This spawns pkg_nofetch if appropriate. The settings
> parameter
> + This spawns pkg_nofetch if appropriate, while avoiding the
> + need to lock a global build directory. The settings parameter
>   is useful only if setcpv has already been called in order
>   to cache metadata. It will be cloned internally, in order to
>   prevent any changes from interfering with the calling code.
> @@ -40,33 +43,42 @@ def spawn_nofetch(portdb, ebuild_path,
> settings=None, fd_pipes=None): to be displayed for problematic
> packages even though they do not set RESTRICT=fetch (bug #336499).
>  
> - This function does nothing if the PORTAGE_PARALLEL_FETCHONLY
> + This class does nothing if the PORTAGE_PARALLEL_FETCHONLY
>   variable is set in the config instance.
>   """
> + __slots__ = ('ebuild_path', 'fd_pipes', 'portdb', 'settings',
> + '_private_tmpdir')
> +
> + def _start(self):
> + settings = self.settings
> + if settings is None:
> + settings = self.portdb.settings
> +
> + if 'PORTAGE_PARALLEL_FETCHONLY' in settings:
> + # parallel-fetch mode
> + self.returncode = os.EX_OK
> + self._async_wait()
> + return
>  
> - if settings is None:
> - settings = config(clone=portdb.settings)
> - else:
> - settings = config(clone=settings)
> -
> - if 'PORTAGE_PARALLEL_FETCHONLY' in settings:
> - return os.EX_OK
> -
> - # We must create our private PORTAGE_TMPDIR before calling
> - # doebuild_environment(), since lots of variables such
> - # as PORTAGE_BUILDDIR refer to paths inside PORTAGE_TMPDIR.
> - portage_tmpdir = settings.get('PORTAGE_TMPDIR')
> - if not portage_tmpdir or not os.access(portage_tmpdir,
> 

Re: [gentoo-portage-dev] [PATCH 1/2] EbuildBuild: eliminate call to digestgen (bug 614116)

2017-04-03 Thread Brian Dolbec
On Sun,  2 Apr 2017 19:36:53 -0700
Zac Medico  wrote:

> Eliminate the call to digestgen in EbuildBuild._fetchonly_exit,
> and make Scheduler._generate_digests call it earlier when
> --fetchonly is enabled. This avoids event loop recursion which
> is not compatible with asyncio (digestgen makes many calls that
> can trigger event loop recursion).
> 
> X-Gentoo-bug: 614116
> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=614116
> ---
>  pym/_emerge/EbuildBuild.py | 5 -
>  pym/_emerge/Scheduler.py   | 3 ---
>  2 files changed, 8 deletions(-)
> 
> diff --git a/pym/_emerge/EbuildBuild.py b/pym/_emerge/EbuildBuild.py
> index 001f55f..11eb1c9 100644
> --- a/pym/_emerge/EbuildBuild.py
> +++ b/pym/_emerge/EbuildBuild.py
> @@ -21,7 +21,6 @@ from _emerge.TaskSequence import TaskSequence
>  import portage
>  from portage import _encodings, _unicode_decode, _unicode_encode, os
>  from portage.package.ebuild.digestcheck import digestcheck
> -from portage.package.ebuild.digestgen import digestgen
>  from portage.package.ebuild.doebuild import _check_temp_dir
>  from portage.package.ebuild._spawn_nofetch import spawn_nofetch
>  
> @@ -168,10 +167,6 @@ class EbuildBuild(CompositeTask):
>   if self.returncode != os.EX_OK:
>   portdb =
> self.pkg.root_config.trees[self._tree].dbapi spawn_nofetch(portdb,
> self._ebuild_path, settings=self.settings)
> - elif 'digest' in self.settings.features:
> - if not digestgen(mysettings=self.settings,
> -
> myportdb=self.pkg.root_config.trees[self._tree].dbapi):
> - self.returncode = 1
>   self.wait()
>  
>   def _pre_clean_exit(self, pre_clean_phase):
> diff --git a/pym/_emerge/Scheduler.py b/pym/_emerge/Scheduler.py
> index 58ff971..079fac7 100644
> --- a/pym/_emerge/Scheduler.py
> +++ b/pym/_emerge/Scheduler.py
> @@ -616,9 +616,6 @@ class Scheduler(PollScheduler):
>   tasks are started.
>   """
>  
> - if '--fetchonly' in self.myopts:
> - return os.EX_OK
> -
>   digest = '--digest' in self.myopts
>   if not digest:
>   for pkgsettings in self.pkgsettings.values():

looks good

-- 
Brian Dolbec 




Re: [gentoo-dev] [RFC] New Manifest hashes and how to enable them

2017-04-03 Thread Hanno Böck
Hi,

On Mon, 3 Apr 2017 22:00:15 +0200
Dirkjan Ochtman  wrote:

> First of all, SHA-256 should be safe for all intents and purposes, and
> for the foreseeable future. This is nothing like Git's usage of SHA-1,
> which was known to be on the way to brokenville for a long time. I
> don't think there is a solid reason for deprecating it now.
> 
> Second, the amount of diversity proposed does not make sense. If
> asked, I would propose we keep SHA-256 as one of the options and
> additionally add a SHA3 variant and a BLAKE2 variant as other options.
> This would provide more than enough diversity. Also totally agreed
> with Vadim on the obscurity of the GOST algorithms.
> 
> But, this is the kind of thing where we really should get input from
> the Security project, so we should get people like Hanno and Kristian
> involved.

As you specifically asked for my opinion:
I think there's no reason to doubt the security of any of the sha2
hashes (including sha256), any of sha3 or blake2 for the forseeable
future. (That means counting in many decades - there isn't even a shred
of evidence sha256 is going to be broken any time soon.)
There's no point in deprecating anything.

I find it unnecessary to introduce additional complexity here and
adding obscurity algorithms like gost sounds really bizarre and
unnecessary. I'd recommend against introducing anything that
requires unusual dependencies.
If anything I'd propose to just change to a single hash functio

-- 
Hanno Böck
https://hboeck.de/

mail/jabber: ha...@hboeck.de
GPG: FE73757FA60E4E21B937579FA5880072BBB51E42



Re: [gentoo-portage-dev] [PATCH] AsynchronousLock: add async_unlock method (bug 614108)

2017-04-03 Thread Zac Medico
On Mon, Apr 3, 2017 at 1:03 PM, Brian Dolbec  wrote:
> On Sun,  2 Apr 2017 15:51:40 -0700
> Zac Medico  wrote:
>
>> Add an async_unlock method, in order to avoid event loop
>> recursion which is incompatible with asyncio.
>>
>> X-Gentoo-bug: 614108
>> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=614108
>> ---
>>  pym/_emerge/AsynchronousLock.py   | 89
>> +--
>> pym/portage/tests/locks/test_asynchronous_lock.py | 15 +++- 2 files
>> changed, 92 insertions(+), 12 deletions(-)
>>
>
> looks fine :)
>

Thanks, pushed:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=916a0733c7201b7a8b22f5262bd5be8cbc8992a6
-- 
Thanks,
Zac



Re: [gentoo-portage-dev] [PATCH] emerge: fix --autounmask-continue to work with --getbinpkg (bug 614474)

2017-04-03 Thread Zac Medico
On Mon, Apr 3, 2017 at 12:57 PM, Brian Dolbec  wrote:
> On Sat,  1 Apr 2017 18:02:03 -0700
> Zac Medico  wrote:
>
>> Fix action_build to populate binarytree instances with remote package
>> metadata following config reload, using a new getbinpkg_refresh=False
>> parameter to prevent redundant fetching of remote package metadata.
>>
>> X-Gentoo-bug: 614474
>> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=614474
>> Fixes: e2d88ef3ff59 ("Add emerge --autounmask-continue option (bug
>> 582624)") ---
>>  pym/_emerge/actions.py   | 15 +++
>>  pym/portage/dbapi/bintree.py | 19 +++
>>  2 files changed, 30 insertions(+), 4 deletions(-)
>>
>
> looks good.  Thanks for migrating that 0 for False

Thanks, pushed:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=f479250c9cb9d82af4d621aa008d4d1e37a28a39
-- 
Thanks,
Zac



Re: [gentoo-portage-dev] [PATCH] AsynchronousLock: add async_unlock method (bug 614108)

2017-04-03 Thread Brian Dolbec
On Sun,  2 Apr 2017 15:51:40 -0700
Zac Medico  wrote:

> Add an async_unlock method, in order to avoid event loop
> recursion which is incompatible with asyncio.
> 
> X-Gentoo-bug: 614108
> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=614108
> ---
>  pym/_emerge/AsynchronousLock.py   | 89
> +--
> pym/portage/tests/locks/test_asynchronous_lock.py | 15 +++- 2 files
> changed, 92 insertions(+), 12 deletions(-)
> 
> diff --git a/pym/_emerge/AsynchronousLock.py
> b/pym/_emerge/AsynchronousLock.py index c0b9b26..6a32d2d 100644
> --- a/pym/_emerge/AsynchronousLock.py
> +++ b/pym/_emerge/AsynchronousLock.py
> @@ -35,7 +35,7 @@ class AsynchronousLock(AsynchronousTask):
>  
>   __slots__ = ('path', 'scheduler',) + \
>   ('_imp', '_force_async', '_force_dummy',
> '_force_process', \
> - '_force_thread')
> + '_force_thread', '_unlock_future')
>  
>   _use_process_by_default = True
>  
> @@ -84,6 +84,11 @@ class AsynchronousLock(AsynchronousTask):
>   return self.returncode
>  
>   def unlock(self):
> + """
> + This method is deprecated in favor of async_unlock,
> since waiting
> + for the child process to respond can trigger event
> loop recursion
> + which is incompatible with asyncio.
> + """
>   if self._imp is None:
>   raise AssertionError('not locked')
>   if isinstance(self._imp, (_LockProcess,
> _LockThread)): @@ -92,6 +97,28 @@ class
> AsynchronousLock(AsynchronousTask): unlockfile(self._imp)
>   self._imp = None
>  
> + def async_unlock(self):
> + """
> + Release the lock asynchronously. Release
> notification is available
> + via the add_done_callback method of the returned
> Future instance. +
> + @returns: Future, result is None
> + """
> + if self._imp is None:
> + raise AssertionError('not locked')
> + if self._unlock_future is not None:
> + raise AssertionError("already unlocked")
> + if isinstance(self._imp, (_LockProcess,
> _LockThread)):
> + unlock_future = self._imp.async_unlock()
> + else:
> + unlockfile(self._imp)
> + unlock_future =
> self.scheduler.create_future()
> +
> self.scheduler.call_soon(unlock_future.set_result, None)
> + self._imp = None
> + self._unlock_future = unlock_future
> + return unlock_future
> +
> +
>  class _LockThread(AbstractPollTask):
>   """
>   This uses the portage.locks module to acquire a lock
> asynchronously, @@ -105,7 +132,7 @@ class
> _LockThread(AbstractPollTask): """
>  
>   __slots__ = ('path',) + \
> - ('_force_dummy', '_lock_obj', '_thread',)
> + ('_force_dummy', '_lock_obj', '_thread',
> '_unlock_future') 
>   def _start(self):
>   self._registered = True
> @@ -132,13 +159,35 @@ class _LockThread(AbstractPollTask):
>   pass
>  
>   def unlock(self):
> + """
> + This method is deprecated in favor of async_unlock,
> for compatibility
> + with _LockProcess.
> + """
> + self._unlock()
> + self._unlock_future.set_result(None)
> +
> + def _unlock(self):
>   if self._lock_obj is None:
>   raise AssertionError('not locked')
>   if self.returncode is None:
>   raise AssertionError('lock not acquired yet')
> + if self._unlock_future is not None:
> + raise AssertionError("already unlocked")
> + self._unlock_future = self.scheduler.create_future()
>   unlockfile(self._lock_obj)
>   self._lock_obj = None
>  
> + def async_unlock(self):
> + """
> + Release the lock asynchronously. Release
> notification is available
> + via the add_done_callback method of the returned
> Future instance. +
> + @returns: Future, result is None
> + """
> + self._unlock()
> +
> self.scheduler.call_soon(self._unlock_future.set_result, None)
> + return self._unlock_future
> +
>   def _unregister(self):
>   self._registered = False
>  
> @@ -156,7 +205,8 @@ class _LockProcess(AbstractPollTask):
>   """
>  
>   __slots__ = ('path',) + \
> - ('_acquired', '_kill_test', '_proc', '_files',
> '_reg_id', '_unlocked')
> + ('_acquired', '_kill_test', '_proc', '_files',
> +  '_reg_id','_unlock_future')
>  
>   def _start(self):
>   in_pr, in_pw = os.pipe()
> @@ -223,13 +273,16 @@ class _LockProcess(AbstractPollTask):
>   return
>  
>   if 

Re: [gentoo-dev] [RFC] New Manifest hashes and how to enable them

2017-04-03 Thread Dirkjan Ochtman
On Mon, Apr 3, 2017 at 7:09 PM, Michał Górny  wrote:
> Your thoughts?

This seems pretty hasty.

First of all, SHA-256 should be safe for all intents and purposes, and
for the foreseeable future. This is nothing like Git's usage of SHA-1,
which was known to be on the way to brokenville for a long time. I
don't think there is a solid reason for deprecating it now.

Second, the amount of diversity proposed does not make sense. If
asked, I would propose we keep SHA-256 as one of the options and
additionally add a SHA3 variant and a BLAKE2 variant as other options.
This would provide more than enough diversity. Also totally agreed
with Vadim on the obscurity of the GOST algorithms.

But, this is the kind of thing where we really should get input from
the Security project, so we should get people like Hanno and Kristian
involved.

Third, I don't much trust the security record of the python libraries
mentioned. cryptography is the best Python library for crypto by far,
and I think we should use it exclusively for anything Python doesn't
provide in the stdlib. The PyCrypto security record is not exactly
stellar IIRC, and since pycryptodome is a fork of it, I don't trust it
that much, either.

But mainly, please, I think we should leave the security-sensitive
decisions to people with more security expertise.

Cheers,

Dirkjan



[gentoo-dev] Tree signing and verification on the user side - status?

2017-04-03 Thread Andreas K. Huettel
Hey all, 

while we're discussing super-strength hash algos, it would be cool to know 
what's still missing for
* rsync-side manifest signing in whatever way
* verification of such signatures in portage / emerge

This is the bigger problem (probably also requiring more work though)...

Cheers, 
Andreas

-- 
Andreas K. Hüttel
dilfri...@gentoo.org
Gentoo Linux developer (council, perl, libreoffice)

signature.asc
Description: This is a digitally signed message part.


Re: [gentoo-portage-dev] [PATCH] emerge: fix --autounmask-continue to work with --getbinpkg (bug 614474)

2017-04-03 Thread Brian Dolbec
On Sat,  1 Apr 2017 18:02:03 -0700
Zac Medico  wrote:

> Fix action_build to populate binarytree instances with remote package
> metadata following config reload, using a new getbinpkg_refresh=False
> parameter to prevent redundant fetching of remote package metadata.
> 
> X-Gentoo-bug: 614474
> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=614474
> Fixes: e2d88ef3ff59 ("Add emerge --autounmask-continue option (bug
> 582624)") ---
>  pym/_emerge/actions.py   | 15 +++
>  pym/portage/dbapi/bintree.py | 19 +++
>  2 files changed, 30 insertions(+), 4 deletions(-)
> 
> diff --git a/pym/_emerge/actions.py b/pym/_emerge/actions.py
> index cc0269d..818fab9 100644
> --- a/pym/_emerge/actions.py
> +++ b/pym/_emerge/actions.py
> @@ -347,6 +347,21 @@ def action_build(emerge_config,
> trees=DeprecationWarning, adjust_configs(emerge_config.opts,
> emerge_config.trees) settings, trees, mtimedb = emerge_config
>  
> + # After config reload, the freshly
> instantiated binarytree
> + # instances need to load remote metadata if
> --getbinpkg
> + # is enabled. Use getbinpkg_refresh=False to
> use cached
> + # metadata, since the cache is already fresh.
> + if "--getbinpkg" in emerge_config.opts:
> + for root_trees in
> emerge_config.trees.values():
> + try:
> +
> root_trees["bintree"].populate(
> +
> getbinpkgs=True,
> +
> getbinpkg_refresh=False)
> + except ParseError as e:
> +
> writemsg("\n\n!!!%s.\nSee make.conf(5) for more info.\n"
> +  %
> e, noiselevel=-1)
> + return 1
> +
>   if "--autounmask-only" in myopts:
>   mydepgraph.display_problems()
>   return 0
> diff --git a/pym/portage/dbapi/bintree.py
> b/pym/portage/dbapi/bintree.py index 12c3d3e..ca90ba8 100644
> --- a/pym/portage/dbapi/bintree.py
> +++ b/pym/portage/dbapi/bintree.py
> @@ -510,8 +510,16 @@ class binarytree(object):
>   except PortageException:
>   pass
>  
> - def populate(self, getbinpkgs=0):
> - "populates the binarytree"
> + def populate(self, getbinpkgs=False, getbinpkg_refresh=True):
> + """
> + Populates the binarytree with package metadata.
> +
> + @param getbinpkgs: include remote packages
> + @type getbinpkgs: bool
> + @param getbinpkg_refresh: attempt to refresh the
> cache
> + of remote package metadata if getbinpkgs is
> also True
> + @type getbinpkg_refresh: bool
> + """
>  
>   if self._populating:
>   return
> @@ -522,13 +530,13 @@ class binarytree(object):
>   pkgindex_lock =
> lockfile(self._pkgindex_file, wantnewlockfile=1)
>   self._populating = True
> - self._populate(getbinpkgs)
> + self._populate(getbinpkgs,
> getbinpkg_refresh=getbinpkg_refresh) finally:
>   if pkgindex_lock:
>   unlockfile(pkgindex_lock)
>   self._populating = False
>  
> - def _populate(self, getbinpkgs=0):
> + def _populate(self, getbinpkgs=False,
> getbinpkg_refresh=True): if (not os.path.isdir(self.pkgdir) and not
> getbinpkgs): return 0
>  
> @@ -832,6 +840,9 @@ class binarytree(object):
>   url = base_url.rstrip("/") +
> "/Packages" f = None
>  
> + if not getbinpkg_refresh and
> local_timestamp:
> + raise
> UseCachedCopyOfRemoteIndex() +
>   try:
>   ttl =
> float(pkgindex.header.get("TTL", 0)) except ValueError:

looks good.  Thanks for migrating that 0 for False

-- 
Brian Dolbec 




[gentoo-dev] Last rites: app-emulation/crossover-office-bin app-emulation/crossover-office-pro-bin

2017-04-03 Thread NP-Hardass
# NP-Hardass  (03 Apr 2017)
# Masked for removal in 30 days. Unable to generate new
# hashes for the manifest, per Bug #612720, Bug #612718
# Upstream has also deprecated these in favor of
# app-emulation/crossover-bin
app-emulation/crossover-office-bin
app-emulation/crossover-office-pro-bin

-- 
NP-Hardass



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-dev] [RFC] New Manifest hashes and how to enable them

2017-04-03 Thread Michał Górny
On wto, 2017-04-04 at 00:32 +0700, Vadim A. Misbakh-Soloviov wrote:
> Good idea, but all the time I read it from first mention until the end of 
> your 
> email, I asked myself: "Who the hell on the Earth need GOST-crypto crap in 
> portage?".
> 
> The only purpose of this crypto algorythms is to use them in Russian 
> government-related structures (includig schools, tho :-/ ) just because of 
> "security through NIH-syndrome".
> 
> So, does it mean, gentoo was (or have plans to be) certified by FSB to be 
> "Russian national OS", or how is it possible on Earth that GOST algos was 
> implemented in portage? :)
> 

I think the point was more like it's unfair to give NSA backdoors
without giving one to KGB as well.

-- 
Best regards,
Michał Górny


signature.asc
Description: This is a digitally signed message part


Re: [gentoo-dev] [RFC] New Manifest hashes and how to enable them

2017-04-03 Thread Robin H. Johnson
On Tue, Apr 04, 2017 at 12:49:16AM +0700, Vadim A. Misbakh-Soloviov wrote:
> > What is the gain of using a secure hash
> > algorithm in the manifests if you can simply replace the manifest with a
> > MITM attack on the rsync update?
> I'd say "the solution is to stop using rsync and use git" (there is git 
> mirror 
> with all the metadata), but...
> Git does not support (correct me, if I'm wrong) resuming a fetch in case of 
> fails (bad connection, slow connection, or the any other reason to stop it 
> and 
> continue later).
Upstream is still working on resumable fetch, and if you need it
already, it can be had via git bundles.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136



Re: [gentoo-dev] [RFC] New Manifest hashes and how to enable them

2017-04-03 Thread David Seifert
On Mon, 2017-04-03 at 19:09 +0200, Michał Górny wrote:
> Therefore, my proposal would be to use the following set once their
> support reaches the stable version of Portage:
> 
>   manifest-hashes = SHA512 SHA3-512 WHIRLPOOL
> 
> 
> Your thoughts?
> 
> 
> 
> [1]:https://bugs.gentoo.org/612716
> [2]:https://bugs.gentoo.org/611568
> 
> -- 
> Best regards,
> Michał Górny

+5 Let's not repeat the git mistake of delaying this until it's too
late.



Re: [gentoo-dev] [RFC] New Manifest hashes and how to enable them

2017-04-03 Thread Vadim A. Misbakh-Soloviov
> What is the gain of using a secure hash
> algorithm in the manifests if you can simply replace the manifest with a
> MITM attack on the rsync update?

I'd say "the solution is to stop using rsync and use git" (there is git mirror 
with all the metadata), but...
Git does not support (correct me, if I'm wrong) resuming a fetch in case of 
fails (bad connection, slow connection, or the any other reason to stop it and 
continue later).

So... We either need GPG manifest signing enabled, or totally move to git and 
ignore all the users with bad internet connection and totally move portage to 
git (hint: we shouldn't), until we invent something else, that can solve all 
of that problems.



Re: [gentoo-dev] [RFC] New Manifest hashes and how to enable them

2017-04-03 Thread Vadim A. Misbakh-Soloviov
Good idea, but all the time I read it from first mention until the end of your 
email, I asked myself: "Who the hell on the Earth need GOST-crypto crap in 
portage?".

The only purpose of this crypto algorythms is to use them in Russian 
government-related structures (includig schools, tho :-/ ) just because of 
"security through NIH-syndrome".

So, does it mean, gentoo was (or have plans to be) certified by FSB to be 
"Russian national OS", or how is it possible on Earth that GOST algos was 
implemented in portage? :)



So, excluding this moment, I voting up for your suggestion ;)



Re: [gentoo-dev] [RFC] New Manifest hashes and how to enable them

2017-04-03 Thread Matthias Maier
>   manifest-hashes = SHA512 SHA3-512 WHIRLPOOL
>
> Your thoughts?

I just want to point out that according to GLEP 63 we only require pgp
signatures with at least sha-256 [1]. Further, our PGP signatures by the
release team are as well either SHA-256/SHA-512.

So using SHA3-512 (or whirlpool for that matter) is nice but it feels a
bit like overdoing it a bit. What about simply SHA512 and calling it a
day?

Further, it might be a good time to finally resolve the issue with our
rsync integrity for users. (What is the gain of using a secure hash
algorithm in the manifests if you can simply replace the manifest with a
MITM attack on the rsync update?)

Best,
Matthias

[1] https://wiki.gentoo.org/wiki/GLEP:63#Specifications_for_GnuPG_keys


signature.asc
Description: PGP signature


[gentoo-dev] [RFC] New Manifest hashes and how to enable them

2017-04-03 Thread Michał Górny
Hi, everyone.

I'd like to open an early discussion and start planning transition to
an updated set of Manifest hashes.


Current state
=

The current hash set includes the three following hashes:
- SHA256 (SHA2),
- SHA512 (SHA2),
- Whirlpool.

Of these three hashes, SHA256 is considered 'required' by Portage. All
current Manifests in Gentoo include it.

SHA512 and Whirlpool are included in the majority of Manifest.
The packages missing them are tracked in [1].

There are still some stale hashes present in some Manifests (e.g.
RMD160).


Supported hash algorithms in package managers
=

The following hashes are supported by the stable version of Portage,
pkgcore and Paludis: MD5, SHA1, SHA256, SHA512, RMD160, WHIRLPOOL.

Support for MD5, SHA1, SHA256 and SHA512 in Portage and pkgcore is
provided unconditionally by all supported Python versions. For
WHIRLPOOL, a fallback implementation is always provided.

The following hashes are supported by the current git version of Portage
and are on the way of being integrated into pkgcore: BLAKE2B, BLAKE2S,
SHA3-256, SHA3-512. They are not currently supported by Paludis.

Support for those algorithms is guaranteed in Python 3.6+. Fallbacks for
older Python versions are supported -- using pygcrypt, pycryptodome
and pysha3 (the last one doesn't include BLAKE2) in Portage; and using
pycryptodome in pkgcore.

Additionally, the current git version of Portage supports
the STREEBOG256 and STREEBOG512 (GOST R 34.11-2012) algorithms. They can
be provided either using pygcrypt or pygost modules.

Of the listed fallbacks, the following limitations need to be noted:

A. pygcrypt requires old cffi, and therefore is not in ::gentoo. I have
contacted upstream and they are working on updating it.

B. pycryptodome blocks pycrypto, and some packages explicitly require
pycrypto (see tracker at [2]).

C. pygost is just horrible code-wise, and I've only added it so that we
can run tests without any special dependencies.


SHA256 deprecation
==

I think the first reasonable change would be to deprecate SHA256. It is
pretty much the same algorithm as SHA512, except for different
parameters. It is weaker than SHA512, and SHA512 is supported on all
existing platforms anyway.

Therefore, I think the following procedure would apply (please correct
me if I'm wrong):

1. wait till all remaining blockers of [1] are fixed,

2. update the required hash from SHA256 to SHA512 in Portage,

3. wait till stable Portage carries the above change + (possibly some
upgrade time?),

4. remove SHA256 from list of hashes included in Manifests.


New hash set


After deprecating SHA256, the hash list would include only SHA512
and WHIRLPOOL. I think the first one should be kept as a portable
required hash, and I'm indifferent to keeping the second one.

Of the new hashes, I think it would be the most reasonable to add SHA3-
512. It is built-in since Python 3.6+, and multiple implementations are
available for older Python versions (cryptography [though I think it
doesn't work with current openssl], pycryptodome, pygcrypt, pysha3).

The alternative would be to use BLAKE2B hash (which provides similar
strength to SHA3-512). However, it provides less fallbacks (no support
in libgcypt nor cryptography).

For diversity, we could also consider including the Streebog hash
(a user has requested it already). However, it seems to be at SHA2 level
and does not have any reasonable implementation except for libgcrypt.

Therefore, my proposal would be to use the following set once their
support reaches the stable version of Portage:

  manifest-hashes = SHA512 SHA3-512 WHIRLPOOL


Your thoughts?



[1]:https://bugs.gentoo.org/612716
[2]:https://bugs.gentoo.org/611568

-- 
Best regards,
Michał Górny


signature.asc
Description: This is a digitally signed message part


[gentoo-dev] Packages up for grasp

2017-04-03 Thread Amy Liffey
Hello,
some packages for grasp who is interested:

app-emulation/vpcs
dev-python/aiohttp-cors
dev-python/kiwisolver
dev-python/python-zipstream
dev-python/raven
net-misc/gns3-converter
net-misc/gns3-gui
net-misc/gns3-server
net-misc/leapcast

Cheers,
Amy Liffey



signature.asc
Description: OpenPGP digital signature