Re: [gentoo-portage-dev] [PATCH 1/3] portage.const: Remove obsolete manifest-hashes comment

[Robin H. Johnson]

On Mon, Nov 06, 2017 at 09:14:56AM +0100, Michał Górny wrote:
> -# Future events:
> -#
> -# After WHIRLPOOL is supported in stable portage for at least 1 year:
> -# - Change MANIFEST2_REQUIRED_HASH to WHIRLPOOL.
> -# - Remove SHA256 from MANIFEST2_HASH_*.
> -# - Set manifest-hashes in gentoo-x86/metadata/layout.conf as follows:
> -# manifest-hashes = SHA512 WHIRLPOOL
Can we please drop SHA256 as a required hash already? It was scheduled
for future removal a long time ago. I don't want that to get lost with
the removal of this comment block.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136


signature.asc
Description: Digital signature

Re: [gentoo-dev] [v1.0.4] GLEP 74: Full-tree verification using Manifest files

[Michał Górny]

Hopefully the last version, after getting all the suggestions
from Robin.

W dniu czw, 26.10.2017 o godzinie 22∶12 +0200, użytkownik Michał Górny
napisał:
> 
> ReST: https://dev.gentoo.org/~mgorny/tmp/glep-0074.rst
> HTML: https://dev.gentoo.org/~mgorny/tmp/glep-0074.html
> impl: https://github.com/mgorny/gemato/
> 

---
GLEP: 74
Title: Full-tree verification using Manifest files
Author: Michał Górny ,
Robin Hugh Johnson ,
Ulrich Müller 
Type: Standards Track
Status: Draft
Version: 1
Created: 2017-10-21
Last-Modified: 2017-11-06
Post-History: 2017-10-26
Content-Type: text/x-rst
Requires: 59, 61
Replaces: 44, 58, 60
---

Abstract


This GLEP extends the Manifest file format to cover full-tree file
integrity and authenticity checks.The format aims to be future-proof,
efficient and provide means of backwards compatibility.


Motivation
==

The Manifest files as defined by GLEP 44 [#GLEP44]_ provide the current
means of verifying the integrity of distfiles and package files
in Gentoo. Combined with OpenPGP signatures, they provide means to
ensure the authenticity of the covered files. However, as noted
in GLEP 57 [#GLEP57]_ they lack the ability to provide full-tree
authenticity verification as they do not cover any files outside
the package directory. In particular, they provide multiple ways
for a third party to inject malicious code into the ebuild environment.

Historically, the topic of providing authenticity coverage for the whole
repository has been mentioned multiple times. The most noteworthy effort
are GLEPs 58 [#GLEP58]_ and 60 [#GLEP60]_ by Robin H. Johnson from 2008.
They were accepted by the Council in 2010 but have never been
implemented. When potential implementation work started in 2017, a new
discussion about the specification arose. It prompted the creation
of a competing GLEP that would provide a redesigned alternative to
the old GLEPs.

This specification is designed with the following goals in mind:

1. It should provide means to ensure the authenticity of the complete
   repository, including preventing the injection of additional files.

2. The format should be universal enough to work both for the Gentoo
   repository and third-party repositories of different characteristics.

3. The Manifest files should be verifiable stand-alone, that is without
   knowing any details about the underlying repository format.


Specification
=

Manifest file format


This specification reuses and extends the Manifest file format defined
in GLEP 44 [#GLEP44]_. For the purpose of it, the *file type* field is
repurposed as a generic *tag* that could also indicate additional
(non-checksum) metadata. Appropriately, those tags can be followed by
other space-separated values.

Unless specified otherwise, the paths used in the Manifest files
are relative to the directory containing the Manifest file. The paths
must not reference the parent directory (``..``).


Manifest file locations and nesting
---

The ``Manifest`` file located in the root directory of the repository
is called top-level Manifest, and it is used to perform the full-tree
verification. In order to verify the authenticity, it must be signed
using OpenPGP, using the armored cleartext format.

The top-level Manifest may reference sub-Manifests contained
in subdirectories of the repository. The sub-Manifests are traditionally
named ``Manifest``; however, the implementation must support arbitrary
names, including the possibility of multiple (split) Manifests
for a single directory. The sub-Manifest can only cover the files inside
the directory tree where it resides.

The sub-Manifest can also be signed using OpenPGP armored cleartext
format. However, the signature verification can be omitted if it is
covered by a signed top-level Manifest.


Directory tree coverage
---

The specification provides three ways of skipping Manifest verification
of specific files and directories (recursively):

1. explicit ``IGNORE`` entries in Manifest files,

2. injected ignore paths via package manager configuration,

3. using names starting with a dot (``.``) which are always skipped.

All files that are not ignored must be covered by at least one
of the Manifests.

A single file may be matched by multiple identical or equivalent
Manifest entries, if and only if the entries have the same semantics,
specify the same size and the checksums common to both entries match.
It is an error for a single file to be matched by multiple entries
of different semantics, file size or checksum values. It is an error
to specify another entry for a file matching ``IGNORE``, or one of its
subdirectories.

The file entries (except for ``IGNORE``) can be specified for regular
files only. Symbolic links are followed when opening files
and traversing directories. It is an error to specify an entry for
a different file 

[gentoo-dev] dev-php/PEAR-HTTP_Download

[Brian Evans]

# Brian Evans 

Re: [gentoo-dev] [v1.0.3] GLEP 74: Full-tree verification using Manifest files

[Robin H. Johnson]

On Sun, Nov 05, 2017 at 10:10:32PM +0100, Michał Górny wrote:
> > Nits: 
> > - please stick to ASCII ellipsis. The unicode ellipsis is unreadable in
> >   some monospace fonts.
> Done. Also replaced '—' for consistency.
I wasn't even aware you had used a different dash, it was rendered
identically here, definitely thanks for fixing that too.

> > Further items inline:
> > > Directory tree coverage
> > > ---
> I've went for something even more explicit:
> | If files or directories that are not otherwise ignored reside
> | on a different filesystem, or symbolic links point to targets
> | on a different filesystem, they must be explicitly excluded
> | via ``IGNORE``.
+1, resolves the concern very well, nice and clear.

> > > Tree layout restrictions
> > > 
> > 'common' in the second sentence seems odd. What about uncommon
> > filenames? Maybe just s/other common filenames/other filenames/.
> Done. The idea was to say 'do not put IGNORE for corner cases which are
> better handled via PM config' but I guess it's not necessary here.
Yes. Generally, IGNORE entries in Manifest should be for files
distributed alongside the Manifest. We're say as common special cases,
that local/distfiles/packages/lost+found are also known for ignore,
since they have previously-defined meaning in the repo (along with the
old timestamp files).

> > > Non-strict Manifest verification
> > > 
> Rewritten to:
> | It is much more common for users to strip whole packages
> | or categories. The ``MISC`` type is not suitable for that,
> | and so a dedicated package manager mechanism needs to be developed
> | instead; possibly combining it with rsync exclusion list. The same
> | mechanism can also handle files that historically used the ``MISC``
> | type.
> But it's merely a rationale, so I'd rather not spend another hour trying
> to cover every corner case in it.
+1. Maybe cover it with a single sentence, "As an example, the package
manager may choose to generate both the rsync exclusion list and
Manifest IGNORE based on a source list"

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136


signature.asc
Description: Digital signature

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Michał Górny]

W dniu pon, 06.11.2017 o godzinie 19∶13 +, użytkownik Robin H.
Johnson napisał:
> +1 overall, just one timeline clarification.
> 
> On Mon, Nov 06, 2017 at 05:58:21PM +0100, Michał Górny wrote:
> > T + 7 days
> > --
> > Set:
> >   manifest-hashes = BLAKE2B SHA512
> >   manifest-required-hashes = SHA512
> > 
> > New Manifest entries will use the new hashes but Portage will keep the
> > old hash set whenever it would need to refetch old distfiles.
> 
> Query:
> Do we need to wait for it to be stable before making this change?
> Shouldn't old stable versions of Portage continue to verify SHA512 fine?
> Mostly I think devs need to be using a new enough Portage that can
> generate the BLAKE2B entries, but it shouldn't impact user Portage
> versions.
> 

Devs are who I'm worried about. Those 7 days should give them enough
time to upgrade their stable Portage.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Mike Gilbert]

On Mon, Nov 6, 2017 at 2:13 PM, Robin H. Johnson  wrote:
> +1 overall, just one timeline clarification.
>
> On Mon, Nov 06, 2017 at 05:58:21PM +0100, Michał Górny wrote:
>> T + 7 days
>> --
>> Set:
>>   manifest-hashes = BLAKE2B SHA512
>>   manifest-required-hashes = SHA512
>>
>> New Manifest entries will use the new hashes but Portage will keep the
>> old hash set whenever it would need to refetch old distfiles.
> Query:
> Do we need to wait for it to be stable before making this change?
> Shouldn't old stable versions of Portage continue to verify SHA512 fine?
> Mostly I think devs need to be using a new enough Portage that can
> generate the BLAKE2B entries, but it shouldn't impact user Portage
> versions.

Quite a few devs use stable versions of Portage and Repoman when committing.

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Robin H. Johnson]

+1 overall, just one timeline clarification.

On Mon, Nov 06, 2017 at 05:58:21PM +0100, Michał Górny wrote:
> T + 7 days
> --
> Set:
>   manifest-hashes = BLAKE2B SHA512
>   manifest-required-hashes = SHA512
> 
> New Manifest entries will use the new hashes but Portage will keep the
> old hash set whenever it would need to refetch old distfiles.
Query:
Do we need to wait for it to be stable before making this change?
Shouldn't old stable versions of Portage continue to verify SHA512 fine?
Mostly I think devs need to be using a new enough Portage that can
generate the BLAKE2B entries, but it shouldn't impact user Portage
versions.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136


signature.asc
Description: Digital signature

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Michał Górny]

Hi,

So here's my proposed plan, after considering all the replies.


Immediately after accepting
---

a. Revbump Portage to add pyblake2 dep (to ensure BLAKE2 is supported
on py<3.6) and request stabilizing this version.

b. Create a git update hook that rejects Manifest entries that contain
SHA512 only, to prevent a bug in current versions of Portage, that
causes it to skip BLAKE2 when no implementation is installed instead
of complaining [optional].


Now, let T = day when the new version is stable on amd64.


T + 7 days
--

Set:

  manifest-hashes = BLAKE2B SHA512
  manifest-required-hashes = SHA512

New Manifest entries will use the new hashes but Portage will keep the
old hash set whenever it would need to refetch old distfiles.



T + 3 months


Set:

  manifest-required-hashes = BLAKE2B

Portage will now request updating hashes for all files, including
old distfiles. We will start proactively updating Manifests here,
and file bugs for fetch-restricted packages.


T + 6 months


All Manifests should use the new hashes by this time. The remaining
fetch-restricted packages should be last-rited.


T + 36 months
-

Set:

  manifest-hashes = BLAKE2B

Remove SHA512 from all Manifests.


-- 
Best regards,
Michał Górny

Re: [gentoo-portage-dev] [PATCH] Make manifest-required-hashes configurable

[Zac Medico]

On 11/06/2017 08:47 AM, Michał Górny wrote:
> W dniu pon, 06.11.2017 o godzinie 08∶44 -0800, użytkownik Zac Medico
> napisał:
>> On 11/06/2017 07:27 AM, Michał Górny wrote:
>>> @@ -148,15 +147,21 @@ class Manifest(object):
>>> self.pkgdir = _unicode_decode(pkgdir).rstrip(os.sep) + os.sep
>>> self.fhashdict = {}
>>> self.hashes = set()
>>> +   self.required_hashes = set()
>>>  
>>> if hashes is None:
>>> hashes = MANIFEST2_HASH_DEFAULTS
>>> +   if required_hashes is None:
>>> +   required_hashes = hashes
>>>  
>>> self.hashes.update(hashes)
>>> self.hashes.difference_update(hashname for hashname in \
>>> list(self.hashes) if hashname not in 
>>> get_valid_checksum_keys())
>>> self.hashes.add("size")
>>> -   self.hashes.add(MANIFEST2_REQUIRED_HASH)
>>> +
>>> +   self.required_hashes.update(required_hashes)
>>> +   self.required_hashes.intersection_update(self.hashes)
>>> +
>>
>> Requiring that presence of digests for all manifest-hashes can be
>> problematic, because when we extend manifest-hashes it means that nobody
>> can regenerate a particular manifest without fetching *all* distfiles
>> referenced in the Manifest. It's much more friendly if we generate the
>> missing digests when the distfiles happen to be available in DISTDIR,
>> which is how the portage currently behaves.
> 
> This doesn't change. The intersection only strips away hashes that are
> not in 'self.hashes'.

Then the bug must have been introduced by changes to behavior of
digestgen in the same patch. Try adding MD5 to manifest-hashes, and then
try generating a manifest with no internet connection and DISTDIR set to
an empty directory. You'll see that it fails with the patch, but
succeeds with the current master branch.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] Make manifest-required-hashes configurable

[Michał Górny]

W dniu pon, 06.11.2017 o godzinie 08∶44 -0800, użytkownik Zac Medico
napisał:
> On 11/06/2017 07:27 AM, Michał Górny wrote:
> > @@ -148,15 +147,21 @@ class Manifest(object):
> > self.pkgdir = _unicode_decode(pkgdir).rstrip(os.sep) + os.sep
> > self.fhashdict = {}
> > self.hashes = set()
> > +   self.required_hashes = set()
> >  
> > if hashes is None:
> > hashes = MANIFEST2_HASH_DEFAULTS
> > +   if required_hashes is None:
> > +   required_hashes = hashes
> >  
> > self.hashes.update(hashes)
> > self.hashes.difference_update(hashname for hashname in \
> > list(self.hashes) if hashname not in 
> > get_valid_checksum_keys())
> > self.hashes.add("size")
> > -   self.hashes.add(MANIFEST2_REQUIRED_HASH)
> > +
> > +   self.required_hashes.update(required_hashes)
> > +   self.required_hashes.intersection_update(self.hashes)
> > +
> 
> Requiring that presence of digests for all manifest-hashes can be
> problematic, because when we extend manifest-hashes it means that nobody
> can regenerate a particular manifest without fetching *all* distfiles
> referenced in the Manifest. It's much more friendly if we generate the
> missing digests when the distfiles happen to be available in DISTDIR,
> which is how the portage currently behaves.

This doesn't change. The intersection only strips away hashes that are
not in 'self.hashes'.

-- 
Best regards,
Michał Górny

Re: [gentoo-portage-dev] [PATCH] Make manifest-required-hashes configurable

[Zac Medico]

On 11/06/2017 07:27 AM, Michał Górny wrote:
> @@ -148,15 +147,21 @@ class Manifest(object):
>   self.pkgdir = _unicode_decode(pkgdir).rstrip(os.sep) + os.sep
>   self.fhashdict = {}
>   self.hashes = set()
> + self.required_hashes = set()
>  
>   if hashes is None:
>   hashes = MANIFEST2_HASH_DEFAULTS
> + if required_hashes is None:
> + required_hashes = hashes
>  
>   self.hashes.update(hashes)
>   self.hashes.difference_update(hashname for hashname in \
>   list(self.hashes) if hashname not in 
> get_valid_checksum_keys())
>   self.hashes.add("size")
> - self.hashes.add(MANIFEST2_REQUIRED_HASH)
> +
> + self.required_hashes.update(required_hashes)
> + self.required_hashes.intersection_update(self.hashes)
> +

Requiring that presence of digests for all manifest-hashes can be
problematic, because when we extend manifest-hashes it means that nobody
can regenerate a particular manifest without fetching *all* distfiles
referenced in the Manifest. It's much more friendly if we generate the
missing digests when the distfiles happen to be available in DISTDIR,
which is how the portage currently behaves.
-- 
Thanks,
Zac

[gentoo-portage-dev] [PATCH] Make manifest-required-hashes configurable

[Michał Górny]

The set of required hashes specify which hashes must be present for
a distfile not to be refetched. It makes little sense to hardcode this
value, and it is mostly useful for transition periods, so make it
configurable via layout.conf and default to all hashes
in manifest-hashes.
---
 pym/portage/_emirrordist/FetchTask.py   |  2 +-
 pym/portage/const.py|  2 +-
 pym/portage/manifest.py | 26 +---
 pym/portage/package/ebuild/digestgen.py |  4 ++--
 pym/portage/repository/config.py| 42 ++---
 pym/portage/tests/ebuild/test_config.py |  1 +
 repoman/pym/repoman/repos.py| 14 +++
 7 files changed, 64 insertions(+), 27 deletions(-)

diff --git a/pym/portage/_emirrordist/FetchTask.py 
b/pym/portage/_emirrordist/FetchTask.py
index 203b8c213..47908cb6b 100644
--- a/pym/portage/_emirrordist/FetchTask.py
+++ b/pym/portage/_emirrordist/FetchTask.py
@@ -20,7 +20,7 @@ from portage.util._async.PipeLogger import PipeLogger
 from portage.util._async.PopenProcess import PopenProcess
 from _emerge.CompositeTask import CompositeTask
 
-default_hash_name = portage.const.MANIFEST2_REQUIRED_HASH
+default_hash_name = portage.const.MANIFEST2_HASH_DEFAULT
 
 # Use --no-check-certificate since Manifest digests should provide
 # enough security, and certificates can be self-signed or whatnot.
diff --git a/pym/portage/const.py b/pym/portage/const.py
index 0af57d0e2..ec877b841 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -207,7 +207,7 @@ EAPI = 6
 HASHING_BLOCKSIZE= 32768
 
 MANIFEST2_HASH_DEFAULTS = frozenset(["SHA256", "SHA512", "WHIRLPOOL"])
-MANIFEST2_REQUIRED_HASH  = "SHA512"
+MANIFEST2_HASH_DEFAULT  = "SHA512"
 
 MANIFEST2_IDENTIFIERS= ("AUX", "MISC", "DIST", "EBUILD")
 
diff --git a/pym/portage/manifest.py b/pym/portage/manifest.py
index 36c82690c..4ec20515e 100644
--- a/pym/portage/manifest.py
+++ b/pym/portage/manifest.py
@@ -26,8 +26,7 @@ from portage import _unicode_encode
 from portage.exception import DigestException, FileNotFound, \
InvalidDataType, MissingParameter, PermissionDenied, \
PortageException, PortagePackageException
-from portage.const import (MANIFEST2_HASH_DEFAULTS,
-   MANIFEST2_IDENTIFIERS, MANIFEST2_REQUIRED_HASH)
+from portage.const import (MANIFEST2_HASH_DEFAULTS, MANIFEST2_IDENTIFIERS)
 from portage.localization import _
 
 _manifest_re = re.compile(
@@ -128,7 +127,7 @@ class Manifest(object):
parsers = (parseManifest2,)
def __init__(self, pkgdir, distdir=None, fetchlist_dict=None,
manifest1_compat=DeprecationWarning, from_scratch=False, 
thin=False,
-   allow_missing=False, allow_create=True, hashes=None,
+   allow_missing=False, allow_create=True, hashes=None, 
required_hashes=None,
find_invalid_path_char=None, strict_misc_digests=True):
""" Create new Manifest instance for package in pkgdir.
Do not parse Manifest file if from_scratch == True (only 
for internal use)
@@ -148,15 +147,21 @@ class Manifest(object):
self.pkgdir = _unicode_decode(pkgdir).rstrip(os.sep) + os.sep
self.fhashdict = {}
self.hashes = set()
+   self.required_hashes = set()
 
if hashes is None:
hashes = MANIFEST2_HASH_DEFAULTS
+   if required_hashes is None:
+   required_hashes = hashes
 
self.hashes.update(hashes)
self.hashes.difference_update(hashname for hashname in \
list(self.hashes) if hashname not in 
get_valid_checksum_keys())
self.hashes.add("size")
-   self.hashes.add(MANIFEST2_REQUIRED_HASH)
+
+   self.required_hashes.update(required_hashes)
+   self.required_hashes.intersection_update(self.hashes)
+
for t in MANIFEST2_IDENTIFIERS:
self.fhashdict[t] = {}
if not from_scratch:
@@ -269,9 +274,11 @@ class Manifest(object):
def checkIntegrity(self):
for t in self.fhashdict:
for f in self.fhashdict[t]:
-   if MANIFEST2_REQUIRED_HASH not in 
self.fhashdict[t][f]:
-   raise MissingParameter(_("Missing %s 
checksum: %s %s") %
-   (MANIFEST2_REQUIRED_HASH, t, f))
+   diff = self.required_hashes.difference(
+   set(self.fhashdict[t][f]))
+   if diff:
+   raise MissingParameter(_("Missing %s 
checksum(s): %s %s") %
+   (' '.join(diff), t, f))
 
def write(self, sign=False, force=False):
""" Write Manifest 

Re: [gentoo-portage-dev] [PATCH 3/3] Remove last traces of Manifest1 code

[Zac Medico]

On 11/06/2017 12:14 AM, Michał Górny wrote:
> ---
>  pym/portage/const.py| 1 -
>  pym/portage/manifest.py | 7 ++-
>  2 files changed, 2 insertions(+), 6 deletions(-)

This patch series looks good. Please merge.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-portage-dev] [PATCH 1/3] portage.const: Remove obsolete manifest-hashes comment

[Michał Górny]

The comment is based upon the wrong assumption that the world is
Portage-centric. We are keeping the manifest-hashes in layout.conf
indefinitely, and we will most likely take BLAKE2 instead of SHA3.
---
 pym/portage/const.py | 23 ---
 1 file changed, 23 deletions(-)

diff --git a/pym/portage/const.py b/pym/portage/const.py
index cbd2b6042..11e94b0a2 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -207,29 +207,6 @@ EAPI = 6
 HASHING_BLOCKSIZE= 32768
 MANIFEST1_HASH_FUNCTIONS = ("MD5", "SHA256", "RMD160")
 
-# Past events:
-#
-# 20120704 - After WHIRLPOOL is supported in stable portage:
-# - Set manifest-hashes in gentoo-x86/metadata/layout.conf as follows:
-# manifest-hashes = SHA256 SHA512 WHIRLPOOL
-# - Add SHA512 and WHIRLPOOL to MANIFEST2_HASH_DEFAULTS.
-# - Remove SHA1 and RMD160 from MANIFEST2_HASH_*.
-#
-# Future events:
-#
-# After WHIRLPOOL is supported in stable portage for at least 1 year:
-# - Change MANIFEST2_REQUIRED_HASH to WHIRLPOOL.
-# - Remove SHA256 from MANIFEST2_HASH_*.
-# - Set manifest-hashes in gentoo-x86/metadata/layout.conf as follows:
-# manifest-hashes = SHA512 WHIRLPOOL
-#
-# After SHA-3 is supported in stable portage:
-# - Set manifest-hashes in gentoo-x86/metadata/layout.conf as follows:
-# manifest-hashes = SHA3 SHA512 WHIRLPOOL
-#
-# After layout.conf settings correspond to defaults in stable portage:
-# - Remove redundant settings from gentoo-x86/metadata/layout.conf.
-
 MANIFEST2_HASH_FUNCTIONS = ("SHA256", "SHA512", "WHIRLPOOL",
"BLAKE2B", "BLAKE2S", "SHA3_256", "SHA3_512",
"STREEBOG256", "STREEBOG512")
-- 
2.15.0

[gentoo-portage-dev] [PATCH 3/3] Remove last traces of Manifest1 code

[Michał Górny]

---
 pym/portage/const.py| 1 -
 pym/portage/manifest.py | 7 ++-
 2 files changed, 2 insertions(+), 6 deletions(-)

diff --git a/pym/portage/const.py b/pym/portage/const.py
index 98b7c88c9..0af57d0e2 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -205,7 +205,6 @@ SUPPORTED_FEATURES   = frozenset([
 EAPI = 6
 
 HASHING_BLOCKSIZE= 32768
-MANIFEST1_HASH_FUNCTIONS = ("MD5", "SHA256", "RMD160")
 
 MANIFEST2_HASH_DEFAULTS = frozenset(["SHA256", "SHA512", "WHIRLPOOL"])
 MANIFEST2_REQUIRED_HASH  = "SHA512"
diff --git a/pym/portage/manifest.py b/pym/portage/manifest.py
index 0a68aa653..36c82690c 100644
--- a/pym/portage/manifest.py
+++ b/pym/portage/manifest.py
@@ -26,7 +26,7 @@ from portage import _unicode_encode
 from portage.exception import DigestException, FileNotFound, \
InvalidDataType, MissingParameter, PermissionDenied, \
PortageException, PortagePackageException
-from portage.const import (MANIFEST1_HASH_FUNCTIONS, MANIFEST2_HASH_DEFAULTS,
+from portage.const import (MANIFEST2_HASH_DEFAULTS,
MANIFEST2_IDENTIFIERS, MANIFEST2_REQUIRED_HASH)
 from portage.localization import _
 
@@ -710,10 +710,7 @@ class Manifest(object):
myfile.close()
for l in lines:
mysplit = l.split()
-   if len(mysplit) == 4 and mysplit[0] in 
MANIFEST1_HASH_FUNCTIONS \
-   and 1 not in rVal:
-   rVal.append(1)
-   elif len(mysplit) > 4 and mysplit[0] in 
MANIFEST2_IDENTIFIERS \
+   if len(mysplit) > 4 and mysplit[0] in 
MANIFEST2_IDENTIFIERS \
and ((len(mysplit) - 3) % 2) == 0 and not 2 in 
rVal:
rVal.append(2)
return rVal
-- 
2.15.0

[gentoo-portage-dev] [PATCH 2/3] Replace static MANIFEST2_HASH_FUNCTIONS with dynamic list

[Michał Górny]

Remove the MANIFEST2_HASH_FUNCTIONS const and replace it with
(deduplicated) calls to get_valid_checksum_keys(). We want Portage
to always complain whenever one of the hashes is not available even
if it is technically supported by the specific Portage version.

Closes: https://bugs.gentoo.org/634812
---
 man/portage.5| 4 ++--
 pym/portage/const.py | 3 ---
 pym/portage/manifest.py  | 6 +++---
 pym/portage/repository/config.py | 5 +++--
 repoman/pym/repoman/repos.py | 3 ++-
 5 files changed, 10 insertions(+), 11 deletions(-)

diff --git a/man/portage.5 b/man/portage.5
index 89dc8ce44..7605d7cfa 100644
--- a/man/portage.5
+++ b/man/portage.5
@@ -1261,8 +1261,8 @@ for every file), "true" (if an entry exists for a file, 
enforce it), or "false"
 .TP
 .BR manifest\-hashes
 List of hashes to generate/check in Manifest files.  Valid hashes depend on the
-current version of portage; see the portage.const.MANIFEST2_HASH_FUNCTIONS
-constant for the current list.
+current version of portage; see the portage.checksum module for the current
+list.
 .TP
 .BR update\-changelog " = [true|" false "]"
 The default setting for repoman's --echangelog option.
diff --git a/pym/portage/const.py b/pym/portage/const.py
index 11e94b0a2..98b7c88c9 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -207,9 +207,6 @@ EAPI = 6
 HASHING_BLOCKSIZE= 32768
 MANIFEST1_HASH_FUNCTIONS = ("MD5", "SHA256", "RMD160")
 
-MANIFEST2_HASH_FUNCTIONS = ("SHA256", "SHA512", "WHIRLPOOL",
-   "BLAKE2B", "BLAKE2S", "SHA3_256", "SHA3_512",
-   "STREEBOG256", "STREEBOG512")
 MANIFEST2_HASH_DEFAULTS = frozenset(["SHA256", "SHA512", "WHIRLPOOL"])
 MANIFEST2_REQUIRED_HASH  = "SHA512"
 
diff --git a/pym/portage/manifest.py b/pym/portage/manifest.py
index 28a77ba87..0a68aa653 100644
--- a/pym/portage/manifest.py
+++ b/pym/portage/manifest.py
@@ -27,7 +27,7 @@ from portage.exception import DigestException, FileNotFound, \
InvalidDataType, MissingParameter, PermissionDenied, \
PortageException, PortagePackageException
 from portage.const import (MANIFEST1_HASH_FUNCTIONS, MANIFEST2_HASH_DEFAULTS,
-   MANIFEST2_HASH_FUNCTIONS, MANIFEST2_IDENTIFIERS, 
MANIFEST2_REQUIRED_HASH)
+   MANIFEST2_IDENTIFIERS, MANIFEST2_REQUIRED_HASH)
 from portage.localization import _
 
 _manifest_re = re.compile(
@@ -152,7 +152,7 @@ class Manifest(object):
if hashes is None:
hashes = MANIFEST2_HASH_DEFAULTS
 
-   
self.hashes.update(hashes.intersection(MANIFEST2_HASH_FUNCTIONS))
+   self.hashes.update(hashes)
self.hashes.difference_update(hashname for hashname in \
list(self.hashes) if hashname not in 
get_valid_checksum_keys())
self.hashes.add("size")
@@ -251,7 +251,7 @@ class Manifest(object):
return myhashdict
 
def _createManifestEntries(self):
-   valid_hashes = set(MANIFEST2_HASH_FUNCTIONS)
+   valid_hashes = set(get_valid_checksum_keys())
valid_hashes.add('size')
mytypes = list(self.fhashdict)
mytypes.sort()
diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
index 902213014..3be0e8bda 100644
--- a/pym/portage/repository/config.py
+++ b/pym/portage/repository/config.py
@@ -11,7 +11,8 @@ import re
 
 import portage
 from portage import eclass_cache, os
-from portage.const import (MANIFEST2_HASH_FUNCTIONS, MANIFEST2_REQUIRED_HASH,
+from portage.checksum import get_valid_checksum_keys
+from portage.const import (MANIFEST2_REQUIRED_HASH,
PORTAGE_BASE_PATH, REPO_NAME_LOC, USER_CONFIG_PATH)
 from portage.eapi import 
eapi_allows_directories_on_profile_level_and_repository_level
 from portage.env.loaders import KeyValuePairFileLoader
@@ -1060,7 +1061,7 @@ def parse_layout_conf(repo_location, repo_name=None):
"layout_filename":layout_filename}),
DeprecationWarning)
unsupported_hashes = manifest_hashes.difference(
-   MANIFEST2_HASH_FUNCTIONS)
+   get_valid_checksum_keys())
if unsupported_hashes:
repo_name = _get_repo_name(repo_location, 
cached=repo_name)
warnings.warn((_("Repository named '%(repo_name)s' has 
a "
diff --git a/repoman/pym/repoman/repos.py b/repoman/pym/repoman/repos.py
index 39f53c180..11a6231de 100644
--- a/repoman/pym/repoman/repos.py
+++ b/repoman/pym/repoman/repos.py
@@ -13,6 +13,7 @@ from repoman._portage import portage
 from portage import os
 from portage import _encodings
 from portage import _unicode_encode
+from portage.checksum import get_valid_checksum_keys
 
 from repoman.errors import err
 from repoman.profile import ProfileDesc, valid_profile_types
@@ -116,7 +117,7 @@ class RepoSettings(object):