Re: [gentoo-portage-dev] Re: [PATCH v2 3/3] _emerge.Ebuild*: delay creating DISTDIR shadow until src_unpack

2018-01-25 Thread Zac Medico
On 01/25/2018 10:42 PM, Michał Górny wrote:
> W dniu czw, 25.01.2018 o godzinie 21∶30 -0800, użytkownik Zac Medico
> napisał:
>> On 01/25/2018 01:11 AM, Michał Górny wrote:
>>> W dniu czw, 25.01.2018 o godzinie 10∶07 +0100, użytkownik Michael
>>> Haubenwallner napisał:
 Hi,

 ${Subject} ringing a bell here:

 dev-db/oracle-instantclient is fetch restricted. As a binary package with
 multiple USE options there's a bunch of files to download - even for
 multiple archs when multilib is active.

 So in pkg_nofetch() I'm telling the user whether a file to download is
 "already here" or "still absent", by testing if $A exists in $DISTDIR.

 With ${Subject}, I'm wondering if DISTDIR is created for pkg_nofetch too.

>>>
>>> You're doing the wrong thing then. DISTDIR is not allowed
>>> in pkg_nofetch().
>>
>> It seems to be a common assumption that it's allowed, this command
>> currently shows 163 results in the gentoo repo:
>>
>> git grep -l pkg_nofetch | xargs grep 'e\(log\|info\).*DISTDIR' | wc -l
>>
>> We should double check with the PMS maintainers to see if they think
>> it's worthy of an exception. Otherwise, we need to announce the issue on
>> the gentoo-dev mailing list.
> 
> PMS maintainers already verified that back during the first run of those
> patches. However, we believe the only reasonable way to get this out of
> pkg_nofetch() is to actually stop it from working, so people would stop
> using it.

Okay, that works for me. The patches looks good. Please merge.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] Re: [PATCH v2 3/3] _emerge.Ebuild*: delay creating DISTDIR shadow until src_unpack

2018-01-25 Thread Michał Górny
W dniu czw, 25.01.2018 o godzinie 21∶30 -0800, użytkownik Zac Medico
napisał:
> On 01/25/2018 01:11 AM, Michał Górny wrote:
> > W dniu czw, 25.01.2018 o godzinie 10∶07 +0100, użytkownik Michael
> > Haubenwallner napisał:
> > > Hi,
> > > 
> > > ${Subject} ringing a bell here:
> > > 
> > > dev-db/oracle-instantclient is fetch restricted. As a binary package with
> > > multiple USE options there's a bunch of files to download - even for
> > > multiple archs when multilib is active.
> > > 
> > > So in pkg_nofetch() I'm telling the user whether a file to download is
> > > "already here" or "still absent", by testing if $A exists in $DISTDIR.
> > > 
> > > With ${Subject}, I'm wondering if DISTDIR is created for pkg_nofetch too.
> > > 
> > 
> > You're doing the wrong thing then. DISTDIR is not allowed
> > in pkg_nofetch().
> 
> It seems to be a common assumption that it's allowed, this command
> currently shows 163 results in the gentoo repo:
> 
> git grep -l pkg_nofetch | xargs grep 'e\(log\|info\).*DISTDIR' | wc -l
> 
> We should double check with the PMS maintainers to see if they think
> it's worthy of an exception. Otherwise, we need to announce the issue on
> the gentoo-dev mailing list.

PMS maintainers already verified that back during the first run of those
patches. However, we believe the only reasonable way to get this out of
pkg_nofetch() is to actually stop it from working, so people would stop
using it.

> Furthermore, you're touching files whose hashes have
> > not been verified which is twice wrong.
> 
> Checking if files exist is not really a security risk, but yes, we
> should conform to the spec.

There's no technical reason why the ebuild wouldn't have done more than
checking if they exist.

-- 
Best regards,
Michał Górny




Re: [gentoo-portage-dev] [PATCH] install-qa-check.d: Scan build log for CMake unused var warnings

2018-01-25 Thread Michał Górny
W dniu czw, 25.01.2018 o godzinie 20∶49 -0800, użytkownik Zac Medico
napisał:
> On 01/25/2018 01:13 AM, Michał Górny wrote:
> > Scan build log and report verbosely CMake warnings about unused
> > variables. This is a quite common problem, yet currently it is hard
> > to notice it since the warning is mixed with src_configure() output.
> > Repeat it verbosely after the install.
> > 
> > This check outputs warnings such as:
> > 
> >  * One or more CMake variables were not used by the project:
> >  *   CMAKE_USER_MAKE_RULES_OVERRIDE
> > ---
> >  bin/install-qa-check.d/90cmake-warnings | 28 
> >  1 file changed, 28 insertions(+)
> >  create mode 100644 bin/install-qa-check.d/90cmake-warnings
> > 
> > diff --git a/bin/install-qa-check.d/90cmake-warnings 
> > b/bin/install-qa-check.d/90cmake-warnings
> > new file mode 100644
> > index 0..36a09851b
> > --- /dev/null
> > +++ b/bin/install-qa-check.d/90cmake-warnings
> > @@ -0,0 +1,28 @@
> > +# Check for CMake invalid option warnings
> > +
> > +cmake_warn_check() {
> > +   if [[ -n ${PORTAGE_LOG_FILE} && -r ${PORTAGE_LOG_FILE} ]] ; then
> > +   local cat=cat
> > +   [[ ${PORTAGE_LOG_FILE} == *.gz ]] && cat=zcat
> > +
> > +   local vars=()
> > +   while read -r l; do
> > +   vars+=( "${l}" )
> > +   done < <( "${cat}" "${PORTAGE_LOG_FILE}" \
> > +   | sed -n -e '/Manually-specified variables were not 
> > used by the project/,/^--/{/^/p}' \
> > +   | sort -u)
> 
> We should probably use LC_ALL=C sort to ensure locale independence.
> 
> Otherwise, this patch looks good.
> 

Done that and merged, thanks!

-- 
Best regards,
Michał Górny




Re: [gentoo-portage-dev] [PATCH 8/8] eshowkw: Always group Prefix keywords last

2018-01-25 Thread Zac Medico
On 01/23/2018 03:25 PM, Alec Warner wrote:
> 
> 
> On Tue, Jan 23, 2018 at 4:48 PM, Michał Górny  > wrote:
> 
> Always group all Prefix keywords after other types of keywords. This
> not only ensures that fbsd sorts first but more importantly stabilizes
> the LHS output between regular and -P variant -- that is, -P always adds
> additional keywords at the end.
> ---
>  pym/gentoolkit/eshowkw/keywords_header.py | 16 ++--
>  1 file changed, 10 insertions(+), 6 deletions(-)
> 
> diff --git a/pym/gentoolkit/eshowkw/keywords_header.py
> b/pym/gentoolkit/eshowkw/keywords_header.py
> index 41b8ba4..1b64bfd 100644
> --- a/pym/gentoolkit/eshowkw/keywords_header.py
> +++ b/pym/gentoolkit/eshowkw/keywords_header.py
> @@ -142,12 +142,16 @@ class keywords_header:
>                                         break
> 
>                 # sort by, in order (to match Bugzilla):
> -               # 1. arch, then ~arch
> -               # 2. profile stability
> -               # 3. short keywords, then long (prefix, fbsd)
> -               # 4. keyword name in reverse component order
> -               normal.sort(key=lambda kw: (kw in
> self.__TESTING_KW_ARCHS,
> -                       levels.get(kw, 99), kw.count('-'),
> list(reversed(kw.split('-')
> +               # 1. non-prefix, then prefix (stable output between
> -P and not)
> +               # 2. arch, then ~arch
> +               # 3. profile stability
> +               # 4. short keywords, then long (prefix, fbsd)
> +               # 5. keyword name in reverse component order
> +               normal.sort(key=lambda kw: (self.__isPrefix(kw),
> +                       kw in self.__TESTING_KW_ARCHS,
> +                       levels.get(kw, 99),
> +                       kw.count('-'),
> +                       list(reversed(kw.split('-')
> 
> 
> I'm a bit sad about this lambda because its ended up a bit long.
> 
> What are your thoughts on splitting it out?

The fact that it's a lambda doesn't bother me so much as the
inefficiency of regenerating the key on every call. I've found this cute
little memodict decorator that will optimize it nicely:

http://code.activestate.com/recipes/578231-probably-the-fastest-memoization-decorator-in-the-/

def memodict(f):
class memodict(dict):
def __missing__(self, key):
ret = self[key] = f(key)
return ret
return memodict().__getitem__

-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-portage-dev] Re: [PATCH v2 3/3] _emerge.Ebuild*: delay creating DISTDIR shadow until src_unpack

2018-01-25 Thread Zac Medico
On 01/25/2018 01:11 AM, Michał Górny wrote:
> W dniu czw, 25.01.2018 o godzinie 10∶07 +0100, użytkownik Michael
> Haubenwallner napisał:
>> Hi,
>>
>> ${Subject} ringing a bell here:
>>
>> dev-db/oracle-instantclient is fetch restricted. As a binary package with
>> multiple USE options there's a bunch of files to download - even for
>> multiple archs when multilib is active.
>>
>> So in pkg_nofetch() I'm telling the user whether a file to download is
>> "already here" or "still absent", by testing if $A exists in $DISTDIR.
>>
>> With ${Subject}, I'm wondering if DISTDIR is created for pkg_nofetch too.
>>
> 
> You're doing the wrong thing then. DISTDIR is not allowed
> in pkg_nofetch().

It seems to be a common assumption that it's allowed, this command
currently shows 163 results in the gentoo repo:

git grep -l pkg_nofetch | xargs grep 'e\(log\|info\).*DISTDIR' | wc -l

We should double check with the PMS maintainers to see if they think
it's worthy of an exception. Otherwise, we need to announce the issue on
the gentoo-dev mailing list.

Furthermore, you're touching files whose hashes have
> not been verified which is twice wrong.

Checking if files exist is not really a security risk, but yes, we
should conform to the spec.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-dev] [News item review] Portage rsync tree verification (v2)

2018-01-25 Thread Alon Bar-Lev
On 26 January 2018 at 00:21, Robin H. Johnson  wrote:
> On Thu, Jan 25, 2018 at 11:55:58PM +0200, Alon Bar-Lev wrote:
>> I did not looked into the detailed implementation, however, please
>> make sure integrity check handles the same cases we have applied to
>> emerge-webrsync in the past, including:
> Gemato is the implementation of GLEP74/MetaManifest, which DOES
> explicitly address both of these concerns.

Good!
Thanks.

>
>> 1. Fast forward only in time, this is required to avoid hacker to
>> redirect into older portage to install vulnerabilities that were
>> approved at that time.
> Replay attacks per #1 are addressed via TIMESTAMP field in MetaManifest.

Interesting, I tried again to understand how it is working without
performing rsync to a temporary directory, compare the timestamp and
reject if unexpected.
Are we doing multiple rsync for the metadata?
Long since I used this insecure rsync...

For me it seems like webrsync and/or squashfs are much easier/faster
to apply integrity into than rsync... :)

Regards,
Alon



Re: [gentoo-dev] [News item review] Portage rsync tree verification (v2)

2018-01-25 Thread Robin H. Johnson
On Thu, Jan 25, 2018 at 11:55:58PM +0200, Alon Bar-Lev wrote:
> I did not looked into the detailed implementation, however, please
> make sure integrity check handles the same cases we have applied to
> emerge-webrsync in the past, including:
Gemato is the implementation of GLEP74/MetaManifest, which DOES
explicitly address both of these concerns.

> 1. Fast forward only in time, this is required to avoid hacker to
> redirect into older portage to install vulnerabilities that were
> approved at that time.
Replay attacks per #1 are addressed via TIMESTAMP field in MetaManifest.

> 2. Content integrity, especially removal, as far as I understand, the
> mechanism will not enable to detect authorized removal of content.
I think you meant 'unauthorized' rather than 'authorized' here.
It will detect files that are expected to exist but are missing.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136


signature.asc
Description: Digital signature


Re: [gentoo-dev] [News item review] Portage rsync tree verification (v2)

2018-01-25 Thread Alon Bar-Lev
Hi,

On 25 January 2018 at 14:35, Michał Górny  wrote:
>
> Starting with sys-apps/portage-2.3.22, Portage enables cryptographic
> verification of the Gentoo rsync repository distributed over rsync
> by default. This aims to prevent malicious third parties from altering
> the contents of the ebuild repository received by our users.



I did not looked into the detailed implementation, however, please
make sure integrity check handles the same cases we have applied to
emerge-webrsync in the past, including:
1. Fast forward only in time, this is required to avoid hacker to
redirect into older portage to install vulnerabilities that were
approved at that time.
2. Content integrity, especially removal, as far as I understand, the
mechanism will not enable to detect authorized removal of content.

Regards,
Alon



Re: [gentoo-dev] [News item review] Portage rsync tree verification (v2)

2018-01-25 Thread R0b0t1
On Thu, Jan 25, 2018 at 3:45 PM, Michał Górny  wrote:
> W dniu czw, 25.01.2018 o godzinie 21∶37 +, użytkownik Robin H.
> Johnson napisał:
>> On Thu, Jan 25, 2018 at 01:35:17PM +0100, Michał Górny wrote:
>> > Title: Portage rsync tree verification
>> > Author: Michał Górny 
>> > Posted: 2018-01-xx
>> > Revision: 1
>> > News-Item-Format: 2.0
>> > Display-If-Installed: >
>> Drop Display-If-Installed, they need to always see this until they know
>> it was bootstrapped.
>
> Well, the idea was that if someone starts with stage that has >2.3.21,
> then he has bootstrapped via verifying the stage signature.
>
>> > Starting with sys-apps/portage-2.3.22, Portage enables cryptographic
>> > verification of the Gentoo rsync repository distributed over rsync
>> > by default.
>>
>> Seems very wordy, suggested cleanup:
>> > > Starting with sys-apps/portage-2.3.22, Portage will verify the Gentoo
>> > > repository after rsync by default.
>> > This aims to prevent malicious third parties from altering
>> > the contents of the ebuild repository received by our users.
>> >
>> > This does not affect users syncing using git and other methods.
>> > Appropriate verification mechanisms for them will be provided
>> > in the future.
>>
>> Note that emerge-webrsync has verification via FEATURES=webrsync-gpg?
>
> I'm sorry, I have never used that. Does it cover full key maintenance
> or rely on user to do the gpg work?
>

It used to be necessary to set up a GnuPG home for portage and pull
the keys in, but now users can emerge app-crypt/gentoo-keys and set
PORTAGE_GPG_DIR="/var/lib/gentoo/gkeys/keyrings/gentoo/release".

>>
>> Rewrite:
>> > > The new verification is intended for users who syncing via rsync.
>> > > Users who sync by emerge-webrsync should see [linkref].
>> > > Verification mechanisms for other methods of sync will be provided in
>> > > future.
>>
>>
>> > On Gentoo installations created using installation media that included
>> > portage-2.3.22, the keys will already be covered by the installation
>> > media signatures. On existing installations, you need to manually
>> > compare the primary key fingerprint (reported by gemato on every sync)
>> > against the official Gentoo keys [1]. An example gemato output is:
>> >   INFO:root:Valid OpenPGP signature found:
>> >   INFO:root:- primary key: 1234567890ABCDEF1234567890ABCDEF12345678
>> >   INFO:root:- subkey: FEDCBA0987654321FEDCBA0987654321FEDCBA09
>>
>> Either we should use real key here, or specifically note this is a fake
>> key output on purpose.
>
> Well, I've assumed most people would be able to figure out that it would
> be quite a coincidence to see such a key id. I wanted to avoid putting
> the real id so that people would actually check that HTTPS site instead
> of relying on the security of news item delivery.
>
> Will send an updated version tomorrow.
>
> --
> Best regards,
> Michał Górny
>
>



Re: [gentoo-dev] [News item review] Portage rsync tree verification (v2)

2018-01-25 Thread Michał Górny
W dniu czw, 25.01.2018 o godzinie 21∶37 +, użytkownik Robin H.
Johnson napisał:
> On Thu, Jan 25, 2018 at 01:35:17PM +0100, Michał Górny wrote:
> > Title: Portage rsync tree verification
> > Author: Michał Górny 
> > Posted: 2018-01-xx
> > Revision: 1
> > News-Item-Format: 2.0
> > Display-If-Installed:  
> Drop Display-If-Installed, they need to always see this until they know
> it was bootstrapped.

Well, the idea was that if someone starts with stage that has >2.3.21,
then he has bootstrapped via verifying the stage signature.

> > Starting with sys-apps/portage-2.3.22, Portage enables cryptographic
> > verification of the Gentoo rsync repository distributed over rsync
> > by default. 
> 
> Seems very wordy, suggested cleanup:
> > > Starting with sys-apps/portage-2.3.22, Portage will verify the Gentoo
> > > repository after rsync by default.
> > This aims to prevent malicious third parties from altering
> > the contents of the ebuild repository received by our users.
> > 
> > This does not affect users syncing using git and other methods.
> > Appropriate verification mechanisms for them will be provided
> > in the future.
> 
> Note that emerge-webrsync has verification via FEATURES=webrsync-gpg?

I'm sorry, I have never used that. Does it cover full key maintenance
or rely on user to do the gpg work?

> 
> Rewrite:
> > > The new verification is intended for users who syncing via rsync.
> > > Users who sync by emerge-webrsync should see [linkref]. 
> > > Verification mechanisms for other methods of sync will be provided in
> > > future.
> 
> 
> > On Gentoo installations created using installation media that included
> > portage-2.3.22, the keys will already be covered by the installation
> > media signatures. On existing installations, you need to manually
> > compare the primary key fingerprint (reported by gemato on every sync)
> > against the official Gentoo keys [1]. An example gemato output is:
> >   INFO:root:Valid OpenPGP signature found:
> >   INFO:root:- primary key: 1234567890ABCDEF1234567890ABCDEF12345678
> >   INFO:root:- subkey: FEDCBA0987654321FEDCBA0987654321FEDCBA09
> 
> Either we should use real key here, or specifically note this is a fake
> key output on purpose.

Well, I've assumed most people would be able to figure out that it would
be quite a coincidence to see such a key id. I wanted to avoid putting
the real id so that people would actually check that HTTPS site instead
of relying on the security of news item delivery.

Will send an updated version tomorrow.

-- 
Best regards,
Michał Górny




Re: [gentoo-dev] [News item review] Portage rsync tree verification

2018-01-25 Thread M. J. Everitt


On 25/01/18 11:01, Kristian Fiskerstrand wrote:
> On 01/25/2018 11:04 AM, Michał Górny wrote:
>
>> The verification is implemented using app-portage/gemato. Currently, 
> ... "implemented in", as opposed to "using"? its implemented using
> various cryptographic primitives, but gemato is the implementation
> itself of sorts.
>
>

"implemented with gemato" ?




signature.asc
Description: OpenPGP digital signature


Re: [gentoo-dev] [News item review] Portage rsync tree verification (v2)

2018-01-25 Thread Robin H. Johnson
On Thu, Jan 25, 2018 at 01:35:17PM +0100, Michał Górny wrote:
> Title: Portage rsync tree verification
> Author: Michał Górny 
> Posted: 2018-01-xx
> Revision: 1
> News-Item-Format: 2.0
> Display-If-Installed:  Starting with sys-apps/portage-2.3.22, Portage enables cryptographic
> verification of the Gentoo rsync repository distributed over rsync
> by default. 
Seems very wordy, suggested cleanup:
|| Starting with sys-apps/portage-2.3.22, Portage will verify the Gentoo
|| repository after rsync by default.

> This aims to prevent malicious third parties from altering
> the contents of the ebuild repository received by our users.
> 
> This does not affect users syncing using git and other methods.
> Appropriate verification mechanisms for them will be provided
> in the future.
Note that emerge-webrsync has verification via FEATURES=webrsync-gpg?

Rewrite:
|| The new verification is intended for users who syncing via rsync.
|| Users who sync by emerge-webrsync should see [linkref]. 
|| Verification mechanisms for other methods of sync will be provided in
|| future.


> On Gentoo installations created using installation media that included
> portage-2.3.22, the keys will already be covered by the installation
> media signatures. On existing installations, you need to manually
> compare the primary key fingerprint (reported by gemato on every sync)
> against the official Gentoo keys [1]. An example gemato output is:
>   INFO:root:Valid OpenPGP signature found:
>   INFO:root:- primary key: 1234567890ABCDEF1234567890ABCDEF12345678
>   INFO:root:- subkey: FEDCBA0987654321FEDCBA0987654321FEDCBA09
Either we should use real key here, or specifically note this is a fake
key output on purpose.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136



Re: [gentoo-dev] [News item review] Portage rsync tree verification (v2)

2018-01-25 Thread Ulrich Mueller
> On Thu, 25 Jan 2018, Michał Górny wrote:

> Here's the updated version:
> ---

> Starting with sys-apps/portage-2.3.22, Portage enables cryptographic
> verification of the Gentoo rsync repository distributed over rsync
> by default.

Looks like there's one "rsync" too much in that sentence.


pgpCx6DDEeinP.pgp
Description: PGP signature


Re: [gentoo-dev] [News item review] Portage rsync tree verification (v2)

2018-01-25 Thread Aaron W. Swenson
On 2018-01-25 13:35, Michał Górny wrote:
> Display-If-Installed: =2.3.22 this same information?

I know we don’t have expires, yet. How about making it


signature.asc
Description: Digital signature


Re: [gentoo-dev] [News item review] Portage rsync tree verification (v2)

2018-01-25 Thread Michał Górny
Here's the updated version:

---
Title: Portage rsync tree verification
Author: Michał Górny 
Posted: 2018-01-xx
Revision: 1
News-Item-Format: 2.0
Display-If-Installed: https://www.gentoo.org/downloads/signatures/
---

-- 
Best regards,
Michał Górny




Re: [gentoo-dev] [News item review] Portage rsync tree verification

2018-01-25 Thread Michał Górny
W dniu czw, 25.01.2018 o godzinie 12∶01 +0100, użytkownik Kristian
Fiskerstrand napisał:
> On 01/25/2018 11:04 AM, Michał Górny wrote:
> > Hi,
> > 
> 
> Thanks for your work on this!
> 
> > This one would be committed once new sys-apps/portage release is
> > wrapped up and hits ~arch.
> > 
> > --- Title: Portage rsync tree verification Author: Michał Górny
> >  Posted: 2018-01-xx Revision: 1 News-Item-Format:
> > 2.0 Display-If-Installed:  > 
> > Starting with sys-apps/portage-2.3.22, Portage enables strong
> > cryptographic verification of the Gentoo rsync tree by default. This
> > aims to prevent malicious third parties from altering the contents of
> > the ebuild repository received by our users.
> 
> Just for sake of it, would remove "strong" here, as it is a description
> and not PR document. Should we be consistent with referencing, so e.g
> the Gentoo ebuild repository as distributed through rsync, or something?
> Atm we seem to be using different terms all of the place, so should try
> to harmonize a bit.

Done.

> 
> > 
> > The verification is implemented using app-portage/gemato. Currently, 
> 
> ... "implemented in", as opposed to "using"? its implemented using
> various cryptographic primitives, but gemato is the implementation
> itself of sorts.

It was supposed to mean that Portage currently uses gemato to
do the verification. 'via using' maybe?

> 
> > the whole repository is verified after syncing. On systems with slow 
> > hard drives, this could take around 2 minutes. If you wish to
> > disable it, you can disable the 'rsync-verify' flag on
> 
> USE flag?

Done.

> 
> > sys-apps/portage or set 'sync-rsync-verify-metamanifest = no' in your
> > repos.conf.
> > 
> > Please note that the verification currently does not prevent Portage 
> > from using the repository after syncing. If 'emerge --sync' fails, do
> > not install any packages and retry syncing. In case of prolonged or
> > frequent verification failures, please make sure to report a bug 
> > including the failing mirror addresses (found in emerge.log).
> > 
> > The verification uses keys provided by the app-crypt/gentoo-keys 
> > package. The keys are refreshed from the keyserver before every use 
> > in order to check for revocation. The post-sync verification ensures 
> > that the key package is verified itself. However, manua
> > verification is required before the first use.
> 
> Maybe some wording around binary keyring? e.g the verification uses
> information from the binary keyring provided by app-crypt/gentoo-keys?
> In particular the reference to "key package" might be misread (and the
> keyring consists of multiple public keyblocks, that includes much more
> information than the cryptographic keys per se)

Done.

> 
> > 
> > On new Gentoo installations including portage-2.3.22, the
> 
> stage3s?

Nah. I need to think how to word it properly. It's about installations
that are created from new stages.

> 
> > verification of the keys will be covered by verifying the
> > installation media and repository snapshot signatures. On existing
> > installations, you need to manually compare the primary key
> > fingerprint (reported by gemato on every sync) against the official
> > Gentoo keys [1]. An example gemato output is:
> > 
> > INFO:root:Valid OpenPGP signature found: INFO:root:- primary key:
> > 1234567890ABCDEF1234567890ABCDEF12345678 INFO:root:- subkey:
> > FEDCBA0987654321FEDCBA0987654321FEDCBA09
> > 
> > The primary key printed must match 'Gentoo Portage Snapshot Signing
> > Key' on the site. Please make sure to also check the certificate
> > used for the secure connection to the site!
> > 
> > [1]:https://www.gentoo.org/downloads/signatures/ ---
> > 
> 
> 

-- 
Best regards,
Michał Górny




[gentoo-dev] Last rites: media-video/qx11grab

2018-01-25 Thread Michael Palimaka
# Michael Palimaka  (25 Jan 2018)
# Requires dead Qt 4. Dead upstream.
# Masked for removal in 30 days. Bug #644532.
media-video/qx11grab



Re: [gentoo-dev] [News item review] Portage rsync tree verification

2018-01-25 Thread Kristian Fiskerstrand
On 01/25/2018 11:04 AM, Michał Górny wrote:
> Hi,
> 

Thanks for your work on this!

> This one would be committed once new sys-apps/portage release is
> wrapped up and hits ~arch.
> 
> --- Title: Portage rsync tree verification Author: Michał Górny
>  Posted: 2018-01-xx Revision: 1 News-Item-Format:
> 2.0 Display-If-Installed:  
> Starting with sys-apps/portage-2.3.22, Portage enables strong
> cryptographic verification of the Gentoo rsync tree by default. This
> aims to prevent malicious third parties from altering the contents of
> the ebuild repository received by our users.

Just for sake of it, would remove "strong" here, as it is a description
and not PR document. Should we be consistent with referencing, so e.g
the Gentoo ebuild repository as distributed through rsync, or something?
Atm we seem to be using different terms all of the place, so should try
to harmonize a bit.

> 
> The verification is implemented using app-portage/gemato. Currently, 

... "implemented in", as opposed to "using"? its implemented using
various cryptographic primitives, but gemato is the implementation
itself of sorts.

> the whole repository is verified after syncing. On systems with slow 
> hard drives, this could take around 2 minutes. If you wish to
> disable it, you can disable the 'rsync-verify' flag on

USE flag?

> sys-apps/portage or set 'sync-rsync-verify-metamanifest = no' in your
> repos.conf.
> 
> Please note that the verification currently does not prevent Portage 
> from using the repository after syncing. If 'emerge --sync' fails, do
> not install any packages and retry syncing. In case of prolonged or
> frequent verification failures, please make sure to report a bug 
> including the failing mirror addresses (found in emerge.log).
> 
> The verification uses keys provided by the app-crypt/gentoo-keys 
> package. The keys are refreshed from the keyserver before every use 
> in order to check for revocation. The post-sync verification ensures 
> that the key package is verified itself. However, manua
> verification is required before the first use.

Maybe some wording around binary keyring? e.g the verification uses
information from the binary keyring provided by app-crypt/gentoo-keys?
In particular the reference to "key package" might be misread (and the
keyring consists of multiple public keyblocks, that includes much more
information than the cryptographic keys per se)

> 
> On new Gentoo installations including portage-2.3.22, the

stage3s?

> verification of the keys will be covered by verifying the
> installation media and repository snapshot signatures. On existing
> installations, you need to manually compare the primary key
> fingerprint (reported by gemato on every sync) against the official
> Gentoo keys [1]. An example gemato output is:
> 
> INFO:root:Valid OpenPGP signature found: INFO:root:- primary key:
> 1234567890ABCDEF1234567890ABCDEF12345678 INFO:root:- subkey:
> FEDCBA0987654321FEDCBA0987654321FEDCBA09
> 
> The primary key printed must match 'Gentoo Portage Snapshot Signing
> Key' on the site. Please make sure to also check the certificate
> used for the secure connection to the site!
> 
> [1]:https://www.gentoo.org/downloads/signatures/ ---
> 


-- 
Kristian Fiskerstrand
OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3



signature.asc
Description: OpenPGP digital signature


[gentoo-dev] Re: [News item review] Portage rsync tree verification

2018-01-25 Thread Duncan
Michał Górny posted on Thu, 25 Jan 2018 11:04:27 +0100 as excerpted:

> Hi,
> 
> This one would be committed once new sys-apps/portage release is wrapped
> up and hits ~arch.
> 
> ---
> Title: Portage rsync tree verification

Might want to put a paragraph in there saying git sync users can ignore, 
or how they can get gpg signature verification as well if its possible. 
(Sufficient to just link it if it's more involved than a single 
paragraph, since this is primarily for rsync users.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman




[gentoo-dev] [News item review] Portage rsync tree verification

2018-01-25 Thread Michał Górny
Hi,

This one would be committed once new sys-apps/portage release is wrapped
up and hits ~arch.

---
Title: Portage rsync tree verification
Author: Michał Górny 
Posted: 2018-01-xx
Revision: 1
News-Item-Format: 2.0
Display-If-Installed: https://www.gentoo.org/downloads/signatures/
---

-- 
Best regards,
Michał Górny




[gentoo-portage-dev] [PATCH] install-qa-check.d: Scan build log for CMake unused var warnings

2018-01-25 Thread Michał Górny
Scan build log and report verbosely CMake warnings about unused
variables. This is a quite common problem, yet currently it is hard
to notice it since the warning is mixed with src_configure() output.
Repeat it verbosely after the install.

This check outputs warnings such as:

 * One or more CMake variables were not used by the project:
 *   CMAKE_USER_MAKE_RULES_OVERRIDE
---
 bin/install-qa-check.d/90cmake-warnings | 28 
 1 file changed, 28 insertions(+)
 create mode 100644 bin/install-qa-check.d/90cmake-warnings

diff --git a/bin/install-qa-check.d/90cmake-warnings 
b/bin/install-qa-check.d/90cmake-warnings
new file mode 100644
index 0..36a09851b
--- /dev/null
+++ b/bin/install-qa-check.d/90cmake-warnings
@@ -0,0 +1,28 @@
+# Check for CMake invalid option warnings
+
+cmake_warn_check() {
+   if [[ -n ${PORTAGE_LOG_FILE} && -r ${PORTAGE_LOG_FILE} ]] ; then
+   local cat=cat
+   [[ ${PORTAGE_LOG_FILE} == *.gz ]] && cat=zcat
+
+   local vars=()
+   while read -r l; do
+   vars+=( "${l}" )
+   done < <( "${cat}" "${PORTAGE_LOG_FILE}" \
+   | sed -n -e '/Manually-specified variables were not 
used by the project/,/^--/{/^/p}' \
+   | sort -u)
+
+   if [[ ${vars} ]]; then
+   eqawarn "One or more CMake variables were not used by 
the project:"
+   local v
+   for v in "${vars[@]}"; do
+   eqawarn "  ${v}"
+   done
+   fi
+   fi
+}
+
+cmake_warn_check
+: # guarantee successful exit
+
+# vim:ft=sh
-- 
2.16.1




Re: [gentoo-portage-dev] Re: [PATCH v2 3/3] _emerge.Ebuild*: delay creating DISTDIR shadow until src_unpack

2018-01-25 Thread Michał Górny
W dniu czw, 25.01.2018 o godzinie 10∶07 +0100, użytkownik Michael
Haubenwallner napisał:
> Hi,
> 
> ${Subject} ringing a bell here:
> 
> dev-db/oracle-instantclient is fetch restricted. As a binary package with
> multiple USE options there's a bunch of files to download - even for
> multiple archs when multilib is active.
> 
> So in pkg_nofetch() I'm telling the user whether a file to download is
> "already here" or "still absent", by testing if $A exists in $DISTDIR.
> 
> With ${Subject}, I'm wondering if DISTDIR is created for pkg_nofetch too.
> 

You're doing the wrong thing then. DISTDIR is not allowed
in pkg_nofetch(). Furthermore, you're touching files whose hashes have
not been verified which is twice wrong.

-- 
Best regards,
Michał Górny




[gentoo-portage-dev] Re: [PATCH v2 3/3] _emerge.Ebuild*: delay creating DISTDIR shadow until src_unpack

2018-01-25 Thread Michael Haubenwallner
Hi,

${Subject} ringing a bell here:

dev-db/oracle-instantclient is fetch restricted. As a binary package with
multiple USE options there's a bunch of files to download - even for
multiple archs when multilib is active.

So in pkg_nofetch() I'm telling the user whether a file to download is
"already here" or "still absent", by testing if $A exists in $DISTDIR.

With ${Subject}, I'm wondering if DISTDIR is created for pkg_nofetch too.

/haubi/

On 01/25/2018 09:50 AM, Michał Górny wrote:
> ---
>  pym/_emerge/EbuildExecuter.py | 4 
>  pym/_emerge/EbuildPhase.py| 6 --
>  2 files changed, 4 insertions(+), 6 deletions(-)
> 
> diff --git a/pym/_emerge/EbuildExecuter.py b/pym/_emerge/EbuildExecuter.py
> index ab79ce901..d387b42be 100644
> --- a/pym/_emerge/EbuildExecuter.py
> +++ b/pym/_emerge/EbuildExecuter.py
> @@ -8,7 +8,6 @@ import portage
>  from portage import os
>  from portage.eapi import eapi_has_src_prepare_and_src_configure, \
>   eapi_exports_replace_vars
> -from portage.package.ebuild.prepare_build_dirs import _prepare_fake_distdir
>  
>  class EbuildExecuter(CompositeTask):
>  
> @@ -25,9 +24,6 @@ class EbuildExecuter(CompositeTask):
>   cleanup = 0
>   portage.prepare_build_dirs(pkg.root, settings, cleanup)
>  
> - alist = settings.configdict["pkg"].get("A", "").split()
> - _prepare_fake_distdir(settings, alist)
> -
>   if eapi_exports_replace_vars(settings['EAPI']):
>   vardb = pkg.root_config.trees['vartree'].dbapi
>   settings["REPLACING_VERSIONS"] = " ".join(
> diff --git a/pym/_emerge/EbuildPhase.py b/pym/_emerge/EbuildPhase.py
> index aa3a66831..d3fada622 100644
> --- a/pym/_emerge/EbuildPhase.py
> +++ b/pym/_emerge/EbuildPhase.py
> @@ -1,4 +1,4 @@
> -# Copyright 1999-2013 Gentoo Foundation
> +# Copyright 1999-2018 Gentoo Foundation
>  # Distributed under the terms of the GNU General Public License v2
>  
>  import gzip
> @@ -12,7 +12,7 @@ from _emerge.MiscFunctionsProcess import 
> MiscFunctionsProcess
>  from _emerge.EbuildProcess import EbuildProcess
>  from _emerge.CompositeTask import CompositeTask
>  from portage.package.ebuild.prepare_build_dirs import (_prepare_workdir,
> - _prepare_fake_filesdir)
> + _prepare_fake_distdir, _prepare_fake_filesdir)
>  from portage.util import writemsg
>  
>  try:
> @@ -171,6 +171,8 @@ class EbuildPhase(CompositeTask):
>   def _start_ebuild(self):
>  
>   if self.phase == "unpack":
> + alist = self.settings.configdict["pkg"].get("A", 
> "").split()
> + _prepare_fake_distdir(self.settings, alist)
>   _prepare_fake_filesdir(self.settings)
>  
>   fd_pipes = self.fd_pipes
> 




[gentoo-portage-dev] [PATCH v2 3/3] _emerge.Ebuild*: delay creating DISTDIR shadow until src_unpack

2018-01-25 Thread Michał Górny
---
 pym/_emerge/EbuildExecuter.py | 4 
 pym/_emerge/EbuildPhase.py| 6 --
 2 files changed, 4 insertions(+), 6 deletions(-)

diff --git a/pym/_emerge/EbuildExecuter.py b/pym/_emerge/EbuildExecuter.py
index ab79ce901..d387b42be 100644
--- a/pym/_emerge/EbuildExecuter.py
+++ b/pym/_emerge/EbuildExecuter.py
@@ -8,7 +8,6 @@ import portage
 from portage import os
 from portage.eapi import eapi_has_src_prepare_and_src_configure, \
eapi_exports_replace_vars
-from portage.package.ebuild.prepare_build_dirs import _prepare_fake_distdir
 
 class EbuildExecuter(CompositeTask):
 
@@ -25,9 +24,6 @@ class EbuildExecuter(CompositeTask):
cleanup = 0
portage.prepare_build_dirs(pkg.root, settings, cleanup)
 
-   alist = settings.configdict["pkg"].get("A", "").split()
-   _prepare_fake_distdir(settings, alist)
-
if eapi_exports_replace_vars(settings['EAPI']):
vardb = pkg.root_config.trees['vartree'].dbapi
settings["REPLACING_VERSIONS"] = " ".join(
diff --git a/pym/_emerge/EbuildPhase.py b/pym/_emerge/EbuildPhase.py
index aa3a66831..d3fada622 100644
--- a/pym/_emerge/EbuildPhase.py
+++ b/pym/_emerge/EbuildPhase.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2013 Gentoo Foundation
+# Copyright 1999-2018 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 import gzip
@@ -12,7 +12,7 @@ from _emerge.MiscFunctionsProcess import MiscFunctionsProcess
 from _emerge.EbuildProcess import EbuildProcess
 from _emerge.CompositeTask import CompositeTask
 from portage.package.ebuild.prepare_build_dirs import (_prepare_workdir,
-   _prepare_fake_filesdir)
+   _prepare_fake_distdir, _prepare_fake_filesdir)
 from portage.util import writemsg
 
 try:
@@ -171,6 +171,8 @@ class EbuildPhase(CompositeTask):
def _start_ebuild(self):
 
if self.phase == "unpack":
+   alist = self.settings.configdict["pkg"].get("A", 
"").split()
+   _prepare_fake_distdir(self.settings, alist)
_prepare_fake_filesdir(self.settings)
 
fd_pipes = self.fd_pipes
-- 
2.16.1




[gentoo-portage-dev] [PATCH v2 1/3] portage.package.ebuild: Move _prepare_fake_distdir to .prepare_build_dirs

2018-01-25 Thread Michał Górny
---
 pym/_emerge/EbuildExecuter.py|  4 +--
 pym/portage/package/ebuild/doebuild.py   | 34 ++--
 pym/portage/package/ebuild/prepare_build_dirs.py | 33 ++-
 3 files changed, 36 insertions(+), 35 deletions(-)

diff --git a/pym/_emerge/EbuildExecuter.py b/pym/_emerge/EbuildExecuter.py
index 5587d4eb0..ab79ce901 100644
--- a/pym/_emerge/EbuildExecuter.py
+++ b/pym/_emerge/EbuildExecuter.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2011 Gentoo Foundation
+# Copyright 1999-2018 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 from _emerge.EbuildPhase import EbuildPhase
@@ -8,7 +8,7 @@ import portage
 from portage import os
 from portage.eapi import eapi_has_src_prepare_and_src_configure, \
eapi_exports_replace_vars
-from portage.package.ebuild.doebuild import _prepare_fake_distdir
+from portage.package.ebuild.prepare_build_dirs import _prepare_fake_distdir
 
 class EbuildExecuter(CompositeTask):
 
diff --git a/pym/portage/package/ebuild/doebuild.py 
b/pym/portage/package/ebuild/doebuild.py
index f75f11a1a..c8df9b744 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -1,4 +1,4 @@
-# Copyright 2010-2015 Gentoo Foundation
+# Copyright 2010-2018 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 from __future__ import unicode_literals
@@ -31,6 +31,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.package.ebuild.digestcheck:digestcheck',
'portage.package.ebuild.digestgen:digestgen',
'portage.package.ebuild.fetch:fetch',
+   'portage.package.ebuild.prepare_build_dirs:_prepare_fake_distdir',
'portage.package.ebuild._ipc.QueryCommand:QueryCommand',
'portage.dep._slot_operator:evaluate_slot_operator_equal_deps',
'portage.package.ebuild._spawn_nofetch:spawn_nofetch',
@@ -1351,37 +1352,6 @@ def _prepare_env_file(settings):
env_extractor.wait()
return env_extractor.returncode
 
-def _prepare_fake_distdir(settings, alist):
-   orig_distdir = settings["DISTDIR"]
-   settings["PORTAGE_ACTUAL_DISTDIR"] = orig_distdir
-   edpath = settings["DISTDIR"] = \
-   os.path.join(settings["PORTAGE_BUILDDIR"], "distdir")
-   portage.util.ensure_dirs(edpath, gid=portage_gid, mode=0o755)
-
-   # Remove any unexpected files or directories.
-   for x in os.listdir(edpath):
-   symlink_path = os.path.join(edpath, x)
-   st = os.lstat(symlink_path)
-   if x in alist and stat.S_ISLNK(st.st_mode):
-   continue
-   if stat.S_ISDIR(st.st_mode):
-   shutil.rmtree(symlink_path)
-   else:
-   os.unlink(symlink_path)
-
-   # Check for existing symlinks and recreate if necessary.
-   for x in alist:
-   symlink_path = os.path.join(edpath, x)
-   target = os.path.join(orig_distdir, x)
-   try:
-   link_target = os.readlink(symlink_path)
-   except OSError:
-   os.symlink(target, symlink_path)
-   else:
-   if link_target != target:
-   os.unlink(symlink_path)
-   os.symlink(target, symlink_path)
-
 def _spawn_actionmap(settings):
features = settings.features
restrict = settings["PORTAGE_RESTRICT"].split()
diff --git a/pym/portage/package/ebuild/prepare_build_dirs.py 
b/pym/portage/package/ebuild/prepare_build_dirs.py
index e3ae318bd..16afc3f98 100644
--- a/pym/portage/package/ebuild/prepare_build_dirs.py
+++ b/pym/portage/package/ebuild/prepare_build_dirs.py
@@ -1,4 +1,4 @@
-# Copyright 2010-2013 Gentoo Foundation
+# Copyright 2010-2018 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 from __future__ import unicode_literals
@@ -409,3 +409,34 @@ def _prepare_fake_filesdir(settings):
if link_target != real_filesdir:
os.unlink(symlink_path)
os.symlink(real_filesdir, symlink_path)
+
+def _prepare_fake_distdir(settings, alist):
+   orig_distdir = settings["DISTDIR"]
+   settings["PORTAGE_ACTUAL_DISTDIR"] = orig_distdir
+   edpath = settings["DISTDIR"] = \
+   os.path.join(settings["PORTAGE_BUILDDIR"], "distdir")
+   portage.util.ensure_dirs(edpath, gid=portage_gid, mode=0o755)
+
+   # Remove any unexpected files or directories.
+   for x in os.listdir(edpath):
+   symlink_path = os.path.join(edpath, x)
+   st = os.lstat(symlink_path)
+   if x in alist and stat.S_ISLNK(st.st_mode):
+   continue
+   if stat.S_ISDIR(st.st_mode):
+   shutil.rmtree(symlink_path)
+   else:
+   

[gentoo-portage-dev] [PATCH v2 2/3] portage.package.ebuild.config: Override DISTDIR unconditionally

2018-01-25 Thread Michał Górny
Ensure that DISTDIR is always defined to the path to the shadow
directory. This ensures that PMS rules for consistent value are
followed, and that no global scope calls should be able to access
the distfile directory. This also ensures that global-scope assignments
(e.g. in PATCHES) do not work around the shadow directory.

Bug: https://bugs.gentoo.org/612972
---
 pym/portage/package/ebuild/config.py | 11 ++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/pym/portage/package/ebuild/config.py 
b/pym/portage/package/ebuild/config.py
index d0225a311..5624e86d3 100644
--- a/pym/portage/package/ebuild/config.py
+++ b/pym/portage/package/ebuild/config.py
@@ -1,4 +1,4 @@
-# Copyright 2010-2016 Gentoo Foundation
+# Copyright 2010-2018 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 from __future__ import unicode_literals
@@ -2848,6 +2848,15 @@ class config(object):
else:
raise AssertionError("C locale did not 
pass the test!")
 
+   try:
+   builddir = mydict["PORTAGE_BUILDDIR"]
+   distdir = mydict["DISTDIR"]
+   except KeyError:
+   pass
+   else:
+   mydict["PORTAGE_ACTUAL_DISTDIR"] = distdir
+   mydict["DISTDIR"] = os.path.join(builddir, "distdir")
+
return mydict
 
def thirdpartymirrors(self):
-- 
2.16.1




[gentoo-portage-dev] [PATCHES v2] DISTDIR shadow updates

2018-01-25 Thread Michał Górny
Hi,

Here's the refurbished patches for DISTDIR shadow updates, with Zac's
feedback implemented. Please review.

--
Best regards,
Michał Górny




Re: [gentoo-portage-dev] [PATCH v4] rsync: Introduce support for running full-tree gemato verification

2018-01-25 Thread Michał Górny
W dniu śro, 24.01.2018 o godzinie 13∶56 -0800, użytkownik Zac Medico
napisał:
> On 01/24/2018 01:36 PM, Michał Górny wrote:
> > Add two new configuration options to rsync repositories:
> > sync-rsync-verify-metamanifest and sync-rsync-openpgp-key-path.
> > The first controls whether gemato verification is run for
> > the repository (defaults to true for ::gentoo, false otherwise),
> > the second makes it possible to override the key path for custom
> > repositories.
> > ---
> >  cnf/repos.conf |  2 ++
> >  man/portage.5  |  9 +
> >  pym/portage/sync/modules/rsync/__init__.py |  4 +++-
> >  pym/portage/sync/modules/rsync/rsync.py| 20 +++-
> >  4 files changed, 33 insertions(+), 2 deletions(-)
> > 
> > v4: also key option to repos.conf
> 
> Looks good.
> 
> I see that the man page currently does not escape hyphens in
> sync-rsync-*, but we can fix those up separately.

Fixed that and pushed. Thanks! Note however that the hyphens
in adjoining entries that I haven't modified are still unescaped.

-- 
Best regards,
Michał Górny