Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure

[Jason Zaman]

On Sat, Jan 27, 2018 at 12:24:39AM +0100, Michał Górny wrote:
> Migrating mirrors to the hashed structure
> -

> The hard link solution allows us to save space on the master mirror.
> Additionally, if ``-H`` option is used by the mirrors it avoids
> transferring existing files again.  However, this option is known
> to be expensive and could cause significant server load.  Without it,
> all mirrors need to transfer a second copy of all the existing files.
> 
> The symbolic link solution could be more reliable if we could rely
> on mirrors using the ``--links`` rsync option.  Without that, symbolic
> links are not transferred at all.

These rsync options might help for mirrors too:
 --compare-dest=DIR  also compare destination files relative to DIR
 --copy-dest=DIR ... and include copies of unchanged files
 --link-dest=DIR hardlink to files in DIR when unchanged

> Using hashed structure for local distfiles
> --
> The hashed structure defined above could also be used for local distfile
> storage as used by the package manager.  For this to work, the package
> manager authors need to ensure that:
> 
> a. The ``${DISTDIR}`` variable in the ebuild scope points to a temporary
>directory where distfiles specific to the package are linked
>in a flat structure.
> 
> b. All tools are updated to support the nested structure.
> 
> c. The package manager provides a tool for users to easily manipulate
>distfiles, in particular to add distfiles for fetch-restricted
>packages into an appropriate subdirectory.
> 
> For extended compatibility, the package manager may support finding
> distfiles in flat and nested structure simultaneously.

trying nested first then falling back to flat would make it easy for
users if they have to download distfiles for fetch-restricted packages
because then the instructions stay as "move it to
/usr/portage/distfiles".
or alternatively the tool could have a mode which will go through all
files in the base dir and move it to where it should be in the nested
tree. then you save everything to the same dir and run edist --fix

> Rationale
> =
> Algorithm for splitting distfiles
> -
> In the original debate that occurred in bug #534528 [#BUG534528]_,
> three possible solutions for splitting distfiles were listed:
> 
> a. using initial portion of filename,
> 
> b. using initial portion of file hash,
> 
> c. using initial portion of filename hash.
> 
> The significant advantage of the filename option was simplicity.  With
> that solution, the users could easily determine the correct subdirectory
> themselves.  However, it's significant disadvantage was very uneven
> shuffling of data.  In particular, the TeΧ Live packages alone count
> almost 23500 distfiles and all use a common prefix, making it impossible
> to split them further.

the filename is the original upstream or the renamed one? eg
SRC_URI="http://foo/foo.tar -> bar.tar" it will be bar.tar?

I think im in favour of using the initial part of the filename anyway.
sure its not balanced but its still a hell of a lot more balanced than
today and its really easy.

Another thing im wondering is if we can just use the same dir layout as
the packages themselves. that would fix texlive since it has a whole lot
of separate packages. eg /usr/portage/distfiles/app-cat/pkg/pkg-1.0.tgz

there is a problem if many packages use the same distfiles (quite
extensive for SELinux, every single of the sec-policy/selinux-* packages
has identical distfiles) so im not sure how to deal with it.

this would also make it easy in future to make the sandbox restrict
access to files outside of that package if we wanted to do that.

> The alternate option of using file hash has the advantage of having
> a more balanced split.  Furthermore, since hashes are stored
> in Manifests using them is zero-cost.  However, this solution has two
> significant disadvantages:
> 
> 1. The hash values are unknown for newly-downloaded distfiles, so
>``repoman`` (or an equivalent tool) would have to use a temporary
>directory before locating the file in appropriate subdirectory.
> 
> 2. User-provided distfiles (e.g. for fetch-restricted packages) with
>hash mismatches would be placed in the wrong subdirectory,
>potentially causing confusing errors.

Not just this, but on principle, I also think you should be able to read
an ebuild and compute the url to download the file from the mirrors
without any extra knowledge (especially downloading the distfile).

> Using filename hashes has proven to provide a similar balance
> to using file hashes.  Furthermore, since filenames are known up front
> this solution does not suffer from the both listed problems.  While
> hashes need to be computed manually, hashing short string should not
> cause any performance problems.
> 
> .. figure:: 

Re: [gentoo-dev] [News item review] Portage rsync tree verification (v2)

[R0b0t1]

On Sat, Jan 27, 2018 at 8:27 AM, Michał Górny  wrote:
> W dniu czw, 25.01.2018 o godzinie 15∶55 -0600, użytkownik R0b0t1
> napisał:
>> On Thu, Jan 25, 2018 at 3:45 PM, Michał Górny  wrote:
>> > W dniu czw, 25.01.2018 o godzinie 21∶37 +, użytkownik Robin H.
>> > Johnson napisał:
>> > > On Thu, Jan 25, 2018 at 01:35:17PM +0100, Michał Górny wrote:
>> > > > Title: Portage rsync tree verification
>> > > > Author: Michał Górny 
>> > > > Posted: 2018-01-xx
>> > > > Revision: 1
>> > > > News-Item-Format: 2.0
>> > > > Display-If-Installed: > > >
>> > > Drop Display-If-Installed, they need to always see this until they know
>> > > it was bootstrapped.
>> >
>> > Well, the idea was that if someone starts with stage that has >2.3.21,
>> > then he has bootstrapped via verifying the stage signature.
>> >
>> > > > Starting with sys-apps/portage-2.3.22, Portage enables cryptographic
>> > > > verification of the Gentoo rsync repository distributed over rsync
>> > > > by default.
>> > >
>> > > Seems very wordy, suggested cleanup:
>> > > > > Starting with sys-apps/portage-2.3.22, Portage will verify the Gentoo
>> > > > > repository after rsync by default.
>> > > >
>> > > > This aims to prevent malicious third parties from altering
>> > > > the contents of the ebuild repository received by our users.
>> > > >
>> > > > This does not affect users syncing using git and other methods.
>> > > > Appropriate verification mechanisms for them will be provided
>> > > > in the future.
>> > >
>> > > Note that emerge-webrsync has verification via FEATURES=webrsync-gpg?
>> >
>> > I'm sorry, I have never used that. Does it cover full key maintenance
>> > or rely on user to do the gpg work?
>> >
>>
>> It used to be necessary to set up a GnuPG home for portage and pull
>> the keys in, but now users can emerge app-crypt/gentoo-keys and set
>> PORTAGE_GPG_DIR="/var/lib/gentoo/gkeys/keyrings/gentoo/release".
>>
>
> In that case I'd rather not announce it until it is integrated properly.
>

What is "properly?" It's referenced in the handbook.

Cheers,
 R0b0t1

[gentoo-dev] Last rites: media-video/qgifer

[Andreas Sturmlechner]

# Andreas Sturmlechner  (28 Jan 2018)
# Dead upstream, depends on deprecated Qt4.
# Masked for removal in 30 days. Bug #644558
media-video/qgifer

Re: [gentoo-dev] time to retire

[Craig]

Hi,

Danke! Man sieht sich vielleicht beim nächsten Congress...oder meld
dich, wenn du mal in Düsseldorf bist. :)

Gruß,

craig


On Sat, 27 Jan 2018 17:21:02 +0100
Sebastian Pipping  wrote:

> Stefan,
> 
> 
> thanks for your work on Gentoo!
> 
> All the best
> 
> 
> 
> Sebastian
> 
> 

Re: [gentoo-dev] Merge 7 Fedora wallpapers packages to single one with slots?

[Alec Warner]

On Sat, Jan 27, 2018 at 11:32 AM, Michael Orlitzky  wrote:

> On 01/27/2018 11:09 AM, Sebastian Pipping wrote:
> > Hi!
> >
> >
> > I noticed that we have 7 packages on Fedora wallpapers with names that
> > only explain themselves to Fedora insiders:
>

So traditionally we follow upstream package naming. If we aim to deviate,
I'd prefer we have strong reasons for it.


> >
> > ...
> >
> > I was thinking that we could merge these packages into a new package
> > "x11-themes/fedora-backgrounds" or so with slots 11 to 16 so that people
> > can still install them in parallel, get slot updates automatically,
> > adding more recent ones does not add more packages, and the package name
> > explains itself.
>

Why not just make x11-themes/fedora-backgrounds, a metapackage that
includes all of the packages?


>
> If you do merge them, then it might be better to use flags for the
> different sub-packages rather than slots. There's no place to describe
> what a slot is for, but having a local USE=solar with a corresponding
> description in metadata.xml is (relatively) discoverable.
>

Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure

[Michael Orlitzky]

On 01/27/2018 02:47 PM, Michał Górny wrote:
>>  *
>>  * Please download fileN from
>>  * wherever fileN can be found
>>  * and move it to $DISTDIR/subdirN
> 
> Do you really believe this to be more friendly than a helper that places
> all the files in correct directories?

Well, it's no worse than what we have now, because it is what we have
now. And if having an "edistadd" helper is easier, then it could check
the computed hash against the manifest before moving it under $DISTDIR.
If the computed hash and manifest hash don't match, a decent error
message can be displayed.

The person who asked for it and the person who are implementing this
prefer hash(filename), so by all means, do that. The two stated reasons
for rejecting hash(filedata) just stood out to me as not very
compelling. Maybe Robin's comment from the bug is a better reason?

> Checksum from the manifest is problematic because not every file in
> distfiles is in a Manifest (there are whitelisted files).

(https://bugs.gentoo.org/534528#c29)

Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure

[Michael Orlitzky]

On 01/27/2018 02:01 PM, Gordon Pettey wrote:
> On Sat, Jan 27, 2018 at 10:48 AM, Michael Orlitzky  wrote:
>> On 01/27/2018 11:42 AM, Gordon Pettey wrote:
>>> Why not use a hash of the file name instead of its contents?...
>>
>> That's the proposal =P
> 
> I'm not following, then. What's all this about a temporary directory
> because of not knowing the hash in advance? The ebuild must specify
> the file name, or src_unpack wouldn't work. There is never a point
> where the file name, and therefore its hash, is unknown.
> 

There were three proposed ways to split up the files: by filename,
hash(filename), and hash(filedata). The winner was hash(filename). The
GLEP lists only two reasons for rejecting hash(filedata), and I was
curious about them.

Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure

[Michał Górny]

W dniu sob, 27.01.2018 o godzinie 13∶24 -0500, użytkownik Michael
Orlitzky napisał:
> On 01/27/2018 01:14 PM, Michał Górny wrote:
> > > 
> > > If we have a tool like edistadd, then I see the problem. But if we were
> > > going to use file-data based hashes, then there would be no need for a
> > > tool in most cases. As a developer, "repoman manifest" would handle it.
> > > As a user, I'm going to see a message like,
> > > 
> > >  Fetch instructions for games-fps/doom3-lms-4:
> > >   * Please download LastManStandingCoop4Multiplatform.zip from:
> > >   * http://www.moddb.com/mods/last-man-standing-coop/downloads
> > >   * and move it to /var/cache/portage/distfiles
> > > 
> > > except instead of $DISTDIR, it would suggest whatever directory is
> > > computed from the hash in the manifest.
> > > 
> > 
> > How would that work if you had 5 different files, every one evaluating
> > to a different directory?
> > 
> 
> for i in range(1,N):
> do-what-you-did-for-the-first-one(i)
> 
> For example,
> 
> Fetch instructions for app-cat/pkg:
>  *
>  * Please download file1 from:
>  * wherever file1 can be found
>  * and move it to $DISTDIR/subdir1
>  *
>  * Please download file2 from
>  * wherever file2 can be found
>  * and move it to $DISTDIR/subdir2
>  *
>  * ...
>  *
>  * Please download fileN from
>  * wherever fileN can be found
>  * and move it to $DISTDIR/subdirN
> 

Do you really believe this to be more friendly than a helper that places
all the files in correct directories?

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure

[Gordon Pettey]

On Sat, Jan 27, 2018 at 10:48 AM, Michael Orlitzky  wrote:
> On 01/27/2018 11:42 AM, Gordon Pettey wrote:
>> Why not use a hash of the file name instead of its contents? That
>> seems like it would be much simpler, and that's not going to reduce
>> the output space for balance...
>
> That's the proposal =P

I'm not following, then. What's all this about a temporary directory
because of not knowing the hash in advance? The ebuild must specify
the file name, or src_unpack wouldn't work. There is never a point
where the file name, and therefore its hash, is unknown.

Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure

[Michael Orlitzky]

On 01/27/2018 01:14 PM, Michał Górny wrote:
>>
>> If we have a tool like edistadd, then I see the problem. But if we were
>> going to use file-data based hashes, then there would be no need for a
>> tool in most cases. As a developer, "repoman manifest" would handle it.
>> As a user, I'm going to see a message like,
>>
>>  Fetch instructions for games-fps/doom3-lms-4:
>>   * Please download LastManStandingCoop4Multiplatform.zip from:
>>   * http://www.moddb.com/mods/last-man-standing-coop/downloads
>>   * and move it to /var/cache/portage/distfiles
>>
>> except instead of $DISTDIR, it would suggest whatever directory is
>> computed from the hash in the manifest.
>>
> 
> How would that work if you had 5 different files, every one evaluating
> to a different directory?
> 

for i in range(1,N):
do-what-you-did-for-the-first-one(i)

For example,

Fetch instructions for app-cat/pkg:
 *
 * Please download file1 from:
 * wherever file1 can be found
 * and move it to $DISTDIR/subdir1
 *
 * Please download file2 from
 * wherever file2 can be found
 * and move it to $DISTDIR/subdir2
 *
 * ...
 *
 * Please download fileN from
 * wherever fileN can be found
 * and move it to $DISTDIR/subdirN

Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure

[Michał Górny]

W dniu sob, 27.01.2018 o godzinie 11∶47 -0500, użytkownik Michael
Orlitzky napisał:
> On 01/27/2018 03:30 AM, Michał Górny wrote:
> > > 
> > > What are we worried about in using a temporary directory? Copying across
> > > filesystem boundaries? Except in rare cases, $DISTDIR itself will be
> > > usable a temporary location (on the same filesystem), won't it?
> > 
> > Why add the extra complexity when there's no need for one? Note that
> > there's also the problem of resuming transfers, so in the end we're
> > talking about permanent temporary directory where we keep unfinished
> > transfers.
> 
> Can't argue with that, but I don't see it as a huge "con."
> 
> 
> > > For the second point, portage is going to tell me where to put the file,
> > > isn't it? Then no matter what garbage I download, won't portage look for
> > > it in the right place, because where-to-put-it is determined using the
> > > same manifest hash that determines where-to-find-it?
> > 
> > No, it won't. Why would it? You're going to call something like:
> > 
> >   edistadd foo.tar.gz bar.tar.gz
> > 
> > ...and it will place the files in the right subdirectories.
> 
> If we have a tool like edistadd, then I see the problem. But if we were
> going to use file-data based hashes, then there would be no need for a
> tool in most cases. As a developer, "repoman manifest" would handle it.
> As a user, I'm going to see a message like,
> 
>  Fetch instructions for games-fps/doom3-lms-4:
>   * Please download LastManStandingCoop4Multiplatform.zip from:
>   * http://www.moddb.com/mods/last-man-standing-coop/downloads
>   * and move it to /var/cache/portage/distfiles
> 
> except instead of $DISTDIR, it would suggest whatever directory is
> computed from the hash in the manifest.
> 

How would that work if you had 5 different files, every one evaluating
to a different directory?


-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Merge 7 Fedora wallpapers packages to single one with slots?

[Sebastian Pipping]

On 27.01.2018 19:06, Sebastian Pipping wrote:
>   11-solar
>   12-constantine
>   13-goddard
>   14-laughlin
>   15-lovelock
>   16-verne

Correction:

  10-solar
  11-leonidas
  12-constantine
  13-goddard
  14-laughlin
  15-lovelock
  16-verne

Re: [gentoo-dev] Merge 7 Fedora wallpapers packages to single one with slots?

[Sebastian Pipping]

Hi,


On 27.01.2018 17:32, Michael Orlitzky wrote:
> If you do merge them, then it might be better to use flags for the
> different sub-packages rather than slots. There's no place to describe
> what a slot is for, but having a local USE=solar with a corresponding
> description in metadata.xml is (relatively) discoverable.

use flags work well if we make a single ebuild offering to install one
to all of these.  That would be natural if the goddard/13 source rpm
included files of constantine/12 and solar/11 as well and so on but it
doesn't seem to.  I would rather go with one ebuild per mayor release
number of Fedora: Needs less use flag configuration as well.

About slot names, if "12" is not good enough we could use

  11-solar
  12-constantine
  13-goddard
  14-laughlin
  15-lovelock
  16-verne

or so for SLOT to have a mapping to names?

Best



Sebastian

Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure

[Michael Orlitzky]

On 01/27/2018 11:42 AM, Gordon Pettey wrote:
> Why not use a hash of the file name instead of its contents? That
> seems like it would be much simpler, and that's not going to reduce
> the output space for balance...

That's the proposal =P

Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure

[Michael Orlitzky]

On 01/27/2018 03:30 AM, Michał Górny wrote:
>>
>> What are we worried about in using a temporary directory? Copying across
>> filesystem boundaries? Except in rare cases, $DISTDIR itself will be
>> usable a temporary location (on the same filesystem), won't it?
> 
> Why add the extra complexity when there's no need for one? Note that
> there's also the problem of resuming transfers, so in the end we're
> talking about permanent temporary directory where we keep unfinished
> transfers.

Can't argue with that, but I don't see it as a huge "con."


>> For the second point, portage is going to tell me where to put the file,
>> isn't it? Then no matter what garbage I download, won't portage look for
>> it in the right place, because where-to-put-it is determined using the
>> same manifest hash that determines where-to-find-it?
> 
> No, it won't. Why would it? You're going to call something like:
> 
>   edistadd foo.tar.gz bar.tar.gz
> 
> ...and it will place the files in the right subdirectories.

If we have a tool like edistadd, then I see the problem. But if we were
going to use file-data based hashes, then there would be no need for a
tool in most cases. As a developer, "repoman manifest" would handle it.
As a user, I'm going to see a message like,

 Fetch instructions for games-fps/doom3-lms-4:
  * Please download LastManStandingCoop4Multiplatform.zip from:
  * http://www.moddb.com/mods/last-man-standing-coop/downloads
  * and move it to /var/cache/portage/distfiles

except instead of $DISTDIR, it would suggest whatever directory is
computed from the hash in the manifest.

Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure

[Gordon Pettey]

Why not use a hash of the file name instead of its contents? That
seems like it would be much simpler, and that's not going to reduce
the output space for balance...

On Sat, Jan 27, 2018 at 5:41 AM, Michał Górny  wrote:
> W dniu sob, 27.01.2018 o godzinie 11∶36 +, użytkownik Roy Bamford
> napisał:
>> On 2018.01.27 08:30, Michał Górny wrote:
>> > W dniu pią, 26.01.2018 o godzinie 20∶48 -0500, użytkownik Michael
>> > Orlitzky napisał:
>> > > On 01/26/2018 06:24 PM, Michał Górny wrote:
>> > > >
>> > > > The alternate option of using file hash has the advantage of
>> >
>> > having
>> > > > a more balanced split.  Furthermore, since hashes are stored
>> > > > in Manifests using them is zero-cost.  However, this solution has
>> >
>> > two
>> > > > significant disadvantages:
>> > > >
>> > > > 1. The hash values are unknown for newly-downloaded distfiles, so
>> > > >``repoman`` (or an equivalent tool) would have to use a
>> >
>> > temporary
>> > > >directory before locating the file in appropriate subdirectory.
>> > > >
>> > > > 2. User-provided distfiles (e.g. for fetch-restricted packages)
>> >
>> > with
>> > > >hash mismatches would be placed in the wrong subdirectory,
>> > > >potentially causing confusing errors.
>> > > >
>> > >
>> > > The filename proposal sounds fine, so this is only academic, but:
>> >
>> > are
>> > > these two points really disadvantages?
>> > >
>> > > What are we worried about in using a temporary directory? Copying
>> >
>> > across
>> > > filesystem boundaries? Except in rare cases, $DISTDIR itself will be
>> > > usable a temporary location (on the same filesystem), won't it?
>> >
>> > Why add the extra complexity when there's no need for one? Note that
>> > there's also the problem of resuming transfers, so in the end we're
>> > talking about permanent temporary directory where we keep unfinished
>> > transfers.
>> >
>> > > For the second point, portage is going to tell me where to put the
>> >
>> > file,
>> > > isn't it? Then no matter what garbage I download, won't portage look
>> >
>> > for
>> > > it in the right place, because where-to-put-it is determined using
>> >
>> > the
>> > > same manifest hash that determines where-to-find-it?
>> >
>> > No, it won't. Why would it? You're going to call something like:
>> >
>> >   edistadd foo.tar.gz bar.tar.gz
>> >
>> > ...and it will place the files in the right subdirectories.
>> >
>> > --
>> > Best regards,
>> > Michał Górny
>> >
>> >
>> >
>> >
>>
>> Michał,
>>
>> How does this work for fetch restricted files and finding other files
>> no longer on the mirrors?
>>
>> Its no longer a download and move it to $DISTFILES, or is it?
>> Whatever it is, users will need to do it unless files in  $DISTFILES
>> are accepted by package managers if they are not found in the main
>> structure.
>
> I've just answered that, and it's in the GLEP also. There will be
> a helper tool to make this easy. Furthermore, I think we may even make
> Portage keep accepting both locations indefinitely.
>
> As for finding files in your distdir, there's no reason why plain:
>
>   find -name 'foo.tar.gz'
>
> wouldn't work.
>
> --
> Best regards,
> Michał Górny
>
>

Re: [gentoo-dev] Merge 7 Fedora wallpapers packages to single one with slots?

[Michael Orlitzky]

On 01/27/2018 11:09 AM, Sebastian Pipping wrote:
> Hi!
> 
> 
> I noticed that we have 7 packages on Fedora wallpapers with names that
> only explain themselves to Fedora insiders:
> 
> ...
> 
> I was thinking that we could merge these packages into a new package
> "x11-themes/fedora-backgrounds" or so with slots 11 to 16 so that people
> can still install them in parallel, get slot updates automatically,
> adding more recent ones does not add more packages, and the package name
> explains itself.

If you do merge them, then it might be better to use flags for the
different sub-packages rather than slots. There's no place to describe
what a slot is for, but having a local USE=solar with a corresponding
description in metadata.xml is (relatively) discoverable.

Re: [gentoo-dev] time to retire

[Sebastian Pipping]

Stefan,


thanks for your work on Gentoo!

All the best



Sebastian

[gentoo-dev] Merge 7 Fedora wallpapers packages to single one with slots?

[Sebastian Pipping]

Hi!


I noticed that we have 7 packages on Fedora wallpapers with names that
only explain themselves to Fedora insiders:


# eix background | fgrep -B3 Fedora
* x11-themes/constantine-backgrounds
 Available versions:  12.1.1.4-r1
 Homepage:https://fedoraproject.org/wiki/F12_Artwork
 Description: Fedora official background artwork
--
* x11-themes/goddard-backgrounds
 Available versions:  13.0.0.3-r1
 Homepage:https://fedoraproject.org/wiki/F13_Artwork
 Description: Fedora official background artwork
--
* x11-themes/laughlin-backgrounds
 Available versions:  14.1.0.3-r1
 Homepage:https://fedoraproject.org/wiki/F14_Artwork
 Description: Fedora official background artwork
--
* x11-themes/leonidas-backgrounds
 Available versions:  11.0.0.2-r1
 Homepage:https://fedoraproject.org/wiki/F11_Artwork
 Description: Fedora official background artwork
--
* x11-themes/lovelock-backgrounds
 Available versions:  14.91.1.1-r1
 Homepage:https://fedoraproject.org/wiki/F15_Artwork
 Description: Fedora official background artwork
--
* x11-themes/solar-backgrounds
 Available versions:  0.92.0.5-r1
 Homepage:https://fedoraproject.org/wiki/F11_Artwork
 Description: Fedora official background artwork
--
* x11-themes/verne-backgrounds
 Available versions:  (~)15.91.0.1-r1
 Homepage:https://fedoraproject.org/wiki/F16_Artwork
 Description: Fedora official background artwork


I was thinking that we could merge these packages into a new package
"x11-themes/fedora-backgrounds" or so with slots 11 to 16 so that people
can still install them in parallel, get slot updates automatically,
adding more recent ones does not add more packages, and the package name
explains itself.

Any objections?

Best



Sebastian

Re: [gentoo-dev] [News item review] Portage rsync tree verification (v3)

[Nils Freydank]

Am Samstag, 27. Januar 2018, 15:26:44 CET schrieb Michał Górny:
> [...]
> 
> The new verification is intended for users who syncing via rsync.
> Verification mechanisms for other methods of sync will be provided
> in future.
s/who syncing/who are syncing/

("who sync via rsync" would sound a bit odd, but should work aswell.)

-- 
GPG fingerprint: '766B 8122 1342 6912 3401 492A 8B54 D7A3 FF3C DB17'
Holgersson

signature.asc
Description: This is a digitally signed message part.

[gentoo-dev] Re: [News item review] Portage rsync tree verification (v3)

[Duncan]

Michał Górny posted on Sat, 27 Jan 2018 15:26:44 +0100 as excerpted:

> The new verification is intended for users who syncing via rsync.
> Verification mechanisms for other methods of sync will be provided in
> future.


s/in future/in the future/

Thanks for adding that paragraph.  It perfectly addresses the question I 
had about the original. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] [News item review] Portage rsync tree verification (v3)

[M. J. Everitt]

On 27/01/18 14:26, Michał Górny wrote [excerpted]:
> The verification is implemented via using app-portage/gemato. Currently,
> the whole repository is verified after syncing.
>
I would drop either 'via' or 'using' - they both are the same
verb/meaning and one is hence redundant.
Just my 2c as a native English speaker :)

MJE



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [News item review] Portage rsync tree verification (v2)

[Michał Górny]

W dniu czw, 25.01.2018 o godzinie 15∶55 -0600, użytkownik R0b0t1
napisał:
> On Thu, Jan 25, 2018 at 3:45 PM, Michał Górny  wrote:
> > W dniu czw, 25.01.2018 o godzinie 21∶37 +, użytkownik Robin H.
> > Johnson napisał:
> > > On Thu, Jan 25, 2018 at 01:35:17PM +0100, Michał Górny wrote:
> > > > Title: Portage rsync tree verification
> > > > Author: Michał Górny 
> > > > Posted: 2018-01-xx
> > > > Revision: 1
> > > > News-Item-Format: 2.0
> > > > Display-If-Installed:  > > 
> > > Drop Display-If-Installed, they need to always see this until they know
> > > it was bootstrapped.
> > 
> > Well, the idea was that if someone starts with stage that has >2.3.21,
> > then he has bootstrapped via verifying the stage signature.
> > 
> > > > Starting with sys-apps/portage-2.3.22, Portage enables cryptographic
> > > > verification of the Gentoo rsync repository distributed over rsync
> > > > by default.
> > > 
> > > Seems very wordy, suggested cleanup:
> > > > > Starting with sys-apps/portage-2.3.22, Portage will verify the Gentoo
> > > > > repository after rsync by default.
> > > > 
> > > > This aims to prevent malicious third parties from altering
> > > > the contents of the ebuild repository received by our users.
> > > > 
> > > > This does not affect users syncing using git and other methods.
> > > > Appropriate verification mechanisms for them will be provided
> > > > in the future.
> > > 
> > > Note that emerge-webrsync has verification via FEATURES=webrsync-gpg?
> > 
> > I'm sorry, I have never used that. Does it cover full key maintenance
> > or rely on user to do the gpg work?
> > 
> 
> It used to be necessary to set up a GnuPG home for portage and pull
> the keys in, but now users can emerge app-crypt/gentoo-keys and set
> PORTAGE_GPG_DIR="/var/lib/gentoo/gkeys/keyrings/gentoo/release".
> 

In that case I'd rather not announce it until it is integrated properly.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [News item review] Portage rsync tree verification (v3)

[Michał Górny]

Next round:

Title: Portage rsync tree verification
Author: Michał Górny 
Posted: 2018-01-xx
Revision: 1
News-Item-Format: 2.0
Display-If-Installed: sys-apps/portage

Starting with sys-apps/portage-2.3.22, Portage will verify the Gentoo
repository after rsync by default.

The new verification is intended for users who syncing via rsync.
Verification mechanisms for other methods of sync will be provided
in future.

This does not affect users syncing using git and other methods.
Appropriate verification mechanisms for them will be provided
in the future.

The verification is implemented via using app-portage/gemato. Currently,
the whole repository is verified after syncing. On systems with slow
hard drives, this could take around 2 minutes. If you wish to disable
it, you can disable the 'rsync-verify' USE flag on sys-apps/portage
or set 'sync-rsync-verify-metamanifest = no' in your repos.conf.

Please note that the verification currently does not prevent Portage
from using the repository after syncing. If 'emerge --sync' fails,
do not install any packages and retry syncing. In case of prolonged
or frequent verification failures, please make sure to report a bug
including the failing mirror addresses (found in emerge.log).

The verification uses information from the binary keyring provided
by the app-crypt/gentoo-keys package. The keys are refreshed
from the keyserver before every use in order to check for revocation.
The post-sync verification ensures that the key package is verified
itself. However, manual verification is required before the first use.

On Gentoo installations created using installation media that included
portage-2.3.22, the keys will already be covered by the installation
media signatures. On existing installations, you need to manually
compare the primary key fingerprint (reported by gemato on every sync)
against the official Gentoo keys [1]. An example gemato output is:

  INFO:root:Valid OpenPGP signature found:
  INFO:root:- primary key: 1234567890ABCDEF1234567890ABCDEF12345678
  INFO:root:- subkey: FEDCBA0987654321FEDCBA0987654321FEDCBA09

Please note that the above snippet does not include the real key id
on purpose. The primary key actually printed by gemato must match
the 'Gentoo Portage Snapshot Signing Key' on the website. Please make
sure to also check the certificate used for the secure connection
to the site!

[1]:https://www.gentoo.org/downloads/signatures/

-- 
Best regards,
Michał Górny

[gentoo-dev] Gentoo sync mirror commits are now OpenPGP-signed

[Michał Górny]

Hi, everyone.

Just a quick announcement: I have enabled experimental signatures
on gentoo-mirror commits for gentoo.git. The server now verifies
developer signatures from gentoo.git (using fingerprints from LDAP)
and if everything looks fine pushes a signed mirror commit.

$ git log --show-signature -1 | cat
commit ec25012d1f9e8c795d8822810970127b13adf2c1
gpg: Signature made Sat Jan 27 14:48:54 2018 CET
gpg:using RSA key F265B6A01DEF32748C6184C79FA394EB86CB7342
gpg: Good signature from "Repository Mirror & CI project (automated signing 
key) " [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:  There is no indication that the signature belongs to the owner.
Primary key fingerprint: C5DF ACC4 F05D 47E4 383C  E4C2 403B C085 18DA F97B
 Subkey fingerprint: F265 B6A0 1DEF 3274 8C61  84C7 9FA3 94EB 86CB 7342
Author: Repository QA checks 
Date:   Sat Jan 27 14:48:54 2018

2018-01-27 13:48:53 UTC


The appropriate public key has been pushed to SKS keyservers.
The current fingerprints can be found on the project page [1]. I will
work on Portage integration later on.

[1]:https://wiki.gentoo.org/wiki/Project:Repository_mirror_and_CI

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure

[Michał Górny]

W dniu sob, 27.01.2018 o godzinie 11∶36 +, użytkownik Roy Bamford
napisał:
> On 2018.01.27 08:30, Michał Górny wrote:
> > W dniu pią, 26.01.2018 o godzinie 20∶48 -0500, użytkownik Michael
> > Orlitzky napisał:
> > > On 01/26/2018 06:24 PM, Michał Górny wrote:
> > > > 
> > > > The alternate option of using file hash has the advantage of
> > 
> > having
> > > > a more balanced split.  Furthermore, since hashes are stored
> > > > in Manifests using them is zero-cost.  However, this solution has
> > 
> > two
> > > > significant disadvantages:
> > > > 
> > > > 1. The hash values are unknown for newly-downloaded distfiles, so
> > > >``repoman`` (or an equivalent tool) would have to use a
> > 
> > temporary
> > > >directory before locating the file in appropriate subdirectory.
> > > > 
> > > > 2. User-provided distfiles (e.g. for fetch-restricted packages)
> > 
> > with
> > > >hash mismatches would be placed in the wrong subdirectory,
> > > >potentially causing confusing errors.
> > > > 
> > > 
> > > The filename proposal sounds fine, so this is only academic, but:
> > 
> > are
> > > these two points really disadvantages?
> > > 
> > > What are we worried about in using a temporary directory? Copying
> > 
> > across
> > > filesystem boundaries? Except in rare cases, $DISTDIR itself will be
> > > usable a temporary location (on the same filesystem), won't it?
> > 
> > Why add the extra complexity when there's no need for one? Note that
> > there's also the problem of resuming transfers, so in the end we're
> > talking about permanent temporary directory where we keep unfinished
> > transfers.
> > 
> > > For the second point, portage is going to tell me where to put the
> > 
> > file,
> > > isn't it? Then no matter what garbage I download, won't portage look
> > 
> > for
> > > it in the right place, because where-to-put-it is determined using
> > 
> > the
> > > same manifest hash that determines where-to-find-it?
> > 
> > No, it won't. Why would it? You're going to call something like:
> > 
> >   edistadd foo.tar.gz bar.tar.gz
> > 
> > ...and it will place the files in the right subdirectories.
> > 
> > -- 
> > Best regards,
> > Michał Górny
> > 
> > 
> > 
> > 
> 
> Michał,
> 
> How does this work for fetch restricted files and finding other files
> no longer on the mirrors?
> 
> Its no longer a download and move it to $DISTFILES, or is it?
> Whatever it is, users will need to do it unless files in  $DISTFILES
> are accepted by package managers if they are not found in the main 
> structure.

I've just answered that, and it's in the GLEP also. There will be
a helper tool to make this easy. Furthermore, I think we may even make
Portage keep accepting both locations indefinitely.

As for finding files in your distdir, there's no reason why plain:

  find -name 'foo.tar.gz'

wouldn't work.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure

[Roy Bamford]

On 2018.01.27 08:30, Michał Górny wrote:
> W dniu pią, 26.01.2018 o godzinie 20∶48 -0500, użytkownik Michael
> Orlitzky napisał:
> > On 01/26/2018 06:24 PM, Michał Górny wrote:
> > > 
> > > The alternate option of using file hash has the advantage of
> having
> > > a more balanced split.  Furthermore, since hashes are stored
> > > in Manifests using them is zero-cost.  However, this solution has
> two
> > > significant disadvantages:
> > > 
> > > 1. The hash values are unknown for newly-downloaded distfiles, so
> > >``repoman`` (or an equivalent tool) would have to use a
> temporary
> > >directory before locating the file in appropriate subdirectory.
> > > 
> > > 2. User-provided distfiles (e.g. for fetch-restricted packages)
> with
> > >hash mismatches would be placed in the wrong subdirectory,
> > >potentially causing confusing errors.
> > > 
> > 
> > The filename proposal sounds fine, so this is only academic, but:
> are
> > these two points really disadvantages?
> > 
> > What are we worried about in using a temporary directory? Copying
> across
> > filesystem boundaries? Except in rare cases, $DISTDIR itself will be
> > usable a temporary location (on the same filesystem), won't it?
> 
> Why add the extra complexity when there's no need for one? Note that
> there's also the problem of resuming transfers, so in the end we're
> talking about permanent temporary directory where we keep unfinished
> transfers.
> 
> > For the second point, portage is going to tell me where to put the
> file,
> > isn't it? Then no matter what garbage I download, won't portage look
> for
> > it in the right place, because where-to-put-it is determined using
> the
> > same manifest hash that determines where-to-find-it?
> 
> No, it won't. Why would it? You're going to call something like:
> 
>   edistadd foo.tar.gz bar.tar.gz
> 
> ...and it will place the files in the right subdirectories.
> 
> -- 
> Best regards,
> Michał Górny
> 
> 
> 
> 

Michał,

How does this work for fetch restricted files and finding other files
no longer on the mirrors?

Its no longer a download and move it to $DISTFILES, or is it?
Whatever it is, users will need to do it unless files in  $DISTFILES
are accepted by package managers if they are not found in the main 
structure.

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods


pgpqad588ZI9H.pgp
Description: PGP signature

Re: [gentoo-portage-dev] [PATCH v2 2/2] doebuild: Remove unnecessary restore of PORTAGE_ACTUAL_DISTDIR

[Zac Medico]

On 01/27/2018 02:02 AM, Michał Górny wrote:
> Now that the DISTDIR override does not leak to global settings instance,
> there is no need to restore it afterwards. Suggested by Zac Medico.
> ---
>  pym/portage/package/ebuild/doebuild.py | 5 -
>  1 file changed, 5 deletions(-)

Both patches look good. Please merge.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-portage-dev] [PATCH v2 2/2] doebuild: Remove unnecessary restore of PORTAGE_ACTUAL_DISTDIR

[Michał Górny]

Now that the DISTDIR override does not leak to global settings instance,
there is no need to restore it afterwards. Suggested by Zac Medico.
---
 pym/portage/package/ebuild/doebuild.py | 5 -
 1 file changed, 5 deletions(-)

diff --git a/pym/portage/package/ebuild/doebuild.py 
b/pym/portage/package/ebuild/doebuild.py
index c8df9b744..17fc8ccf4 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -1250,11 +1250,6 @@ def doebuild(myebuild, mydo, _unused=DeprecationWarning, 
settings=None, debug=0,
 
mysettings.pop("REPLACING_VERSIONS", None)
 
-   # Make sure that DISTDIR is restored to it's normal value 
before we return!
-   if "PORTAGE_ACTUAL_DISTDIR" in mysettings:
-   mysettings["DISTDIR"] = 
mysettings["PORTAGE_ACTUAL_DISTDIR"]
-   del mysettings["PORTAGE_ACTUAL_DISTDIR"]
-
if logfile and not returnpid:
try:
if os.stat(logfile).st_size == 0:
-- 
2.16.1

[gentoo-portage-dev] [PATCH v2 1/2] prepare_build_dirs: Fix duplicate override of DISTDIR for 'ebuild'

[Michał Górny]

Fix the duplicate override of DISTDIR that happens when using
the 'ebuild' tool to run ebuilds. In this case, both the 'config'
code and the 'prepare_build_dirs' code attempt to replace DISTDIR,
and the shadow distdir symlinks end up pointing to themselves.

As suggested by Zac Medico, after removing this override DISTDIR
in settings will always be the real directory, so there is no need
for a PORTAGE_ACTUAL_DISTDIR check here.
---
 pym/portage/package/ebuild/prepare_build_dirs.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/pym/portage/package/ebuild/prepare_build_dirs.py 
b/pym/portage/package/ebuild/prepare_build_dirs.py
index 16afc3f98..21e2aa87c 100644
--- a/pym/portage/package/ebuild/prepare_build_dirs.py
+++ b/pym/portage/package/ebuild/prepare_build_dirs.py
@@ -412,9 +412,7 @@ def _prepare_fake_filesdir(settings):
 
 def _prepare_fake_distdir(settings, alist):
orig_distdir = settings["DISTDIR"]
-   settings["PORTAGE_ACTUAL_DISTDIR"] = orig_distdir
-   edpath = settings["DISTDIR"] = \
-   os.path.join(settings["PORTAGE_BUILDDIR"], "distdir")
+   edpath = os.path.join(settings["PORTAGE_BUILDDIR"], "distdir")
portage.util.ensure_dirs(edpath, gid=portage_gid, mode=0o755)
 
# Remove any unexpected files or directories.
-- 
2.16.1

Re: [gentoo-portage-dev] [PATCH] prepare_build_dirs: Fix duplicate override of DISTDIR for 'ebuild'

[Zac Medico]

On 01/27/2018 01:38 AM, Zac Medico wrote:
> On 01/27/2018 01:01 AM, Michał Górny wrote:
>> Fix the duplicate override of DISTDIR that happens when using
>> the 'ebuild' tool to run ebuilds. In this case, both the 'config'
>> code and the 'prepare_build_dirs' code attempt to replace DISTDIR,
>> and the shadow distdir symlinks end up pointing to themselves.
>>
>> Apparently the _prepare_fake_distdir() is passed settings containing
>> the original DISTDIR by emerge, while ebuild passes an instance
>> contaning the DISTDIR override. Try PORTAGE_ACTUAL_DISTDIR first
>> to account for both cases.
>> ---
>>  pym/portage/package/ebuild/prepare_build_dirs.py | 8 
>>  1 file changed, 4 insertions(+), 4 deletions(-)
>>
>> diff --git a/pym/portage/package/ebuild/prepare_build_dirs.py 
>> b/pym/portage/package/ebuild/prepare_build_dirs.py
>> index 16afc3f98..0b6a3a4df 100644
>> --- a/pym/portage/package/ebuild/prepare_build_dirs.py
>> +++ b/pym/portage/package/ebuild/prepare_build_dirs.py
>> @@ -411,10 +411,10 @@ def _prepare_fake_filesdir(settings):
>>  os.symlink(real_filesdir, symlink_path)
>>  
>>  def _prepare_fake_distdir(settings, alist):
>> -orig_distdir = settings["DISTDIR"]
>> -settings["PORTAGE_ACTUAL_DISTDIR"] = orig_distdir
>> -edpath = settings["DISTDIR"] = \
>> -os.path.join(settings["PORTAGE_BUILDDIR"], "distdir")
> 
> Above you've removed the only code that overrode settings["DISTDIR"],
> which is good.
> 
>> +orig_distdir = settings.get("PORTAGE_ACTUAL_DISTDIR",
>> +settings.get("DISTDIR"))
>> +assert orig_distdir
> 
> Since settings["DISTDIR"] is no longer overridden, now this is all
> that's needed:
> 
> orig_distdir = settings["DISTDIR"]

Also, we don't need this code in the doebuild function anymore:

if "PORTAGE_ACTUAL_DISTDIR" in mysettings:
mysettings["DISTDIR"] = mysettings["PORTAGE_ACTUAL_DISTDIR"]
del mysettings["PORTAGE_ACTUAL_DISTDIR"]
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-portage-dev] [PATCH] prepare_build_dirs: Fix duplicate override of DISTDIR for 'ebuild'

[Zac Medico]

On 01/27/2018 01:01 AM, Michał Górny wrote:
> Fix the duplicate override of DISTDIR that happens when using
> the 'ebuild' tool to run ebuilds. In this case, both the 'config'
> code and the 'prepare_build_dirs' code attempt to replace DISTDIR,
> and the shadow distdir symlinks end up pointing to themselves.
> 
> Apparently the _prepare_fake_distdir() is passed settings containing
> the original DISTDIR by emerge, while ebuild passes an instance
> contaning the DISTDIR override. Try PORTAGE_ACTUAL_DISTDIR first
> to account for both cases.
> ---
>  pym/portage/package/ebuild/prepare_build_dirs.py | 8 
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/pym/portage/package/ebuild/prepare_build_dirs.py 
> b/pym/portage/package/ebuild/prepare_build_dirs.py
> index 16afc3f98..0b6a3a4df 100644
> --- a/pym/portage/package/ebuild/prepare_build_dirs.py
> +++ b/pym/portage/package/ebuild/prepare_build_dirs.py
> @@ -411,10 +411,10 @@ def _prepare_fake_filesdir(settings):
>   os.symlink(real_filesdir, symlink_path)
>  
>  def _prepare_fake_distdir(settings, alist):
> - orig_distdir = settings["DISTDIR"]
> - settings["PORTAGE_ACTUAL_DISTDIR"] = orig_distdir
> - edpath = settings["DISTDIR"] = \
> - os.path.join(settings["PORTAGE_BUILDDIR"], "distdir")

Above you've removed the only code that overrode settings["DISTDIR"],
which is good.

> + orig_distdir = settings.get("PORTAGE_ACTUAL_DISTDIR",
> + settings.get("DISTDIR"))
> + assert orig_distdir

Since settings["DISTDIR"] is no longer overridden, now this is all
that's needed:

orig_distdir = settings["DISTDIR"]


> + edpath = os.path.join(settings["PORTAGE_BUILDDIR"], "distdir")
>   portage.util.ensure_dirs(edpath, gid=portage_gid, mode=0o755)
>  
>   # Remove any unexpected files or directories.
> 

-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-portage-dev] [PATCH] prepare_build_dirs: Fix duplicate override of DISTDIR for 'ebuild'

[Michał Górny]

Fix the duplicate override of DISTDIR that happens when using
the 'ebuild' tool to run ebuilds. In this case, both the 'config'
code and the 'prepare_build_dirs' code attempt to replace DISTDIR,
and the shadow distdir symlinks end up pointing to themselves.

Apparently the _prepare_fake_distdir() is passed settings containing
the original DISTDIR by emerge, while ebuild passes an instance
contaning the DISTDIR override. Try PORTAGE_ACTUAL_DISTDIR first
to account for both cases.
---
 pym/portage/package/ebuild/prepare_build_dirs.py | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/pym/portage/package/ebuild/prepare_build_dirs.py 
b/pym/portage/package/ebuild/prepare_build_dirs.py
index 16afc3f98..0b6a3a4df 100644
--- a/pym/portage/package/ebuild/prepare_build_dirs.py
+++ b/pym/portage/package/ebuild/prepare_build_dirs.py
@@ -411,10 +411,10 @@ def _prepare_fake_filesdir(settings):
os.symlink(real_filesdir, symlink_path)
 
 def _prepare_fake_distdir(settings, alist):
-   orig_distdir = settings["DISTDIR"]
-   settings["PORTAGE_ACTUAL_DISTDIR"] = orig_distdir
-   edpath = settings["DISTDIR"] = \
-   os.path.join(settings["PORTAGE_BUILDDIR"], "distdir")
+   orig_distdir = settings.get("PORTAGE_ACTUAL_DISTDIR",
+   settings.get("DISTDIR"))
+   assert orig_distdir
+   edpath = os.path.join(settings["PORTAGE_BUILDDIR"], "distdir")
portage.util.ensure_dirs(edpath, gid=portage_gid, mode=0o755)
 
# Remove any unexpected files or directories.
-- 
2.16.1

Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure

[Michał Górny]

W dniu pią, 26.01.2018 o godzinie 20∶48 -0500, użytkownik Michael
Orlitzky napisał:
> On 01/26/2018 06:24 PM, Michał Górny wrote:
> > 
> > The alternate option of using file hash has the advantage of having
> > a more balanced split.  Furthermore, since hashes are stored
> > in Manifests using them is zero-cost.  However, this solution has two
> > significant disadvantages:
> > 
> > 1. The hash values are unknown for newly-downloaded distfiles, so
> >``repoman`` (or an equivalent tool) would have to use a temporary
> >directory before locating the file in appropriate subdirectory.
> > 
> > 2. User-provided distfiles (e.g. for fetch-restricted packages) with
> >hash mismatches would be placed in the wrong subdirectory,
> >potentially causing confusing errors.
> > 
> 
> The filename proposal sounds fine, so this is only academic, but: are
> these two points really disadvantages?
> 
> What are we worried about in using a temporary directory? Copying across
> filesystem boundaries? Except in rare cases, $DISTDIR itself will be
> usable a temporary location (on the same filesystem), won't it?

Why add the extra complexity when there's no need for one? Note that
there's also the problem of resuming transfers, so in the end we're
talking about permanent temporary directory where we keep unfinished
transfers.

> For the second point, portage is going to tell me where to put the file,
> isn't it? Then no matter what garbage I download, won't portage look for
> it in the right place, because where-to-put-it is determined using the
> same manifest hash that determines where-to-find-it?

No, it won't. Why would it? You're going to call something like:

  edistadd foo.tar.gz bar.tar.gz

...and it will place the files in the right subdirectories.

-- 
Best regards,
Michał Górny