Re: [gentoo-portage-dev] Normaliser function for distfiles

[Rich Freeman]

On Tue, May 17, 2022 at 8:32 AM Markus Walter  wrote:
>
> On Tue, 17 May 2022 14:14:57 +0200,
> Rich Freeman wrote:
> >
> > On Mon, May 16, 2022 at 1:37 PM Markus Walter  
> > wrote:
> > >
> > > My use case is the following: I would like to improve the gs-elpa program
> > > and provide a precomputed overlay for melpa. However the melpa distfiles 
> > > are
> > > rebuilt everyday and cause checksum failures. However the only thing
> > > changing are the timestamps. Hence if a normaliser program could simply 
> > > set
> > > all timestamps to some predefined value (say 1.1.1970) then this problem
> > > should vanish.
> > >
> >
> > Wouldn't a simpler solution be to just have an ebuild setting that
> > tells the package manager to not check the timestamp?
>
> The timestamps are inside archive files thus changing the overall file
> hash. This happens during distfile download, where some more sophisticated
> replace all timestamps function would be necessary than just ignoring one
> timestamp.

Ah, apologies.  Totally missed that.  Yeah, obviously if the
timestamps INSIDE the archive are changing your only solution is to
edit the contents.  I thought it was just the external timestamp
changing (I'm not sure if portage even checks that, or if wget changes
the mtime on files it downloads).

-- 
Rich

Re: [gentoo-portage-dev] Normaliser function for distfiles

[Rich Freeman]

On Mon, May 16, 2022 at 1:37 PM Markus Walter  wrote:
>
> My use case is the following: I would like to improve the gs-elpa program
> and provide a precomputed overlay for melpa. However the melpa distfiles are
> rebuilt everyday and cause checksum failures. However the only thing
> changing are the timestamps. Hence if a normaliser program could simply set
> all timestamps to some predefined value (say 1.1.1970) then this problem
> should vanish.
>

Wouldn't a simpler solution be to just have an ebuild setting that
tells the package manager to not check the timestamp?

-- 
Rich

Re: [gentoo-portage-dev] Re: [RFC] Improving Gentoo package format

[Rich Freeman]

On Sun, Nov 11, 2018 at 1:02 PM M. J. Everitt  wrote:
>
> If you can really present a decent argument for replicating the
> functionality of other distros like Debian, Arch, Ubuntu etc then let's
> here it. For now, the strength of Gentoo is being able to fully customise a
> system to your own requirements, not being trapped by some distro
> maintainer's arbitrary choices. Play to your USP's and strengths rather
> than chasing rainbows ..
>

Why do we support binary packages at all?  Simple: compiling packages
is expensive, and if you happen to already have them compiled, fully
customized to your own requirements, then there is no point in
recompiling them.  You're just spending a ton of resources to build
the exact same files you already have.

The only change I'm suggesting is that portage could take all the
configuration you're already supplying, and then optionally go see if
somebody you trust has already built the package that meets your
requirements.  If so, then it would be downloaded and installed,
otherwise it would just compile from source.

You get the exact same files installed on your system either way.

-- 
Rich

Re: [gentoo-portage-dev] Re: [RFC] Improving Gentoo package format

[Rich Freeman]

On Sun, Nov 11, 2018 at 12:31 PM M. J. Everitt  wrote:
>
> Binpkgs are also a popular component of a few downstream distro's based on
> Gentoo (thinking pentoo right now as an easy example).
>
> So we don't want to break existing users of this format without considering
> the ramifications for these scenarios, as you'll have some very grumpy devs...
>

I'd argue that they'd be more important for Gentoo if they were more
useful.  IMO the main limitation with them is the inability to
auto-download them from a repository, detecting the binpkg USE flags
BEFORE downloading.  This is why I suggested hashing the USE flags or
similar and sticking that in the filename.

Obviously you can't host a repository with all the USE combinations.
However, you could have a reference repo and the package manager could
check it before doing a build.  If you get a hit then you can install
the binpkg.  If you don't then you can do a source build.

Portage already checks the USE flags inside the binpkg before merging
it and by default doesn't use a non-matching binpkg.  The problem with
the current approach is:
1.  You have to download the package to check this (could be a big file).
2.  You can't host multiple versions of a binpkg with different USE
flags since the filenames collide.

I suggested a content hash because you can use it for an arbitrary
amount of metadata, vs having to cram arch/USE/multilib and I'm sure
something I'm missing into a filename.  Make the hash as short as is
economical - it isn't like we have THAT many permutations, the PM can
still check the internal metadata, and this isn't a security feature.

-- 
Rich

Re: [gentoo-portage-dev] [RFC] Improving Gentoo package format

[Rich Freeman]

On Sun, Nov 11, 2018 at 3:29 AM Michał Górny  wrote:
>
> On Sat, 2018-11-10 at 09:37 -0500, Alec Warner wrote:
> > On Sat, Nov 10, 2018 at 8:09 AM Michał Górny  wrote:
> >
> > >
> > > + If we can maintain reasonable level of vdb compatibility, the user can
> > > even emergency-install a package without causing too much hassle (as it
> > > will be recorded in vdb); ideally Portage would detect this vdb entry
> > > and support fixing the install afterwards.
> > >

IMO just overwriting vdb doesn't seem like a great idea.  If somebody
does do a plain untar then the package manager has no ability to
sanitize anything going in, which means dealing with a potential mess
after-the-fact.  Users wouldn't think to go messing with /var/db/pkg
on their own, but they certainly will be tempted to just untar a file.

Perhaps a package with the same name/version was already installed,
but the new files aren't the same as the old files.  Now we have
orphans because the package manager never had a chance to clean up and
lost all its state regarding what was already there.

Plus, this is basically in-band signaling and that is the sort of
thing that usually ends up being regretted sooner or later.

I'm not sure if vdb is entirely optimal, but if we wanted to stick
with that metadata format, why not just stick it in a separate
tarball?

>
> Are you saying it's better that user emergency-installs a package
> without recording it in vdb, and ends up with mess of collisions
> and untracked files?
>

IMO this is no different than a user unpacking any other random
tarball that goes and creates orphans.  I think the better solution is
some kind of tool to cleanly install a tarball, assuming it doesn't
already exist.  Short of turning /usr into a squashfs or whatever I'm
not sure any distro has a great solution for keeping users from
bypassing the package manager entirely.

> > In short; why should we event a new format?
>
> Because nobody knows how to use 'ar', compared to how almost every
> Gentoo user can use 'tar' immediately?  Of course we could alternatively
> just use a nested tarball but I wanted to keep the possibility
> of actually being able to 'tar -xf' it without having to extract nested
> archives.
>

IMO a nested tarball would be a better solution.  I agree that ar is
obscure, and I don't see how it adds any value.  If we were talking
about something going into a bootloader then optimizing for unpacking
efficiency might be a concern, but there is no reason not to use more
standard tools, unless there was something about the .deb format
itself we wanted to completely preserve (seems unlikely).

Overall though, I definitely think that a better binary format makes a
lot of sense, and I think you're on the right track.

One thing you didn't touch on is file naming.  Right now two binary
packages with different USE/etc configurations are going to collide.
Would it make sense to toss in some kind of content hash of certain
metadata in the filename or something so that it would be much simpler
to host and auto-fetch binary packages?  I realize this is going
beyond your initial scope, but if we wanted to do this it would be a
good time to do so.

-- 
Rich

Re: [gentoo-portage-dev] Tools-Portage Lead

[Rich Freeman]

On Tue, Oct 13, 2015 at 1:47 PM, Paul Varner  wrote:
> Reading GLEP-39, it is not clear to me if a sub-project needs to have
> their leads elected or not.

No interest in interfering with whatever you guys come up with, but as
far as I'm aware subprojects are just projects as far as GLEP 39 goes
- the hierarchy is intended to be more for organization than anything.
So, you guys are welcome to and should of course consider selecting a
lead.

The motivation of the request was less about trying to dictate how
projects run themselves and more about having somebody people can
reach out to with questions/etc, and also to have a way to get a sense
for how alive projects are.

-- 
Rich

Re: [gentoo-portage-dev] Re: Dynamic USE dependencies

[Rich Freeman]

On Mon, Apr 6, 2015 at 8:02 AM, Martin Vaeth mar...@mvath.de wrote:
 My suggestion is something in between - less invasive
 (and, in particular, less time consuming)
 than your suggestion to recalculate the USE-settings
 with every emerge, but more automatic than the current state.

Keep in mind that keeping track of past decisions made by portage does
not require user-editable config files in /etc.  It just requires a
cache of some kind, much as we do with installed packages/etc.

That said, portage still has to spend time basically re-validating the
consistency of the entire system because we allow the use of overlays
and other situations that don't guarantee that portage will have some
kind of consistent pre-calculated depgraph handed to it.  If we
required all repositories to have some kind of pre-generated cache in
them and ensured that this was always up-to-date and better controlled
the kinds of dependency changes we made, then maybe there might be an
opportunity to offload some of the work to the repository level
instead of doing it on every Gentoo system.  Still, unless we banned
overlays I'm not sure how much even this would buy you, since you'd
have many of these that need to be merged somehow.

From Zac's email and other discussions in the past it seems like we're
basically committed to doing all these calculations all the time
anyway, so we shouldn't be too shy about taking advantage of them.

-- 
Rich

Re: [gentoo-portage-dev] Re: Dynamic USE dependencies

[Rich Freeman]

On Sun, Apr 5, 2015 at 11:47 AM, Martin Vaeth mar...@mvath.de wrote:
 One suggestion around this problem would be to use different
 directories for these two types of use-flags, say
 package.use and package.use.needed.

I still think we need a better long-term solution, but the workaround
is simpler than that.

I try to keep files in directories for package.keywords and
package.use.  One starts with a z and is a dumping ground for
auto-whatever.  The other files are where I try to remember to store
personal preferences.  So, in theory at any time I can wipe the z-file
and run emerge and watch it wrestle with re-creating it.

However, I think a cleaner solution would be better...

-- 
Rich

Re: [gentoo-portage-dev] Dynamic USE dependencies

[Rich Freeman]

On Thu, Apr 2, 2015 at 2:03 PM, Kent Fredric kentfred...@gmail.com wrote:

 So I'm basically having trouble with groking the logic you're proposing of
 add a new use flag - implied change of useflag - rebuild when
 useflags change - but don't rebuild for this useflag change using some
 kind of magic


I'm fine with rebuilding a package anytime the IUSE changes.  I just
don't want to see a package built with +a -b rebuilt as -a +b because
either configuration works, so it gets rebuilt every time you run
emerge.  If multiple configurations work, just use the one that is
already installed.

emerge --newuse is already going to rebuild any package that has IUSE
change anyway.

-- 
Rich

Re: [gentoo-portage-dev] Re: Dynamic USE dependencies

[Rich Freeman]

On Thu, Apr 2, 2015 at 10:10 PM, Duncan 1i5t5.dun...@cox.net wrote:
 Rich Freeman posted on Thu, 02 Apr 2015 12:32:41 -0400 as excerpted:

 Out of curiosity, what is keeping us from having USE flag dependencies
 handled dynamically, in the same way that package dependencies are? If
 portage can figure out that I need libxml2 installed even if I don't put
 it in /var/lib/portage/world, why can't it figure out that I need it
 built with USE=icu even if I don't put that in /etc/portage/package.use?

 Because USE flags are binary, with not-yes implying no, while world
 set listings are binary, with not-yes implying do it anyway if needed?

That doesn't need to be true.  Not listing a flag in your config can
just mean do it anyway if needed.  Package USE dependencies already
work this way - you can depend on a flag being set, on it being unset,
or not mention the flag at all which means the package doesn't care.

 Changing the implied meaning of not-yes to match in both cases could
 certainly be done, but while I'm not opposed if the justification really
 is there, I think there's the potential here for a rather massive change
 in base assumptions, and if that is indeed what's involved, I believe
 it'd call for equally massive justifications.

There is no question that it would be a change in behavior.  However,
I wouldn't anticipate a lot of changes.

If you stuck -* in your make.conf then this change would have no
affect at all, since you've explicitly set the configuration of every
use flag.

Anything already in package.use would still stick, since that is also
configuration.  If you wiped your package.use clean then there would
be a possible change, but that is something the user has control over,
and presumably for them it is a change for the better if they're
making it.

Things would still default to their profile/package defaults as they
already do if you don't stick anything in your config files.  However,
future package installs that require a USE change would just make the
change if you didn't explicitly bar that configuration.  It wouldn't
modify your config files.


 And I'm worried that the suggestion here is going further down that
 emerge writing its own config path, without (IMO) appropriate safeguards.

I think this is exactly the opposite.  It would get rid of the need to
have portage go modifying package.use at all in many cases.  You'd
still need to modify configuration (perhaps with automatic help) if
you had explicitly set something which conflicts with what you want to
install (such as USE=-* globally).  However, for users who just go
with the profile defaults they wouldn't have to stick nearly as much
stuff in package.use just to change a flag setting they never
expressed a personal preference for one way or the other.

The current configuration forces you to use config files to capture
settings you care about, and also ones you don't actually care about,
and unless you're careful you'll have trouble telling these settings
apart later.  It is like sticking every installed package in your
world file.

-- 
Rich

[gentoo-portage-dev] Dynamic USE dependencies

[Rich Freeman]

Out of curiosity, what is keeping us from having USE flag dependencies
handled dynamically, in the same way that package dependencies are?
If portage can figure out that I need libxml2 installed even if I
don't put it in /var/lib/portage/world, why can't it figure out that I
need it built with USE=icu even if I don't put that in
/etc/portage/package.use?

I was concerned that it might be more work in calculating the
dependencies, but then I was thinking that portage probably already
does all this work just to validate that the current configuration is
still consistent.

I fully appreciate that there could be USE blocks, just as there can
be package blocks, and that resolving these could require hints such
as adding some USE settings to config files, or doing oneshot installs
(perhaps the dynamic configuration would be set up to preserve
whatever is installed unless it conflicts - just as is done with
virtuals today).

-- 
Rich

Re: [gentoo-portage-dev] Dynamic USE dependencies

[Rich Freeman]

On Thu, Apr 2, 2015 at 12:56 PM, Kent Fredric kentfred...@gmail.com wrote:

 On 3 April 2015 at 05:32, Rich Freeman ri...@gentoo.org wrote:

 Out of curiosity, what is keeping us from having USE flag dependencies
 handled dynamically, in the same way that package dependencies are?
 If portage can figure out that I need libxml2 installed even if I
 don't put it in /var/lib/portage/world, why can't it figure out that I
 need it built with USE=icu even if I don't put that in
 /etc/portage/package.use?

 I'd say its more a concept issue than an application issue.

 USE flags often signify a need for code to be recompiled to grant the
 feature.

 How do you disambiguate between USE flags that *do* need a recompile to
 enable their power, and those that *dont* need a recompile to enable their
 power.

Why is this necessary?  If a USE flag changes, just rebuild the application.


 Or even clarify to portage that, The older version of X that didn't have
 IUSE=foo, actually had feature foo, but just didn't have the use flag vs
 The older version of X that didn't have IUSE=foo, didnt have feature foo or
 the IUSE.

Do what --newuse does.  If a flag is added/removed, then rebuild the
application.


 Is this new useflag defaulted on because it already existed, or is it
 defaulted on because its a new feature and its awesome

 Is this new useflag defaulted off because it didnt already exist, or did it
 exist and were disabling the feature because its bad


I'd suggest that the algorithm is:

1.  When installing a new or updated package:
1a. set all the flags in accordance with configuration (explicit
set/unset in profiles, make.conf, package.use)
1b. if these explicitly conflict with package dependencies then fail
with an immediate error (that is, config says USE=foo, and package dep
says atom[-foo] and so on
1c. additionally set flags as needed to satisfy package dependencies
when they don't conflict with explicit configuration

2.  When evaluating the dep tree to see if a package needs to be
rebuilt, just keep the existing USE configuration unless it conflicts
with explicit configuration or a package dependency.

So, the goal would be to avoid rebuilding stuff just because we can.
However, we would rebuild things if a user wants a flag to be set
differently, or if there is now a dependency problem.


-- 
Rich