Re: [gentoo-portage-dev] Normaliser function for distfiles
On Tue, 17 May 2022 16:35:11 +0200, Michał Górny wrote: > This would require a new EAPI. We don't really want more Portage- > specific hacks that are going to break for everyone not using Portage or > the very specific Portage version. > > I'm not saying that it's not doable but I'm not convinced the added > complexity is really worth the effort, especially given that this looks > like a very special corner case. In the end, fixing Melpa is the right > thing to do. > > For a start, you'd have to ensure that the "normalizer script" (or its > dependencies, if you put it in the repo) is available at the time of > fetching. This pretty much goes back to the problem of "fetch > dependencies", and requires a major design change in Portage that > reduces separation between fetching and installing that we have now. > I mean, right now Portage pretty much assumes that you can do > a `--fetchonly` with no extra packages necessary. > > The "normalizer" wouldn't be trivial either. In the end, we're talking > about getting 100% consistent results on all platforms, over > a reasonably long timeframe. Oh, what a chain of complexity. Yeah, I'll go try fixing melpa first. Markus -- For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much---the wheel, New York, wars and so on---while all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man---for precisely the same reasons. (Douglas Adams, The Hitchhikers Guide to the Galaxy.)
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Mon, 2022-05-16 at 19:37 +0200, Markus Walter wrote: > Hello all, > > is it possible to do the following: after fetching a distfile portage runs > an external normaliser program specified in an ebuild before checking the > hash? > > My use case is the following: I would like to improve the gs-elpa program > and provide a precomputed overlay for melpa. However the melpa distfiles are > rebuilt everyday and cause checksum failures. However the only thing > changing are the timestamps. Hence if a normaliser program could simply set > all timestamps to some predefined value (say 1.1.1970) then this problem > should vanish. > This would require a new EAPI. We don't really want more Portage- specific hacks that are going to break for everyone not using Portage or the very specific Portage version. I'm not saying that it's not doable but I'm not convinced the added complexity is really worth the effort, especially given that this looks like a very special corner case. In the end, fixing Melpa is the right thing to do. For a start, you'd have to ensure that the "normalizer script" (or its dependencies, if you put it in the repo) is available at the time of fetching. This pretty much goes back to the problem of "fetch dependencies", and requires a major design change in Portage that reduces separation between fetching and installing that we have now. I mean, right now Portage pretty much assumes that you can do a `--fetchonly` with no extra packages necessary. The "normalizer" wouldn't be trivial either. In the end, we're talking about getting 100% consistent results on all platforms, over a reasonably long timeframe. -- Best regards, Michał Górny
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Tue, May 17, 2022 at 8:32 AM Markus Walter wrote: > > On Tue, 17 May 2022 14:14:57 +0200, > Rich Freeman wrote: > > > > On Mon, May 16, 2022 at 1:37 PM Markus Walter > > wrote: > > > > > > My use case is the following: I would like to improve the gs-elpa program > > > and provide a precomputed overlay for melpa. However the melpa distfiles > > > are > > > rebuilt everyday and cause checksum failures. However the only thing > > > changing are the timestamps. Hence if a normaliser program could simply > > > set > > > all timestamps to some predefined value (say 1.1.1970) then this problem > > > should vanish. > > > > > > > Wouldn't a simpler solution be to just have an ebuild setting that > > tells the package manager to not check the timestamp? > > The timestamps are inside archive files thus changing the overall file > hash. This happens during distfile download, where some more sophisticated > replace all timestamps function would be necessary than just ignoring one > timestamp. Ah, apologies. Totally missed that. Yeah, obviously if the timestamps INSIDE the archive are changing your only solution is to edit the contents. I thought it was just the external timestamp changing (I'm not sure if portage even checks that, or if wget changes the mtime on files it downloads). -- Rich
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Tue, 17 May 2022 14:14:57 +0200, Rich Freeman wrote: > > On Mon, May 16, 2022 at 1:37 PM Markus Walter > wrote: > > > > My use case is the following: I would like to improve the gs-elpa program > > and provide a precomputed overlay for melpa. However the melpa distfiles are > > rebuilt everyday and cause checksum failures. However the only thing > > changing are the timestamps. Hence if a normaliser program could simply set > > all timestamps to some predefined value (say 1.1.1970) then this problem > > should vanish. > > > > Wouldn't a simpler solution be to just have an ebuild setting that > tells the package manager to not check the timestamp? The timestamps are inside archive files thus changing the overall file hash. This happens during distfile download, where some more sophisticated replace all timestamps function would be necessary than just ignoring one timestamp. Markus -- For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much---the wheel, New York, wars and so on---while all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man---for precisely the same reasons. (Douglas Adams, The Hitchhikers Guide to the Galaxy.)
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Mon, May 16, 2022 at 1:37 PM Markus Walter wrote: > > My use case is the following: I would like to improve the gs-elpa program > and provide a precomputed overlay for melpa. However the melpa distfiles are > rebuilt everyday and cause checksum failures. However the only thing > changing are the timestamps. Hence if a normaliser program could simply set > all timestamps to some predefined value (say 1.1.1970) then this problem > should vanish. > Wouldn't a simpler solution be to just have an ebuild setting that tells the package manager to not check the timestamp? -- Rich
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Tue, 17 May 2022 05:39:27 +0200, Robin H. Johnson wrote: > I don't know what 'gs-elpa' & 'Melpa' are, but maybe talking to upstream > would be good here, and improving that behavior. That's actually not too bad a plan. Thanks for this somewhat obvious suggestion, which I however failed to see. I'll try that. Although reproducibility is not a tenet there, they may take a patch which improves the situation. Melpa is the largest (not totally sure) repository for Emacs add-ons. And gs-elpa is the g-sorcery integration for Gentoo. Markus -- For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much---the wheel, New York, wars and so on---while all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man---for precisely the same reasons. (Douglas Adams, The Hitchhikers Guide to the Galaxy.)
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Tue, 17 May 2022 05:21:02 +0200, Zac Medico wrote: > The only usable hook that we currently have for this is FETCHCOMMAND and > RESUMCOMMAND in make.conf. You can replace them with a script that does the > normal thing and then sets the timestamp. The default values are found in > /usr/share/portage/config/make.globals. Thanks for pointing this out, but I fear this will not easily solve the problem for new users. Markus -- For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much---the wheel, New York, wars and so on---while all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man---for precisely the same reasons. (Douglas Adams, The Hitchhikers Guide to the Galaxy.)
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Mon, May 16, 2022 at 07:37:40PM +0200, Markus Walter wrote: > Hello all, > > is it possible to do the following: after fetching a distfile portage runs > an external normaliser program specified in an ebuild before checking the > hash? > > My use case is the following: I would like to improve the gs-elpa program > and provide a precomputed overlay for melpa. However the melpa distfiles are > rebuilt everyday and cause checksum failures. However the only thing > changing are the timestamps. Hence if a normaliser program could simply set > all timestamps to some predefined value (say 1.1.1970) then this problem > should vanish. I don't know what 'gs-elpa' & 'Melpa' are, but maybe talking to upstream would be good here, and improving that behavior. If the file contents or non-timestamp metadata change, absolutely the timestamps should change. But otherwise, the timestamp should NOT change. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature
Re: [gentoo-portage-dev] Normaliser function for distfiles
On 5/16/22 10:37, Markus Walter wrote: Hello all, is it possible to do the following: after fetching a distfile portage runs an external normaliser program specified in an ebuild before checking the hash? My use case is the following: I would like to improve the gs-elpa program and provide a precomputed overlay for melpa. However the melpa distfiles are rebuilt everyday and cause checksum failures. However the only thing changing are the timestamps. Hence if a normaliser program could simply set all timestamps to some predefined value (say 1.1.1970) then this problem should vanish. Thanks in advance Markus The only usable hook that we currently have for this is FETCHCOMMAND and RESUMCOMMAND in make.conf. You can replace them with a script that does the normal thing and then sets the timestamp. The default values are found in /usr/share/portage/config/make.globals. -- Thanks, Zac OpenPGP_signature Description: OpenPGP digital signature
