Re: [buildd-tools-devel] Bug#843773: Bug#843773: misleading timestamps in binnmus
On Sun, Jan 01, 2017 at 05:40:44PM +0100, Guillem Jover wrote: > The only correct "solution" I see while keeping the current mess, would > be to declare binNMU versions a globally shared resource across all > architectures (in and out of archive!), trigger them globally for all > architectures (or replay them for late comers) As someone who tried to make unofficial jessie for x32, I say: hell yeah, oh so much please do this! Or better yet, just drop that concept altogether and instead make binNMUs automated _sourceful_ uploads. For someone who's trying to simulate testing/stable at home, inconsistent binNMU versions create such a mess that, despite jessie-x32 being mostly done, you didn't hear it announced, I didn't bother to build security updates as planned, it's a bitch to upgrade to unstable, and I'm not trying that again for stretch. So here's another reason for your idea, and why binNMUs are bad. Meow! -- Autotools hint: to do a zx-spectrum build on a pdp11 host, type: ./configure --host=zx-spectrum --build=pdp11
Re: [buildd-tools-devel] Bug#843773: Bug#843773: misleading timestamps in binnmus
[ Had this half-drafted, but had not found the time to finish it up
until now. ]
Hi!
On Mon, 2016-11-14 at 13:52:18 +, Ian Jackson wrote:
> Johannes Schauer writes ("Re: [buildd-tools-devel] Bug#843773: Bug#843773:
> misleading timestamps in binnmus"):
> > Instead, file conflicts might be created from files with
> > content that depends on SOURCE_DATE_EPOCH.
>
> tl;dr:
>Analysis. Revised proposal:
>Introduce BUILD_DATE_EPOCH (= time of sbuild binnmu invocation)
>and use it for file timestamps (only (for now))
Thanks for the analysis. Although I see few problems with it.
* binNMUs are not co-installable anyway, and although I've got code
to make them so on the dpkg side, other parties who have to deal
with dependency resolution are less than enthused on the prospect
of having to cover that new invariant.
* Even if we made binNMUs co-installable, they would contain files
with different metadata, which is currently not considered when
checking for the same-ness of the files, but I think it does
make sense to consider in the future.
* Even if we didn't consider the metadata part of the criteria for
same-ness, it still will need to be stored in the dpkg db, but the
filesystem is a shared resource and only one of the instances can
be present so the divergent metadata will make verification somewhat
worthless, or way more complex to implement or reason about.
* Even if any of the above are ignored/solved somehow, the binNMUs
will contain different metadata, which means you could end up
with timestamps on disk going back and forth in time for the same
content. Say you install one of each on several consecutive days
libfoo_1+b2_arch-a, libfoo_1+b1_arch-b and libfoo_1+b3_arch-c,
which might also upset some backup solutions.
And this would be yet another reason why allowing coexistence of
Multi-Arch:same and binNMUs that are not in lockstep for all
architectures would be a bad idea.
In any case the concept of binNMUs as being pure rebuilds is simply
an illusion. They are sourceful updates where the source is discarded,
and not only is the environment they are built against different, their
contents will be at least different just because of the different
version and different changelog entry.
So, yes, again IMO Multi-Arch:same plus ref-counted files are broken by
design, and they are pretty much incompatible with binNMUs. I stated
this in the monumental multiarch thread here some years ago. There
was rough consensus that this was either not a problem or one where
its benefits outweighted the potential problems. Fine, but then do not
expect miracles out of the current situation we got into. :)
So, I think this has been already implemented in sbuild and pbuilder,
but indeed, my proposal would be for now to ignore all this, and just
emit a changelog entry with the current build date for each binNMU.
When combined with the M-A:same and ref-counted files requirements, this
*is* still conceptually and practically wrong, because binNMUs are not
triggered for all architectures in lockstep, and are not triggered for
the same reasons (a b1_arch-y might not exist for the same reasons a
b1_arch-z).
The only correct "solution" I see while keeping the current mess, would
be to declare binNMU versions a globally shared resource across all
architectures (in and out of archive!), trigger them globally for all
architectures (or replay them for late comers), and use the binNMU
trigger date for the changelog entry (and ideally also the email address
of the person or role who triggered the binNMU, instead of the buildd
doing the build).
(Well the *only* correct solution would be IMO to ban ref-counted files,
but I don't think that will gain much favor this time around either,
although I think I might make it possible to disable it at dpkg build
time for those downstreams that do not want anything to do with this
insanity. :)
Thanks,
Guillem
Re: [buildd-tools-devel] Bug#843773: Bug#843773: misleading timestamps in binnmus
On Thu, Dec 01, 2016 at 04:51:39PM +0100, Johannes Schauer wrote: > Hi, > > Quoting Wouter Verhelst (2016-12-01 16:24:16) > > On Mon, Nov 14, 2016 at 06:10:57PM +0100, Johannes Schauer wrote: > > > But maybe to talk about this option: what would speak against changing the > > > "nmu" command of wanna-build to also add an option that allows setting a > > > timestamp, or even let wanna-build generate that timestamp itself (from > > > the > > > time it processes the "nmu" command) and then pass it to sbuild via a > > > not-yet-existing --binNMU-timestamp option? > > > > Wanna-build has a "State-Change" date: > > > > wouter@wuiet:~$ wanna-build -A powerpc --info nbd > > nbd: > > Package : nbd > > Version : 1:3.14-4 > > Builder : buildd_powerpc-porpora > > State : Installed > > Section : admin > > Priority: source > > Installed-Version : 1:3.14-4 > > Previous-State : Uploaded > > State-Change: 2016-11-21 23:13:18.744533 > > Build-time : 9255 > > CalculatedPri : 50 > > component : main > > Distribution: sid > > Notes : out-of-date > > Old-Failed : 1:2.9.23-1 > > fails test suite > > State-Days : 9 > > State-Time : 835808 > > Success-build-time : 366 > > > > Why not use that? > > I don't know wanna-build but this timestamp seems to be architecture specific > (I see "powerpc" in your paste above)? > > Instead, sbuild should be called with the same input timestamp on all > architectures when an nmu is to be built. Hmm, yes. That doesn't fit then. -- < ron> I mean, the main *practical* problem with C++, is there's like a dozen people in the world who think they really understand all of its rules, and pretty much all of them are just lying to themselves too. -- #debian-devel, OFTC, 2016-02-12
Re: [buildd-tools-devel] Bug#843773: Bug#843773: misleading timestamps in binnmus
Hi, Quoting Wouter Verhelst (2016-12-01 16:24:16) > On Mon, Nov 14, 2016 at 06:10:57PM +0100, Johannes Schauer wrote: > > But maybe to talk about this option: what would speak against changing the > > "nmu" command of wanna-build to also add an option that allows setting a > > timestamp, or even let wanna-build generate that timestamp itself (from the > > time it processes the "nmu" command) and then pass it to sbuild via a > > not-yet-existing --binNMU-timestamp option? > > Wanna-build has a "State-Change" date: > > wouter@wuiet:~$ wanna-build -A powerpc --info nbd > nbd: > Package : nbd > Version : 1:3.14-4 > Builder : buildd_powerpc-porpora > State : Installed > Section : admin > Priority: source > Installed-Version : 1:3.14-4 > Previous-State : Uploaded > State-Change: 2016-11-21 23:13:18.744533 > Build-time : 9255 > CalculatedPri : 50 > component : main > Distribution: sid > Notes : out-of-date > Old-Failed : 1:2.9.23-1 > fails test suite > State-Days : 9 > State-Time : 835808 > Success-build-time : 366 > > Why not use that? I don't know wanna-build but this timestamp seems to be architecture specific (I see "powerpc" in your paste above)? Instead, sbuild should be called with the same input timestamp on all architectures when an nmu is to be built. Thanks! cheers, josch signature.asc Description: signature
Re: [buildd-tools-devel] Bug#843773: Bug#843773: misleading timestamps in binnmus
Hi, (Sorry for piping in so late to the party here) On Mon, Nov 14, 2016 at 06:10:57PM +0100, Johannes Schauer wrote: > But maybe to talk about this option: what would speak against changing the > "nmu" command of wanna-build to also add an option that allows setting a > timestamp, or even let wanna-build generate that timestamp itself (from the > time it processes the "nmu" command) and then pass it to sbuild via a > not-yet-existing --binNMU-timestamp option? Wanna-build has a "State-Change" date: wouter@wuiet:~$ wanna-build -A powerpc --info nbd nbd: Package : nbd Version : 1:3.14-4 Builder : buildd_powerpc-porpora State : Installed Section : admin Priority: source Installed-Version : 1:3.14-4 Previous-State : Uploaded State-Change: 2016-11-21 23:13:18.744533 Build-time : 9255 CalculatedPri : 50 component : main Distribution: sid Notes : out-of-date Old-Failed : 1:2.9.23-1 fails test suite State-Days : 9 State-Time : 835808 Success-build-time : 366 Why not use that? -- < ron> I mean, the main *practical* problem with C++, is there's like a dozen people in the world who think they really understand all of its rules, and pretty much all of them are just lying to themselves too. -- #debian-devel, OFTC, 2016-02-12
Re: [buildd-tools-devel] Bug#843773: Bug#843773: misleading timestamps in binnmus
Ian Jackson wrote: [...] > I'm not a fan of the idea of merely adding 1 second per binnmu. That > would mean that making a second binnmu correctly would involve looking > in the archive to see what the previous binnmu timestamp was. [...] The reference point would be the last source change according to debian/changelog, not the time of the last binNMU before the current one. For the +5 binNMU you add 5 seconds/minutes/whatever to the source entry in debian/changelog. cu Andreas -- `What a good friend you are to him, Dr. Maturin. His other friends are so grateful to you.' `I sew his ears on from time to time, sure'
Re: [buildd-tools-devel] Bug#843773: Bug#843773: misleading timestamps in binnmus
Hi, thanks for having this discussion! On Mon, Nov 14, 2016 at 06:10:57PM +0100, Johannes Schauer wrote: > Quoting Ian Jackson (2016-11-14 17:33:55) > > Can I ask you the converse question: what same-timestamp proposal do > > you think is best and why ? > > I found Guillem's suggestion the most sensible and as far as I understand the > matter also the easiest to implement: > > Quoting Guillem Jover (2016-11-09 00:18:25) > > So the actual problem is that the last timestamp gets reused for the > > binNMUs, which seems totally bogus to me. This needs to be fixed in > > whatever is injecting the binNMU entries on the buildds. > > but that proposal did not get any attention… I'm not sure what his actual proposal was. To me it seems a binNMU should change SOURCE_DATE_EPOCH, as debian/changelog gets modified by changelog.$arch, so it's actually a different source which is being build. What do you think? -- cheers, Holger signature.asc Description: Digital signature
Re: [buildd-tools-devel] Bug#843773: Bug#843773: misleading timestamps in binnmus
Hi, Quoting Ian Jackson (2016-11-14 17:33:55) > Unless the timestamp is of the binnmu request, plumbing to try to get > the same timestamp will be difficult. > > I'm not a fan of the idea of merely adding 1 second per binnmu. That > would mean that making a second binnmu correctly would involve looking > in the archive to see what the previous binnmu timestamp was. It > would also mean that the timestamps would be quite blatant lies: for > example, there would be files claiming to have been generated with > compiler X at time T, where compiler X did not exist at time T. > > If the timestamp is of the binnmu request then I guess it will all > work, but the extra plumbing seems unnecessary. > > > I also don't see why it's a problem that a package might only be > > rebuilt on some architectures. If only some architectures of a > > M-A:same package get a binNMU, then they are not co-installable > > anyways. > > I think you're right that this isn't a problem. > > Can I ask you the converse question: what same-timestamp proposal do > you think is best and why ? I found Guillem's suggestion the most sensible and as far as I understand the matter also the easiest to implement: Quoting Guillem Jover (2016-11-09 00:18:25) > So the actual problem is that the last timestamp gets reused for the > binNMUs, which seems totally bogus to me. This needs to be fixed in > whatever is injecting the binNMU entries on the buildds. but that proposal did not get any attention so I somehow assumed that there's probably a reason not to do so and thus I suggested an alternative: choose the new timestamp by using the binNMU number and add that many number of seconds. A +b17 binNMU would add 17 seconds to the original changelog timestamp. Thus, no archive lookups would be required. But maybe to talk about this option: what would speak against changing the "nmu" command of wanna-build to also add an option that allows setting a timestamp, or even let wanna-build generate that timestamp itself (from the time it processes the "nmu" command) and then pass it to sbuild via a not-yet-existing --binNMU-timestamp option? With that solution we would not have to change anything other than wanna-build and sbuild. And I would take care of the latter. Thanks! cheers, josch signature.asc Description: signature
Re: [buildd-tools-devel] Bug#843773: Bug#843773: misleading timestamps in binnmus
Johannes Schauer writes ("Re: [buildd-tools-devel] Bug#843773: Bug#843773:
misleading timestamps in binnmus"):
> I want to understand why passing the same timestamp to all
> architectures is an inferior solution to your proposal.
This is a sensible question. Thanks for helping to explore all the
issues.
TBH I'm not completely sure that it is, but:
Unless the timestamp is of the binnmu request, plumbing to try to get
the same timestamp will be difficult.
I'm not a fan of the idea of merely adding 1 second per binnmu. That
would mean that making a second binnmu correctly would involve looking
in the archive to see what the previous binnmu timestamp was. It
would also mean that the timestamps would be quite blatant lies: for
example, there would be files claiming to have been generated with
compiler X at time T, where compiler X did not exist at time T.
If the timestamp is of the binnmu request then I guess it will all
work, but the extra plumbing seems unnecessary.
> I also don't see why it's a problem that a package might only be
> rebuilt on some architectures. If only some architectures of a
> M-A:same package get a binNMU, then they are not co-installable
> anyways.
I think you're right that this isn't a problem.
Can I ask you the converse question: what same-timestamp proposal do
you think is best and why ?
Regards,
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Re: [buildd-tools-devel] Bug#843773: Bug#843773: misleading timestamps in binnmus
Hi, Quoting Ian Jackson (2016-11-14 14:52:18) >I don't think it is possible to make the binnmu timestamp the same >across architectures. For example, a package might be rebuilt only >on some architectures. I don't think we want to change that. In >particular, even if we were prepared to change that in Debian, we >should not impose always-binnmu-all-arches as a rule on all our >downstreams. Thank you for your analysis and your proposal. I want to understand why passing the same timestamp to all architectures is an inferior solution to your proposal. I also don't see why it's a problem that a package might only be rebuilt on some architectures. If only some architectures of a M-A:same package get a binNMU, then they are not co-installable anyways. Thanks! cheers, josch signature.asc Description: signature
Re: [buildd-tools-devel] Bug#843773: Bug#843773: misleading timestamps in binnmus
Johannes Schauer writes ("Re: [buildd-tools-devel] Bug#843773: Bug#843773:
misleading timestamps in binnmus"):
> Instead, file conflicts might be created from files with
> content that depends on SOURCE_DATE_EPOCH.
tl;dr:
Analysis. Revised proposal:
Introduce BUILD_DATE_EPOCH (= time of sbuild binnmu invocation)
and use it for file timestamps (only (for now))
We have a difficulty now. There are, for a MA-Same package, two
relevant timestamps:
(i) The time of the original source package changelog. I'm going to
call this the `source timestamp'.
This has to be used for the embedded timestamps of all shared
files in MA-same packages.
Note that it is technically not correct to use the source timestamp
for embedded timestamps of these files. Consider a package which
is binnmu'd on all architectures because a documentation rebuild is
needed. In practice, though, such timestamps are generally either
in hidden places and ignored by almost anything, or displayed to
users in a kind of context where users are already often exposed to
out-of-date timestamps (or where users might already expect the
timestamp to relate to the content of the documentation, rather
than the time it was last reformatted).
It would be tolerable although technically not quite correct, for
the reasons I have just discussed, to use the source timestamp for
embedded timestamps of arch-specific files in MA-Same packages, and
embedded timestamps of all files in non-MA packages.
(ii) The time of the binnmu build (or perhaps the time, when the
binnmu request was generated and the request message chosen etc.)
I will call this the `binnmu timestamp'.
This should be used (at least) for all file timestamps of any
arch-specific files (to avoid the bug I reported at the start of
this thread).
This timestamp can safely be used for _file timestamps_ of all
files, without disturbing reproducibility.
It would be correct to use this timestamp for embedded timestamps
in arch-specific files, since (in the usual case), arch-specific
files will change on such a rebuild. There is no reason not to use
the binnmu timestamp in these cases, except for the effort of
causing this to happen.
It would be correct to use this timestamp for embedded timestamps
in MA-same shared files iff the shared files were (intentionally)
changed by the rebuild. We don't have a mechanism to say, right
now, whether that is the case.
I don't think it is possible to make the binnmu timestamp the same
across architectures. For example, a package might be rebuilt only
on some architectures. I don't think we want to change that. In
particular, even if we were prepared to change that in Debian, we
should not impose always-binnmu-all-arches as a rule on all our
downstreams.
Both of these timestamps are in principle available to dpkg, dh, et
al.
I suggest the following scheme:
* The binnmu timestamp will be set to the current time when the
binnmu changelog entry is generated. (The whole binnmu changelog
entry must in any case be reused when a build is to be reproduced,
so there is no reproducibility hazard here.)
* For file timestamps of generated files, we will use the binnmu
timestamp.
* For all embedded timestamps in shared files, we will use the
source timestamp.
* For embedded timestamps, we will initially use the source
timestamp; but, we will treat the misleading timestamp as a minor
bug and intend, eventually (ie some time next century) arrange for
it to be set to the binnmu timestamp, where possible.
We would implement this as follows:
1. dpkg-buildpackage will be changed to set BUILD_DATE_EPOCH to the
binnmu timestamp, if there is one. It will set it to the source
timestamp otherwise. dpkg-buildpackage will conversely be changed
to set SOURCE_DATE_EPOCH to the _source_ changelog timestamp, even
if there is also a binnmu changelog entry.
2. dpkg-deb will be changed to honour BUILD_DATE_EPOCH instead of
SOURCE_DATE_EPOCH, if it is set, when clamping file timestamps.
3. sbuild should be changed to set the binnmu changelog timestamp to
the answer from time(2), by default.
4. Eventually, if anyone cares, we will teach tools that generate
arch-specific files, and include timestamps, to honour
BUILD_DATE_EPOCH (either by adjusting those tools directly, or by
adjusting the way they are called by dh, or whatever). We could
also arrange for SOURCE_DATE_EPOCH to be set to the binnmu
timestamp iff the package generates no MA-same .debs, or other
crazy things. None of these future changes are essential for our
tools to function correct; they only fix the minor cosmetic issue
of misleading embedded timestamps.
Ian.
;4~
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.
Re: [buildd-tools-devel] Bug#843773: Bug#843773: misleading timestamps in binnmus
Hi, Quoting Ximin Luo (2016-11-10 18:13:00) > Holger Levsen wrote: > > On Thu, Nov 10, 2016 at 08:59:48AM -0200, Johannes Schauer wrote: > > > One solution would be to increase SOURCE_DATE_EPOCH by 1 second for every > > > binNMU to a package. > > > > > > Any other ideas? > > set SOURCE_DATE_EPOCH to the creation time of that changelog.$arch > > entry? > I'm tending towards the latter suggestion because it's simpler. There's no > need to stick to a +1 second scheme etc, and it might mislead people into > thinking they can do calculations with this - such as reversing the original > timestamp of the sourceful-upload. maybe I'm misunderstanding the problem but for the sake getting a better picture, let me write a summary of the conversation so far: Currently, buildds (using sbuild) create binNMU changelog entries by copying the date from the last changelog entry. See for example libghc-yesod-auth-oauth-dev. 16 binNMUs and the same timestamp is used for all of them. In his original post, Ian complains that despite the package version being incremented, the file mtimes are not. This is because we now use the timestamp from the latest changelog entry for the mtimes of the binary package contents so that the binary packages become reproducible. If we want to have later mtimes for the files in binNMU binary packages compared the ones in the last version, then the question becomes: which mechanism should we use to set this timestamp? I do not think that reproducibility is an issue here. No matter which mechanism we choose to determine the timestamp of the new changelog entry, that timestamp will be recorded in the Binary-Only-Changes field of the generated .buildinfo file. The content of that field can then be used by package builders to generate the new debian/changelog which in turn means that at the time that dpkg-buildpackage is run, SOURCE_DATE_EPOCH will be set to a deterministic value: the value of the last changelog entry which we obtain from the Binary-Only-Changes field. Back to the question of how we should determine the initial timestamp for the new binNMU. One way would be to just use the timestamp of the time at which the binNMU changelog entry is created by sbuild on the buildds. The problem with this approach is, that binNMUs will be done at slightly different times on each architecture and thus a different changelog timestamp will be used per architecture. This does not cause Multi-Arch:same file conflicts because of the changelog itself because these are stored in architecture-specific paths for binNMUs (if dh_installchangelogs is used). Instead, file conflicts might be created from files with content that depends on SOURCE_DATE_EPOCH. Because each architecture will generate their own timestamp, SOURCE_DATE_EPOCH will be different on each. If the content of shared files depends on the set timestamp, then these packages will not be co-installable anymore. One solultion to this problem would be to mandate that files shared across Multi-Arch:same binary packages must be independent of SOURCE_DATE_EPOCH but that might impose too much of a burden of writing and then maintaining the appropriate patches for upstreams that refuse to become independent of SOURCE_DATE_EPOCH. Moving the architecture-invariant but SOURCE_DATE_EPOCH dependent files to Architecture:all packages is not a solution either because we might want the packages containing these files be able to transport their architecture to their dependencies. Thus I think it would be best if the changelog timestamp for the same binNMU version would be equal across all architectures. One way to achieve this would be by using a unified scheme to generate the timestamps, for example by incrementing the original timestamp by one second for each binNMU. Another way to achieve this would be to require a timestamp to be supplied (or generated) every time a binNMU is scheduled. That timestamp would then be passed to all buildds and be used by sbuild to generate the new entry in debian/changelog. What do you think? Thanks! cheers, josch signature.asc Description: signature
