Re: Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
On Tue, 19 Jan 2021 at 05:02:29 +0100, Guillem Jover wrote: > On Mon, 2021-01-11 at 00:48:42 +1100, Stuart Prescott wrote: > > SOURCE_BASE_DIR has a delightful symmetry to SOURCE_DATE_EPOCH and the > > algorithm for use is similar: if it is in the environment then use it, if > > not, > > fall back to the previous behaviour. > > Yes! And as this was triggered by reproducible builds, and as you > mention due to the similarities, it would nice to publish a “spec” > once we reach some agreement, given that would end up being part of > a neutral project, with more chances for upstream adoption. One problem that I can see with SOURCE_BASE_DIR (compared with, for example, what GLib does) is: what is "the package"? Many codebases get used both as an independent package in their own right, and as a bundled/vendored subproject in larger projects. Even if *in Debian* we don't like vendoring projects that have their own independent releases, *outside Debian* people do this all the time. There is explicit support for this in at least Autotools and Meson, and it's used both for vendoring subprojects that are released independently and for dealing with subprojects that are logically separate but never actually released separately. For instance, in Debian we think of gcc-10 as a single, monolithic source package, but it's really made up of three upstream tarballs (gcc, gm2 and newlib) containing 35 Autotools build systems (estimated by counting files named configure.ac). Obviously it isn't reasonable to expect dpkg or debhelper to know that they should somehow set a different SOURCE_BASE_DIR when recursing into the content of each upstream tarball, or when recursing into each of those 35 Autotools build systems. Similarly, we think of gnome-shell as a single monolithic package, but as of 3.38.3 it consists of a top-level Meson project with 4 subprojects (that do not have any independent existence as source releases, although I think at least one of them is a non-API-stable "copylib" that exists in more than one GNOME package). We also think of GTK as a single package, but when built in non-Debian contexts, it supports building external dependencies like GLib and Cairo as subprojects instead of using a system copy. In Debian, by policy we don't do this, but people who are building GTK apps for Windows absolutely do want to take this approach, and GTK upstream needs to support both. In the case of a vendorable dependency like GLib, this means that when you're building GLib, SOURCE_BASE_DIR might be set to the top of GLib (when building it as a package in its own right), or it might be set to the top of GTK (in which GLib is a subprojects/glib subdirectory). The thing I would always want to know, as a GLib developer who wants to load a resource into my unit test, is the top of GLib. GLib's G_TEST_SRCDIR/G_TEST_BUILDDIR sidestep this by being defined in terms of the current directory, rather than some top-level package. This means it isn't really feasible for dpkg to implement them - they have to be something that the upstream build system provides, because it's the upstream build system that does the recursion. Constructs like Autotools' $(top_srcdir), $(abs_builddir), etc. and Meson's meson.project_source_root(), meson.current_build_dir() etc. are very suitable for implementing things like G_TEST_SRCDIR/G_TEST_BUILDDIR, and they are defined in ways that make sense for that upstream build system, taking into account how it deals with subprojects. smcv
Re: Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
Hi!
On Mon, 2021-01-11 at 00:48:42 +1100, Stuart Prescott wrote:
> Sune Stolborg Vuorela wrote:
> > Can you provide some kind of hook in the environment so we at least can work
> > around it in the users, so that the internal functions (where the output of
> > __FILE__ is forwarded to) can glue e.g. the content of an environment
> > variable in front of the usage of the __FILE__ content to try to
> > reconstruct it at runtime, so that all packages doesn't need to do export
> > SOURCE_BASE_DIR=`pwd`; make test, but it will just work if we patch Qt and
> > similar libraries ?
Sure, that sounds doable, but as always given that we have not yet
decided to obsolete supporting debian/rules as a build entry point,
any such support would either need to be conditional from the upstream
or packaging side, or something like debhelper would also need to
guarantee it is set up. :/
> SOURCE_BASE_DIR has a delightful symmetry to SOURCE_DATE_EPOCH and the
> algorithm for use is similar: if it is in the environment then use it, if
> not,
> fall back to the previous behaviour.
Yes! And as this was triggered by reproducible builds, and as you
mention due to the similarities, it would nice to publish a “spec”
once we reach some agreement, given that would end up being part of
a neutral project, with more chances for upstream adoption.
> At its crudest level, a package's debian/rules could include SOURCE_BASE_DIR=
> $(shell pwd); however, putting this into the build environment saves source
> changes in multiple packages, sets us up for future use across the entire
> ecosystem, signals to upstreams that this is a broad standard and not a mere
> hack in d/rules, and also saves maintainers from thinking about nasty corner
> cases to do with current directories with makefile invocation.
>
> Looking to the future, tests using SOURCE_BASE_DIR to locate test data would
> allow build-time tests to be repurposed as autopkgtest tests too, as the
> autopkgtest could tell the tests where the test input data is to be found.
> This would be a substantial improvement on the current situation where the
> tests can only be run at build time and not on ci.debian.net.
>
> By dpkg making SOURCE_BASE_DIR 'real', there is a distinct prospect of Debian
> carrying Qt5 patches to use it (thanks Sune!) and for Qt6 to include them
> upstream.
I agree with all of this, barring my initial comment above. I guess I
need to go back to the thread proposing to obsolete debian/rules as a
supported build entry point. :)
I think Simon's remark might be worth having a thought about too. But
as a starting point I prepared the attached patch, which sets the
variable from dpkg-buildpackage and pkg-info.mk. Once there's
agreement on the variable(s), I'm fine merging this, but whether to
include this for bullseye would need release-team approval at this
point I guess.
Thanks,
Guillem
From 33f7213e34dad402d5159b3a82bf2efc9454222f Mon Sep 17 00:00:00 2001
From: Guillem Jover
Date: Sat, 16 Jan 2021 14:23:55 +0100
Subject: [PATCH] dpkg-buildpackage: Add SOURCE_BASE_DIR support
This is a generic variable that can be used by upstream projects wanting
to get at the source root directory, independently of out tree builds,
and in a portable way in contrast to making assumptions about for
example the content of the C __FILE__ preprocessor macro.
Proposed-by: Sune Stolborg Vuorela
---
man/dpkg-buildpackage.pod | 5 +
scripts/dpkg-buildpackage.pl | 3 +++
scripts/mk/pkg-info.mk| 4
scripts/t/dpkg_buildpackage.t | 8
t-func/atlocal.in | 3 +++
5 files changed, 23 insertions(+)
diff --git a/man/dpkg-buildpackage.pod b/man/dpkg-buildpackage.pod
index e648a8063..d6bf58749 100644
--- a/man/dpkg-buildpackage.pod
+++ b/man/dpkg-buildpackage.pod
@@ -668,6 +668,11 @@ B.
This variable is set to the Unix timestamp since the epoch of the
latest entry in I, if it is not already defined.
+=item B
+
+This variable is set to the absolute pathname of the unpacked source
+directory, if it is not already defined (since dpkg 1.20.8).
+
=back
=head1 FILES
diff --git a/scripts/dpkg-buildpackage.pl b/scripts/dpkg-buildpackage.pl
index aacb83118..dbff851ce 100755
--- a/scripts/dpkg-buildpackage.pl
+++ b/scripts/dpkg-buildpackage.pl
@@ -23,6 +23,7 @@
use strict;
use warnings;
+use Cwd;
use File::Temp qw(tempdir);
use File::Basename;
use File::Copy;
@@ -460,6 +461,8 @@ if ($changedby) {
#
Re: Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
On Mon, 11 Jan 2021 at 00:48:42 +1100, Stuart Prescott wrote: > Looking to the future, tests using SOURCE_BASE_DIR to locate test data would > allow build-time tests to be repurposed as autopkgtest tests too, as the > autopkgtest could tell the tests where the test input data is to be found. Prior art that would be worth looking at: GLib's test infrastructure uses $G_TEST_SRCDIR and $G_TEST_BUILDDIR. These are usually used via a function g_test_build_filename() whose first argument is a constant (enum value), either G_TEST_DIST or G_TEST_BUILT, indicating whether the test data is found in the source directory or the build directory. If $G_TEST_SRCDIR and $G_TEST_BUILDDIR aren't set, g_test_build_filename() defaults both to the directory where the currently-running test executable was found (by looking at argv[0], /proc/self/exe or similar), which is enough to make it unnecessary to set those variables for autopkgtests in at least some cases. If you intend to propose this as a feature in Qt or other libraries, it's probably best to distinguish between the equivalent of G_TEST_SRCDIR and the equivalent of G_TEST_BUILDDIR, so that tests in srcdir != builddir builds can use test data and helper scripts that are distributed in git/tarballs (GLib code would use G_TEST_DIST), and also use test data and helper executables that are generated by the build (GLib code would use G_TEST_BUILT). $G_TEST_SRCDIR and $G_TEST_BUILDDIR are usually defined in terms of the directory containing the test (so typically you'd have something like G_TEST_SRCDIR=$HOME/src/glib/gobject/tests and G_TEST_BUILDDIR=$HOME/src/glib/_build/gobject/tests for tests in the gobject/tests subdirectory of a srcdir != builddir build), but it would also be reasonable to have a pair of variables that is defined in terms of the root directory, like the abs_top_srcdir=$HOME/src/glib and abs_top_builddir=$HOME/src/glib/_build defined by Autotools. In the non-GLib world, dbus has a similar setup, although instead of separating the two sets of test data, it copies test files from the equivalent of $G_TEST_SRCDIR into the equivalent of $G_TEST_BUILDDIR, so that the tests can look in one location for everything. smcv
Re: Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
Sune Stolborg Vuorela wrote: > Can you provide some kind of hook in the environment so we at least can work > around it in the users, so that the internal functions (where the output of > __FILE__ is forwarded to) can glue e.g. the content of an environment > variable in front of the usage of the __FILE__ content to try to > reconstruct it at runtime, so that all packages doesn't need to do export > SOURCE_BASE_DIR=`pwd`; make test, but it will just work if we patch Qt and > similar libraries ? +1 for the request for the build system to provide for a reference point for where the source is to be found. SOURCE_BASE_DIR has a delightful symmetry to SOURCE_DATE_EPOCH and the algorithm for use is similar: if it is in the environment then use it, if not, fall back to the previous behaviour. At its crudest level, a package's debian/rules could include SOURCE_BASE_DIR= $(shell pwd); however, putting this into the build environment saves source changes in multiple packages, sets us up for future use across the entire ecosystem, signals to upstreams that this is a broad standard and not a mere hack in d/rules, and also saves maintainers from thinking about nasty corner cases to do with current directories with makefile invocation. Looking to the future, tests using SOURCE_BASE_DIR to locate test data would allow build-time tests to be repurposed as autopkgtest tests too, as the autopkgtest could tell the tests where the test input data is to be found. This would be a substantial improvement on the current situation where the tests can only be run at build time and not on ci.debian.net. By dpkg making SOURCE_BASE_DIR 'real', there is a distinct prospect of Debian carrying Qt5 patches to use it (thanks Sune!) and for Qt6 to include them upstream. regards Stuart -- Stuart Prescotthttp://www.nanonanonano.net/ stu...@nanonanonano.net Debian Developer http://www.debian.org/ stu...@debian.org GPG fingerprint90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
On Saturday, January 9, 2021 12:49:50 AM CET Guillem Jover wrote: > Disabling this option in these few places gives you room to possibly > look for a fix, or not, w/o the pressure of the freeze, while not > affecting the other packages. Hi Guillem Can you provide some kind of hook in the environment so we at least can work around it in the users, so that the internal functions (where the output of __FILE__ is forwarded to) can glue e.g. the content of an environment variable in front of the usage of the __FILE__ content to try to reconstruct it at runtime, so that all packages doesn't need to do export SOURCE_BASE_DIR=`pwd`; make test, but it will just work if we patch Qt and similar libraries ? /Sune -- I didn’t stop pretending when I became an adult, it’s just that when I was a kid I was pretending that I fit into the rules and structures of this world. And now that I’m an adult, I pretend that those rules and structures exist. - zefrank
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
On 2021-01-09, Lisandro Damián Nicanor Pérez Meyer wrote: > Note: in case we do not agree on this topic this will be the text I'll > send to the > tech-ctte. Thanks for taking the time to draft some text. If we can come closer to agreement on the proposed text, that would probably take a bit of the load off of the tech-ctte. Hopefully some of my comments move in that direction, although I also have added some counterpoint text as well... > Let me start by noting that I have nothing against reproducibility. In fact > quite the opposite: I love the idea... as long as it's properly implemented. It is even "recommended" in Debian Policy policy to build reproducibly with different build paths. > The problem here is that __FILE__ is a public, well defined API with very > valid > use cases (more on this below), even if the current implementation of all the > major compilers go against reproducibility. At least two major compilers, GCC and Clang provide the -ffile-prefix-map feature to do exactly that(which is what dpkg is enabling), so it seems a bit overstated to say that all major compilers "go against" reproducibility. They do not enable reproducibility by default. > So if we want to mangle __FILE__ > (thus breaking API) this should be done by an opt-in, and **never** by opting > out. Else we risk breaking a valid implementation, be it already on the > archive > any new package added afterwards. While I understand that may be how you feel, it would be appreciated if we could use something a little less loaded than "mangle". perhaps: So if we want to enable features using __FILE__ in a way which arguably breaks API > Even more: we library maintainers continuously ask our upstreams to keep their > API as stable as possible, and if not possible following some rules > (SONAME change, for example). I will present an option to do the same > ourselves > without breaking API/ABI. > > # __FILE__ is a public, well defined API > > During the course of #876901 many reasons were used to both say that __FILE__ > is or not a well defined API. In fact one of the evidences used where the > compiler's documentation. For example > https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html > (enphasis mine): > > This macro expands to the name of the current input file, in the form of a C > string constant. **This is the path by which the preprocessor opened > the file**, > not the short name specified in ‘#include’ or as the input file name > argument. > For example, "/usr/local/include/myheader.h" is a possible expansion of this > macro. > > This definition says that it's up to the preprocessor to define the path. So, > what has been the behaviour of all major compilers during all these years? > Using > the full path. The proof of this is Qt itself. Check > https://sources.debian.org/src/qtbase-opensource-src/5.15.2+dfsg-2/src/testlib/qtestcase.h/#L216 > > This code has been working on **every** platform Qt works without any change. > Qt is compiled in a myriad of OSs, using a myriad of compilers. They all do > the exact same thing. So developers depend on a very stable definition > of an API. > > And we all know that breaking API is bad, except if done carefully (read > below). > > # Doing the right thing > > This is just an idea of "doing the right thing", like bumping SONAME > on a library. > It definitely doesn't have to be the only one. I understand your position is fundamentally about "Doing the right thing" but I would say that we are all trying to do the right thing. > I understand that the reproducibility people do not want to consider > providing the > same build path. While this is arguable I do understand that > reproducibility without > depending on the build path is a good thing. So I've tried to come up with a > path to achieve this. > ## New macro and warning (if they do not exist already) > > This would be the first step. The idea is to provide a new macro that, > by definition > and documentation, it can be mangled with the help of the build > system, much as you > are currently doing with __FILE__ now. > > The second step in this is to make compilers create a reproducibility warning > if > some code uses __FILE__. This will have three effects effects: We haven't proposed an alternate macro, which would surely take years at best, possibly decades. That doesn't seem too realistic. > - discouraging it's use > - creating awareness on reproducibility. We've discouraged the use of __FILE__ for years, have done plenty of outreach on the subject of reproducibility, and gotten traction with various upstream projects. > - making other distros jump into reproducibility in a much easier way. Arguably some distros (e.g. archlinux) are passing us by when it comes to real-world reproducibility; I'm not sure Debian is the example by which all other distros should be measured anymore. Which is good in some ways, but somewhat disappointing to see Debian start something
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
Hi (again)! On Sat, 9 Jan 2021 at 17:53, Vagrant Cascadian wrote: > > On 2021-01-09, Lisandro Damián Nicanor Pérez Meyer wrote: > > Oh, I have sadly forgotten to mention another thing. > > > > On Sat, 9 Jan 2021 at 15:53, Lisandro Damián Nicanor Pérez Meyer > > wrote: > >> # __FILE__ is a public, well defined API > > > > According to: > > Adrian Bunks mentions it in > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876901#10 > > Holger Levsen in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876901#42 > > Mattia Rizzolo on > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876901#192 > > > > The ability of gcc to change __FILE__ was a patch that, at the time of > > those writings, wasn't yet accepted. > > That is no longer the case. The fixfilepath feature enabled in dpkg only > uses features (e.g. -ffile-prefix-map) available in both upstream GCC > (>=9? or 8? ~2018) and Clang (>= 10), possibly other compilers as > well. There are no special patches to toolchains needed to enable this > feature. This is actually good to read! > > Even more, Ximion Luo wrote on > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876901#212 > > > > The GCC patch (neither the previous nor the planned version) does > > not change the default behaviour of __FILE__, and was never intended > > to. Instead, it gives users the ability to rewrite __FILE__, more > > specifically a prefix of it. > > > > makes it clear that the default behaviour is not changed. So if the > > patch got accepted (did it get accepted?) it was only to allow > > reproducible people to break API in order to get reproducibility. > > Since then an alternate implementation was implemented that the > reproducible=+fixfilepath feature in dpkg takes advantage of, in order > to implement this feature in another distribution, if I recall > correctly. > > It didn't address all the cases of the old GCC patches that Ximin > submitted, but the Reproducible Builds team decided in 2018 to make use > of upstream supported features only and dropped all of our patches to > GCC. > > The notable difference is that it not longer makes use of an environment > variable; it requires passing the -ffile-prefix-map option to the > compiler. The dpkg patch simply adds this to the default relevent *FLAGS > variables. > > (For historical completeness, though somewhat an aside to the topic at > hand, the -ffile-prefix-map approach does not address all the cases of > the former patches to GCC as the compiler flags sometimes end up in > various shipped artifacts in *some* cases, though certainly not all.) Again, good to read. > > This in itself, if something has not changed in the meantime, marks > > another reference that __FILE__ is a public, well defined API. > > I think reading #876901 demonstrates that the case can be made either > way; it not as well defined as you make it out to be. As stated in my previous mail I was wrong here, as demonstrated by Mattias. I even just updated the base Qt bug accordingly. Thanks for being so informative too! -- Lisandro Damián Nicanor Pérez Meyer http://perezmeyer.com.ar/ http://perezmeyer.blogspot.com/
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
On 2021-01-09, Lisandro Damián Nicanor Pérez Meyer wrote: > Oh, I have sadly forgotten to mention another thing. > > On Sat, 9 Jan 2021 at 15:53, Lisandro Damián Nicanor Pérez Meyer > wrote: >> # __FILE__ is a public, well defined API > > According to: > Adrian Bunks mentions it in > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876901#10 > Holger Levsen in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876901#42 > Mattia Rizzolo on https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876901#192 > > The ability of gcc to change __FILE__ was a patch that, at the time of > those writings, wasn't yet accepted. That is no longer the case. The fixfilepath feature enabled in dpkg only uses features (e.g. -ffile-prefix-map) available in both upstream GCC (>=9? or 8? ~2018) and Clang (>= 10), possibly other compilers as well. There are no special patches to toolchains needed to enable this feature. > Even more, Ximion Luo wrote on > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876901#212 > > The GCC patch (neither the previous nor the planned version) does > not change the default behaviour of __FILE__, and was never intended > to. Instead, it gives users the ability to rewrite __FILE__, more > specifically a prefix of it. > > makes it clear that the default behaviour is not changed. So if the > patch got accepted (did it get accepted?) it was only to allow > reproducible people to break API in order to get reproducibility. Since then an alternate implementation was implemented that the reproducible=+fixfilepath feature in dpkg takes advantage of, in order to implement this feature in another distribution, if I recall correctly. It didn't address all the cases of the old GCC patches that Ximin submitted, but the Reproducible Builds team decided in 2018 to make use of upstream supported features only and dropped all of our patches to GCC. The notable difference is that it not longer makes use of an environment variable; it requires passing the -ffile-prefix-map option to the compiler. The dpkg patch simply adds this to the default relevent *FLAGS variables. (For historical completeness, though somewhat an aside to the topic at hand, the -ffile-prefix-map approach does not address all the cases of the former patches to GCC as the compiler flags sometimes end up in various shipped artifacts in *some* cases, though certainly not all.) > This in itself, if something has not changed in the meantime, marks > another reference that __FILE__ is a public, well defined API. I think reading #876901 demonstrates that the case can be made either way; it not as well defined as you make it out to be. live well, vagrant signature.asc Description: PGP signature
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
Hi! On Sat, 9 Jan 2021 at 16:39, Samuel Thibault wrote: > > Lisandro Damián Nicanor Pérez Meyer, le sam. 09 janv. 2021 15:53:41 -0300, a > ecrit: > > # __FILE__ is a public, well defined API > > ? My copy of C11 says > > “ > __FILE__ The presumed name of the current source file (a character string > literal) > ” > > that's not so well-defined. I would not expect it to necessarily > contain the path to it. And the same goes for C99, so I was wrong at this point. Again, sorry for the noise.
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
Hi!
On Sat, 9 Jan 2021 at 16:52, Mattia Rizzolo wrote:
>
> On Sat, Jan 09, 2021 at 08:37:48PM +0100, Samuel Thibault wrote:
> > Lisandro Damián Nicanor Pérez Meyer, le sam. 09 janv. 2021 15:53:41 -0300,
> > a ecrit:
> > > # __FILE__ is a public, well defined API
> >
> > ? My copy of C11 says
> >
> > “
> > __FILE__ The presumed name of the current source file (a character string
> > literal)
> > ”
> >
> > that's not so well-defined. I would not expect it to necessarily
> > contain the path to it.
>
> In fact, empirically I've seen that __FILE__ is expanded to whatever
> path gets passed to the compiler.
> You can easily see how stuff build with cmake get a full path in there,
> whereas stuff built with make gets a path relative to the Makefile.
Mattias has just showed me that in fact __FILE__ is too malleable:
mattia@warren /tmp/tmp % cat test.c
#include
int main() {
printf("%s\n", __FILE__);
}
mattia@warren /tmp/tmp % gcc test.c ; ./a.out
test.c
mattia@warren /tmp/tmp % gcc ./test.c ; ./a.out
./test.c
mattia@warren /tmp/tmp % gcc /tmp/tmp/./test.c ; ./a.out
/tmp/tmp/./test.c
So yes, the cornerstone of my assumption was totally wrong. My
apologies for all the noise, all I can say is: today I learned
something new.
--
Lisandro Damián Nicanor Pérez Meyer
http://perezmeyer.com.ar/
http://perezmeyer.blogspot.com/
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
On Sat, Jan 09, 2021 at 08:37:48PM +0100, Samuel Thibault wrote: > Lisandro Damián Nicanor Pérez Meyer, le sam. 09 janv. 2021 15:53:41 -0300, a > ecrit: > > # __FILE__ is a public, well defined API > > ? My copy of C11 says > > “ > __FILE__ The presumed name of the current source file (a character string > literal) > ” > > that's not so well-defined. I would not expect it to necessarily > contain the path to it. In fact, empirically I've seen that __FILE__ is expanded to whatever path gets passed to the compiler. You can easily see how stuff build with cmake get a full path in there, whereas stuff built with make gets a path relative to the Makefile. -- regards, Mattia Rizzolo GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`. More about me: https://mapreri.org : :' : Launchpad user: https://launchpad.net/~mapreri `. `'` Debian QA page: https://qa.debian.org/developer.php?login=mattia `- signature.asc Description: PGP signature
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
Lisandro Damián Nicanor Pérez Meyer, le sam. 09 janv. 2021 15:53:41 -0300, a ecrit: > # __FILE__ is a public, well defined API ? My copy of C11 says “ __FILE__ The presumed name of the current source file (a character string literal) ” that's not so well-defined. I would not expect it to necessarily contain the path to it. > ## New macro and warning (if they do not exist already) > > This would be the first step. That would not work long-term-wise. One of the issues with __FILE__ is that it's used by assert(), and thus references to __FILE__ are popping up in various software (that is the largest __FILE__usage I have seen in my packages). And that's written explicitly in C11 about assert: “ the latter are respectively the values of the preprocessing macros __FILE__ ” so we can't make assert() use something else than __FILE__, so something else than __FILE__ cannot provide large reproducibility. Samuel
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
Oh, I have sadly forgotten to mention another thing. On Sat, 9 Jan 2021 at 15:53, Lisandro Damián Nicanor Pérez Meyer wrote: > # __FILE__ is a public, well defined API According to: Adrian Bunks mentions it in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876901#10 Holger Levsen in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876901#42 Mattia Rizzolo on https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876901#192 The ability of gcc to change __FILE__ was a patch that, at the time of those writings, wasn't yet accepted. Even more, Ximion Luo wrote on https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876901#212 The GCC patch (neither the previous nor the planned version) does not change the default behaviour of __FILE__, and was never intended to. Instead, it gives users the ability to rewrite __FILE__, more specifically a prefix of it. makes it clear that the default behaviour is not changed. So if the patch got accepted (did it get accepted?) it was only to allow reproducible people to break API in order to get reproducibility. This in itself, if something has not changed in the meantime, marks another reference that __FILE__ is a public, well defined API.
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
Note: in case we do not agree on this topic this will be the text I'll send to the tech-ctte. Let me start by noting that I have nothing against reproducibility. In fact quite the opposite: I love the idea... as long as it's properly implemented. The problem here is that __FILE__ is a public, well defined API with very valid use cases (more on this below), even if the current implementation of all the major compilers go against reproducibility. So if we want to mangle __FILE__ (thus breaking API) this should be done by an opt-in, and **never** by opting out. Else we risk breaking a valid implementation, be it already on the archive any new package added afterwards. Even more: we library maintainers continuously ask our upstreams to keep their API as stable as possible, and if not possible following some rules (SONAME change, for example). I will present an option to do the same ourselves without breaking API/ABI. # __FILE__ is a public, well defined API During the course of #876901 many reasons were used to both say that __FILE__ is or not a well defined API. In fact one of the evidences used where the compiler's documentation. For example https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html (enphasis mine): This macro expands to the name of the current input file, in the form of a C string constant. **This is the path by which the preprocessor opened the file**, not the short name specified in ‘#include’ or as the input file name argument. For example, "/usr/local/include/myheader.h" is a possible expansion of this macro. This definition says that it's up to the preprocessor to define the path. So, what has been the behaviour of all major compilers during all these years? Using the full path. The proof of this is Qt itself. Check https://sources.debian.org/src/qtbase-opensource-src/5.15.2+dfsg-2/src/testlib/qtestcase.h/#L216 This code has been working on **every** platform Qt works without any change. Qt is compiled in a myriad of OSs, using a myriad of compilers. They all do the exact same thing. So developers depend on a very stable definition of an API. And we all know that breaking API is bad, except if done carefully (read below). # Doing the right thing This is just an idea of "doing the right thing", like bumping SONAME on a library. It definitely doesn't have to be the only one. I understand that the reproducibility people do not want to consider providing the same build path. While this is arguable I do understand that reproducibility without depending on the build path is a good thing. So I've tried to come up with a path to achieve this. ## New macro and warning (if they do not exist already) This would be the first step. The idea is to provide a new macro that, by definition and documentation, it can be mangled with the help of the build system, much as you are currently doing with __FILE__ now. The second step in this is to make compilers create a reproducibility warning if some code uses __FILE__. This will have three effects effects: - discouraging it's use - creating awareness on reproducibility. - making other distros jump into reproducibility in a much easier way. ## Mangling by opting-in, or what to do in the meantime Of course we want to make things reproducible as fast as possible too. This is why mangling __FILE__ was considered and implemented. But this is breaking API, no matter how you look at it. But that doesn't means it can not be used: if a package can be built mangling __FILE__ in order to get a reproducible way then the maintainer should really consider opting *in* for the change, but never ever should the maintainer be forced to opt *out* using a broken API. In fact I consider all the "fixes" of opting in a bug, and forcing this change a couple of weeks before freeze even worse. So I propose you to undo the change in dpkg and, after the release, starting to work on letting maintainers opting in. Regards, Lisandro. -- Lisandro Damián Nicanor Pérez Meyer http://perezmeyer.com.ar/ http://perezmeyer.blogspot.com/
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
On 2021-01-08, Lisandro Damián Nicanor Pérez Meyer wrote: > On Fri, 8 Jan 2021 at 21:15, Lisandro Damián Nicanor Pérez Meyer > wrote: > In fact most of those packages would not be unreproducible if the > environment would be the same as the original build. That includes the > build path. True, that is a fairly simple workaround. Which is why we do not vary the build path when testing bullseye for tests.reproducible-builds.org. But we do vary build paths when testing experimental and unstable on tests.reproducible-builds.org, as it helps identify cases where build paths are an issue. It would also help greatly as we move towards verification builds of packages in the archive to not have to worry about build paths as much. > I do understand that it is *desirable* to be able to reproducibly > build a package no matter the build path, but that's just desirable. According to Debian Policy it is recommended: "It is recommended that packages produce bit-for-bit identical binaries even if most environment variables and build paths are varied. It is intended for this stricter standard to replace the above when it is easier for packages to meet it." > The real fix here is to do the right thing, ie, provide the very same > environment as the original build, including the build path. That does sound like a workaround more than a real fix. > So again, mangling __FILE__ should not be a default. I'll agree to disagree. I will admit that a change of defaults in dpkg this close to freeze does seem a bit on the late side of things. Still, The process has been going on for months, announced in accordance with the process for getting defaults changes into dpkg. Bugs with trivial patches were filed in October. Unfortunately, most of the affected packages seem to disproportionately affect packages maintained by the KDE team. I did what I could to mitigate that impact by actually building each and every one of the affected packages to ensure that the opt-out patches worked correctly. Most of those have been applied already. live well, vagrant signature.asc Description: PGP signature
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
On Fri, 8 Jan 2021 at 21:15, Lisandro Damián Nicanor Pérez Meyer wrote: > > Hi! Explicitely CCing my bug, since it seems it missed my previous reply. > > On Fri, 8 Jan 2021 at 20:49, Guillem Jover wrote: > > > > On Fri, 2021-01-08 at 19:23:13 -0300, Lisandro Damián Nicanor Pérez Meyer > > wrote: > > > [snip] > > > > We did a full archive rebuild testing this change, and I provided > > > > patches to all known affected packages several months ago. It is a > > > > one-line change in debian/rules in most cases: > > > > > > > > > > > > https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=reproducible-builds%40lists.alioth.debian.org=fixfilepath > > > > > > > > It seems there are less than 10 packages left that have not applied the > > > > patch. > > > > > > > > Longer-term, it would be nice to be able to fix QFINDTESTDATA to be > > > > compatible, sure. > > > > > > >From a couple of "fixes": > > > > > > -export DEB_BUILD_MAINT_OPTIONS = hardening=+all > > > +# Disable fixfilepath feature, as it triggers build failures when > > > +# enabled. > > > +export DEB_BUILD_MAINT_OPTIONS = hardening=+all reproducible=-fixfilepath > > > > > > That's not a fix but hiding the dirt under the carpet. You are not > > > fixing the root issue nor the reproducibility one. > > > > I'm not sure I understand this objection. Reverting the patch from > > dpkg would do the same but at a global scale, which would make many > > packages that would benefit from the new default, not reproducible, > > and would still "hide the dirt under the carpet" for the very few > > that would otherwise need the option disabled. In fact most of those packages would not be unreproducible if the environment would be the same as the original build. That includes the build path. I do understand that it is *desirable* to be able to reproducibly build a package no matter the build path, but that's just desirable. The real fix here is to do the right thing, ie, provide the very same environment as the original build, including the build path. So again, mangling __FILE__ should not be a default. -- Lisandro Damián Nicanor Pérez Meyer http://perezmeyer.com.ar/ http://perezmeyer.blogspot.com/
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
Hi! Explicitely CCing my bug, since it seems it missed my previous reply. On Fri, 8 Jan 2021 at 20:49, Guillem Jover wrote: > > On Fri, 2021-01-08 at 19:23:13 -0300, Lisandro Damián Nicanor Pérez Meyer > wrote: > > [snip] > > > We did a full archive rebuild testing this change, and I provided > > > patches to all known affected packages several months ago. It is a > > > one-line change in debian/rules in most cases: > > > > > > > > > https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=reproducible-builds%40lists.alioth.debian.org=fixfilepath > > > > > > It seems there are less than 10 packages left that have not applied the > > > patch. > > > > > > Longer-term, it would be nice to be able to fix QFINDTESTDATA to be > > > compatible, sure. > > > > >From a couple of "fixes": > > > > -export DEB_BUILD_MAINT_OPTIONS = hardening=+all > > +# Disable fixfilepath feature, as it triggers build failures when > > +# enabled. > > +export DEB_BUILD_MAINT_OPTIONS = hardening=+all reproducible=-fixfilepath > > > > That's not a fix but hiding the dirt under the carpet. You are not > > fixing the root issue nor the reproducibility one. > > I'm not sure I understand this objection. Reverting the patch from > dpkg would do the same but at a global scale, which would make many > packages that would benefit from the new default, not reproducible, > and would still "hide the dirt under the carpet" for the very few > that would otherwise need the option disabled. __FILE__ has defined behaviour: the path the preprocessor finds. The only place where __FILE__ is mangled, to the best of my knowledge, is Debian. So expecting that any developer out there makes code that runs with this Debian specific variation is unreasonable. In other words: breaking perfectly valid code *by default*, even in the name of reproducibility, is not right. > > Disabling this option in these few places gives you room to possibly > look for a fix, or not, w/o the pressure of the freeze, while not > affecting the other packages. But still breaking perfectly valid code. > As stated in the thread in d-d, I still fail to see the weight of the > objections, TBH. And given this I'm planning on closing the bug in > dpkg. In that case we at least should agree that we disagree, so I would ask the tech-ctte to review this case. -- Lisandro Damián Nicanor Pérez Meyer http://perezmeyer.com.ar/ http://perezmeyer.blogspot.com/
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
On Fri, 2021-01-08 at 19:23:13 -0300, Lisandro Damián Nicanor Pérez Meyer wrote: > [snip] > > We did a full archive rebuild testing this change, and I provided > > patches to all known affected packages several months ago. It is a > > one-line change in debian/rules in most cases: > > > > > > https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=reproducible-builds%40lists.alioth.debian.org=fixfilepath > > > > It seems there are less than 10 packages left that have not applied the > > patch. > > > > Longer-term, it would be nice to be able to fix QFINDTESTDATA to be > > compatible, sure. > > >From a couple of "fixes": > > -export DEB_BUILD_MAINT_OPTIONS = hardening=+all > +# Disable fixfilepath feature, as it triggers build failures when > +# enabled. > +export DEB_BUILD_MAINT_OPTIONS = hardening=+all reproducible=-fixfilepath > > That's not a fix but hiding the dirt under the carpet. You are not > fixing the root issue nor the reproducibility one. I'm not sure I understand this objection. Reverting the patch from dpkg would do the same but at a global scale, which would make many packages that would benefit from the new default, not reproducible, and would still "hide the dirt under the carpet" for the very few that would otherwise need the option disabled. Disabling this option in these few places gives you room to possibly look for a fix, or not, w/o the pressure of the freeze, while not affecting the other packages. As stated in the thread in d-d, I still fail to see the weight of the objections, TBH. And given this I'm planning on closing the bug in dpkg. Thanks, Guillem
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
The code that relies on __FILE__ can be found at https://sources.debian.org/src/qtbase-opensource-src/5.15.2+dfsg-2/src/testlib/qtestcase.h/#L216 So maybe we can try and get a fix there and push this change in the next stable release, after appropriately tested. -- Lisandro Damián Nicanor Pérez Meyer http://perezmeyer.com.ar/ http://perezmeyer.blogspot.com/
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
Hi! [snip] > > We did a full archive rebuild testing this change, and I provided > patches to all known affected packages several months ago. It is a > one-line change in debian/rules in most cases: > > > https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=reproducible-builds%40lists.alioth.debian.org=fixfilepath > > It seems there are less than 10 packages left that have not applied the > patch. > > Longer-term, it would be nice to be able to fix QFINDTESTDATA to be > compatible, sure. >From a couple of "fixes": -export DEB_BUILD_MAINT_OPTIONS = hardening=+all +# Disable fixfilepath feature, as it triggers build failures when +# enabled. +export DEB_BUILD_MAINT_OPTIONS = hardening=+all reproducible=-fixfilepath That's not a fix but hiding the dirt under the carpet. You are not fixing the root issue nor the reproducibility one. -- Lisandro Damián Nicanor Pérez Meyer http://perezmeyer.com.ar/ http://perezmeyer.blogspot.com/
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
On 2021-01-08, Lisandro Damián Nicanor Pérez Meyer wrote: > On Fri, 13 Nov 2020 at 17:40, Vagrant Cascadian > wrote: >> >> On 2020-11-13, Sune Vuorela wrote: >> > On 2020-10-27, Vagrant Cascadian wrote: >> >> Though, of course, identifying the exact reproducibility problem would >> >> be preferable. One of the common issues is test suites relying on the >> >> behavior of __FILE__ returning a full path to find fixtures or other >> >> test data. >> > >> > has QFIND_TESTDATA been adapted to work with this, or are we just >> > "lucky" that most packages don't actually build and run test suites? >> >> Yes, QFINDTESTDATA is one of the primary (only?) issues with test suites >> found in about 20-30 packages in the archive, according to the >> archive-wide rebuild that was performed. For most of those packages >> patches have been submitted, and some are already either fixed or marked >> as pending. > > But QFINDTESTDATA is using __FILE__ in a valid way. It might not be > what you are expecting, but still a valid usage. > > See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876901 We have > discussed this before. > > >> If it could be fixed at the core for QFINDTESTDATA, that would be nicer >> than fixing 20-30 packages individually, though we're not there right >> now. > > In that case I would expect a valid patch from the people interested > in enabling this. In the meantime the dpkg change broke a very valid > usage. Inconvenient for reproducibility? yes, probably, but still very > valid. We did a full archive rebuild testing this change, and I provided patches to all known affected packages several months ago. It is a one-line change in debian/rules in most cases: https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=reproducible-builds%40lists.alioth.debian.org=fixfilepath It seems there are less than 10 packages left that have not applied the patch. Longer-term, it would be nice to be able to fix QFINDTESTDATA to be compatible, sure. live well, vagrant signature.asc Description: PGP signature
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
On Fri, 8 Jan 2021 at 10:29, Lisandro Damián Nicanor Pérez Meyer wrote: > > Hi! Sorry for this late reply, but it just came into my view. > > On Fri, 13 Nov 2020 at 17:40, Vagrant Cascadian > wrote: > > > > On 2020-11-13, Sune Vuorela wrote: > > > On 2020-10-27, Vagrant Cascadian wrote: > > >> Though, of course, identifying the exact reproducibility problem would > > >> be preferable. One of the common issues is test suites relying on the > > >> behavior of __FILE__ returning a full path to find fixtures or other > > >> test data. > > > > > > has QFIND_TESTDATA been adapted to work with this, or are we just > > > "lucky" that most packages don't actually build and run test suites? > > > > Yes, QFINDTESTDATA is one of the primary (only?) issues with test suites > > found in about 20-30 packages in the archive, according to the > > archive-wide rebuild that was performed. For most of those packages > > patches have been submitted, and some are already either fixed or marked > > as pending. > > But QFINDTESTDATA is using __FILE__ in a valid way. It might not be > what you are expecting, but still a valid usage. > > See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876901 We have > discussed this before. > > > > If it could be fixed at the core for QFINDTESTDATA, that would be nicer > > than fixing 20-30 packages individually, though we're not there right > > now. > > In that case I would expect a valid patch from the people interested > in enabling this. In the meantime the dpkg change broke a very valid > usage. Inconvenient for reproducibility? yes, probably, but still very > valid. > > Oh, and changing this just before freeze is also something not > acceptable considering I have seen no valid path proposed for > QFINDTESTDATA yet. You are breaking something that it's expected to > work. > > So please revert the change in dpkg. After this release we can > definitely wait for someone producing a patch for QFINDTESTDATA, test > it and go ahead wth the change. I have just filed #979570 for this. -- Lisandro Damián Nicanor Pérez Meyer http://perezmeyer.com.ar/ http://perezmeyer.blogspot.com/
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
Hi! Sorry for this late reply, but it just came into my view. On Fri, 13 Nov 2020 at 17:40, Vagrant Cascadian wrote: > > On 2020-11-13, Sune Vuorela wrote: > > On 2020-10-27, Vagrant Cascadian wrote: > >> Though, of course, identifying the exact reproducibility problem would > >> be preferable. One of the common issues is test suites relying on the > >> behavior of __FILE__ returning a full path to find fixtures or other > >> test data. > > > > has QFIND_TESTDATA been adapted to work with this, or are we just > > "lucky" that most packages don't actually build and run test suites? > > Yes, QFINDTESTDATA is one of the primary (only?) issues with test suites > found in about 20-30 packages in the archive, according to the > archive-wide rebuild that was performed. For most of those packages > patches have been submitted, and some are already either fixed or marked > as pending. But QFINDTESTDATA is using __FILE__ in a valid way. It might not be what you are expecting, but still a valid usage. See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876901 We have discussed this before. > If it could be fixed at the core for QFINDTESTDATA, that would be nicer > than fixing 20-30 packages individually, though we're not there right > now. In that case I would expect a valid patch from the people interested in enabling this. In the meantime the dpkg change broke a very valid usage. Inconvenient for reproducibility? yes, probably, but still very valid. Oh, and changing this just before freeze is also something not acceptable considering I have seen no valid path proposed for QFINDTESTDATA yet. You are breaking something that it's expected to work. So please revert the change in dpkg. After this release we can definitely wait for someone producing a patch for QFINDTESTDATA, test it and go ahead wth the change. -- Lisandro Damián Nicanor Pérez Meyer http://perezmeyer.com.ar/ http://perezmeyer.blogspot.com/
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
On Sat, 2020-11-14 at 11:15:03 -0800, Vagrant Cascadian wrote: > On 2020-11-14, Sune Vuorela wrote: > > Unfortunately, only like 10% of the relevant packages have test suites > > enabled and run, because gettings things to work reliable is sometimes > > hard. > So, based on your estimate and the current packages known to be > affected, Debian might have an additional 300 packages that might > someday enable test suites. That is ~1% of the archive that would need > to make a one-line change in debian/rules if the maintainers enable test > suites for those packages. > > Are there any templates or documentation used for such packages that > might be able to facilitate the process? It looks like if this cannot be fixed in the test suite themselves for now, then refactoring this setting into a dh sequencer could cover it automatically for all these cases? > > Adding more hurdles does not help. > > I think this is a hurdle we do not need. > > To me, a one-line change in packaging seems like a quite small hurdle in > the short-term, but clearly you do not agree. This being just a default change that can be overridden with a one line modification, seems to me also to be quite unobtrusive. More so when we have things like hardening that people are overriding all over the place w/o much of an issue. > So it really comes down to applying opt-in patches for hundreds (maybe > thousands) of packages, or an opt-out change for somewhere in the > ballpark of tens or hundreds of packages. Yes, that's my view too. I had the patch queued in dpkg for my next push, but I've pulled it out for now given the raised concern, but I'm not seeing a very strong case here to not include it again TBH. > Long-term, of course it would be more ideal to fix QFINDTESTDATA to be > compatible with -ffile-prefix-map/-fmacro-prefix-map compiler flags > being used to strip the build path from the compiled outputs; this would > solve the issue for potentially hundreds of packages and would make the > issue essentially moot. Yes. Thanks, Guillem
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
On 2020-11-14, Sune Vuorela wrote: > On 2020-11-13, Vagrant Cascadian wrote: >> If it could be fixed at the core for QFINDTESTDATA, that would be nicer >> than fixing 20-30 packages individually, though we're not there right >> now. > > Unfortunately, only like 10% of the relevant packages have test suites > enabled and run, because gettings things to work reliable is sometimes > hard. That is a a bit of a surprise! So, based on your estimate and the current packages known to be affected, Debian might have an additional 300 packages that might someday enable test suites. That is ~1% of the archive that would need to make a one-line change in debian/rules if the maintainers enable test suites for those packages. Are there any templates or documentation used for such packages that might be able to facilitate the process? > Adding more hurdles does not help. > I think this is a hurdle we do not need. To me, a one-line change in packaging seems like a quite small hurdle in the short-term, but clearly you do not agree. So it really comes down to applying opt-in patches for hundreds (maybe thousands) of packages, or an opt-out change for somewhere in the ballpark of tens or hundreds of packages. Long-term, of course it would be more ideal to fix QFINDTESTDATA to be compatible with -ffile-prefix-map/-fmacro-prefix-map compiler flags being used to strip the build path from the compiled outputs; this would solve the issue for potentially hundreds of packages and would make the issue essentially moot. live well, vagrant signature.asc Description: PGP signature
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
On 2020-11-13, Sune Vuorela wrote: > On 2020-10-27, Vagrant Cascadian wrote: >> Though, of course, identifying the exact reproducibility problem would >> be preferable. One of the common issues is test suites relying on the >> behavior of __FILE__ returning a full path to find fixtures or other >> test data. > > has QFIND_TESTDATA been adapted to work with this, or are we just > "lucky" that most packages don't actually build and run test suites? Yes, QFINDTESTDATA is one of the primary (only?) issues with test suites found in about 20-30 packages in the archive, according to the archive-wide rebuild that was performed. For most of those packages patches have been submitted, and some are already either fixed or marked as pending. If it could be fixed at the core for QFINDTESTDATA, that would be nicer than fixing 20-30 packages individually, though we're not there right now. > I don't think that we should make it harder for maintainers to enable > test suites on their packages Similarly, I don't think we should make it harder to make hundreds of packages more reproducible just to avoid a trivial workaround. We're talking about a one-line change in debian/rules in a small number of packages, as opposed to the 3 packages that need no changes. > when we can just record the filepath in > the build info and still have things reproducible. Sure, rebuilding packages in the same path is a relatively easy workaround to the build path issue. It does have some drawbacks: * More complicated infrastructure to recreate the build environment. * The .buildinfo files need to include the build path by default, which I believe is enabled for most buildd machines, but not the default for dpkg-genbuildinfo (and there may be some privacy concerns to making it the default). So, it will take some work to make this change in Debian and because it seems to affect QT related test suites, it affects the teams working with QT more than the rest of Debian. Because of that, I did individual builds for all the affected packages, and submitted patches to workaround this issue to try and minimize the extra work needed by any affected maintainers or teams. I still would like for Debian to move forward with this change. live well, vagrant signature.asc Description: PGP signature
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
On 2020-10-27, Vagrant Cascadian wrote: > Though, of course, identifying the exact reproducibility problem would > be preferable. One of the common issues is test suites relying on the > behavior of __FILE__ returning a full path to find fixtures or other > test data. has QFIND_TESTDATA been adapted to work with this, or are we just "lucky" that most packages don't actually build and run test suites? I don't think that we should make it harder for maintainers to enable test suites on their packages, when we can just record the filepath in the build info and still have things reproducible. /Sune
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
Hi On Mon, 9 Nov 2020 at 22:06, Vagrant Cascadian wrote: > Given no objections or concerns of any kind raised in the last two > weeks, I've submitted a bug against dpkg: > > https://bugs.debian.org/974087 There was a query from one of our upstreams in #972294 to which I have not seen a response. Regards Graham
Re: Updating dpkg-buildflags to enable reproducible=+fixfilepath by default
On 2020-10-27, Vagrant Cascadian wrote: > The dpkg-buildflags feature reproducible=+fixfilepath was added to dpkg > in 2018. ... > The result of enabling this feature by default will be to make several > hundred packages reproducible with varying build-path and reduce the > differences in many other packages, making it easier to identify other > more nuanced reproducibility issues. > > It would be great to see the reproducible=+fixfilepath feature enabled > by default in dpkg-buildflags, and we would like to proceed forward with > this soon unless we hear any major concerns or other outstanding issues. Given no objections or concerns of any kind raised in the last two weeks, I've submitted a bug against dpkg: https://bugs.debian.org/974087 live well, vagrant signature.asc Description: PGP signature
