[Reproducible-builds] Bug#792856: libsynthesis: please make the build reproducible
Source: libsynthesis
Version: 3.4.0.47.4-2
Severity: wishlist
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: locale
X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org
Hi!
While working on the reproducible builds effort [1], we have noticed
that libsynthesis could not be built reproducibly.
Some file lists are sorted differently depending on the locale.
The attached patch fixes this by setting LC_ALL to C before sorting.
Regards,
Reiner
[1]: https://wiki.debian.org/ReproducibleBuilds
diff --git a/debian/patches/series b/debian/patches/series
new file mode 100644
index 000..428d147
--- /dev/null
+++ b/debian/patches/series
@@ -0,0 +1 @@
+sort_file_lists.patch
diff --git a/debian/patches/sort_file_lists.patch b/debian/patches/sort_file_lists.patch
new file mode 100644
index 000..9d55779
--- /dev/null
+++ b/debian/patches/sort_file_lists.patch
@@ -0,0 +1,52 @@
+Author: Reiner Herrmann rei...@reiner-h.de
+Description: Make sorting independent of locale
+ Sort behaves differently depending on the configured locale.
+ To get reproducible results, sort the file lists
+ with the C locale.
+
+Index: libsynthesis-3.4.0.47.4/src/gen-makefile-am.sh
+===
+--- libsynthesis-3.4.0.47.4.orig/src/gen-makefile-am.sh
libsynthesis-3.4.0.47.4/src/gen-makefile-am.sh
+@@ -74,7 +74,7 @@ LIBSYNTHESIS_SOURCES=`find ${ENGINE_SOUR
+ sysync_SDK/DB_Interfaces/text_db \
+ \( -name '*.cpp' -o -name '*.[ch]' \) |
+ grep -v -E -f EXCLUDE_FILES |
+-sort`
++LC_ALL=C sort`
+ LIBSYNTHESIS_SOURCES=`echo $LIBSYNTHESIS_SOURCES`
+
+ # files to be included in both libsynthesis and libsynthesissdk;
+@@ -84,7 +84,7 @@ cat SERVER_FILES CLIENT_FILES EXTRA_FILE
+ LIBSYNTHESISSDK_SOURCES_BOTH=`find sysync_SDK/Sources \
+ \( -name '*.cpp' -o -name '*.c' \) |
+ grep -v -E -f EXCLUDE_FILES |
+-sort`
++LC_ALL=C sort`
+ LIBSYNTHESISSDK_SOURCES_BOTH=`echo $LIBSYNTHESISSDK_SOURCES_BOTH`
+
+ # files only needed in libsynthesissdk
+@@ -93,7 +93,7 @@ LIBSYNTHESISSDK_SOURCES_ONLY=`find sysyn
+ \( -name '*.cpp' -o -name '*.c' \) |
+ grep -E -f SDK_FILES |
+ grep -v -E -f EXCLUDE_FILES |
+-sort`
++LC_ALL=C sort`
+ LIBSYNTHESISSDK_SOURCES_ONLY=`echo $LIBSYNTHESISSDK_SOURCES_ONLY`
+
+ # files needed in libsmltk
+@@ -101,12 +101,12 @@ LIBSMLTK_SOURCES=`find syncml_tk \
+ \( -name '*.cpp' -o -name '*.[ch]' \) \
+ \! \( -path syncml_tk/src/sml/\*/palm/\* -o \
+-path syncml_tk/src/sml/\*/win/\* \) |
+- sort`
++ LC_ALL=C sort`
+ LIBSMLTK_SOURCES=`echo $LIBSMLTK_SOURCES`
+
+ # header files required for using libsynthesissdk,
+ # with synthesis/ prefix
+-LIBSYNTHESISSDK_HEADERS=`find sysync_SDK/Sources -name '*.h' | sed -e 's;.*/;synthesis/;' | sort`
++LIBSYNTHESISSDK_HEADERS=`find sysync_SDK/Sources -name '*.h' | sed -e 's;.*/;synthesis/;' | LC_ALL=C sort`
+ LIBSYNTHESISSDK_HEADERS=`echo $LIBSYNTHESISSDK_HEADERS`
+
+ sed -e s;@LIBSYNTHESIS_SOURCES@;$LIBSYNTHESIS_SOURCES; \
signature.asc
Description: OpenPGP digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [U-Boot] [PATCH] build: create time and date independent binary
Hi Paul, sorry for the late reply. On Samstag, 13. Juni 2015, Paul Kocialkowski wrote: you've seen https://reproducible.debian.net/u-boot ? This seems very minimalistic, but it's good to see U-Boot was given some attention already! :-) but maybe you can explain why u-boot needs more reproducibility testing than what there currently is. i'm definitly interested and not opposed, even though I think there shoukd be good reasons to treat some software specially. The point is to make U-Boot reproducible for all possible targets, not only the few ones that are supported by U-Boot. I think your sentence is missing some word…?!? I think this requires some extra infrastructure. In that sense, it is very similar to Coreboot. (also please note that we currently only have amd64 hw to run our tests on.) The problem is the same as Coreboot, which uses its own toolchain to build images. We don't need to have native armhf builds for U-Boot, testing with the armhf toolchain that is in Debian should be enough. I see. I understand, this works out nicely because all the work on Coreboot will be inherited by Libreboot. However, on U-Boot, the work to bring reproducible builds has to take place initially. I know for a fact that parts of the code use things like __FILE__ or timestamps. Ah. That makes sense. For U-Boot, it will certainly make sense for the distributions packaging it. I'm the main developer of Replicant, the fully free version of Android, and it would definitely be useful to have a smartphone on which we can trust that the bootloader can be built in a reproducible manner (and checked after being installed). indeed! All this said, if you send me patches, I will probably deploy them as I'm very curious and more reproducibility efforts are good :-) We can can always decide to remove or move them later. I wish to make all contributions upstream. What would really help at first would be to have all targets built regularly to see where work is needed. This is where I think the Debian infrastructure could help, in a similar way as what was started for Coreboot. can you point me to a how to explaining this or tell me those steps, starting with git clone...? cheers, Holger signature.asc Description: This is a digitally signed message part. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [U-Boot] [PATCH] build: create time and date independent binary
On 2015-07-19, Holger Levsen wrote: All this said, if you send me patches, I will probably deploy them as I'm very curious and more reproducibility efforts are good :-) We can can always decide to remove or move them later. I wish to make all contributions upstream. What would really help at first would be to have all targets built regularly to see where work is needed. This is where I think the Debian infrastructure could help, in a similar way as what was started for Coreboot. FWIW, I was planning on including this patch to u-boot in the next upload to Debian: https://anonscm.debian.org/cgit/collab-maint/u-boot.git/tree/debian/patches/use-date-from-debian-changelog.patch?h=experimental-2015.07 I *think* that actually makes u-boot build reproducibly with Debian's reproducible builds toolchain when SOURCE_DATE_EPOCH is set, but I haven't tested it fully. I might have missed some other sources of non-determinism... Hoping to get some armhf buildd nodes up an running soonish... although it should also be buildable with the cross-toolchains, if the reproducible buildds coulld be made to support that. live well, vagrant signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#792867: FTCBFS: uses wrong architecture Python
Source: bsdmainutils Version: 9.0.6 Tags: patch User: helm...@debian.org Usertags: rebootstrap Dear bsdmainutils maintainers, While trying to cross build bsdmainutils, I noticed that it was installing the host architecture python and python-hdate. It then tries to execute that Python as a build tool, which fails, because the host architecture is usually not executable during cross compilation. Since python and python-hdate are only used as build tools, the correct solution is to always install them for the build architecture. This can be reflected in the Build-Depends by annotating those packages with :native. The attached patch implements that proposal. Please consider applying it. The :native is supported in apt and dpkg since wheezy. If in doubt, don't hesitate to ask for details on the identified problem or the proposed solution. As an aside note, I question the use of python-hdate to compute the Jewish calendar: * Since the build date is used to select the year of the calendar file, the package is not reproducible. (Ccing the reproducible people) * The last build happened in 2014, so the installed calendar is no longer useful for day to day use (neither in stable nor in unstable). Helmut diff -Nru bsdmainutils-9.0.6/debian/changelog bsdmainutils-9.0.6+nmu1/debian/changelog --- bsdmainutils-9.0.6/debian/changelog 2014-10-17 15:42:56.0 +0200 +++ bsdmainutils-9.0.6+nmu1/debian/changelog2015-07-19 15:37:08.0 +0200 @@ -1,3 +1,11 @@ +bsdmainutils (9.0.6+nmu1) UNRELEASED; urgency=medium + + * Non-maintainer upload. + * Add :native annotationes to Python-related Build-Depends to facilitate +cross dependency satisfiability (Closes: #-1) + + -- Helmut Grohne hel...@subdivi.de Sun, 19 Jul 2015 15:36:38 +0200 + bsdmainutils (9.0.6) unstable; urgency=medium * Add Ubuntu 13.04 release date to calendar.ubuntu. diff -Nru bsdmainutils-9.0.6/debian/control bsdmainutils-9.0.6+nmu1/debian/control --- bsdmainutils-9.0.6/debian/control 2014-08-15 11:49:55.0 +0200 +++ bsdmainutils-9.0.6+nmu1/debian/control 2015-07-19 15:36:33.0 +0200 @@ -3,7 +3,7 @@ Priority: important Maintainer: Debian Bsdmainutils Team pkg-bsdmainut...@teams.debian.net Uploaders: Giacomo Catenazzi c...@debian.org, Michael Meskes mes...@debian.org -Build-Depends: debhelper (= 7), libncurses5-dev, quilt (= 0.40), python, python-hdate +Build-Depends: debhelper (= 7), libncurses5-dev, quilt (= 0.40), python:native, python-hdate:native Standards-Version: 3.9.5 Vcs-Git: git://anonscm.debian.org/bsdmainutils/bsdmainutils.git Vcs-Browser: http://anonscm.debian.org/gitweb/?p=bsdmainutils/bsdmainutils.git ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [U-Boot] [PATCH] build: create time and date independent binary
Hi Paul, On Sonntag, 19. Juli 2015, Paul Kocialkowski wrote: sorry for the late reply. No problem, I haven't kept you or the list posted on this either. I got a chance to discuss the issue with Lunar during RMLL 2015 and we came up with a nice way of doing things, using SOURCE_DATE_EPOCH. oh, nice! you've seen https://reproducible.debian.net/u-boot ? As far as I understood, this was only for the U-Boot tools. yes There seem to be two solutions to this: * Including a script (e.g. the one from coreboot) to build the toolchain as part of the build process * Using native builds with visualization on the servers that run the builds for the reproducible task force I tend to prefer the second one since it only requires a one-time setup cost, while the other one, that requires to build toolchains for each test build, implies a considerably longer build time for each test. these hosts also need maintenance so I actually prefer the first. The basics for building U-Boot are the following (e.g. for the Cubieboard2 target) git clone git://git.denx.de/u-boot.git cd u-boot make -C $SRC O=$DST CROSS_COMPILE=$CROSS_COMPILE Cubieboard2_defconfig make -C $SRC O=$DST CROSS_COMPILE=$CROSS_COMPILE aint there a makefile with a proper target? Also how to build these cross compilers? Let me know if you need more indications on this. yes, please. best a complete runnable script, aka a _working_ 5-10 liner, not the fancy one with variations, rebuilds and running debbindiff. Just the above 4 lines in working - (those 4 lines dont work cause the variables are not defined...) cheers, Holger signature.asc Description: This is a digitally signed message part. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [U-Boot] [PATCH] build: create time and date independent binary
Le dimanche 19 juillet 2015 à 17:47 +0200, Holger Levsen a écrit : There seem to be two solutions to this: * Including a script (e.g. the one from coreboot) to build the toolchain as part of the build process * Using native builds with visualization on the servers that run the builds for the reproducible task force I tend to prefer the second one since it only requires a one-time setup cost, while the other one, that requires to build toolchains for each test build, implies a considerably longer build time for each test. these hosts also need maintenance so I actually prefer the first. I understand. We could raise the topic on the U-Boot mailing list and see whether adding the scripts to build the cross-compilers tu U-Boot would be doable or not. Otherwise, we can just grab those scripts aside the U-Boot source code for the reproducible task, so that we don't have to wait for a long discussion to conclude. The basics for building U-Boot are the following (e.g. for the Cubieboard2 target) git clone git://git.denx.de/u-boot.git cd u-boot make -C $SRC O=$DST CROSS_COMPILE=$CROSS_COMPILE Cubieboard2_defconfig make -C $SRC O=$DST CROSS_COMPILE=$CROSS_COMPILE aint there a makefile with a proper target? Also how to build these cross compilers? I am using those scripts, that I wrote: http://git.paulk.fr/gitweb/?p=embedded-freedom-scripts.git;a=tree;f=u-boot;h=f735bc015b056c3b63a0144703d7f7168382ca59;hb=HEAD That can help but is not really usable as-is. Let me know if you need more indications on this. yes, please. best a complete runnable script, aka a _working_ 5-10 liner, not the fancy one with variations, rebuilds and running debbindiff. Just the above 4 lines in working - (those 4 lines dont work cause the variables are not defined...) I could do that, but we have to figure out the toolchain issue first, I suppose. -- Paul Kocialkowski, Replicant developer Replicant is a fully free Android distribution running on several devices, a free software mobile operating system putting the emphasis on freedom and privacy/security. Website: http://www.replicant.us/ Blog: http://blog.replicant.us/ Wiki/tracker/forums: http://redmine.replicant.us/ signature.asc Description: This is a digitally signed message part ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [U-Boot] [PATCH] build: create time and date independent binary
Hi, On Sonntag, 19. Juli 2015, Paul Kocialkowski wrote: I *think* that actually makes u-boot build reproducibly with Debian's reproducible builds toolchain when SOURCE_DATE_EPOCH is set, but I haven't tested it fully. I might have missed some other sources of non-determinism... We came up with something similar last week. This is not actually sufficient, but close. I will submit what I have to the U-Boot mailing list soon (hopefully, later today), feel free to review it there and import it to Debian. cool! Native builds don't seem realistic given the many different architectures that U-Boot supports, unless done with virtualisation (see my other email from today). [...] Either way, we can always try to add a toolchain build script to U-Boot. I agree, I also think we should test uboot with such a script, similar like we do for coreboot. That wouldnt need armhf buildds. That said, making use of these armhf buildds Vagrant is planning to setup would also useful to test Debian packages (incl uboot) on armhf. cheers, Holger signature.asc Description: This is a digitally signed message part. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Not Serious!
[resend with the correct list address] Hi, for the record, Mattia reproduced this with a plain unstable pbuilder setup, not using our (=reproducible) modified toolchain packages. Colin, I'm still curious why you downgraded the original bug report to minor. As I see it this is definitly at least important or serious - and important only if the bug doesnt occur with sbuild and debuild… cheers, Holger signature.asc Description: This is a digitally signed message part. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [U-Boot] [PATCH] build: create time and date independent binary
Le dimanche 19 juillet 2015 à 06:14 -0700, Vagrant Cascadian a écrit : On 2015-07-19, Holger Levsen wrote: All this said, if you send me patches, I will probably deploy them as I'm very curious and more reproducibility efforts are good :-) We can can always decide to remove or move them later. I wish to make all contributions upstream. What would really help at first would be to have all targets built regularly to see where work is needed. This is where I think the Debian infrastructure could help, in a similar way as what was started for Coreboot. FWIW, I was planning on including this patch to u-boot in the next upload to Debian: https://anonscm.debian.org/cgit/collab-maint/u-boot.git/tree/debian/patches/use-date-from-debian-changelog.patch?h=experimental-2015.07 I *think* that actually makes u-boot build reproducibly with Debian's reproducible builds toolchain when SOURCE_DATE_EPOCH is set, but I haven't tested it fully. I might have missed some other sources of non-determinism... We came up with something similar last week. This is not actually sufficient, but close. I will submit what I have to the U-Boot mailing list soon (hopefully, later today), feel free to review it there and import it to Debian. Hoping to get some armhf buildd nodes up an running soonish... although it should also be buildable with the cross-toolchains, if the reproducible buildds coulld be made to support that. Native builds don't seem realistic given the many different architectures that U-Boot supports, unless done with virtualisation (see my other email from today). Now that I think about it, maybe virtualisation would require simply too many VMs to setup. Either way, we can always try to add a toolchain build script to U-Boot. -- Paul Kocialkowski, Replicant developer Replicant is a fully free Android distribution running on several devices, a free software mobile operating system putting the emphasis on freedom and privacy/security. Website: http://www.replicant.us/ Blog: http://blog.replicant.us/ Wiki/tracker/forums: http://redmine.replicant.us/ signature.asc Description: This is a digitally signed message part ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#792867: FTCBFS: uses wrong architecture Python
clone 792867 -1 clone 792867 -2 retitle -1 bsdmainutils: unreproducible as current year is used while building retitle -2 bsdmainutils: jewish calendar unsuable, needs a rebuild user reproducible-builds@lists.alioth.debian.org usertag -2 + timestamps thanks Hi, I'll leave 792867 alone and just comment on the cloning. On Sonntag, 19. Juli 2015, Helmut Grohne wrote: As an aside note, I question the use of python-hdate to compute the Jewish calendar: * Since the build date is used to select the year of the calendar file, the package is not reproducible. (Ccing the reproducible people) this is clone -1, though please note that this is not yet made visible on reproducible.debian.net, the package appears to be reproducible there. * The last build happened in 2014, so the installed calendar is no longer useful for day to day use (neither in stable nor in unstable). this is clone -2, the severity maybe should be raised to important. cheers, Holger signature.asc Description: This is a digitally signed message part. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [U-Boot] [PATCH] build: create time and date independent binary
sorry for the late reply. No problem, I haven't kept you or the list posted on this either. I got a chance to discuss the issue with Lunar during RMLL 2015 and we came up with a nice way of doing things, using SOURCE_DATE_EPOCH. On Samstag, 13. Juni 2015, Paul Kocialkowski wrote: you've seen https://reproducible.debian.net/u-boot ? This seems very minimalistic, but it's good to see U-Boot was given some attention already! As far as I understood, this was only for the U-Boot tools. but maybe you can explain why u-boot needs more reproducibility testing than what there currently is. i'm definitly interested and not opposed, even though I think there shoukd be good reasons to treat some software specially. The point is to make U-Boot reproducible for all possible targets, not only the few ones that are supported by U-Boot. I think your sentence is missing some word…?!? not only for the few one that are supported by U-Boot maybe? Looks good otherwise! I think this requires some extra infrastructure. In that sense, it is very similar to Coreboot. (also please note that we currently only have amd64 hw to run our tests on.) The problem is the same as Coreboot, which uses its own toolchain to build images. We don't need to have native armhf builds for U-Boot, testing with the armhf toolchain that is in Debian should be enough. I see. There seem to be two solutions to this: * Including a script (e.g. the one from coreboot) to build the toolchain as part of the build process * Using native builds with visualization on the servers that run the builds for the reproducible task force I tend to prefer the second one since it only requires a one-time setup cost, while the other one, that requires to build toolchains for each test build, implies a considerably longer build time for each test. I understand, this works out nicely because all the work on Coreboot will be inherited by Libreboot. However, on U-Boot, the work to bring reproducible builds has to take place initially. I know for a fact that parts of the code use things like __FILE__ or timestamps. Ah. Not all the targets are using that though, and the target we used during RMLL required only timestamp changes to become reproducible (it was the Cubieboard2 IIRC). Still, I have seen that code around, so it must be used somewhere, so it should be fixed, too. All this said, if you send me patches, I will probably deploy them as I'm very curious and more reproducibility efforts are good :-) We can can always decide to remove or move them later. I wish to make all contributions upstream. What would really help at first would be to have all targets built regularly to see where work is needed. This is where I think the Debian infrastructure could help, in a similar way as what was started for Coreboot. can you point me to a how to explaining this or tell me those steps, starting with git clone...? The basics for building U-Boot are the following (e.g. for the Cubieboard2 target) git clone git://git.denx.de/u-boot.git cd u-boot make -C $SRC O=$DST CROSS_COMPILE=$CROSS_COMPILE Cubieboard2_defconfig make -C $SRC O=$DST CROSS_COMPILE=$CROSS_COMPILE I usually also pass ARCH=$ARCH to make but that's useless, apparently. Let me know if you need more indications on this. -- Paul Kocialkowski, Replicant developer Replicant is a fully free Android distribution running on several devices, a free software mobile operating system putting the emphasis on freedom and privacy/security. Website: http://www.replicant.us/ Blog: http://blog.replicant.us/ Wiki/tracker/forums: http://redmine.replicant.us/ signature.asc Description: This is a digitally signed message part ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
