Re: [PATCH] libtool: Sort output of 'find' to enable deterministic builds.
On 2018-06-29 17:09, Jeff Law wrote: > In the immediate term, applying the patch to both instances seems wise. > > Bernhard, do you have commit privs? no, and I dont really want privs, since I expect to be doing only a few patches for gcc. Can you (or someone else) please commit the patch?
[PATCH] libtool: Sort output of 'find' to enable deterministic builds.
so that gcc builds in a reproducible way
in spite of indeterministic filesystem readdir order
See https://reproducible-builds.org/ for why this is good.
While working on the reproducible builds effort, I found that
when building the gcc8 package for openSUSE, there were differences
between each build in resulting binaries like gccgo, cc1obj and cpp
because the order of objects in libstdc++.a varied based on
the order of entries returned by the filesystem.
Two remaining issues are with timestamps in the ada build
and with profiledbootstrap that only is reproducible if all inputs
in the profiling run remain constant (and make -j breaks it too)
Testcases:
none included because patch is trivial and it would need to compare builds on
2 filesystems.
Bootstrapping and testing:
tested successfully with gcc8 on x86_64
[gcc]
2018-06-19 Bernhard M. Wiedemann
libtool: Sort output of 'find' to enable deterministic builds.
---
pulled in libtool commit 74c8993c178a1386ea5e2363a01d919738402f30
because a full update appears to be too troublesome after 8+ years
of divergence, but we still really want that fix.
See also https://gcc.gnu.org/ml/gcc/2017-10/msg00060.html
---
libtool.m4 | 8
ltmain.sh | 4 ++--
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/libtool.m4 b/libtool.m4
index 24d13f344..940faaa16 100644
--- a/libtool.m4
+++ b/libtool.m4
@@ -6005,20 +6005,20 @@ if test "$_lt_caught_CXX_error" != yes; then
_LT_TAGVAR(prelink_cmds, $1)='tpldir=Template.dir~
rm -rf $tpldir~
$CC --prelink_objects --instantiation_dir $tpldir $objs
$libobjs $compile_deplibs~
- compile_command="$compile_command `find $tpldir -name \*.o |
$NL2SP`"'
+ compile_command="$compile_command `find $tpldir -name \*.o |
sort | $NL2SP`"'
_LT_TAGVAR(old_archive_cmds, $1)='tpldir=Template.dir~
rm -rf $tpldir~
$CC --prelink_objects --instantiation_dir $tpldir
$oldobjs$old_deplibs~
- $AR $AR_FLAGS $oldlib$oldobjs$old_deplibs `find $tpldir -name
\*.o | $NL2SP`~
+ $AR $AR_FLAGS $oldlib$oldobjs$old_deplibs `find $tpldir -name
\*.o | sort | $NL2SP`~
$RANLIB $oldlib'
_LT_TAGVAR(archive_cmds, $1)='tpldir=Template.dir~
rm -rf $tpldir~
$CC --prelink_objects --instantiation_dir $tpldir
$predep_objects $libobjs $deplibs $convenience $postdep_objects~
- $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
$tpldir -name \*.o | $NL2SP` $postdep_objects $compiler_flags ${wl}-soname
${wl}$soname -o $lib'
+ $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
$tpldir -name \*.o | sort | $NL2SP` $postdep_objects $compiler_flags
${wl}-soname ${wl}$soname -o $lib'
_LT_TAGVAR(archive_expsym_cmds, $1)='tpldir=Template.dir~
rm -rf $tpldir~
$CC --prelink_objects --instantiation_dir $tpldir
$predep_objects $libobjs $deplibs $convenience $postdep_objects~
- $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
$tpldir -name \*.o | $NL2SP` $postdep_objects $compiler_flags ${wl}-soname
${wl}$soname ${wl}-retain-symbols-file ${wl}$export_symbols -o $lib'
+ $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
$tpldir -name \*.o | sort | $NL2SP` $postdep_objects $compiler_flags
${wl}-soname ${wl}$soname ${wl}-retain-symbols-file ${wl}$export_symbols -o
$lib'
;;
*) # Version 6 and above use weak symbols
_LT_TAGVAR(archive_cmds, $1)='$CC -shared $pic_flag
$predep_objects $libobjs $deplibs $postdep_objects $compiler_flags ${wl}-soname
${wl}$soname -o $lib'
diff --git a/ltmain.sh b/ltmain.sh
index 9503ec85d..79f9ba89a 100644
--- a/ltmain.sh
+++ b/ltmain.sh
@@ -2917,7 +2917,7 @@ func_extract_archives ()
darwin_file=
darwin_files=
for darwin_file in $darwin_filelist; do
- darwin_files=`find unfat-$$ -name $darwin_file -print | $NL2SP`
+ darwin_files=`find unfat-$$ -name $darwin_file -print | sort |
$NL2SP`
$LIPO -create -output "$darwin_file" $darwin_files
done # $darwin_filelist
$RM -rf unfat-$$
@@ -2932,7 +2932,7 @@ func_extract_archives ()
func_extract_an_archive "$my_xdir" "$my_xabs"
;;
esac
- my_oldobjs="$my_oldobjs "`find $my_xdir -name \*.$objext -print -o -name
\*.lo -print | $NL2SP`
+ my_oldobjs="$my_oldobjs "`find $my_xdir -name \*.$objext -print -o -name
\*.lo -print | sort | $NL2SP`
done
func_extract_archives_result="$my_oldobjs"
--
2.13.7
[PATCH] libtool: Sort output of 'find' to enable deterministic builds.
so that gcc builds in a reproducible way
in spite of indeterministic filesystem readdir order
See https://reproducible-builds.org/ for why this is good.
While working on the reproducible builds effort, I found that
when building the gcc8 package for openSUSE, there were differences
between each build in resulting binaries like gccgo, cc1obj and cpp
because the order of objects in libstdc++.a varied based on
the order of entries returned by the filesystem.
Two remaining issues are with timestamps in the ada build
and with profiledbootstrap that only is reproducible if all inputs
in the profiling run remain constant
Testcases:
none included because patch is trivial and it would need to compare builds on
2 filesystems.
Bootstrapping and testing:
tested successfully with gcc8 on x86_64
[gcc]
2018-06-19 Bernhard M. Wiedemann
libtool: Sort output of 'find' to enable deterministic builds.
---
pulled in libtool commit 74c8993c178a1386ea5e2363a01d919738402f30
because a full update appears to be too troublesome after 8+ years
but we still really want that fix.
See also https://gcc.gnu.org/ml/gcc/2017-10/msg00060.html
---
libtool.m4 | 8
ltmain.sh | 4 ++--
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/libtool.m4 b/libtool.m4
index 24d13f344..940faaa16 100644
--- a/libtool.m4
+++ b/libtool.m4
@@ -6005,20 +6005,20 @@ if test "$_lt_caught_CXX_error" != yes; then
_LT_TAGVAR(prelink_cmds, $1)='tpldir=Template.dir~
rm -rf $tpldir~
$CC --prelink_objects --instantiation_dir $tpldir $objs
$libobjs $compile_deplibs~
- compile_command="$compile_command `find $tpldir -name \*.o |
$NL2SP`"'
+ compile_command="$compile_command `find $tpldir -name \*.o |
sort | $NL2SP`"'
_LT_TAGVAR(old_archive_cmds, $1)='tpldir=Template.dir~
rm -rf $tpldir~
$CC --prelink_objects --instantiation_dir $tpldir
$oldobjs$old_deplibs~
- $AR $AR_FLAGS $oldlib$oldobjs$old_deplibs `find $tpldir -name
\*.o | $NL2SP`~
+ $AR $AR_FLAGS $oldlib$oldobjs$old_deplibs `find $tpldir -name
\*.o | sort | $NL2SP`~
$RANLIB $oldlib'
_LT_TAGVAR(archive_cmds, $1)='tpldir=Template.dir~
rm -rf $tpldir~
$CC --prelink_objects --instantiation_dir $tpldir
$predep_objects $libobjs $deplibs $convenience $postdep_objects~
- $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
$tpldir -name \*.o | $NL2SP` $postdep_objects $compiler_flags ${wl}-soname
${wl}$soname -o $lib'
+ $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
$tpldir -name \*.o | sort | $NL2SP` $postdep_objects $compiler_flags
${wl}-soname ${wl}$soname -o $lib'
_LT_TAGVAR(archive_expsym_cmds, $1)='tpldir=Template.dir~
rm -rf $tpldir~
$CC --prelink_objects --instantiation_dir $tpldir
$predep_objects $libobjs $deplibs $convenience $postdep_objects~
- $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
$tpldir -name \*.o | $NL2SP` $postdep_objects $compiler_flags ${wl}-soname
${wl}$soname ${wl}-retain-symbols-file ${wl}$export_symbols -o $lib'
+ $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
$tpldir -name \*.o | sort | $NL2SP` $postdep_objects $compiler_flags
${wl}-soname ${wl}$soname ${wl}-retain-symbols-file ${wl}$export_symbols -o
$lib'
;;
*) # Version 6 and above use weak symbols
_LT_TAGVAR(archive_cmds, $1)='$CC -shared $pic_flag
$predep_objects $libobjs $deplibs $postdep_objects $compiler_flags ${wl}-soname
${wl}$soname -o $lib'
diff --git a/ltmain.sh b/ltmain.sh
index 9503ec85d..79f9ba89a 100644
--- a/ltmain.sh
+++ b/ltmain.sh
@@ -2917,7 +2917,7 @@ func_extract_archives ()
darwin_file=
darwin_files=
for darwin_file in $darwin_filelist; do
- darwin_files=`find unfat-$$ -name $darwin_file -print | $NL2SP`
+ darwin_files=`find unfat-$$ -name $darwin_file -print | sort |
$NL2SP`
$LIPO -create -output "$darwin_file" $darwin_files
done # $darwin_filelist
$RM -rf unfat-$$
@@ -2932,7 +2932,7 @@ func_extract_archives ()
func_extract_an_archive "$my_xdir" "$my_xabs"
;;
esac
- my_oldobjs="$my_oldobjs "`find $my_xdir -name \*.$objext -print -o -name
\*.lo -print | $NL2SP`
+ my_oldobjs="$my_oldobjs "`find $my_xdir -name \*.$objext -print -o -name
\*.lo -print | sort | $NL2SP`
done
func_extract_archives_result="$my_oldobjs"
--
2.13.7
Re: [PATCH] ltmain.sh: Sort input file list
On 2018-06-11 17:06, Joseph Myers wrote: > If we're not doing a general update from upstream libtool, I think we > should use the upstream ltmain.sh fix (libtool commit > 74c8993c178a1386ea5e2363a01d919738402f30, it looks like), or follow it as > close as possible, rather than having our own variant. right. Either variant should be fine. Cherry-picking it, is the safe variant.
[PATCH] ltmain.sh: Sort input file list
so that gcc builds in a reproducible way in spite of indeterministic filesystem readdir order See https://reproducible-builds.org/ for why this is good. While working on the reproducible builds effort, I found that when building the gcc8 package for openSUSE, there were differences between each build in resulting binaries like gccgo, cc1obj and cpp because the order of objects in libstdc++.a varied based on the order of entries returned by the filesystem. Two remaining issues are with timestamps in the ada build and with profiledbootstrap that only is reproducible if all inputs in the profiling run remain constant (which is not the case) Testcases: none included because patch is trivial and it would need to compare builds on 2 filesystems. ChangeLog: ltmain.sh: sort list of object files to make build reproducible Bootstrapping and testing: tested successfully with gcc8 on x86_64 --- ltmain.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ltmain.sh b/ltmain.sh index 9503ec85d..96311a12c 100644 --- a/ltmain.sh +++ b/ltmain.sh @@ -2932,7 +2932,7 @@ func_extract_archives () func_extract_an_archive "$my_xdir" "$my_xabs" ;; esac - my_oldobjs="$my_oldobjs "`find $my_xdir -name \*.$objext -print -o -name \*.lo -print | $NL2SP` + my_oldobjs="$my_oldobjs "`find $my_xdir -name \*.$objext -print -o -name \*.lo -print | LC_ALL=C sort | $NL2SP` done func_extract_archives_result="$my_oldobjs" -- 2.13.7
