Re: [PATCH] libtool: Sort output of 'find' to enable deterministic builds.
On 07/05/2018 08:53 AM, Bernhard M. Wiedemann wrote: > On 2018-06-29 17:09, Jeff Law wrote: >> In the immediate term, applying the patch to both instances seems wise. >> >> Bernhard, do you have commit privs? > > no, and I dont really want privs, since I expect to be doing only a few > patches for gcc. > Can you (or someone else) please commit the patch? Installed on the trunk. I did not update the libgo copies since we're a downstream consumer. jeff
Re: [PATCH] libtool: Sort output of 'find' to enable deterministic builds.
On 2018-06-29 17:09, Jeff Law wrote: > In the immediate term, applying the patch to both instances seems wise. > > Bernhard, do you have commit privs? no, and I dont really want privs, since I expect to be doing only a few patches for gcc. Can you (or someone else) please commit the patch?
Re: [PATCH] libtool: Sort output of 'find' to enable deterministic builds.
On 6/29/18, Ian Lance Taylor wrote: > On Fri, Jun 29, 2018 at 8:43 AM, Jakub Jelinek wrote: >> On Fri, Jun 29, 2018 at 09:09:38AM -0600, Jeff Law wrote: >>> > Btw, running find to search for libtool.m4/ltmain.sh I find extra >>> > copies in >>> > >>> > ./libgo/config/ltmain.sh >>> > ./libgo/config/libtool.m4 >>> > >>> > which are nearly identical besides appearantly patched in GO support? >>> > >>> > Can we consolidate those and/or do we need to patch those as well? >>> Ideally consolidate. The README indicates that directory is supposed >>> "temporarily until Go support is added to autoconf and libtool". >> >> Can it be done when these files are not owned by GCC, but copied from >> upstream? > > I sent the libtool patches upstream years ago, but I'm not sure what > happened to them. > > Ian > Try pinging them; when there are libtool maintainers, they sometimes miss stuff.
Re: [PATCH] libtool: Sort output of 'find' to enable deterministic builds.
On Fri, Jun 29, 2018 at 8:43 AM, Jakub Jelinek wrote: > On Fri, Jun 29, 2018 at 09:09:38AM -0600, Jeff Law wrote: >> > Btw, running find to search for libtool.m4/ltmain.sh I find extra copies in >> > >> > ./libgo/config/ltmain.sh >> > ./libgo/config/libtool.m4 >> > >> > which are nearly identical besides appearantly patched in GO support? >> > >> > Can we consolidate those and/or do we need to patch those as well? >> Ideally consolidate. The README indicates that directory is supposed >> "temporarily until Go support is added to autoconf and libtool". > > Can it be done when these files are not owned by GCC, but copied from > upstream? I sent the libtool patches upstream years ago, but I'm not sure what happened to them. Ian
Re: [PATCH] libtool: Sort output of 'find' to enable deterministic builds.
On Fri, Jun 29, 2018 at 09:09:38AM -0600, Jeff Law wrote: > > Btw, running find to search for libtool.m4/ltmain.sh I find extra copies in > > > > ./libgo/config/ltmain.sh > > ./libgo/config/libtool.m4 > > > > which are nearly identical besides appearantly patched in GO support? > > > > Can we consolidate those and/or do we need to patch those as well? > Ideally consolidate. The README indicates that directory is supposed > "temporarily until Go support is added to autoconf and libtool". Can it be done when these files are not owned by GCC, but copied from upstream? Jakub
Re: [PATCH] libtool: Sort output of 'find' to enable deterministic builds.
On 06/29/2018 02:13 AM, Richard Biener wrote: > On Mon, Jun 25, 2018 at 1:39 PM Bernhard M. Wiedemann > wrote: >> >> so that gcc builds in a reproducible way >> in spite of indeterministic filesystem readdir order >> >> See https://reproducible-builds.org/ for why this is good. >> >> While working on the reproducible builds effort, I found that >> when building the gcc8 package for openSUSE, there were differences >> between each build in resulting binaries like gccgo, cc1obj and cpp >> because the order of objects in libstdc++.a varied based on >> the order of entries returned by the filesystem. >> >> Two remaining issues are with timestamps in the ada build >> and with profiledbootstrap that only is reproducible if all inputs >> in the profiling run remain constant (and make -j breaks it too) >> >> Testcases: >> none included because patch is trivial and it would need to compare builds >> on 2 filesystems. >> >> Bootstrapping and testing: >> tested successfully with gcc8 on x86_64 > > Looks ok to me. > > Btw, running find to search for libtool.m4/ltmain.sh I find extra copies in > > ./libgo/config/ltmain.sh > ./libgo/config/libtool.m4 > > which are nearly identical besides appearantly patched in GO support? > > Can we consolidate those and/or do we need to patch those as well? Ideally consolidate. The README indicates that directory is supposed "temporarily until Go support is added to autoconf and libtool". So, assuming autoconf/libtool have updated appropriately upstream, then we "just" need to get ourselves up-to-date and I think that directory of stuff can go away. In the immediate term, applying the patch to both instances seems wise. Bernhard, do you have commit privs? jeff
Re: [PATCH] libtool: Sort output of 'find' to enable deterministic builds.
On Mon, Jun 25, 2018 at 1:39 PM Bernhard M. Wiedemann
wrote:
>
> so that gcc builds in a reproducible way
> in spite of indeterministic filesystem readdir order
>
> See https://reproducible-builds.org/ for why this is good.
>
> While working on the reproducible builds effort, I found that
> when building the gcc8 package for openSUSE, there were differences
> between each build in resulting binaries like gccgo, cc1obj and cpp
> because the order of objects in libstdc++.a varied based on
> the order of entries returned by the filesystem.
>
> Two remaining issues are with timestamps in the ada build
> and with profiledbootstrap that only is reproducible if all inputs
> in the profiling run remain constant (and make -j breaks it too)
>
> Testcases:
> none included because patch is trivial and it would need to compare builds
> on 2 filesystems.
>
> Bootstrapping and testing:
> tested successfully with gcc8 on x86_64
Looks ok to me.
Btw, running find to search for libtool.m4/ltmain.sh I find extra copies in
./libgo/config/ltmain.sh
./libgo/config/libtool.m4
which are nearly identical besides appearantly patched in GO support?
Can we consolidate those and/or do we need to patch those as well?
Thanks,
Richard.
> [gcc]
> 2018-06-19 Bernhard M. Wiedemann
>
> libtool: Sort output of 'find' to enable deterministic builds.
>
> ---
> pulled in libtool commit 74c8993c178a1386ea5e2363a01d919738402f30
> because a full update appears to be too troublesome after 8+ years
> of divergence, but we still really want that fix.
>
> See also https://gcc.gnu.org/ml/gcc/2017-10/msg00060.html
> ---
> libtool.m4 | 8
> ltmain.sh | 4 ++--
> 2 files changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/libtool.m4 b/libtool.m4
> index 24d13f344..940faaa16 100644
> --- a/libtool.m4
> +++ b/libtool.m4
> @@ -6005,20 +6005,20 @@ if test "$_lt_caught_CXX_error" != yes; then
> _LT_TAGVAR(prelink_cmds, $1)='tpldir=Template.dir~
> rm -rf $tpldir~
> $CC --prelink_objects --instantiation_dir $tpldir $objs
> $libobjs $compile_deplibs~
> - compile_command="$compile_command `find $tpldir -name \*.o |
> $NL2SP`"'
> + compile_command="$compile_command `find $tpldir -name \*.o |
> sort | $NL2SP`"'
> _LT_TAGVAR(old_archive_cmds, $1)='tpldir=Template.dir~
> rm -rf $tpldir~
> $CC --prelink_objects --instantiation_dir $tpldir
> $oldobjs$old_deplibs~
> - $AR $AR_FLAGS $oldlib$oldobjs$old_deplibs `find $tpldir -name
> \*.o | $NL2SP`~
> + $AR $AR_FLAGS $oldlib$oldobjs$old_deplibs `find $tpldir -name
> \*.o | sort | $NL2SP`~
> $RANLIB $oldlib'
> _LT_TAGVAR(archive_cmds, $1)='tpldir=Template.dir~
> rm -rf $tpldir~
> $CC --prelink_objects --instantiation_dir $tpldir
> $predep_objects $libobjs $deplibs $convenience $postdep_objects~
> - $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
> $tpldir -name \*.o | $NL2SP` $postdep_objects $compiler_flags ${wl}-soname
> ${wl}$soname -o $lib'
> + $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
> $tpldir -name \*.o | sort | $NL2SP` $postdep_objects $compiler_flags
> ${wl}-soname ${wl}$soname -o $lib'
> _LT_TAGVAR(archive_expsym_cmds, $1)='tpldir=Template.dir~
> rm -rf $tpldir~
> $CC --prelink_objects --instantiation_dir $tpldir
> $predep_objects $libobjs $deplibs $convenience $postdep_objects~
> - $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
> $tpldir -name \*.o | $NL2SP` $postdep_objects $compiler_flags ${wl}-soname
> ${wl}$soname ${wl}-retain-symbols-file ${wl}$export_symbols -o $lib'
> + $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
> $tpldir -name \*.o | sort | $NL2SP` $postdep_objects $compiler_flags
> ${wl}-soname ${wl}$soname ${wl}-retain-symbols-file ${wl}$export_symbols -o
> $lib'
> ;;
> *) # Version 6 and above use weak symbols
> _LT_TAGVAR(archive_cmds, $1)='$CC -shared $pic_flag
> $predep_objects $libobjs $deplibs $postdep_objects $compiler_flags
> ${wl}-soname ${wl}$soname -o $lib'
> diff --git a/ltmain.sh b/ltmain.sh
> index 9503ec85d..79f9ba89a 100644
> --- a/ltmain.sh
> +++ b/ltmain.sh
> @@ -2917,7 +2917,7 @@ func_extract_archives ()
> darwin_file=
> darwin_files=
> for darwin_file in $darwin_filelist; do
> - darwin_files=`find unfat-$$ -name $darwin_file -print | $NL2SP`
> + darwin_files=`find unfat-$$ -name $darwin_file -print | sort |
> $NL2SP`
> $LIPO -create -output "$darwin_file" $darwin_files
> done # $darwin_filelist
> $RM -rf unfat-$$
> @@ -2932,7 +2932,7 @@ func_extract_archives ()
> func_extract_an_archive "$my_xdir" "$my_xabs"
[PATCH] libtool: Sort output of 'find' to enable deterministic builds.
so that gcc builds in a reproducible way
in spite of indeterministic filesystem readdir order
See https://reproducible-builds.org/ for why this is good.
While working on the reproducible builds effort, I found that
when building the gcc8 package for openSUSE, there were differences
between each build in resulting binaries like gccgo, cc1obj and cpp
because the order of objects in libstdc++.a varied based on
the order of entries returned by the filesystem.
Two remaining issues are with timestamps in the ada build
and with profiledbootstrap that only is reproducible if all inputs
in the profiling run remain constant (and make -j breaks it too)
Testcases:
none included because patch is trivial and it would need to compare builds on
2 filesystems.
Bootstrapping and testing:
tested successfully with gcc8 on x86_64
[gcc]
2018-06-19 Bernhard M. Wiedemann
libtool: Sort output of 'find' to enable deterministic builds.
---
pulled in libtool commit 74c8993c178a1386ea5e2363a01d919738402f30
because a full update appears to be too troublesome after 8+ years
of divergence, but we still really want that fix.
See also https://gcc.gnu.org/ml/gcc/2017-10/msg00060.html
---
libtool.m4 | 8
ltmain.sh | 4 ++--
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/libtool.m4 b/libtool.m4
index 24d13f344..940faaa16 100644
--- a/libtool.m4
+++ b/libtool.m4
@@ -6005,20 +6005,20 @@ if test "$_lt_caught_CXX_error" != yes; then
_LT_TAGVAR(prelink_cmds, $1)='tpldir=Template.dir~
rm -rf $tpldir~
$CC --prelink_objects --instantiation_dir $tpldir $objs
$libobjs $compile_deplibs~
- compile_command="$compile_command `find $tpldir -name \*.o |
$NL2SP`"'
+ compile_command="$compile_command `find $tpldir -name \*.o |
sort | $NL2SP`"'
_LT_TAGVAR(old_archive_cmds, $1)='tpldir=Template.dir~
rm -rf $tpldir~
$CC --prelink_objects --instantiation_dir $tpldir
$oldobjs$old_deplibs~
- $AR $AR_FLAGS $oldlib$oldobjs$old_deplibs `find $tpldir -name
\*.o | $NL2SP`~
+ $AR $AR_FLAGS $oldlib$oldobjs$old_deplibs `find $tpldir -name
\*.o | sort | $NL2SP`~
$RANLIB $oldlib'
_LT_TAGVAR(archive_cmds, $1)='tpldir=Template.dir~
rm -rf $tpldir~
$CC --prelink_objects --instantiation_dir $tpldir
$predep_objects $libobjs $deplibs $convenience $postdep_objects~
- $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
$tpldir -name \*.o | $NL2SP` $postdep_objects $compiler_flags ${wl}-soname
${wl}$soname -o $lib'
+ $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
$tpldir -name \*.o | sort | $NL2SP` $postdep_objects $compiler_flags
${wl}-soname ${wl}$soname -o $lib'
_LT_TAGVAR(archive_expsym_cmds, $1)='tpldir=Template.dir~
rm -rf $tpldir~
$CC --prelink_objects --instantiation_dir $tpldir
$predep_objects $libobjs $deplibs $convenience $postdep_objects~
- $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
$tpldir -name \*.o | $NL2SP` $postdep_objects $compiler_flags ${wl}-soname
${wl}$soname ${wl}-retain-symbols-file ${wl}$export_symbols -o $lib'
+ $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
$tpldir -name \*.o | sort | $NL2SP` $postdep_objects $compiler_flags
${wl}-soname ${wl}$soname ${wl}-retain-symbols-file ${wl}$export_symbols -o
$lib'
;;
*) # Version 6 and above use weak symbols
_LT_TAGVAR(archive_cmds, $1)='$CC -shared $pic_flag
$predep_objects $libobjs $deplibs $postdep_objects $compiler_flags ${wl}-soname
${wl}$soname -o $lib'
diff --git a/ltmain.sh b/ltmain.sh
index 9503ec85d..79f9ba89a 100644
--- a/ltmain.sh
+++ b/ltmain.sh
@@ -2917,7 +2917,7 @@ func_extract_archives ()
darwin_file=
darwin_files=
for darwin_file in $darwin_filelist; do
- darwin_files=`find unfat-$$ -name $darwin_file -print | $NL2SP`
+ darwin_files=`find unfat-$$ -name $darwin_file -print | sort |
$NL2SP`
$LIPO -create -output "$darwin_file" $darwin_files
done # $darwin_filelist
$RM -rf unfat-$$
@@ -2932,7 +2932,7 @@ func_extract_archives ()
func_extract_an_archive "$my_xdir" "$my_xabs"
;;
esac
- my_oldobjs="$my_oldobjs "`find $my_xdir -name \*.$objext -print -o -name
\*.lo -print | $NL2SP`
+ my_oldobjs="$my_oldobjs "`find $my_xdir -name \*.$objext -print -o -name
\*.lo -print | sort | $NL2SP`
done
func_extract_archives_result="$my_oldobjs"
--
2.13.7
[PATCH] libtool: Sort output of 'find' to enable deterministic builds.
so that gcc builds in a reproducible way
in spite of indeterministic filesystem readdir order
See https://reproducible-builds.org/ for why this is good.
While working on the reproducible builds effort, I found that
when building the gcc8 package for openSUSE, there were differences
between each build in resulting binaries like gccgo, cc1obj and cpp
because the order of objects in libstdc++.a varied based on
the order of entries returned by the filesystem.
Two remaining issues are with timestamps in the ada build
and with profiledbootstrap that only is reproducible if all inputs
in the profiling run remain constant
Testcases:
none included because patch is trivial and it would need to compare builds on
2 filesystems.
Bootstrapping and testing:
tested successfully with gcc8 on x86_64
[gcc]
2018-06-19 Bernhard M. Wiedemann
libtool: Sort output of 'find' to enable deterministic builds.
---
pulled in libtool commit 74c8993c178a1386ea5e2363a01d919738402f30
because a full update appears to be too troublesome after 8+ years
but we still really want that fix.
See also https://gcc.gnu.org/ml/gcc/2017-10/msg00060.html
---
libtool.m4 | 8
ltmain.sh | 4 ++--
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/libtool.m4 b/libtool.m4
index 24d13f344..940faaa16 100644
--- a/libtool.m4
+++ b/libtool.m4
@@ -6005,20 +6005,20 @@ if test "$_lt_caught_CXX_error" != yes; then
_LT_TAGVAR(prelink_cmds, $1)='tpldir=Template.dir~
rm -rf $tpldir~
$CC --prelink_objects --instantiation_dir $tpldir $objs
$libobjs $compile_deplibs~
- compile_command="$compile_command `find $tpldir -name \*.o |
$NL2SP`"'
+ compile_command="$compile_command `find $tpldir -name \*.o |
sort | $NL2SP`"'
_LT_TAGVAR(old_archive_cmds, $1)='tpldir=Template.dir~
rm -rf $tpldir~
$CC --prelink_objects --instantiation_dir $tpldir
$oldobjs$old_deplibs~
- $AR $AR_FLAGS $oldlib$oldobjs$old_deplibs `find $tpldir -name
\*.o | $NL2SP`~
+ $AR $AR_FLAGS $oldlib$oldobjs$old_deplibs `find $tpldir -name
\*.o | sort | $NL2SP`~
$RANLIB $oldlib'
_LT_TAGVAR(archive_cmds, $1)='tpldir=Template.dir~
rm -rf $tpldir~
$CC --prelink_objects --instantiation_dir $tpldir
$predep_objects $libobjs $deplibs $convenience $postdep_objects~
- $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
$tpldir -name \*.o | $NL2SP` $postdep_objects $compiler_flags ${wl}-soname
${wl}$soname -o $lib'
+ $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
$tpldir -name \*.o | sort | $NL2SP` $postdep_objects $compiler_flags
${wl}-soname ${wl}$soname -o $lib'
_LT_TAGVAR(archive_expsym_cmds, $1)='tpldir=Template.dir~
rm -rf $tpldir~
$CC --prelink_objects --instantiation_dir $tpldir
$predep_objects $libobjs $deplibs $convenience $postdep_objects~
- $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
$tpldir -name \*.o | $NL2SP` $postdep_objects $compiler_flags ${wl}-soname
${wl}$soname ${wl}-retain-symbols-file ${wl}$export_symbols -o $lib'
+ $CC -shared $pic_flag $predep_objects $libobjs $deplibs `find
$tpldir -name \*.o | sort | $NL2SP` $postdep_objects $compiler_flags
${wl}-soname ${wl}$soname ${wl}-retain-symbols-file ${wl}$export_symbols -o
$lib'
;;
*) # Version 6 and above use weak symbols
_LT_TAGVAR(archive_cmds, $1)='$CC -shared $pic_flag
$predep_objects $libobjs $deplibs $postdep_objects $compiler_flags ${wl}-soname
${wl}$soname -o $lib'
diff --git a/ltmain.sh b/ltmain.sh
index 9503ec85d..79f9ba89a 100644
--- a/ltmain.sh
+++ b/ltmain.sh
@@ -2917,7 +2917,7 @@ func_extract_archives ()
darwin_file=
darwin_files=
for darwin_file in $darwin_filelist; do
- darwin_files=`find unfat-$$ -name $darwin_file -print | $NL2SP`
+ darwin_files=`find unfat-$$ -name $darwin_file -print | sort |
$NL2SP`
$LIPO -create -output "$darwin_file" $darwin_files
done # $darwin_filelist
$RM -rf unfat-$$
@@ -2932,7 +2932,7 @@ func_extract_archives ()
func_extract_an_archive "$my_xdir" "$my_xabs"
;;
esac
- my_oldobjs="$my_oldobjs "`find $my_xdir -name \*.$objext -print -o -name
\*.lo -print | $NL2SP`
+ my_oldobjs="$my_oldobjs "`find $my_xdir -name \*.$objext -print -o -name
\*.lo -print | sort | $NL2SP`
done
func_extract_archives_result="$my_oldobjs"
--
2.13.7
