[Bug go/104832] gccgo / libgo Reproducibility Problem
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104832 --- Comment #7 from Toolybird --- I just stumbled across this openSUSE bug report which may or may not be related: https://bugzilla.opensuse.org/show_bug.cgi?id=1188621 Interesting that it mentions filesystem readdir which I hadn't considered as a possible factor. For the record, I'm building on ext4.
[Bug go/104832] gccgo / libgo Reproducibility Problem
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104832 --- Comment #6 from Toolybird --- Created attachment 52588 --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=52588=edit archive/zip.o differences
[Bug go/104832] gccgo / libgo Reproducibility Problem
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104832 --- Comment #5 from Toolybird --- Created attachment 52587 --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=52587=edit image/gif.o differences No worries. I'm attaching 2 diffs pertaining to "image/gif.o" and "archive/zip.o". Thanks
[Bug go/104832] gccgo / libgo Reproducibility Problem
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104832 --- Comment #3 from Toolybird --- Thanks for the quick patch! Things are a bit different now but unfortunately the problem persists. It appears there is an element of randomness to this problem as the list of affected object files has changed. Some have disappeared, some new news ones have appeared. Here's a diff of the list between my first run and now with your patch: gcc-build/x86_64-pc-linux-gnu/libgo/archive/.libs/zip.o gcc-build/x86_64-pc-linux-gnu/libgo/archive/zip.o -gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/bug.o -gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/clean.o +gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/cache.o gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/envcmd.o gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/get.o gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/help.o +gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/list.o +gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/load.o +gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/modcmd.o gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/modfetch.o gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/modget.o -gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/test.o +gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/modload.o gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/workcmd.o gcc-build/x86_64-pc-linux-gnu/libgo/cmd/go/internal/work.o -gcc-build/x86_64-pc-linux-gnu/libgo/crypto/.libs/x509.o -gcc-build/x86_64-pc-linux-gnu/libgo/crypto/tls.o -gcc-build/x86_64-pc-linux-gnu/libgo/debug/.libs/macho.o -gcc-build/x86_64-pc-linux-gnu/libgo/fmt.o -gcc-build/x86_64-pc-linux-gnu/libgo/go/ast.o +gcc-build/x86_64-pc-linux-gnu/libgo/golang.org/x/mod/modfile.o +gcc-build/x86_64-pc-linux-gnu/libgo/golang.org/x/tools/go/analysis/passes/printf.o +gcc-build/x86_64-pc-linux-gnu/libgo/go/.libs/build.o gcc-build/x86_64-pc-linux-gnu/libgo/html/.libs/template.o gcc-build/x86_64-pc-linux-gnu/libgo/html/template.o gcc-build/x86_64-pc-linux-gnu/libgo/image/gif.o gcc-build/x86_64-pc-linux-gnu/libgo/image/.libs/gif.o +gcc-build/x86_64-pc-linux-gnu/libgo/image/png.o gcc-build/x86_64-pc-linux-gnu/libgo/index/suffixarray.o -gcc-build/x86_64-pc-linux-gnu/libgo/.libs/fmt.o -gcc-build/x86_64-pc-linux-gnu/libgo/.libs/os.o gcc-build/x86_64-pc-linux-gnu/libgo/.libs/runtime.o -gcc-build/x86_64-pc-linux-gnu/libgo/.libs/syscall.o gcc-build/x86_64-pc-linux-gnu/libgo/net/http.o gcc-build/x86_64-pc-linux-gnu/libgo/net/.libs/http.o gcc-build/x86_64-pc-linux-gnu/libgo/net/.libs/mail.o gcc-build/x86_64-pc-linux-gnu/libgo/net/.libs/rpc.o gcc-build/x86_64-pc-linux-gnu/libgo/net/mail.o gcc-build/x86_64-pc-linux-gnu/libgo/net/rpc.o -gcc-build/x86_64-pc-linux-gnu/libgo/os.o -gcc-build/x86_64-pc-linux-gnu/libgo/syscall.o -gcc-build/x86_64-pc-linux-gnu/libgo/testing.o +gcc-build/x86_64-pc-linux-gnu/libgo/runtime/.libs/pprof.o +gcc-build/x86_64-pc-linux-gnu/libgo/runtime.o +gcc-build/x86_64-pc-linux-gnu/libgo/runtime/pprof.o gcc-build/x86_64-pc-linux-gnu/libgo/text/.libs/template.o -gcc-build/x86_64-pc-linux-gnu/libgo/text/template.o Is it possible that ASLR is a factor here? That was indeed the case for another bug fixed by Jakub last year [1] Thanks again. [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101407
[Bug go/104832] New: gccgo / libgo Reproducibility Problem
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104832 Bug ID: 104832 Summary: gccgo / libgo Reproducibility Problem Product: gcc Version: unknown Status: UNCONFIRMED Severity: normal Priority: P3 Component: go Assignee: ian at airs dot com Reporter: toolybird at tuta dot io CC: cmang at google dot com Target Milestone: --- TL;DR -> Different binaries are produced depending on whether gccgo is already installed on the host or not. Background: I'm looking into broad issues of reproducibility in the context of bootstrapping whole toolchains. I first noticed this problem with gcc-11.2.0 but it's still present on GCC trunk. To reproduce, you will need to have the same version of gccgo installed on your host in /usr. For example, with current GCC trunk: $ cat gcc/BASE-VER $ 12.0.1 Step 1: bootstrap GCC with `--enable-languages=c,c++,go --prefix=/usr --libexecdir=/usr/lib' then install into a temp DESTDIR. Step 2: Uninstall gccgo from the host, or you can fudge it with: $ sudo mv /usr/lib/go /usr/lib/go.XX Step 3: bootstrap GCC again with `--enable-languages=c,c++,go --prefix=/usr --libexecdir=/usr/lib' then install into a 2nd temp DESTDIR. Step 4: diff / compare the binaries in the DESTDIRs and you will see differences. Delving deeper and diffing the gcc-build dirs, there are a number of object files affected. A list sorted with `ls -altr' is at the end of this report. Here is a demonstration with "libgo/fmt.o" (although I suspect the real problem is further up the chain): $ cd gcc-build/x86_64-pc-linux-gnu/libgo $ /build/gcc/src/gcc-build/./gcc/gccgo -B/build/gcc/src/gcc-build/./gcc/ -minline-all-stringops -O2 -g -I . -c -fgo-pkgpath=fmt ../../../gcc/libgo/go/fmt/doc.go ../../../gcc/libgo/go/fmt/errors.go ../../../gcc/libgo/go/fmt/format.go ../../../gcc/libgo/go/fmt/print.go ../../../gcc/libgo/go/fmt/scan.go -o fmt-test1.o $ sudo mv /usr/lib/go /usr/lib/go.XX $ /build/gcc/src/gcc-build/./gcc/gccgo -B/build/gcc/src/gcc-build/./gcc/ -minline-all-stringops -O2 -g -I . -c -fgo-pkgpath=fmt ../../../gcc/libgo/go/fmt/doc.go ../../../gcc/libgo/go/fmt/errors.go ../../../gcc/libgo/go/fmt/format.go ../../../gcc/libgo/go/fmt/print.go ../../../gcc/libgo/go/fmt/scan.go -o fmt-test2.o $ diff fmt-test1.o fmt-test2.o Binary files fmt-test1.o and fmt-test2.o differ The differences are in the ".go_export" section of the ELF file. Using "strings" helps to visualize the differences: $ strings fmt-test1.o > 1.txt $ strings fmt-test2.o > 2.txt $ diff -u 1.txt 2.txt --- 1.txt 2022-03-08 11:07:01.577545105 +1100 +++ 2.txt 2022-03-08 11:07:06.364263769 +1100 @@ -860,7 +860,7 @@ import utf8 unicode/utf8 "unicode/utf8" init fmt fmt..import errors errors..import cpu internal_1cpu..import oserror internal_1oserror..import poll internal_1poll..import reflectlite internal_1reflectlite..import testlog internal_1testlog..import io io..import fs io_1fs..import math math..import os os..import path path..import reflect reflect..import runtime runtime..import sort sort..import strconv strconv..import sync sync..import syscall syscall..import time time..import unicode unicode..import abi ~internal_1abi bytealg ~internal_1bytealg fmtsort ~internal_1fmtsort goarch ~internal_1goarch goexperiment ~internal_1goexperiment goos ~internal_1goos itoa ~internal_1itoa race ~internal_1race execenv ~internal_1syscall_1execenv unix ~internal_1syscall_1unix unsafeheader ~internal_1unsafeheader bits ~math_1bits atomic ~runtime_1internal_1atomic math ~runtime_1internal_1math sys ~runtime_1internal_1sys atomic ~sync_1atomic utf8 ~unicode_1utf8 init_graph 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 11 0 12 0 13 0 14 0 15 0 16 0 17 0 18 0 19 1 2 1 5 1 13 3 1 3 2 3 5 3 13 4 1 4 2 4 3 4 5 4 7 4 13 4 16 4 17 4 18 5 2 5 13 6 2 6 13 6 16 7 1 7 2 7 5 7 13 7 16 8 1 8 2 8 3 8 5 8 7 8 11 8 13 8 14 8 16 8 17 8 18 9 2 10 1 10 2 10 3 10 4 10 5 10 6 10 7 10 8 10 11 10 13 10 14 10 16 10 17 10 18 11 1 11 2 11 5 11 13 12 1 12 2 12 5 12 9 12 13 12 15 12 16 12 19 13 2 14 2 14 5 14 13 15 1 15 2 15 5 15 9 15 13 16 2 16 13 17 1 17 2 17 3 17 5 17 13 17 16 18 1 18 2 18 3 18 5 18 13 18 16 18 17 -types 41 7 29 30 29 27 25 28 1067 3172 30 335 26 30 34 35 30 88 92 47 88 47 34 34 18 18 19 115 171 21 22 195 60 46 83 61 73 295 73 44 74 22 +types 41 7 29 30 29 27 25 28 1067 3172 30 335 26 30 34 35 30 88 92 47 88 47 38 34 18 18 19 115 171 21 22 195 60 46 83 61 73 295 73 44 74 22 type 1 "Formatter" type 2 "GoStringer" type 3 "ScanState" @@ -972,7 +972,7 @@ type 18 (? , ? ) type 19 (? , ? , ? , ? , ? ) type 20 (? , ? ) -type 21 (? ) +type 21 (? ) (? ) type 22 (? ) type 23 * type 24 * @@ -1067,7 +1067,7 @@ $ret10 = $convert(, $false) //577 return //577 } //0 -checksum 22D2C8F1A933DC5F8B8E981917B8ADA04DEB2FC1 +checksum A916C151E089AA7EC237CBEBBE2B293A9595478C ScanState's Read should not be called. Use ReadRune fmt:
[Bug go/101407] non-determinism in -fdump-go-spec
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101407 --- Comment #4 from Toolybird --- Your patch solves the problem for me. Thank you very much!
[Bug go/101407] non-determinism in -fdump-go-spec
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101407 --- Comment #2 from Toolybird --- > Will try to bisect Well, that was a complete waste of time. There seems to an element of randomness to the problem. It turns out that GCC 10 is also affected as I was able to trigger it all the way back to basepoints/gcc-10 I give up for the moment. Hopefully someone who cares about binary reproducibility sees this, is able to replicate and fix.
[Bug go/101407] non-determinism in -fdump-go-spec
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101407 --- Comment #1 from Toolybird --- The bug is present on trunk. Will try to bisect...
[Bug go/101407] New: non-determinism in -fdump-go-spec
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101407
Bug ID: 101407
Summary: non-determinism in -fdump-go-spec
Product: gcc
Version: 11.1.0
Status: UNCONFIRMED
Severity: normal
Priority: P3
Component: go
Assignee: ian at airs dot com
Reporter: toolybird at tuta dot io
CC: cmang at google dot com
Target Milestone: ---
I'm seeing a reproducibility issue when building gcc-go in GCC 11. Every build
run produces a different binary.
I've narrowed down the test case to essentially this:
$ gcc -fdump-go-spec=tmp-1.go -S -o sysinfo.s sysinfo.c
$ gcc -fdump-go-spec=tmp-2.go -S -o sysinfo.s sysinfo.c
$ diff -u tmp-1.go tmp-2.go
--- tmp-1.go2021-07-11 07:26:58.512916883 +1000
+++ tmp-2.go2021-07-11 07:27:07.976340655 +1000
@@ -8519,10 +8519,10 @@
const _PRIxFAST8 = "x"
const ___POSIX_FADV_DONTNEED = 4
const _IPPROTO_MTP = 92
-type ___dirstream struct {}
-type ___va_list_tag struct {}
type _iface struct {}
type ___locale_data struct {}
type __IO_marker struct {}
type __IO_codecvt struct {}
type __IO_wide_data struct {}
+type ___dirstream struct {}
+type ___va_list_tag struct {}
The file in question is libgo/sysinfo.c
The output seems to differ each time. Occasionally it is the same.
This didn't happen with GCC 10. I suppose I should try the trunk but haven't
done so yet.
Any thoughts? Thanks.
[Bug driver/101383] GCC 11 Reproducibility Issue
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101383 --- Comment #6 from Toolybird --- The patch you posted to gcc-patches solves my problem. Thank you for the quick fix!
[Bug other/101383] New: GCC 11 Reproducibility Issue
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101383 Bug ID: 101383 Summary: GCC 11 Reproducibility Issue Product: gcc Version: 11.1.0 Status: UNCONFIRMED Severity: normal Priority: P3 Component: other Assignee: unassigned at gcc dot gnu.org Reporter: toolybird at tuta dot io Target Milestone: --- Performing 2 consecutive runs of a normal 3-stage bootstrap results in differing cc1 and cc1plus binaries. This appears to be a regression since GCC 10 does not exhibit the problem. The issue can be pinpointed to cc1-checksum.c and cc1plus-checksum.c I know very little about this stuff but I suspect the root cause is due to the move to DWARF 5 Please note: This is a standard `make' which means default BOOT_CFLAGS of "-g -O2". Comparing the saved object files between runs with `readelf -w' reveals differences with GCC 11 whereas with GCC 10 they are identical. For example with GCC 11: $ cmp -lb gcc-build.1st/prev-gcc/c/c-lang.o gcc-build.2nd/prev-gcc/c/c-lang.o 1932 130 X114 L 1933 62 2 60 0 1934 171 y142 b 1935 127 W 71 9 1936 150 h153 k 1937 121 Q126 V 1943 130 X114 L 1944 62 2 60 0 1945 171 y142 b 1946 127 W 71 9 1947 150 h153 k 1948 121 Q126 V 2066 130 X114 L 2067 62 2 60 0 2068 171 y142 b 2069 127 W 71 9 2070 150 h153 k 2071 121 Q126 V $ readelf -w gcc-build.1st/prev-gcc/c/c-lang.o > 1.txt $ readelf -w gcc-build.2nd/prev-gcc/c/c-lang.o > 2.txt $ diff -u 1.txt 2.txt --- 1.txt 2021-07-09 17:20:09.504532430 +1000 +++ 2.txt 2021-07-09 17:20:26.174675615 +1000 @@ -34,8 +34,8 @@ The File Name Table (offset 0x30, lines 2, columns 2): EntryDir Name - 00 (indirect line string, offset: 0xa): ccX2yWhQ.s - 11 (indirect line string, offset: 0x15): ccX2yWhQ.s + 00 (indirect line string, offset: 0xa): ccL0b9kV.s + 11 (indirect line string, offset: 0x15): ccL0b9kV.s Line Number Statements: [0x003a] Extended opcode 2: set Address to 0x0 @@ -55,8 +55,8 @@ Contents of the .debug_line_str section: - 0x 2f746d70 002f746d 70006363 58327957 /tmp./tmp.ccX2yW - 0x0010 68512e73 00636358 32795768 512e7300 hQ.s.ccX2yWhQ.s. + 0x 2f746d70 002f746d 70006363 4c306239 /tmp./tmp.ccL0b9 + 0x0010 6b562e73 0063634c 3062396b 562e7300 kV.s.ccL0b9kV.s. <...snip...> >From looking at the Makefiles, it appears `cc1-checksum.c' is copied from the previous stage for some reason, which I think might be pertinent. It also seems like different flags are used between stages 2 and 3: -DIN_GCC_FRONTEND -g -O2 -fno-checking -gtoggle -DIN_GCC -DIN_GCC_FRONTEND -g -O2 -fchecking=1 -DIN_GCC Any comments? Thanks.
