[Bug bootstrap/93398] PGO bootstrap is not reproducible
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93398 Martin Liška changed: What|Removed |Added Resolution|WORKSFORME |--- Status|RESOLVED|REOPENED --- Comment #10 from Martin Liška --- It's so easy to end up with a different binary for GCC: @@ -174,7 +174,7 @@ .//objdir/gcc/c/c-decl.gcda:01ab: 0:COUNTERS average 0 counts .//objdir/gcc/c/c-decl.gcda:01ad: 0:COUNTERS ior 0 counts .//objdir/gcc/c/c-decl.gcda:01af: 2:COUNTERS time_profiler 1 counts -.//objdir/gcc/c/c-decl.gcda: 0: 2419 +.//objdir/gcc/c/c-decl.gcda: 0: 2420 .//objdir/gcc/c/c-decl.gcda: 0100: 3:FUNCTION ident=1446792123, lineno_checksum=0x069e6627, cfg_checksum=0x4c361918 .//objdir/gcc/c/c-decl.gcda:01a1: 28:COUNTERS arcs 14 counts .//objdir/gcc/c/c-decl.gcda: 0: 29954 2 2 28793 0 0 873 0 @@ -184,7 +184,7 @@ .//objdir/gcc/c/c-decl.gcda:01ab: 0:COUNTERS average 0 counts .//objdir/gcc/c/c-decl.gcda:01ad: 0:COUNTERS ior 0 counts .//objdir/gcc/c/c-decl.gcda:01af: 2:COUNTERS time_profiler 1 counts -.//objdir/gcc/c/c-decl.gcda: 0: 2421 +.//objdir/gcc/c/c-decl.gcda: 0: 2422 .//objdir/gcc/c/c-decl.gcda: 0100: 3:FUNCTION ident=1033776594, lineno_checksum=0x0314bcea, cfg_checksum=0xa707611a .//objdir/gcc/c/c-decl.gcda:01a1: 16:COUNTERS arcs 8 counts .//objdir/gcc/c/c-decl.gcda: 0: 50 16 0 50 0 50 0 0 So a slightly different tp_first_run can cause different function order: $ objfolderdiff.py /home/marxin/BIG/buildbot/buildworker/marxinbox-gcc-reproducible/base/objdir/gcc/ /home/marxin/BIG/buildbot/buildworker/marxinbox-gcc-reproducible/next/objdir/gcc/ -d 103/ 624: c/c-decl.o: different --- /home/marxin/BIG/buildbot/buildworker/marxinbox-gcc-reproducible/base/objdir/gcc/c/c-decl.o.s.txt 2020-04-07 17:43:18.280993473 +0200 +++ /home/marxin/BIG/buildbot/buildworker/marxinbox-gcc-reproducible/next/objdir/gcc/c/c-decl.o.s.txt 2020-04-07 17:43:18.280993473 +0200 @@ -34570,15 +34570,6 @@ 36: 89 f8 mov%edi,%eax 38: c3 retq -Disassembly of section .text._Z17ggc_cleared_allocI9lang_typeEPT_v: - - <_Z17ggc_cleared_allocI9lang_typeEPT_v>: - 0: b9 01 00 00 00 mov$0x1,%ecx - 5: 31 d2 xor%edx,%edx - 7: 31 f6 xor%esi,%esi - 9: bf 20 00 00 00 mov$0x20,%edi - e: e9 00 00 00 00 jmpq 13 - Disassembly of section .text.hot._Z15hash_table_mod2jj: <_Z15hash_table_mod2jj>: @@ -34618,6 +34609,15 @@ } 3c: c3 retq +Disassembly of section .text._Z17ggc_cleared_allocI9lang_typeEPT_v: + + <_Z17ggc_cleared_allocI9lang_typeEPT_v>: + 0: b9 01 00 00 00 mov$0x1,%ecx + 5: 31 d2 xor%edx,%edx + 7: 31 f6 xor%esi,%esi + 9: bf 20 00 00 00 mov$0x20,%edi + e: e9 00 00 00 00 jmpq 13 + Disassembly of section .text.hot._ZN2wi5neg_pI16generic_wide_intI20wide_int_ref_storageILb1ELb0EbRKT_6signop: <_ZN2wi5neg_pI16generic_wide_intI20wide_int_ref_storageILb1ELb0EbRKT_6signop>:
[Bug bootstrap/93398] PGO bootstrap is not reproducible
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93398 --- Comment #9 from Jan Hubicka --- Also forgot to mention, it is also quite possible that some of the small divergences propagate themselves into differences in the histogram and thus differences in hot/cold decisions in completely unrelated code...
[Bug bootstrap/93398] PGO bootstrap is not reproducible
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93398 --- Comment #8 from Jan Hubicka --- The divergence for non-randomization seems under control. But it also seems that the divergence for randomized build is way too common to be explained by differences in number of conflicts for hashtables hasing pointers. That could probably be worked around by adding no_instrument attribute to the hashtable and pointer hashers Honza
[Bug bootstrap/93398] PGO bootstrap is not reproducible
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93398 --- Comment #7 from Martin Liška --- Created attachment 47697 --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=47697=edit GCOV dump diff There's still some difference, but rather small.
[Bug bootstrap/93398] PGO bootstrap is not reproducible
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93398 Martin Liška changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |WORKSFORME --- Comment #6 from Martin Liška --- Ok, disabled ASLR helped here. Binaries are identical.
[Bug bootstrap/93398] PGO bootstrap is not reproducible
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93398 --- Comment #5 from Martin Liška --- And there's also a difference in TOPN counters: diff -u a b --- a 2020-01-23 10:30:56.527098370 +0100 +++ b 2020-01-23 10:30:42.035189528 +0100 @@ -1,21 +1,21 @@ -== Stats for after == +== Stats for before == stats for indirect_call: total: 9210 - invalid: 527 + invalid: 523 tracked values: 0 values: 7746 times (84.10%) 1 values: 827 times (8.98%) 2 values: 52 times (0.56%) 3 values: 51 times (0.55%) -4 values:7 times (0.08%) +4 values: 11 times (0.12%) stats for topn: total: 1513 - invalid: 129 + invalid: 128 tracked values: 0 values: 1310 times (86.58%) 1 values: 49 times (3.24%) 2 values:9 times (0.59%) 3 values:4 times (0.26%) -4 values: 12 times (0.79%) +4 values: 13 times (0.86%)
[Bug bootstrap/93398] PGO bootstrap is not reproducible
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93398 --- Comment #4 from Martin Liška --- (In reply to Jakub Jelinek from comment #2) > Couldn't e.g. address space randomization of the instrumented binaries > (compiler itself) matter? I must confirm that the machine where I run it has ASLR enabled. I'm rerunning that with no randomization.
[Bug bootstrap/93398] PGO bootstrap is not reproducible
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93398 --- Comment #3 from Richard Biener --- (In reply to Jakub Jelinek from comment #2) > Couldn't e.g. address space randomization of the instrumented binaries > (compiler itself) matter? I bet some hash-tables could end up with different access patterns because of that. Also we use "randomness" (for temporary filenames, etc.) which also can result in some counter differences in routines processing those. But yes, individual inspection of the differences might uncover real issues (but for a start I'd disable address-space randomization).
[Bug bootstrap/93398] PGO bootstrap is not reproducible
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93398 Jakub Jelinek changed: What|Removed |Added CC||jakub at gcc dot gnu.org --- Comment #2 from Jakub Jelinek --- Couldn't e.g. address space randomization of the instrumented binaries (compiler itself) matter?
[Bug bootstrap/93398] PGO bootstrap is not reproducible
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93398 Martin Liška changed: What|Removed |Added Status|UNCONFIRMED |NEW Last reconfirmed||2020-01-23 Ever confirmed|0 |1 --- Comment #1 from Martin Liška --- Ok, I see the difference in the following areas: 1) sum_max is a bit different: insn-extract.gcda: a100: 2:OBJECT_SUMMARY runs=1780, sum_max=867278542 insn-extract.gcda: a100: 2:OBJECT_SUMMARY runs=1780, sum_max=867190916 2) some arc counters are different: cfgexpand.gcda: 0100: 3:FUNCTION ident=79635253, lineno_checksum=0x420a8035, cfg_checksum=0x4c361918 cfgexpand.gcda:01a1: 28:COUNTERS arcs 14 counts cfgexpand.gcda: 0: 377050 12136 12136 176111 0 0 187454 0 cfgexpand.gcda: 8: 185946 0 0 0 0 377050 cfgexpand.gcda: 0100: 3:FUNCTION ident=79635253, lineno_checksum=0x420a8035, cfg_checksum=0x4c361918 cfgexpand.gcda:01a1: 28:COUNTERS arcs 14 counts cfgexpand.gcda: 0: 377050 12136 12136 176261 0 0 186497 0 cfgexpand.gcda: 8: 186779 0 0 0 0 377050 That leads to a difference in 1) 3) time_profiler is sometimes slightly different 4) some functions have a different lineno_checksum: tree-ssa-reassoc.gcda: 0100: 3:FUNCTION ident=123001947, lineno_checksum=0xb99798cf, cfg_checksum=0xa43083b8 tree-ssa-reassoc.gcda: 0100: 3:FUNCTION ident=123001947, lineno_checksum=0x8e05ef3c, cfg_checksum=0xa43083b8 I'll investigate that, it should not happen. 5) some ior counters are different: c-family/c-format.gcda:01ad: 4:COUNTERS ior 2 counts c-family/c-format.gcda: 0: 140734601582032 0 c-family/c-format.gcda:01ad: 4:COUNTERS ior 2 counts c-family/c-format.gcda: 0: 140727851425584 0 These probably track some pointer alignment.
