Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:  closed
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810, tbb-backported  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by gk):

 * keywords:  tbb-rbm, TorBrowserTeam201810R, GeorgKoppen201810, tbb-
 backport => tbb-rbm, TorBrowserTeam201810R, GeorgKoppen201810, tbb-
 backported


Comment:

 Backported to our stable branch (`maint-8.0`) (commit
 dd1d00a2fed85c8bb22c4f85bbc45da00d0e8c05).

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:  closed
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810, tbb-backport|
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by alexcrichton):

 Ok, sounds good to me!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:  closed
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810, tbb-backport|
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by gk):

 Replying to [comment:60 alexcrichton]:
 > Oh so if that's all true, another question, what version of rustc is
 this using?

 The one at 6b7b6b63a928479a29d9fc1282e553e409c66934 with a bunch of
 patches for the LLVM bisecting.

 > And additionally, y'all are seeing reproducibility issues on the current
 nightly release of Rust, right? (latest LLVM we're using as well as latest
 rustc). IIRC though those issues haven't been reduced yet and it's
 suspected that this one is the cause?

 I *think* we could be lucky in that it got fixed more or less recently
 during the upgrade to LLVM 7. The result is not conclusive yet as there
 popped another reproducibility issue up (see: comment:28).

 So, I'd suggest waiting with filing an LLVM bug until I have bisected that
 other issue.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:  closed
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810, tbb-backport|
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by gk):

 Replying to [comment:59 alexcrichton]:
 > Ok so just to make sure I understand, LLVM is completely deterministic
 up to and including r304593 - https://reviews.llvm.org/D33320. When you go
 one more commit to r304594 - https://reviews.llvm.org/D32653 - this LLVM
 is no longer deterministic. The commit in question here that makes LLVM
 nondeterministic is enabling r301649 - https://reviews.llvm.org/D31085 - a
 heuristic for something.

 Yes.

 > This was all tested with an adapted script, where you're compiling a
 fixed version of rustc against a varying version of LLVM. The rustc linked
 with LLVM from r304593 is deterministic and the rustc linked with LLVM
 form r304594 is nondeterministic. The script is then an adaptation of the
 comment you mentioned.

 Yes.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:  closed
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810, tbb-backport|
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by alexcrichton):

 Oh so if that's all true, another question, what version of rustc is this
 using?

 And additionally, y'all are seeing reproducibility issues on the current
 nightly release of Rust, right? (latest LLVM we're using as well as latest
 rustc). IIRC though those issues haven't been reduced yet and it's
 suspected that this one is the cause?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:  closed
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810, tbb-backport|
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by alexcrichton):

 Ok so just to make sure I understand, LLVM is completely deterministic up
 to and including r304593 - https://reviews.llvm.org/D33320. When you go
 one more commit to r304594 - https://reviews.llvm.org/D32653 - this LLVM
 is no longer deterministic. The commit in question here that makes LLVM
 nondeterministic is enabling r301649 - https://reviews.llvm.org/D31085 - a
 heuristic for something.

 This was all tested with an adapted script, where you're compiling a fixed
 version of rustc against a varying version of LLVM. The rustc linked with
 LLVM from r304593 is deterministic and the rustc linked with LLVM form
 r304594 is nondeterministic. The script is then an adaptation of the
 comment you mentioned.

 Does that all sound right? If so I think that's definitely enough to open
 an issue on LLVM itself and try to get the ball rolling there! It'd be
 maximally useful to have a standalone test case, but I sort of suspect
 this may not have an easy standalone test case and is related to how rustc
 is using LLVM on multiple threads internally, which LLVM's CLI tools don't
 do.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:  closed
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810, tbb-backport|
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by gk):

 Replying to [comment:57 alexcrichton]:
 > The first commit you mentioned - https://reviews.llvm.org/D33320 - looks
 like it's just some minor renamings?

 Oops, right. That's the last *good* commit. The problem is r304594 (which
 is enabling r301649)

 > The second though - https://reviews.llvm.org/D32653 - definitely looks
 more suspicious. Since the second commit is turning a flag on by default,
 could you take a "good LLVM" just before that commit, turning the flag on,
 and seeing if it has the same reproducibility issue?

 I can do that with r305493 (i.e. the last good commit) but given that
 r305494 is just flipping that feature on by default (and removing unused
 code) I doubt this will give us new insights. Let me know, though, if you
 think otherwise.

 > Otherwise, do you have a standalone LLVM test case you were testing
 with? Or was it largely always through rustc? I don't mind helping out to
 file a bug in LLVM!

 I don't have a standalone LLVM testcase, alas. Not sure how I would write
 a proper one for this usecase. :( I bisected through rustc adapting your
 test script in https://github.com/rust-
 lang/rust/issues/52044#issuecomment-402349038.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:  closed
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810, tbb-backport|
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by alexcrichton):

 The first commit you mentioned - https://reviews.llvm.org/D33320 - looks
 like it's just some minor renamings? The second though -
 https://reviews.llvm.org/D32653 - definitely looks more suspicious. Since
 the second commit is turning a flag on by default, could you take a "good
 LLVM" just before that commit, turning the flag on, and seeing if it has
 the same reproducibility issue?

 Otherwise, do you have a standalone LLVM test case you were testing with?
 Or was it largely always through rustc? I don't mind helping out to file a
 bug in LLVM!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:  closed
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810, tbb-backport|
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by gk):

 Finally, after bisecting compiler crashes (aka
 https://bugs.llvm.org/show_bug.cgi?id=33917), too, I got the problematic
 commit. It is r304593 (together with r301649). Let me know if I can be of
 any help if that does not make sense on first glance. :) I am happy to
 help understanding what is going wrong here.

 That said, I'll start figuring out why the issue vanished as I could not
 reproduce it anymore taking a nightly from mid-September (I found that
 different IRs showed up after the switch to LLVM 7, see: comment:28 which
 is replacing the issue in this bug with another reproducibility problem).

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:  closed
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810, tbb-backport|
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by gk):

 * status:  needs_review => closed
 * keywords:  tbb-rbm, TorBrowserTeam201810R, GeorgKoppen201810 => tbb-rbm,
 TorBrowserTeam201810R, GeorgKoppen201810, tbb-backport
 * resolution:   => fixed


Comment:

 Replying to [comment:54 boklm]:
 > Other than this issue with `mar-tools-mac64.zip` (which seems
 unrelated), the patch looks good to me.

 I opened #28102 for that.

 Thanks for the review and, Alex, thanks for your help! I merged the patch
 to `master` (commit 0166d3d70e81c6c1072e03f6b950c8b2eb181343), marking it
 for backport as we want Stylo enabled for macOS and potentially fix the
 reproducibility issue on Linux.

 I'll update this ticket with my bisecting results once I am done, so that
 we have a chance to understand what actually went wrong and how it got
 fixed later on.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:
 |  needs_review
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by boklm):

 Other than this issue with `mar-tools-mac64.zip` (which seems unrelated),
 the patch looks good to me.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:
 |  needs_review
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by boklm):

 Replying to [comment:52 boklm]:

 > However, I also got two non matching `mar-tools-mac64.zip` files.
 > https://people.torproject.org/~boklm/tmp/bug_26475/1/mar-tools-
 mac64.zip.asc
 > https://people.torproject.org/~boklm/tmp/bug_26475/1/mar-tools-mac64.zip
 > https://people.torproject.org/~boklm/tmp/bug_26475/2/mar-tools-
 mac64.zip.asc
 > https://people.torproject.org/~boklm/tmp/bug_26475/2/mar-tools-mac64.zip

 If you want to compare with your build, those `mar-tools-mac64.zip` have
 been generated with `make testbuild` on branch `bug_26475_v2`, with
 `testbuild` configured to do `alpha` builds.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:
 |  needs_review
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by boklm):

 Replying to [comment:48 gk]:

 >
 > `bug_26475_v2` (https://gitweb.torproject.org/user/gk/tor-browser-
 build.git/commit/?h=bug_26475_v2=e2da10a29f46762a8561e30c36915f8568d18c71)
 has the proposed workaround.

 After doing two separates `testbuild` using this branch, I got two non
 matching `TorBrowser-8.5a3-osx64_en-US.dmg` files. After checking the
 content of the dmg files, only the snowflake files are different, so this
 is probably #27827. So it seems the issue from this ticket is fixed.

 However, I also got two non matching `mar-tools-mac64.zip` files.
 https://people.torproject.org/~boklm/tmp/bug_26475/1/mar-tools-
 mac64.zip.asc
 https://people.torproject.org/~boklm/tmp/bug_26475/1/mar-tools-mac64.zip
 https://people.torproject.org/~boklm/tmp/bug_26475/2/mar-tools-
 mac64.zip.asc
 https://people.torproject.org/~boklm/tmp/bug_26475/2/mar-tools-mac64.zip

 I don't know if the non-matching mar-tools is caused by this patch, or
 something else.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:
 |  needs_review
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by gk):

 Replying to [comment:50 alexcrichton]:
 > Glad that ended up working out! From my testing oh-so-long-ago at this
 point I suspected this was an LLVM issue, and disabling ThinLTO
 effectively serializes all of LLVM's work, so I wonder if this is a race
 somewhere in LLVM between some shared data structure across threads which
 affects ordering...

 Maybe. I am still bisecting LLVM (what introduced the issue and what fixed
 it as the latter is not clear either) which is... non-trivial. I'll update
 this ticket with my findings.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:
 |  needs_review
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by alexcrichton):

 Glad that ended up working out! From my testing oh-so-long-ago at this
 point I suspected this was an LLVM issue, and disabling ThinLTO
 effectively serializes all of LLVM's work, so I wonder if this is a race
 somewhere in LLVM between some shared data structure across threads which
 affects ordering...

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:
 |  needs_review
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by gk):

 Replying to [comment:48 gk]:
 > I made some testbuilds and tested the compiler against the adapted
 script from https://github.com/rust-
 lang/rust/issues/52044#issuecomment-402349038 and I got matching builds
 and the script is happy now.

 FWIW: I am actually not sure if I solved #27937 with the patch as I was
 never able to reproduce the Linux issue. Looking at the diff my best guess
 is that it might not be `panic` but still LTO related. And the idea is to
 have the patch up for review squash both cases.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:
 |  needs_review
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tbb-rbm, TorBrowserTeam201810R,  |  Actual Points:
  GeorgKoppen201810  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by gk):

 * keywords:  tbb-rbm, TorBrowserTeam201810, GeorgKoppen201810 => tbb-rbm,
 TorBrowserTeam201810R, GeorgKoppen201810
 * status:  new => needs_review


Comment:

 I have a patch that sets `codegen-units` to `1` which seems the way to
 disably Thin LTO. Alas, there is no direct option for disabling LTO yet in
 1.26.1. Moreover, using the `--set` configure option to adjust the
 `codegen-units` does not seem to work as the value gets interpreted as a
 string but we need an integer. So, I resorted to good ol' `sed`.

 I made some testbuilds and tested the compiler against the adapted script
 from https://github.com/rust-lang/rust/issues/52044#issuecomment-402349038
 and I got matching builds and the script is happy now.

 `bug_26475_v2` (https://gitweb.torproject.org/user/gk/tor-browser-
 build.git/commit/?h=bug_26475_v2=e2da10a29f46762a8561e30c36915f8568d18c71)
 has the proposed workaround.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:  new
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tbb-rbm, TorBrowserTeam201810,   |  Actual Points:
  GeorgKoppen201810  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by alexcrichton):

 One way I might be able to still help as well is narrowing down *where*
 the nondeterminism is introduced. If you use `-C save-temps` when
 compiling the compiler should spew dozens of files all over the place.
 Each of these files in theory represents the various stages of compilation
 and provides snapshots into the compiler's pipeline. If you could find the
 set of files that are nondeterministic (we know it's at least the object
 files!) then that may help narrow this down as well!

 If a 16 CGU libstd + full crate LTO is the issue it sounds like this may
 be an issue with the LLVM "linker", but that's just a guess!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:  new
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tbb-rbm, TorBrowserTeam201810,   |  Actual Points:
  GeorgKoppen201810  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by gk):

 Replying to [comment:45 alexcrichton]:
 > Oh sorry about this, that's a good discovery! Those settings definitely
 engage a lot of LLVM infrastructure that's not otherwise engaged, which
 could help explain why something nondeterministic is coming out the other
 end.
 >
 > The settings in `bootstrap` are pretty confusing, but what's happening
 here is either rustc is compiled with 16 codegen units (each crate turns
 into 16 object files) which are then optimized as a set with ThinLTO or
 each compiler crate is compiled into one object file and no ThinLTO is
 used.
 >
 > To clarify is the compiler's own binary nondeterministic when the
 compiler's crates are built with 16 CGUs + ThinLTO? Or is the compiler's
 binary deterministic but its output deterministic?

 When the compiler's crates (in fact just the macOS libstd + code it
 depends on) are built with 16 CGUs + ThinLTO its *output* is non-
 deterministic but *only* if I use `-C lto` (if I drop `-C lto` all is
 fine, too). The binary might, too, be non-deterministic, I have not
 checked. We are currently not concerned with getting a reproducibly built
 rust compiler. Right now, just the output that compiler gives us matters.

 > In terms of impact of these settings:
 >
 > * 1 CGU builds are basically equivalent to 16 CGUs + ThinLTO. The 1 CGU
 build is slower to compile the compiler (less opportunities for
 parallelism), but is likely 1%-ish faster than the 16 CGUs + ThinLTO.
 > * The 16 CGUs + ThinLTO is the default setting (as you've found)
 > * If 16 CGUs are used without ThinLTO, the resulting compiler is
 probably horrendously slow (lots of missed inlining opportunities).
 >
 > Or put another way, if you disable ThinLTO you'll want to be sure to
 also compile with one codegen unit, which should happen in rustbuild via
 the above block.

 Okay, thanks for those explanations, really helpful.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:  new
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tbb-rbm, TorBrowserTeam201810,   |  Actual Points:
  GeorgKoppen201810  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by alexcrichton):

 Oh sorry about this, that's a good discovery! Those settings definitely
 engage a lot of LLVM infrastructure that's not otherwise engaged, which
 could help explain why something nondeterministic is coming out the other
 end.

 The settings in `bootstrap` are pretty confusing, but what's happening
 here is either rustc is compiled with 16 codegen units (each crate turns
 into 16 object files) which are then optimized as a set with ThinLTO or
 each compiler crate is compiled into one object file and no ThinLTO is
 used.

 To clarify is the compiler's own binary nondeterministic when the
 compiler's crates are built with 16 CGUs + ThinLTO? Or is the compiler's
 binary deterministic but its output deterministic?

 In terms of impact of these settings:

 * 1 CGU builds are basically equivalent to 16 CGUs + ThinLTO. The 1 CGU
 build is slower to compile the compiler (less opportunities for
 parallelism), but is likely 1%-ish faster than the 16 CGUs + ThinLTO.
 * The 16 CGUs + ThinLTO is the default setting (as you've found)
 * If 16 CGUs are used without ThinLTO, the resulting compiler is probably
 horrendously slow (lots of missed inlining opportunities).

 Or put another way, if you disable ThinLTO you'll want to be sure to also
 compile with one codegen unit, which should happen in rustbuild via the
 above block.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:  new
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tbb-rbm, TorBrowserTeam201810,   |  Actual Points:
  GeorgKoppen201810  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by gk):

 Replying to [comment:33 gk]:
 > 2) There are additional pieces getting compiled in/used during the LLVM
 compilation done during the rust build that are causing the problem.

 That's what happening in this case. The relevant part is
 {{{
 if self.config.rust_codegen_units.is_none() &&
self.build.is_rust_llvm(compiler.host) &&
self.config.rust_thinlto {
 cargo.env("RUSTC_THINLTO", "1");
 } else if self.config.rust_codegen_units.is_none() {
 // Generally, if ThinLTO has been disabled for some
 reason, we
 // want to set the codegen units to 1. However, we
 shouldn't do
 // this if the option was specifically set by the user.
 cargo.env("RUSTC_CODEGEN_UNITS", "1");
 }
 }}}
 in `builder.rs`.

 For some reason `RUSTC_THINLTO` is only set if one does not specify a LLVM
 with `--llvm-root` and that is the difference I hit. With it set the
 reproducibility problem emerges (even for LLVMs provided by `--llvm-root`
 if I rip out `self.build.is_rust_llvm(compiler.host) &&`) without it not.

 So, to sum up so far: Compiling with `-C lto` is fine unless
 `RUSTC_THINLTO` is used for compiling rust for macOS. Or to be more
 precise: unless `RUSTC_THINLTO` is used for the apple target libstd and
 related libraries. I could not pinpoint the exact lib that is causing this
 issue, though, yet (I can't easily replace them one by one as otherwise
 rustc is complaining about libstd being in need of recompilation).

 I guess more bisecting is next. :) Alex, do you think we could just avoid
 setting `RUSTC_THINLTO` for now when compiling the rust compiler? Or does
 that have any serious, known downsides?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:  new
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tbb-rbm, TorBrowserTeam201810,   |  Actual Points:
  GeorgKoppen201810  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by gk):

 Okay, a further piece in the puzzle: The problem goes away if I drop `-C
 lto` in the test script and only use

 `rustc /dev/stdin -O -C panic=abort -C codegen-units=1 --emit llvm-ir,obj
 --crate-type staticlib --out-dir a --target x86_64-apple-darwin`

 I tried that because I saw
 {{{
 if self.config.rust_codegen_units.is_none() &&
self.build.is_rust_llvm(compiler.host) &&
self.config.rust_thinlto {
 cargo.env("RUSTC_THINLTO", "1");
 } else if self.config.rust_codegen_units.is_none() {
 // Generally, if ThinLTO has been disabled for some
 reason, we
 // want to set the codegen units to 1. However, we
 shouldn't do
 // this if the option was specifically set by the user.
 cargo.env("RUSTC_CODEGEN_UNITS", "1");
 }
 }}}
 in `builder.rs`.

 I wonder how bad it would be if we just dropped that when building Rust
 code (see: https://dxr.mozilla.org/mozilla-
 esr60/source/config/rules.mk#863) while trying to track the underlying
 issue down.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26475 [Applications/Tor Browser]: ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0 (was: ESR60-based .dmg images are not built reprodu

[Tor Bug Tracker & Wiki]

#26475: ESR60-based Tor Browser bundles are not built reproducibly with Stylo
enabled using rustc > 1.25.0
-+-
 Reporter:  gk   |  Owner:  tbb-
 |  team
 Type:  defect   | Status:  new
 Priority:  Immediate|  Milestone:
Component:  Applications/Tor Browser |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tbb-rbm, GeorgKoppen201809,  |  Actual Points:
  TorBrowserTeam201810   |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by gk):

 * priority:  Very High => Immediate


Old description:

> Contrary to our testing results the .dmg bundles we produce for 8.0a9 are
> differing on two machines.

New description:

 Contrary to our testing results the .dmg bundles we produce for 8.0a9 are
 differing on two machines. And it turns out Linux bundles are affected as
 well (see: #27937 for the original Linux bug report).

--

Comment:

 #27937 is a duplicate: We have similar issues on Linux even though it
 seems they are even harder to reproduce. I assume for now the underlying
 issue is the same for both platforms.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs