Re: Status: ASan builds on Windows

[Chris Peterson]

On 12/15/15 4:36 PM, Emanuel Hoogeveen wrote:

Not to derail the topic, but I wonder how many crashes would be more actionable 
if they were caught earlier by assertions. I believe Chrome Canary has 
assertions enabled for their testing audience. I don't know how pleasant this 
would be for Nightly users (like myself), since Nightly right now is usually 
pretty stable, but turning MOZ_ASSERT into MOZ_RELEASE_ASSERT (at build time) 
for a select audience would at least be easy to do.


You can selectively enable release asserts in pre-release channels using 
MOZ_DIAGNOSTIC_ASSERT. It's defined as MOZ_RELEASE_ASSERT in Nightly and 
Aurora and debug-only MOZ_ASSERT in Beta and Release.


Instead of blindly enabling possibly-flaky MOZ_ASSERTs in Nightly, we 
could start by converting some important asserts to 
MOZ_DIAGNOSTIC_ASSERT (or even MOZ_RELEASE_ASSERT).


https://mxr.mozilla.org/mozilla-central/ident?i=MOZ_DIAGNOSTIC_ASSERT
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Status: ASan builds on Windows

[Emanuel Hoogeveen]

Not to derail the topic, but I wonder how many crashes would be more actionable 
if they were caught earlier by assertions. I believe Chrome Canary has 
assertions enabled for their testing audience. I don't know how pleasant this 
would be for Nightly users (like myself), since Nightly right now is usually 
pretty stable, but turning MOZ_ASSERT into MOZ_RELEASE_ASSERT (at build time) 
for a select audience would at least be easy to do.

Of course, having a mechanism to get a stack for assertions without actually 
crashing would be even better, though I don't know how we'd collect such 
information (Health Report?). That's bug 1209131.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Status: ASan builds on Windows

[Ehsan Akhgari]

> On Dec 15, 2015, at 9:36 AM, Nicolas B. Pierron 
>  wrote:
> 2/ Current Blockers:
> 
> 2.1/ Windows Clang ASan builds are not stable yet.

Have we considered doing this on Linux and Mac?  Windows would probably be the 
last platform where we should try this, given that we build Nightly with a 
different toolchain.

> Apparently we have a few issues opened against clang, and we should push 
> these forward to make sure that we can have a stable build.  Part of the 
> stability issues might be related to the way our build system is working as 
> we are linking crt library both statically and dynamically inside Gecko.

Asan on Windows doesn’t support static linking to the CRT at all.  I thought 
that we stopped using static linking in bug 1194834.  Is this still an issue?  
(I haven’t done one of these builds in quite a while so I don’t know for sure.)

> It seems that part of the issue might be related to the fact that Clang 
> default to cl.exe when Clang fails at compiling, and a guess from Raymond was 
> that we have ABI issues caused by a miss conversion of the command line 
> arguments.

I don’t think that would be a problem.  In theory you should be able to link 
object files with and without ASan turned on into the same executable.

I think this is the order in which we need to proceed:

1. Get clang-cl builds on the infrastructure.
2. Make sure that these builds run and don’t crash on startup.  We have 
previously been there 

 but things have regressed since.  #1 helps with keeping this working once we 
get there again.
3. See if we can run _some_ tests on the infra.  Helps #1 and #2 not regress.
4. Get clang-cl builds with ASan on the infrasturcture.
5. See if we can run _some_ tests on these builds too.
6. Profit.

So far all of this is being blocked on having people to work on it.  I need #1 
for other things so I’m hoping to get that done somehow in 2016.

> If you are interested to talk more, or help fix clang issues, you should 
> contact the following persons:
> 
> - Raymond Forbes [:rforbes]
> - Nathan Froyd [:froydnj]
> - Jeff Muizelaar [:jrmuizel]
> - Ehsan Akhgari [:eshan]
> 
> Also, we have few Clang contributors within Mozilla who might have a bit more 
> knowledge about the persons to ask to get momentum on the clang issues, such 
> as:
> 
> - Jakob Stoklund Olesen [:jolesen]
> - Dan Gohman [:sunfish]

I have written some clang-cl fixes before and we have a pretty healthy 
relationship with the Google compiler team who are working on clang-cl so if we 
get to a point where we have specific issues to fix on the clang side, I’m sure 
we can get them fixed.  Most of the problems however are probably on our side.

> And for our build system issues on Windows, we should contact:
> 
> - Mike Hommey [:glandium]

I believe *most* of the build system integration has been finished.

> 2.2/ Enable Crash reporter on ASan builds.
> 
> The crash reporter is currently disabled on ASan builds, we need to figure 
> out why, one hypothesis (I do not recall the author) was that we have issues 
> with the SEGV handler.

This part is platform dependent (there is no SEGV on Windows ;-) ).  In theory, 
we should be able to use vectored exception handling 

 to catch the exception no matter how breakpad handles it.

> Another issue, is that the crash reporter does not capture the stderr stream. 
>  Thus we might have to make additional patches to clang to give it the 
> ability to spew the reports in a file, that we can later attach to the bug 
> report.

You should be able to get access to the error message by registering a callback 
through __asan_set_error_report_callback().  Messing directly with stderr 
stream shouldn’t be needed.

> I don't know who is the right person, but I briefly discuss this issue with 
> Ted Mielczarek [:ted].
> 
> 2.4/ Soccoro interface to display multiple backtraces.
> 
> Currently Soccoro only has support to display one backtrace.  On the other 
> hand ASan builds might report 2 / 3 backtraces (allocation site, deallocation 
> site / crash site).  We would have to extend Soccoro interface to be able to 
> search for the different backtraces, such that we can make buckets of crashes 
> which have identical allocation sites / deallocation sites.

Another thing to think about is symbolifying the stack.  Hopefully we can use 
the existing offline symbolification that breakpad does on the server side.

> 2.5/ Release Management.
> 
> ASan builds have a x2 overhead, and this implies that we have to ship 
> different binaries, ASan is not a simple toggle (as far as I know).
> 
> The performance impact is too high to ship ASan builds by default (Lawrence 
> Mandel).  And as this implies that we have to ship a new version of Firefox, 
> we would have to let people opt-in for a short while on nightly before making 
> t

Re: Status: ASan builds on Windows

[Andrew McCreight]

I'd also note that Google at least used to ship a similar kind of thing to
their Canary users a few times a week, called SyzygyASAN:

http://blog.chromium.org/2013/05/testing-chromium-syzyasan-lightweight.html

I haven't found a ton of detail about whether they still do it but that
blog post does say "In the last three weeks, we’ve found 150 new bugs in
Chromium, several of which could lead to security vulnerabilities."

We already have a couple of extra checks in Nightly, like compartment
checking assertions, that have turned up issues we would not have found
otherwise. Even a single report can sometimes identify an issue, and allow
us to fix it. So I would bet that having more people run builds with even
more checking will turn up things to fix.

Andrew

On Tue, Dec 15, 2015 at 6:36 AM, Nicolas B. Pierron <
nicolas.b.pier...@mozilla.com> wrote:

> Hi,
>
> 0/ Context
>
> As part of the JS engine a lot of issues which are reported on crash-stats
> are rarely actionable.  I personally waited a few times on equivalent
> fuzz-bugs, in hope to fix a crash-stat issue by luck.  One of the proposal
> we made in Orlando was to investigate if we could improve the reports that
> we get from crash-stats.
>
> Among the idea, was the idea to ship ASan builds to nightly users.
>
> This email is a summary of the problems which are blocking.  I also listed
> the persons involved in the discussion, and which might be contacted to
> help solving these issues.
>
> 1/ Would ASan be useful?
>
> While asking Bob Clary [:bc], he reported that even if he did not measure
> the effectiveness of ASan reports, he once noticed that an ASan bug got
> marked as a duplicate a of 12 non-ASan crashes.
>
> As Bob & Decoder mentioned, even if we do not ship ASan builds to users,
> we should at least use Windows ASan builds with Bughunter, and fuzzing
> automation.
>
> 2/ Current Blockers:
>
> 2.1/ Windows Clang ASan builds are not stable yet.
>
> Apparently we have a few issues opened against clang, and we should push
> these forward to make sure that we can have a stable build.  Part of the
> stability issues might be related to the way our build system is working as
> we are linking crt library both statically and dynamically inside Gecko.
>
> It seems that part of the issue might be related to the fact that Clang
> default to cl.exe when Clang fails at compiling, and a guess from Raymond
> was that we have ABI issues caused by a miss conversion of the command line
> arguments.
>
> If you are interested to talk more, or help fix clang issues, you should
> contact the following persons:
>
>  - Raymond Forbes [:rforbes]
>  - Nathan Froyd [:froydnj]
>  - Jeff Muizelaar [:jrmuizel]
>  - Ehsan Akhgari [:eshan]
>
> Also, we have few Clang contributors within Mozilla who might have a bit
> more knowledge about the persons to ask to get momentum on the clang
> issues, such as:
>
>  - Jakob Stoklund Olesen [:jolesen]
>  - Dan Gohman [:sunfish]
>
> And for our build system issues on Windows, we should contact:
>
>  - Mike Hommey [:glandium]
>
> 2.2/ Enable Crash reporter on ASan builds.
>
> The crash reporter is currently disabled on ASan builds, we need to figure
> out why, one hypothesis (I do not recall the author) was that we have
> issues with the SEGV handler.
>
> Another issue, is that the crash reporter does not capture the stderr
> stream.  Thus we might have to make additional patches to clang to give it
> the ability to spew the reports in a file, that we can later attach to the
> bug report.
>
> I don't know who is the right person, but I briefly discuss this issue
> with Ted Mielczarek [:ted].
>
> 2.4/ Soccoro interface to display multiple backtraces.
>
> Currently Soccoro only has support to display one backtrace.  On the other
> hand ASan builds might report 2 / 3 backtraces (allocation site,
> deallocation site / crash site).  We would have to extend Soccoro interface
> to be able to search for the different backtraces, such that we can make
> buckets of crashes which have identical allocation sites / deallocation
> sites.
>
> 2.5/ Release Management.
>
> ASan builds have a x2 overhead, and this implies that we have to ship
> different binaries, ASan is not a simple toggle (as far as I know).
>
> The performance impact is too high to ship ASan builds by default
> (Lawrence Mandel).  And as this implies that we have to ship a new version
> of Firefox, we would have to let people opt-in for a short while on nightly
> before making them fallback to the normal nightly, or suggest this ASan
> builds on supports.mozilla.org to investigate.
>
> The person to contact might be Sylvestre Ledru [:Sylvestre], based on
> Lawrence suggestion.
>
> --
> Nicolas B. Pierron
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.

Re: Status: ASan builds on Windows

[Ted Mielczarek]

On Tue, Dec 15, 2015, at 09:36 AM, Nicolas B. Pierron wrote:
> The crash reporter is currently disabled on ASan builds, we need to
> figure 
> out why, one hypothesis (I do not recall the author) was that we have
> issues 
> with the SEGV handler.

That was my hypothesis. I remember talking to decoder about this at some
point. ASan wants to handle exceptions internally, but Breakpad
registers a signal handler, and I think they don't play nice.

> 2.5/ Release Management.
> 
> ASan builds have a x2 overhead, and this implies that we have to ship 
> different binaries, ASan is not a simple toggle (as far as I know).
> 
> The performance impact is too high to ship ASan builds by default
> (Lawrence 
> Mandel).  And as this implies that we have to ship a new version of
> Firefox, 
> we would have to let people opt-in for a short while on nightly before 
> making them fallback to the normal nightly, or suggest this ASan builds
> on 
> supports.mozilla.org to investigate.

Additionally, ASan builds are incredibly different than what we're
shipping to users now--we'd be building them with an entirely different
toolchain. Since part of the point of nightly builds is to have users
testing the bleeding edge form of what we ship, this would take that
away. We could have an opt-in population, but we definitely could not
make them the default.

-Ted
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Status: ASan builds on Windows

[Nicolas B. Pierron]

Hi,

0/ Context

As part of the JS engine a lot of issues which are reported on crash-stats 
are rarely actionable.  I personally waited a few times on equivalent 
fuzz-bugs, in hope to fix a crash-stat issue by luck.  One of the proposal 
we made in Orlando was to investigate if we could improve the reports that 
we get from crash-stats.


Among the idea, was the idea to ship ASan builds to nightly users.

This email is a summary of the problems which are blocking.  I also listed 
the persons involved in the discussion, and which might be contacted to help 
solving these issues.


1/ Would ASan be useful?

While asking Bob Clary [:bc], he reported that even if he did not measure 
the effectiveness of ASan reports, he once noticed that an ASan bug got 
marked as a duplicate a of 12 non-ASan crashes.


As Bob & Decoder mentioned, even if we do not ship ASan builds to users, we 
should at least use Windows ASan builds with Bughunter, and fuzzing automation.


2/ Current Blockers:

2.1/ Windows Clang ASan builds are not stable yet.

Apparently we have a few issues opened against clang, and we should push 
these forward to make sure that we can have a stable build.  Part of the 
stability issues might be related to the way our build system is working as 
we are linking crt library both statically and dynamically inside Gecko.


It seems that part of the issue might be related to the fact that Clang 
default to cl.exe when Clang fails at compiling, and a guess from Raymond 
was that we have ABI issues caused by a miss conversion of the command line 
arguments.


If you are interested to talk more, or help fix clang issues, you should 
contact the following persons:


 - Raymond Forbes [:rforbes]
 - Nathan Froyd [:froydnj]
 - Jeff Muizelaar [:jrmuizel]
 - Ehsan Akhgari [:eshan]

Also, we have few Clang contributors within Mozilla who might have a bit 
more knowledge about the persons to ask to get momentum on the clang issues, 
such as:


 - Jakob Stoklund Olesen [:jolesen]
 - Dan Gohman [:sunfish]

And for our build system issues on Windows, we should contact:

 - Mike Hommey [:glandium]

2.2/ Enable Crash reporter on ASan builds.

The crash reporter is currently disabled on ASan builds, we need to figure 
out why, one hypothesis (I do not recall the author) was that we have issues 
with the SEGV handler.


Another issue, is that the crash reporter does not capture the stderr 
stream.  Thus we might have to make additional patches to clang to give it 
the ability to spew the reports in a file, that we can later attach to the 
bug report.


I don't know who is the right person, but I briefly discuss this issue with 
Ted Mielczarek [:ted].


2.4/ Soccoro interface to display multiple backtraces.

Currently Soccoro only has support to display one backtrace.  On the other 
hand ASan builds might report 2 / 3 backtraces (allocation site, 
deallocation site / crash site).  We would have to extend Soccoro interface 
to be able to search for the different backtraces, such that we can make 
buckets of crashes which have identical allocation sites / deallocation sites.


2.5/ Release Management.

ASan builds have a x2 overhead, and this implies that we have to ship 
different binaries, ASan is not a simple toggle (as far as I know).


The performance impact is too high to ship ASan builds by default (Lawrence 
Mandel).  And as this implies that we have to ship a new version of Firefox, 
we would have to let people opt-in for a short while on nightly before 
making them fallback to the normal nightly, or suggest this ASan builds on 
supports.mozilla.org to investigate.


The person to contact might be Sylvestre Ledru [:Sylvestre], based on 
Lawrence suggestion.


--
Nicolas B. Pierron
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform