On Wed, Mar 9, 2016 at 10:45 AM, <twsm...@mozilla.com> wrote: > This seems to be the general response whenever this topic is brought up, > with good reason. I don't think modularizing Gecko just for the sake of it > makes much sense. What I think the take away from this should be is: when > developing a new component or maintaining an old one keep in mind that > making it accessible to fuzzers is valuable. In the short term this may > been making gtests that we can leverage. > > Decoder has started work on support for gtests to help use it as an > alternative to the full browser. I believe he does have a couple simple > fuzzers working atm however I'm not sure what they have been finding. I > have tried myself with varying levels of success. > > I think a good starting point for this work would be media and image > processing (codecs, parsers demuxers, decoders, etc...). We (fuzzing team) > have already talked to the media team about this.
FWIW, the Servo team is interested in using the Gecko media stack, so there is already some momentum behind modularizing that. I agree that it makes sense for large and complex leaf subsystems with minimal coupling to the rest of Gecko (media is a good example, possibly the best). > Christoph and I recently fuzzed the new BMP decoder and had many > situations where we hit issues that were not reproducible that would have > been easily reproducible in a shell. The issues were in the image > processing code but obscured by the rest of the system. The main issue in > these cases was threading/timing. > > Maybe others have ideas for how we can make code more fuzz-able? > > > On Wednesday, March 9, 2016 at 10:15:16 AM UTC-8, Bobby Holley wrote: > > Can you elaborate on which Gecko components you're hoping to fuzz > > separately? A lot of the core is pretty heavily-intertwined, so I'm > pretty > > skeptical that we'd ever be able to separate out DOM, style, and layout > > from each other (for example). There are basically two barriers: > > (1) These components are enormous, and were not built to be very modular. > > Any such efforts would require a huge amount of engineering resources, > > which we would probably not spend just to make the component fuzzable. > > (2) The performance cost of adding an abstraction layer between > > tightly-coupled components in C++11 would probably be prohibitive (the > > situation is different for Rust/Servo because of Traits - we could > > potentially do this with C++ Concepts/Modules in around a decade). > > > > This is all to say that I think a general call to "modularize Gecko" > isn't > > really helpful. But if there are specific leaf-y components that you want > > to fuzz separately but can't, that might be a good starting point. > > > > On Wed, Mar 9, 2016 at 9:54 AM, <twsm...@mozilla.com> wrote: > > > > > On Wednesday, March 9, 2016 at 9:38:55 AM UTC-8, Nicolas B. Pierron > wrote: > > > > This discussion is a follow-up discussion to some emails sent > privately > > > by > > > > accident. > > > > > > > > If you have not followed, I will quote David Bryant: > > > > > Improving release quality is one of the three fundamental goals > > > Platform > > > > > Engineering committed to this year. To this end, lmandel built a > > > Bugzilla > > > > > dashboard that allows us to track regressions found in any given > > > release > > > > > cycle. This dashboard [...] can > > > > > also be found at: http://mozilla.github.io/releasehealth/ > > > > > > > > To David's email, I answered the following: > > > > > > > > ---------- > > > > tl;dr: If we want to improve the quality of our products we should > > > > split Gecko in standalone programs which are fuzzing-friendly. > > > > > > > > One thing which strikes me, is the ratio of regressions per component > > > > that we have for each versions, and more over who are the persons > > > > opening these bugs: > > > > - Release: > > > > > > > > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox44&f2=OP&f3=cf_status_firefox43&f4=cf_status_firefox43&f5=cf_status_firefox43&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898533&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > > > > - Beta: > > > > > > > > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox45&f2=OP&f3=cf_status_firefox44&f4=cf_status_firefox44&f5=cf_status_firefox44&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898534&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > > > > - Aurora: > > > > > > > > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox46&f2=OP&f3=cf_status_firefox45&f4=cf_status_firefox45&f5=cf_status_firefox45&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898536&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > > > > - Nightly: > > > > > > > > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox47&f2=OP&f3=cf_status_firefox46&f4=cf_status_firefox46&f5=cf_status_firefox46&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898528&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > > > > > > > > To be more precise: > > > > - The small number of regression we have in the JS engine on the > > > > release channel, versus the Extremely Huge number of regressions we > > > > have on nightly. > > > > - And the fact that (almost) all the bugs opened against the JS > > > > engine are opened by our fuzzing team. > > > > > > > > What I want to remark is the fact that our automated fuzzing is > better > > > > at finding recently introduced regressions. And as far as I know, > > > > Alice is not a bot. > > > > > > > > From what I know, the reason fuzzing team is so efficient on the JS > > > > engine is because we have a *standalone* JS shell. > > > > The *standalone* JS shell is also the reason why our build time is > > > > below 2 minutes as opposed to 18 minutes. > > > > > > > > So, I think that if we want to improve our quality we should focus on > > > > making fuzzing-friendly standalone programs for the different > > > > components of the platform. > > > > Thus reducing, the compilation time, reducing the test suite time, > and > > > > improving the ability of the fuzzing team to find recently added > > > > regressions. > > > > > > > > Maybe I am wrong, in which case the other alternative might be to > > > > staff the JS Team to get rid of all these nightly issues before they > > > > ride the train to release. > > > > ---------- > > > > > > > > To which I got the following replies: > > > > > > > > On Wed, Mar 9, 2016 at 3:05 PM, Kyle Huey wrote: > > > > > The ratio of engineers to code in the js engine is so much higher > than > > > > > the rest of the product that I'm not sure this is a sensible > > > comparison. > > > > > The js engine also doesn't depend on things like 3rd party gfx > > > drivers ... > > > > > > > > On Wed, Mar 9, 2016 at 3:05 PM, Olli Pettay wrote: > > > > > Fuzzing captures only a fraction of issues. > > > > > > > > On Wed, Mar 9, 2016 at 3:42 PM, Chris Hofmann wrote: > > > > > On Wed, Mar 9, 2016 at 7:05 AM, Kyle Huey wrote: > > > > >> > > > > >> The ratio of engineers to code in the js engine is so much higher > > > than the > > > > >> rest of the product that I'm not sure this is a sensible > > > comparison. The js > > > > >> engine also doesn't depend on things like 3rd party gfx drivers > ... > > > > > > > > > > This is probably not the only step that we need to take to > > > substantially > > > > > improve quality so setting up a place to have those discussions is > > > good. It > > > > > really is worth some time and effort to brainstorm about all the > > > things we > > > > > might do to raise the bar, poke some holes in those ideas, then > > > decide on > > > > > and push forward on a few more in the next few quarters. > > > > > > > > On Wed, Mar 9, 2016 at 5:23 PM, Al Billings wrote: > > > > > On 3/9/16 6:58 AM, Nicolas B. Pierron wrote: > > > > >> So, I think that if we want to improve our quality we should > focus on > > > > >> making fuzzing-friendly standalone programs for the different > > > > >> components of the platform. > > > > >> Thus reducing, the compilation time, reducing the test suite > time, > > > and > > > > >> improving the ability of the fuzzing team to find recently added > > > > >> regressions. > > > > > The fuzzing team has asked for this from different teams for > various > > > > > components, with different degrees of support and resistance, > > > depending > > > > > on the component and team. > > > > > > > > > > It is work to do this but it changes the fuzzing efficiency and > > > ability > > > > > to find focused issues when we have this level of support, such > as we > > > > > have with the JS shell. It allows the team to fuzz directly on a > > > > > component and not have to find a way to reach it through Firefox > and > > > all > > > > > the additional noise/assertions/etc that it churns up. > > > > > > > > -- > > > > Nicolas B. Pierron > > > > > > I think Nicolas is right on the mark! JS shell is a good example of > using > > > a shell for fuzzing and I think media and graphics could also get a > good > > > deal out of a shell. > > > > > > To get a bit more specific breaking things in to smaller pieces for > > > fuzzing give us (the fuzzing team) a major advantage. In addition to > what > > > Nicolas mentioned, it makes it much easier to use different sanitizers > > > (Address, Memory, Undefined behavior, etc...) It also makes it > possible to > > > use coverage guided fuzzing which is a major advancement in fuzzing. I > have > > > been doing a lot of work fuzzing 3rd party libraries that we use in > Firefox > > > and there have been many times I wished I could have use the same > tools as > > > easily on our own code. That said the fuzzing on the browser as a > whole it > > > also important as well. > > > > > > More dev support for the fuzzing means more bugs found and sooner! > > > > > > _______________________________________________ > > > dev-platform mailing list > > > dev-platform@lists.mozilla.org > > > https://lists.mozilla.org/listinfo/dev-platform > > > > > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
