This seems to be the general response whenever this topic is brought up, with good reason. I don't think modularizing Gecko just for the sake of it makes much sense. What I think the take away from this should be is: when developing a new component or maintaining an old one keep in mind that making it accessible to fuzzers is valuable. In the short term this may been making gtests that we can leverage.
Decoder has started work on support for gtests to help use it as an alternative to the full browser. I believe he does have a couple simple fuzzers working atm however I'm not sure what they have been finding. I have tried myself with varying levels of success. I think a good starting point for this work would be media and image processing (codecs, parsers demuxers, decoders, etc...). We (fuzzing team) have already talked to the media team about this. Christoph and I recently fuzzed the new BMP decoder and had many situations where we hit issues that were not reproducible that would have been easily reproducible in a shell. The issues were in the image processing code but obscured by the rest of the system. The main issue in these cases was threading/timing. Maybe others have ideas for how we can make code more fuzz-able? On Wednesday, March 9, 2016 at 10:15:16 AM UTC-8, Bobby Holley wrote: > Can you elaborate on which Gecko components you're hoping to fuzz > separately? A lot of the core is pretty heavily-intertwined, so I'm pretty > skeptical that we'd ever be able to separate out DOM, style, and layout > from each other (for example). There are basically two barriers: > (1) These components are enormous, and were not built to be very modular. > Any such efforts would require a huge amount of engineering resources, > which we would probably not spend just to make the component fuzzable. > (2) The performance cost of adding an abstraction layer between > tightly-coupled components in C++11 would probably be prohibitive (the > situation is different for Rust/Servo because of Traits - we could > potentially do this with C++ Concepts/Modules in around a decade). > > This is all to say that I think a general call to "modularize Gecko" isn't > really helpful. But if there are specific leaf-y components that you want > to fuzz separately but can't, that might be a good starting point. > > On Wed, Mar 9, 2016 at 9:54 AM, <twsm...@mozilla.com> wrote: > > > On Wednesday, March 9, 2016 at 9:38:55 AM UTC-8, Nicolas B. Pierron wrote: > > > This discussion is a follow-up discussion to some emails sent privately > > by > > > accident. > > > > > > If you have not followed, I will quote David Bryant: > > > > Improving release quality is one of the three fundamental goals > > Platform > > > > Engineering committed to this year. To this end, lmandel built a > > Bugzilla > > > > dashboard that allows us to track regressions found in any given > > release > > > > cycle. This dashboard [...] can > > > > also be found at: http://mozilla.github.io/releasehealth/ > > > > > > To David's email, I answered the following: > > > > > > ---------- > > > tl;dr: If we want to improve the quality of our products we should > > > split Gecko in standalone programs which are fuzzing-friendly. > > > > > > One thing which strikes me, is the ratio of regressions per component > > > that we have for each versions, and more over who are the persons > > > opening these bugs: > > > - Release: > > > > > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox44&f2=OP&f3=cf_status_firefox43&f4=cf_status_firefox43&f5=cf_status_firefox43&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898533&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > > > - Beta: > > > > > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox45&f2=OP&f3=cf_status_firefox44&f4=cf_status_firefox44&f5=cf_status_firefox44&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898534&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > > > - Aurora: > > > > > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox46&f2=OP&f3=cf_status_firefox45&f4=cf_status_firefox45&f5=cf_status_firefox45&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898536&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > > > - Nightly: > > > > > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox47&f2=OP&f3=cf_status_firefox46&f4=cf_status_firefox46&f5=cf_status_firefox46&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898528&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > > > > > > To be more precise: > > > - The small number of regression we have in the JS engine on the > > > release channel, versus the Extremely Huge number of regressions we > > > have on nightly. > > > - And the fact that (almost) all the bugs opened against the JS > > > engine are opened by our fuzzing team. > > > > > > What I want to remark is the fact that our automated fuzzing is better > > > at finding recently introduced regressions. And as far as I know, > > > Alice is not a bot. > > > > > > From what I know, the reason fuzzing team is so efficient on the JS > > > engine is because we have a *standalone* JS shell. > > > The *standalone* JS shell is also the reason why our build time is > > > below 2 minutes as opposed to 18 minutes. > > > > > > So, I think that if we want to improve our quality we should focus on > > > making fuzzing-friendly standalone programs for the different > > > components of the platform. > > > Thus reducing, the compilation time, reducing the test suite time, and > > > improving the ability of the fuzzing team to find recently added > > > regressions. > > > > > > Maybe I am wrong, in which case the other alternative might be to > > > staff the JS Team to get rid of all these nightly issues before they > > > ride the train to release. > > > ---------- > > > > > > To which I got the following replies: > > > > > > On Wed, Mar 9, 2016 at 3:05 PM, Kyle Huey wrote: > > > > The ratio of engineers to code in the js engine is so much higher than > > > > the rest of the product that I'm not sure this is a sensible > > comparison. > > > > The js engine also doesn't depend on things like 3rd party gfx > > drivers ... > > > > > > On Wed, Mar 9, 2016 at 3:05 PM, Olli Pettay wrote: > > > > Fuzzing captures only a fraction of issues. > > > > > > On Wed, Mar 9, 2016 at 3:42 PM, Chris Hofmann wrote: > > > > On Wed, Mar 9, 2016 at 7:05 AM, Kyle Huey wrote: > > > >> > > > >> The ratio of engineers to code in the js engine is so much higher > > than the > > > >> rest of the product that I'm not sure this is a sensible > > comparison. The js > > > >> engine also doesn't depend on things like 3rd party gfx drivers ... > > > > > > > > This is probably not the only step that we need to take to > > substantially > > > > improve quality so setting up a place to have those discussions is > > good. It > > > > really is worth some time and effort to brainstorm about all the > > things we > > > > might do to raise the bar, poke some holes in those ideas, then > > decide on > > > > and push forward on a few more in the next few quarters. > > > > > > On Wed, Mar 9, 2016 at 5:23 PM, Al Billings wrote: > > > > On 3/9/16 6:58 AM, Nicolas B. Pierron wrote: > > > >> So, I think that if we want to improve our quality we should focus on > > > >> making fuzzing-friendly standalone programs for the different > > > >> components of the platform. > > > >> Thus reducing, the compilation time, reducing the test suite time, > > and > > > >> improving the ability of the fuzzing team to find recently added > > > >> regressions. > > > > The fuzzing team has asked for this from different teams for various > > > > components, with different degrees of support and resistance, > > depending > > > > on the component and team. > > > > > > > > It is work to do this but it changes the fuzzing efficiency and > > ability > > > > to find focused issues when we have this level of support, such as we > > > > have with the JS shell. It allows the team to fuzz directly on a > > > > component and not have to find a way to reach it through Firefox and > > all > > > > the additional noise/assertions/etc that it churns up. > > > > > > -- > > > Nicolas B. Pierron > > > > I think Nicolas is right on the mark! JS shell is a good example of using > > a shell for fuzzing and I think media and graphics could also get a good > > deal out of a shell. > > > > To get a bit more specific breaking things in to smaller pieces for > > fuzzing give us (the fuzzing team) a major advantage. In addition to what > > Nicolas mentioned, it makes it much easier to use different sanitizers > > (Address, Memory, Undefined behavior, etc...) It also makes it possible to > > use coverage guided fuzzing which is a major advancement in fuzzing. I have > > been doing a lot of work fuzzing 3rd party libraries that we use in Firefox > > and there have been many times I wished I could have use the same tools as > > easily on our own code. That said the fuzzing on the browser as a whole it > > also important as well. > > > > More dev support for the fuzzing means more bugs found and sooner! > > > > _______________________________________________ > > dev-platform mailing list > > dev-platform@lists.mozilla.org > > https://lists.mozilla.org/listinfo/dev-platform > > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
