My intuition about fuzz testing is that since we are searching an exponentially-sized search space in random order we will find 90% of the bugs with the first 10% of the effort (or some similar power law). We should burn a large amount of CPU on it when we first introduce it, and thereafter burn a small amount each nightly test run.
No need to ask INFRA for a VM. Just use your own personal machine overnight. Julian > On Sep 12, 2018, at 6:01 AM, Michael Mior <[email protected]> wrote: > > True, although 23 days over the lifetime of the project still isn't very > much. Definitely better than nothing though. If we take a bit of a hit in > CI runtime and catch some bugs, I'm for it :) > >> It would be great, however we need to have a fuzzer first :) > > My setup fuzzing the parser with afl seems to be working well, although > it's quite slow and never found any crashes so probably not really worth it. > > -- > Michael Mior > [email protected] > > > Le mer. 12 sept. 2018 à 08:40, Vladimir Sitnikov < > [email protected]> a écrit : > >>> My only concern is that may betoo short and unlikely to find any bugs >> >> Remember: each time it starts from a random point. >> Apache Jenkins / Calcite-Master has 800+ builds now. >> Travis / Calcite has 2300+ builds now. >> >> Just to clarify: current Travis configuration is 4 jobs (Java 8, 9, 10, 11) >> They take ~15 minutes to complete. >> We can add one more job that would be dedicated to fuzz testing, and we >> could configure it for 1..15 minutes. >> >> 2300 builds * 15 minutes is 23 days worth of fuzzing. >> >>> It seems like we could also possibly request a VM >>> from INFRA to run fuzz testing full time >> >> It would be great, however we need to have a fuzzer first :) >> >> Vladimir >>
