On 201022 1739, Daniel P. Berrangé wrote:
> On Thu, Oct 22, 2020 at 12:24:16PM -0400, Alexander Bulekov wrote:
> > +CC Prasad
> >
> > On 201022 1219, Alexander Bulekov wrote:
> > > Hello,
> > > QEMU was accepted into Google's oss-fuzz continuous-fuzzing platform [1]
> > > earlier this year. The fuzzers currently running on oss-fuzz are based on
> > > my
> > > 2019 Google Summer of Code Project, which leveraged libfuzzer, qtest and
> > > libqos
> > > to provide a framework for writing virtual-device fuzzers. At the moment,
> > > there
> > > are a handful of fuzzers upstream and running on oss-fuzz(located in
> > > tests/qtest/fuzz/). They fuzz only a few devices and serve mostly as
> > > examples.
> > >
> > > If everything goes well, soon a generic fuzzer [2] will land upstream,
> > > which
> > > allows us to fuzz many configurations of QEMU, without any device-specific
> > > code. To date this fuzzer has led to ~50 bug reports on launchpad. Once
> > > the
> > > generic-fuzzer lands upstream, OSS-Fuzz will automatically start fuzzing a
> > > bunch [3] of fuzzer configurations, and it is likely to find bugs.
> > > Others will
> > > also be able to send simple patches to add additional device
> > > configurations for
> > > fuzzing.
> > >
> > > The oss-fuzz process looks roughly like this:
> > > 1. oss-fuzz fuzzes QEMU
> > > 2. When oss-fuzz finds a bug, it reports it to a few [4] people that
> > > have
> > > access to reports and reproducers.
> > > 3. If a fix is merged upstream, oss-fuzz will figure this out and
> > > mark the
> > > bug as fixed and make the report public 30 days later.
> > > 3. After 90 days the bug(fixed or not) becomes public, so anyone can
> > > view
> > > it here https://bugs.chromium.org/p/oss-fuzz/issues/list
> > >
> > > The oss-fuzz reports look like this:
> > > https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=23701&q=qemu&can=2
> > >
> > > This means that when oss-fuzz find new bugs, the relevant developers do
> > > not
> > > know about them unless someone with access files a separate report to the
> > > list/launchpad. So far this hasn't been a problem, since oss-fuzz has
> > > only been
> > > running some small example fuzzers. Once [2] lands upstream, we should
> > > see a significant uptick in oss-fuzz reports, and I hope that we can
> > > develop a
> > > process to ensure these bugs are properly dealt with. One option we have
> > > is to
> > > make the reports public immediately and send notifications to
> > > qemu-devel. This is the approach taken by some other projects on
> > > oss-fuzz, such as LLVM. Though its not on oss-fuzz, bugs found by
> > > syzkaller in the kernel, are also automatically sent to a public list.
> > > The question is:
> > >
> > > What approach should we take for dealing with bugs found on oss-fuzz?
>
> If we assume that a non-negligible number of fuzz bugs will be exploitable
> by a malicious guest OS to break out into the host, then I think it is
> likely undesirable to make them public immediately without at least a basic
> human triage step to catch possibly serious security issues.
>
> Still a large % are likely to be low impact / not urgent to deal with so
> we want a low overhead way to handle the fuzz output, which doesn't create
> a bottleneck on a small number of people.
>
> Overall my feeling is that we want to be able to farm out triage to the
> respective subsystem maintainers, who can then decide whether the bug
> needs notifying to the security team, or can be made public immediately.
>
> I think ideally we would be doing triage in QEMU's own bug tracker, so
> we don't need to have maintainers create accounts on a 3rd party tracker
> to see reports.
>
> Is is practical to identify a primary affected source file from the fuzz
> crash report with any level reliablility such that we could file a private
> launchpad bug, automatically CC'ing a subsystem maintainer (and the security
> team) ?
Hi Daniel,
As far as I know, there is currently no API for accessing oss-fuzz
results. We could use email-based scripts to parse the automated reports
(e.g. [1]) and follow the links to automatically download crash-traces
and reproducers. However, accessing those requires a login through
google.com which might be tough to script against in a reliable way.
Assuming we have found a way to automatically download the binary fuzzer
reproducer, we should be able to automatically convert it into a qtest
reproducer that we could send to the right people.
There are a few approaches I can think of to automatically identify the
maintainers to CC:
1. Walk the stack-trace until we find the line likely responsible for
the bug. This can be tricky, since the buggy line is often not the
first line. E.g. from [2]
#3 __GI___assert_fail assert.c:101
#4 iov_from_buf_full util/iov.c:40
#5 iov_from_buf iov.h:49
#6 net_tx_pkt_update_ip_checksums hw/net/net_tx_pkt.c:139
#7 e1000e_setup_tx_offloads hw/net/e1000e_core.c:638
Do we cc the iov maintainer, the net_tx_pkt maintainer, or the e1000e
maintainer? Might be tough to figure out automatically.
2. git bisect until we find the commit where the reproducer started to
crash and CC everyone who signed-off or reviewed that commit and who is
in MAINTAINERS. This is also not a silver bullet, since reproducers
might stop working due to reasons unrelated to the bug: e.g. the MMIO
range for a device changed, so the qtest reproducer now interacts with
the wrong addresses, or the bug was obscured by another bug.
[1] https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=23701&q=qemu&can=2
[2] https://bugs.launchpad.net/qemu/+bug/1878250
-Alex
>
>
> Regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
>
