On Sun, Apr 8, 2018 at 8:02 PM, Theodore Y. Ts'o <ty...@mit.edu> wrote: > On Sun, Apr 08, 2018 at 03:18:39PM +0200, Dmitry Vyukov wrote: >> >> But note that syzkaller is under active development, so pre-canned >> binaries may not always work. Mismatching binary may not understand >> all syscalls, fail to parse program, interpret arguments differently, >> execute program differently, setup a different environment for the >> test, etc. Now a C program captures all of this, because code that >> transforms syzkaller programs into C is versioned along with the rest >> of the system. >> Strictly saying, for syzkaller reproducers one needs to use the exact >> syzkaller revision listed along with the reproducer, see for example: >> https://syzkaller.appspot.com/bug?id=3fb9c4777053e79a6d2a65ac3738664c87629a21 >> The "#syz test" styzbot command does this. Using a different syzkaller >> revision may or may not work. > > Thanks for the warning. I assume you try to maintain backwards > compatibility where possible? It might be nice if you could add some > kind of explicit versioning scheme --- perhaps with a major/minor > version scheme where the syz-executor needs to have the same major > number, and a minor number >= the minor version number of the test?
We try to not break backwards compatibility without a reason. Preserving full backwards compatibility within a single binary is extremely hard. It's like asking kernel to support each and every ever existed version of every in-memory data structure and all of the non-functional aspects (like any fluctuations in performance). If one could give us several additional FTEs for this, then it might be doable. But even then I don't think it's the best use of the FTE time because version control system already gives us exactly this -- exact behavior on a past revision. On top of this, the backward compatibility support will sure have bugs too. In the best case we will spent time debugging why a new version does not precisely model behavior of an old version. In the worst case you will test something and think that the bug is fixed, but it's just that the new version does not behave exactly as the old one. On top of this, this still does not give us forward compatibility, something that one wants in majority of cases with an old pre-canned binary. On top of this, the binaries will be huge because they will need to capture exact versions of all system call descriptions (and the simplest option for this is keeping copies all versions), a 87 MiB image definitely won't be enough to hold this, the binary will be somewhere between gigs and tens of gigs. > One of the reasons why the C program is not so useful for me is that > in the Repeat:true case, the C program repeats forever. So for > example, I translate Repeat:true to -repeat=100. See: > > https://github.com/tytso/xfstests-bld/blob/master/kvm-xfstests/test-appliance/files/usr/local/bin/run-syz > > I suppose I could just abort the test after N minutes and assume if > the kernel hasn't crashed, that it's probably not going to. But some > way that the C program can be given an argument or an environment > variable to control how number of loops it will run might be useful. > And some kind of hint as how reliable the repro would be (e.g,. some > indication that you should try to run it at least N times to get a > failure at least 95% of the time). I think: timeout -s KILL 450 ./a.out is the solution. Repro logic runs programs for at most 7.5 minutes, so 450 should be good. Re env var. There are opposite views too. People complain that syzkaller C repros are mess (which they are). Currently they complain minimal amount of code to reproduce the bugs. If we also start staffing some aux logic in them, it won't be helpful. timeout command looks just as good.
