On Mon, 29 May 2023, Kent Overstreet wrote:
> On Mon, May 29, 2023 at 04:59:40PM -0400, Mikulas Patocka wrote:
> > Hi
> >
> > I improved the dm-flakey device mapper target, so that it can do random
> > corruption of read and write bios - I uploaded it here:
> > https://people.redhat.com/~mpatocka/testcases/bcachefs/dm-flakey.c
> >
> > I set up dm-flakey, so that it corrupts 10% of read bios and 10% of write
> > bios with this command:
> > dmsetup create flakey --table "0 `blockdev --getsize /dev/ram0` flakey
> > /dev/ram0 0 0 1 4 random_write_corrupt 100000000 random_read_corrupt
> > 100000000"
>
> I've got some existing ktest tests for error injection:
> https://evilpiepirate.org/git/ktest.git/tree/tests/bcachefs/single_device.ktest#n200
> https://evilpiepirate.org/git/ktest.git/tree/tests/bcachefs/replication.ktest#n491
>
> I haven't looked at dm-flakey before, I take it you're silently
> corrupting data instead of just failing the IOs like these tests do?
Yes, silently corrupting.
When I tried to simulate I/O errors with dm-flakey, bcachefs worked
correcly - there were no errors returned to userspace and no crashes.
Perhaps, it should treat metadata checksum errors in the same way as disk
failures?
> Let's add what you're doing to ktest, and see if we can merge it with
> the existing tests.
> Good catches on all of them. Darrick's been on me to get fuzz testing
> going, looks like it's definitely needed :)
>
> However, there's two things I want in place first before I put much
> effort into fuzz testing:
>
> - Code coverage analysis. ktest used to have integrated code coverage
> analysis, where you'd tell it a subdirectory of the kernel tree
> (doing code coverage analysis for the entire kernel is impossibly
> slow) and it would run tests and then give you the lcov output.
>
> However, several years ago something about kbuild changed, and the
> method ktest was using for passing in build flags for a specific
> subdir on the command line stopped working. I would like to track
> down someone who understands kbuild and get this working again.
>
> - Fault injection
>
> Years and years ago, when I was still at Google and this was just
> bcache, we had fault injection that worked like dynamic debug: you
> could call dynamic_fault("type of fault") anywhere in your code,
> and it returned a bool indicating whether that fault had been enabled
> - and faults were controllable at runtime via debugfs, we had tests
> that iterated over e.g. faults in the initialization path, or memory
> allocation failures, and flipped them on one by one and ran
> $test_workload.
>
> The memory allocation profiling stuff that Suren and I have been
> working on includes code tagging, which is for (among other things) a
> new and simplified implementation of dynamic fault injection, which
> I'm going to push forward again once the memory allocation profiling
> stuff gets merged.
>
> The reason I want this stuff is because fuzz testing tends to be a
> heavyweight, scattershot approach.
>
> I want to be able to look at the code coverage analysis first to e.g.
> work on a chunk of code at a time and make sure it's tested thoroughly,
> instead of jumping around in the code at random depending on what fuzz
> testing finds, and when we are fuzz testing I want to be able to add
> fault injection points and write unit tests so that we can have much
> more targeted, quicker to run tests going forward.
>
> Can I get you interested in either of those things? I'd really love to
> find someone to hand off or collaborate with on the fault injection
> stuff in particular.
I'd like to know how do you want to do coverage analysis? By instrumenting
each branch and creating a test case that tests that the branch goes both
ways?
I know that people who write spacecraft-grade software do such tests, but
I can't quite imagine how would that work in a filesystem.
"grep -w if fs/bcachefs/*.[ch] | wc -l" shows that there are 5828
conditions. That's one condition for every 15.5 lines.
Mikulas
--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel