Hello, As some of you might have heard, we have been rewriting significantly some key components of Kdenlive over the past couple years, in an effort to improve stability, testability, and separation of concerns between the view and the model.
As part of this effort, on top of traditional unit-tests (none existed before the rewrite), I have been looking at fuzzing as a way to push the code to its limit. For those who are not familiar with it, fuzzing is a testing technique that essentially consists in generating a lot of random inputs and feeding them to a target program in an attempt to generate unintended behaviours (memoryleak, crash,...). This technique has an excellent track record of finding bugs and vulnerabilites, even in well tested softwares (see eg. https://llvm.org/docs/LibFuzzer.html#trophies) In the case of Kdenlive, we generate a lot of operations that the user could execute on the timeline, and simulate them in the model, to check for robustness over unexpected action sequences. Note that this kind of testing doesn't explicitly test for correctness (it doesn't check whether a clip is correctly moved after a move operation for example), but try to get the software to crash, if there is a way to make that happen. Also, we have some internal checking mechanisms that periodically ensure that the model is in a sane state, and the invariants are respected. So if things go wrong with respect to that, it will also be caught by the fuzzer. This approach has already led to the discovery of several bugs in kdenlive, although the parts covered by the fuzzer are still fairly limited. Are any other projects in KDE using that technique? In order for it to be useful, it pretty much needs to be run continuously (I've been running it a bit on my laptop, but it's not very practical). Some companies offer such service for free for OSS, under some conditions: see for eg: https://github.com/google/oss-fuzz and https://fuzzbuzz.io/pricing, although I'm not sure whether Kdenlive would meet the requirements for either of those. Is there any way we could do this inside the KDE infrastructure instead? Ideally, this are the requirements for such a service: - Continuous running - Git hook to get the latest code - Automatic bug deduplication and reporting - (optional) Coverage report to ensure the fuzzing is effective. Google open-sourced most of its tooling for fuzzing at scale on a cluster, so there is no need to start from scratch. What are your thoughts? Best regards, Nicolas Carion
