On Wed, Jul 5, 2023 at 5:17 PM Sam Parker-Haynes <sam.par...@arm.com> wrote:
> Ah, thanks! Using wami, I now also see a memory in the module. > > From my limited viewing of the code, the huge offsets that are generated > for the memory operations look suspicious. I will try to get some concrete > numbers for successful accesses vs traps. > > For analysing the memory afterwards, I was hoping that memcmp would be > fast enough... With the current setup, of only checking the return from > main, it would suggest there could be many paths that don't affect the > result. > Ack, maybe it's fast enough. Feel free to test and send over a CL. Note that we limit the memory size to 32 pages, so the maximum memory to check is 2MB large. > > But now I'm wondering in how many cases the target would generate a store > - if we're generating the inputs, how many instructions require a 'void' > input? > Generating a `void` should not be that uncommon. It's the initial thing we generate for functions that do not return anything, and when generating any other type we always have the option to generate one or several instructions producing `void` before or after the instruction generating the requested type. Look for uses of `WasmGenerator::sequence`: https://source.chromium.org/chromium/chromium/src/+/refs/heads/main:v8/test/fuzzer/wasm-compile.cc;l=972;drc=e510fa298843c8ff43b538742e4fa16306da265f > On Wednesday, July 5, 2023 at 3:57:15 PM UTC+1 Clemens Backes wrote: > >> On Wed, Jul 5, 2023 at 4:20 PM Sam Parker-Haynes <sam.p...@arm.com> >> wrote: >> >>> Thanks all, I will try those other disassemblers. I wanted to use >>> --wasm-fuzzer-gen-test but it seems I can't do that when running the proper >>> fuzzer instance - and I just want to dump out a load of generated cases for >>> inspection. >>> >>> Could you point me to where the memory is added? I'm still unfamiliar >>> with the API and all I can find is this >>> <https://source.chromium.org/chromium/chromium/src/+/main:v8/test/fuzzer/wasm-fuzzer-common.cc;drc=1a55cc981dec262baa26083bff009de93da979aa;l=156> >>> which, >>> naively, suggests there isn't any and I'd like to inspect it. >> >> >> The memory is added implicitly by the `WasmModuleBuilder`: >> https://source.chromium.org/chromium/chromium/src/+/refs/heads/main:v8/src/wasm/wasm-module-builder.cc;l=739;drc=e510fa298843c8ff43b538742e4fa16306da265f >> >> Ensuring that more memory accesses are in-bounds sounds like a good >> strategy to make the fuzzer execute more code before trapping. >> Regarding comparing the memory contents of the reference run and the >> regular run, I am not sure if that's worth it as it would probably make the >> fuzzer significantly slower. Changes in memory content can also be observed >> in other ways, e.g. by simply reading back the memory content and returning >> it from the main function. So the fuzzer has a chance to find differences >> in memory contents even without explicitly checking for that. That only >> works if the execution does return regularly instead of trapping. It might >> be worth checking how often we actually trap instead of returning. If we >> trap in most runs, then we should indeed do something about this. >> >> >>> >>> I like the idea of using modulo memory size for memory arguments! From >>> what I've seen memory arguments are generated just like any other I32 >>> input, except that specific effort is made around alignment. I'd like to >>> ensure that more memory accesses are in bounds to, hopefully, make >>> generated programs more useful. >>> >>> On Wednesday, July 5, 2023 at 3:13:08 PM UTC+1 Matthias Liedtke wrote: >>> >>>> To extend Andreas' comments a bit: >>>> >>>> 2) If the mjsunit format isn't readable enough (immediate support is >>>> somewhat limited on the test generator), you can also decode the module >>>> from --dump-wasm-module using wami, e.g. wami --full-wat >>>> the-wasm-file.wasm. Wami is part of the V8 repository and also gets updated >>>> with new wasm features in v8. >>>> 3) If the fuzzer "runs out" of input bytes, it will just fill with 0s. >>>> So every randomized decision will take the 0-branch. This also helps to >>>> prevent endless recursions, so it's by design. >>>> As a first experiment (assuming a linux environment) you could try >>>> `head -c 255 /dev/urandom > fuzzer_input ; >>>> ./out/x64.optdebug/v8_simple_wasm_compile_fuzzer fuzzer_input >>>> --wasm-fuzzer-gen-test > test.js`. With 255 random bytes it should be able >>>> to generate meaningful instructions. Some bytes will already be used for >>>> static decisions or for decisions that are not part of the function body >>>> (types, globals, signatures, ...), so if the random input should have a >>>> certain size to create meaningful fuzzing examples. Note that the max input >>>> size is 512 bytes >>>> <https://source.chromium.org/chromium/chromium/src/+/main:v8/test/fuzzer/wasm-fuzzer-common.h;l=50>, >>>> so the input shouldn't be larger than that either! >>>> 4) There might be some possible gains in making the fuzzer smarter on >>>> generating better addresses etc., so that it doesn't trap that often. It is >>>> a bit more difficult however because the fuzzer will generate other >>>> instructions to generate an offset (e.g. a local.get) and it is hard to >>>> predict which value range is generated by that. One could possibly generate >>>> a few more instructions (a modulo with the memory size?) with a certain >>>> probability to reduce the chance of oob memory accesses. >>>> >>>> Best regards, >>>> Matthias >>>> >>>> On Wed, Jul 5, 2023 at 3:52 PM 'Andreas Haas' via v8-dev < >>>> v8-...@googlegroups.com> wrote: >>>> >>>>> >>>>> >>>>> On Wed, Jul 5, 2023 at 3:21 PM Sam Parker-Haynes <sam.p...@arm.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I'm trying to find a codegen bug in aarch64, so I've been looking at >>>>>> the wasm_compile_fuzzer in the hope that it can help me. I have a number >>>>>> of >>>>>> questions about the current behaviour of the fuzz target. (sorry in >>>>>> advance >>>>>> for the list!) >>>>>> >>>>>> 1) What set of commands is best to use? I've noticed on the default >>>>>> setting a single, constant, instruction is generated and I'm not sure how >>>>>> useful that is. I've currently using -len_control=10 to get to the, >>>>>> hopefully, juicy stuff quickly. >>>>>> >>>>> Typically I use `v8_wasm_compile_fuzzer` without parameters. Libfuzzer >>>>> should automatically adjust the input size. >>>>> >>>>>> >>>>>> 2) Viewing the generated modules is difficult. I'm using `DumpModule` >>>>>> to output any valid module and there seems to be two error types that >>>>>> prevent my available tools from working. A common output from the WABT >>>>>> tools is: `error: unexpected type form (got -0x30)`. wasm-objdump tries >>>>>> harder but then often falls over with `error: expected valid local type`. >>>>>> I'm using the latest version of WABT, does anyone know what type(s) the >>>>>> fuzz target generates that could cause this issue? >>>>>> >>>>> >>>>> If you compile `v8_simple_wasm_compile_fuzzer` in V8, then you can run >>>>> it with `v8_simple_wasm_compile_fuzzer --wasm-fuzzer-gen-test`, which will >>>>> generate an mjsunit test that is equivalent to the generated test. >>>>> >>>>> >>>>>> 3) For the modules that I have successfully viewed, I've often >>>>>> noticed long chains of the same operation, i32.eqz being a very popular >>>>>> one. Is there any explanation for this? In general, I still haven't got >>>>>> my >>>>>> head around how the input from libfuzzer is used to generate the >>>>>> module... >>>>>> >>>>>> The fuzzer builds the module top down. It generates the outermost >>>>> instruction randomly, and then generates inputs for the instruction >>>>> randomly, but of the correct type. For those inputs it also generates the >>>>> inputs of the correct types, and so on. Maybe `i32.eqz` gets generated >>>>> with >>>>> a `0`, which makes it more likely by the fuzzer because the fuzzer >>>>> generates long strings of `0`? >>>>> >>>>> >>>>>> 4) Is there any memory attached to the instance when it runs? And if >>>>>> there is, there doesn't seem to be an attempt to ensure addresses are in >>>>>> range. So, do most of the memory operations just crash the program? The >>>>>> differential testing seems to only test that the return of `main` is >>>>>> equal, >>>>>> but what about the contents of memory? >>>>>> >>>>>> A memory is attached, I think there is some logic to prefer useful >>>>> memory addresses, but I don't know the details/current state. The fuzzer >>>>> could also realize that low memory indices test better, and prefer low >>>>> indices. >>>>> >>>>> >>>>>> Thanks! >>>>>> Sam >>>>>> >>>>>> -- >>>>>> -- >>>>>> v8-dev mailing list >>>>>> v8-...@googlegroups.com >>>>>> http://groups.google.com/group/v8-dev >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "v8-dev" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to v8-dev+un...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/v8-dev/2ad6b36e-9a5b-4dba-9d76-abb2e95d9f58n%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/v8-dev/2ad6b36e-9a5b-4dba-9d76-abb2e95d9f58n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Andreas Haas >>>>> >>>>> Software Engineer >>>>> >>>>> ah...@google.com >>>>> >>>>> >>>>> Google Germany GmbH >>>>> >>>>> Erika-Mann-Straße 33 >>>>> >>>>> 80636 München >>>>> >>>>> >>>>> Geschäftsführer: Paul Manicle, Liana Sebastian >>>>> >>>>> Registergericht und -nummer: Hamburg, HRB 86891 >>>>> >>>>> Sitz der Gesellschaft: Hamburg >>>>> >>>>> >>>>> Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise >>>>> erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes >>>>> weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich >>>>> bitte >>>>> wissen, dass die E-Mail an die falsche Person gesendet wurde. >>>>> >>>>> >>>>> >>>>> This e-mail is confidential. If you received this communication by >>>>> mistake, please don't forward it to anyone else, please erase all copies >>>>> and attachments, and please let me know that it has gone to the wrong >>>>> person. >>>>> >>>>> -- >>>>> -- >>>>> v8-dev mailing list >>>>> v8-...@googlegroups.com >>>>> http://groups.google.com/group/v8-dev >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "v8-dev" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to v8-dev+un...@googlegroups.com. >>>>> >>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/v8-dev/CAELSTvcdk2Kng_R9u4qn-eyEtr4if66k0zwQYSJ9SxXVCaNbmg%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/v8-dev/CAELSTvcdk2Kng_R9u4qn-eyEtr4if66k0zwQYSJ9SxXVCaNbmg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> >>>> >>>> -- >>>> >>>> Matthias Liedtke >>>> >>>> Software Engineer >>>> >>>> mlie...@google.com >>>> >>>> Google Germany GmbH >>>> >>>> Erika-Mann-Straße 33 >>>> >>>> 80636 München >>>> >>>> Geschäftsführer: Paul Manicle, Liana Sebastian >>>> >>>> Registergericht und -nummer: Hamburg, HRB 86891 >>>> >>>> Sitz der Gesellschaft: Hamburg >>>> >>>> Diese E-Mail ist vertraulich. Falls Sie diese fälschlicherweise >>>> erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes >>>> weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte >>>> wissen, dass die E-Mail an die falsche Person gesendet wurde. >>>> >>>> >>>> >>>> This e-mail is confidential. If you received this communication by >>>> mistake, please don't forward it to anyone else, please erase all copies >>>> and attachments, and please let me know that it has gone to the wrong >>>> person. >>>> >>>> -- >>> -- >>> v8-dev mailing list >>> v8-...@googlegroups.com >>> http://groups.google.com/group/v8-dev >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "v8-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to v8-dev+un...@googlegroups.com. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/v8-dev/6ecf0e1f-6040-4264-9fee-4409574f9895n%40googlegroups.com >>> <https://groups.google.com/d/msgid/v8-dev/6ecf0e1f-6040-4264-9fee-4409574f9895n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> >> Clemens Backes >> >> Software Engineer >> >> clem...@google.com >> >> Google Germany GmbH >> >> Erika-Mann-Straße 33 >> >> 80636 München >> >> Geschäftsführer: Paul Manicle, Liana Sebastian >> >> Registergericht und -nummer: Hamburg, HRB 86891 >> >> Sitz der Gesellschaft: Hamburg >> >> Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten >> haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, >> löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, >> dass die E-Mail an die falsche Person gesendet wurde. >> >> >> This e-mail is confidential. If you received this communication by >> mistake, please don't forward it to anyone else, please erase all copies >> and attachments, and please let me know that it has gone to the wrong >> person. >> >> >> -- > -- > v8-dev mailing list > v8-dev@googlegroups.com > http://groups.google.com/group/v8-dev > --- > You received this message because you are subscribed to the Google Groups > "v8-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to v8-dev+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/v8-dev/6bc78704-adce-4c7d-9842-2ee2e9344cf5n%40googlegroups.com > <https://groups.google.com/d/msgid/v8-dev/6bc78704-adce-4c7d-9842-2ee2e9344cf5n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Clemens Backes Software Engineer cleme...@google.com Google Germany GmbH Erika-Mann-Straße 33 80636 München Geschäftsführer: Paul Manicle, Liana Sebastian Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde. This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person. -- -- v8-dev mailing list v8-dev@googlegroups.com http://groups.google.com/group/v8-dev --- You received this message because you are subscribed to the Google Groups "v8-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/CAGO%3DqhA8GKfLXuDc3ymrNv2Ht5CJttFmZ7hGfy_ogKYYjAb-yw%40mail.gmail.com.
