On Tue, Feb 11, 2025 at 12:17 PM Andy Shevchenko
<[email protected]> wrote:
>
> On Tue, Feb 11, 2025 at 10:57:11AM -0500, Tamir Duberstein wrote:
> > On Tue, Feb 11, 2025 at 10:54 AM Andy Shevchenko
> > <[email protected]> wrote:
> > > On Tue, Feb 11, 2025 at 10:47:03AM -0500, Tamir Duberstein wrote:
> > > > On Tue, Feb 11, 2025 at 10:40 AM Andy Shevchenko
> > > > <[email protected]> wrote:
> > > > > On Tue, Feb 11, 2025 at 10:13:36AM -0500, Tamir Duberstein wrote:
>
> ...
>
> > > > > > - Use original test assertions as KUNIT_*_EQ_MSG produces
> > > > > > hard-to-parse
> > > > > > messages. The new failure output is:
> > > > >
> > > > > It would be good if you put into cover letter, or even in the
> > > > > respectful patch
> > > > > the example of the error report for the old code and new code that it
> > > > > will be
> > > > > clear how it changes.
> > > > >
> > > > > > vsscanf("0 1e 3e43 31f0 0 0 5797 9c70", "%1hx %2hx %4hx %4hx
> > > > > > %1hx %1hx %4hx %4hx", ...) expected 837828163 got 1044578334
> > > > > > not ok 1 " "
> > > > > > # numbers_list_field_width_val_width: ASSERTION FAILED at
> > > > > > lib/scanf_kunit.c:92
> > > > > > vsscanf("dc2:1c:0:3531:2621:5172:1:7",
> > > > > > "%3hx:%2hx:%1hx:%4hx:%4hx:%4hx:%1hx:%1hx", ...) expected 892403712
> > > > > > got 28
> > > > > > not ok 2 ":"
> > > > > > # numbers_list_field_width_val_width: ASSERTION FAILED at
> > > > > > lib/scanf_kunit.c:92
> > > > > > vsscanf("e083,8f6e,b,70ca,1,1,aab1,10e4",
> > > > > > "%4hx,%4hx,%1hx,%4hx,%1hx,%1hx,%4hx,%4hx", ...) expected 1892286475
> > > > > > got 757614
> > > > > > not ok 3 ","
> > > > > > # numbers_list_field_width_val_width: ASSERTION FAILED at
> > > > > > lib/scanf_kunit.c:92
> > > > > > vsscanf("2e72-8435-1-2fc-7cbd-c2f1-7158-2b41",
> > > > > > "%4hx-%4hx-%1hx-%3hx-%4hx-%4hx-%4hx-%4hx", ...) expected 50069505
> > > > > > got 99381
> > > > > > not ok 4 "-"
> > > > > > # numbers_list_field_width_val_width: ASSERTION FAILED at
> > > > > > lib/scanf_kunit.c:92
> > > > > > vsscanf("403/0/17/1/11e7/1/1fe8/34ba",
> > > > > > "%3hx/%1hx/%2hx/%1hx/%4hx/%1hx/%4hx/%4hx", ...) expected 65559 got
> > > > > > 1507328
> > > > > > not ok 5 "/"
> > > > > > # numbers_list_field_width_val_width: pass:0 fail:5 skip:0
> > > > > > total:5
> > > > > > not ok 4 numbers_list_field_width_val_width
> > > > > > # numbers_slice: ASSERTION FAILED at lib/scanf_kunit.c:92
> > > > > > vsscanf("3c87eac0f4afa1f9231da52",
> > > > > > "%1hx%4hx%4hx%4hx%1hx%4hx%4hx%1hx", ...) expected 1257942031 got
> > > > > > 2886715518
> > > >
> > > > Makes sense. As you can see the error report for the new code is
> > > > included here. I'll add the old code's error report if I have to
> > > > respin v8.
> > >
> > > At a bare minimum. can you add in the reply to this email?
> >
> > Oh, sure:
> >
> > On Tue, Feb 11, 2025 at 6:54 AM Petr Mladek <[email protected]> wrote:
> > >
> > > [...]
> > >
> > > [ 383.100048] test_scanf: vsscanf("1574 9 64ca 935b 7 142d ff58 0",
> > > "%4hx %1hx %4hx %4hx %1hx %4hx %4hx %1hx", ...) expected 2472240330 got
> > > 1690959881
> > > [ 383.102843] test_scanf: vsscanf("f12:2:d:2:c166:1:36b:1906",
> > > "%3hx:%1hx:%1hx:%1hx:%4hx:%1hx:%3hx:%4hx", ...) expected 131085 got 851970
> > > [ 383.105376] test_scanf: vsscanf("4,b2fe,3,593,6,0,3bde,0",
> > > "%1hx,%4hx,%1hx,%3hx,%1hx,%1hx,%4hx,%1hx", ...) expected 93519875 got
> > > 242430
> > > [ 383.105659] test_scanf: vsscanf("6-1-2-1-d9e6-f-93e-e567",
> > > "%1hx-%1hx-%1hx-%1hx-%4hx-%1hx-%3hx-%4hx", ...) expected 65538 got 131073
> > > [ 383.106127] test_scanf: vsscanf("72d6/35/e88d/1/0/6c8c/7/1",
> > > "%4hx/%2hx/%4hx/%1hx/%1hx/%4hx/%1hx/%1hx", ...) expected 125069 got
> > > 3901554741
> > > [ 383.106235] test_scanf: vsscanf("c9bea1b8122113e9a168df573",
> > > "%4hx%4hx%1hx%4hx%4hx%1hx%4hx%3hx", ...) expected 571539457 got 106936
>
> > > [ 383.106398] test_scanf: failed 6 out of 2545 tests
>
> Is it me who cut something or the above missing this information (total
> tests)?
> If the latter, how are we supposed to answer to the question if the failed
> test
> is from new bunch of cases I hypothetically added or regression of the
> existing
> ones? Without this it seems like I need to go through all failures. OTOH it
> may
> be needed anyway as failing test case needs an investigation.
I assume you mean missing from the new output. Yeah, KUnit doesn't do
this counting. Instead you get the test name in the failure message:
> > > > > > vsscanf("0 1e 3e43 31f0 0 0 5797 9c70", "%1hx %2hx %4hx %4hx
> > > > > > %1hx %1hx %4hx %4hx", ...) expected 837828163 got 1044578334
> > > > > > not ok 1 " "
> > > > > > # numbers_list_field_width_val_width: ASSERTION FAILED at
> > > > > > lib/scanf_kunit.c:92
I think maybe you're saying: what if I add a new assertion (rather
than a new test case), and I start getting failure reports - how do I
know if the reporter is running old or new test code?
In an ideal world the message above would give you all the information
you need by including the line number from the test. This doesn't
quite work out in this case because of the various test helper
functions; you end up with a line number in the test helper rather
than in the test itself. We could fix that by passing around __FILE__
and __LINE__ (probably by wrapping the test helpers in a macro). What
do you think?
