Luis, I got the patch but I can't test it now as we have LAVA maintenance/migration happening this week. I'll get back to it when migration is done.
milosz On 3 July 2015 at 11:56, Luis Araujo <luis.ara...@collabora.co.uk> wrote: > Hello Milosz, > > Thanks for going checking this issue that far and pointing me to the cause > of it. > > I glanced over your code and this certainly will involve quite few > operations to properly > get reported the exact missing test defs from LAVA. I also noticed that one > of the > problem for lqa not reporting the 'mmtests' is this particular case was > about not > properly handling duplicated tests names from the bundles. > > I put a simple solution that should handle the below case better, with the > patch from my branch at: > > https://git.collabora.com/cgit/user/araujo/lqa.git/log/?h=WIP-fix-find-missing-tests-duplicate > > Even though we don't detect the same exact missing tests, we are mainly > interested > at the beginning to have an idea of what went wrong with any of the tests. > > Let me know if you still don't get missing results with that patch, or maybe > if > you find other case giving such a failure. > > Cheers, > > Luis > > > On 07/02/2015 11:18 PM, Milosz Wasilewski wrote: >> >> Luis, >> >> I have the exact case that fails for restricted job. Here it is: >> ./lqa -c ./examples/lqa.yaml analyse 415150 >> Generating lqa report for job(s): 415150 >> >> Report for job(s) (Thu Jul 02 16:16:09 2015): >> 415150 >> 1 test job(s) ran: 1 complete (0 fully successful, 1 wih failures), 0 >> incomplete >> >> * --- Failed Jobs --- * >> >> (F) Jobs with failed tests: >> >> 415150: >> https://ci.linaro.org/jenkins/job/linux-linaro-stable-lsk-v3.18/hwpack=beaglebone,label=linaro-kernel-ci/12/ >> >> ==================================================================================================================== >> 116 passed, 4 failed, 0 skipped, 0 unknown >> FAILED | lava:test_kernel_exception_1 >> FAILED | lava:test_kernel_exception_1 >> FAILED | lava:lava_test_shell >> FAILED | lava:test_kernel_exception_1 >> Job: https://validation.linaro.org/scheduler/job/415150 >> Bundle: >> https://validation.linaro.org/dashboard/permalink/bundle/d54379110280ae1d33c9cd45dcfb02fbe608153f/ >> >> Hope that helps (one instance of mmtest is missing - compare results >> with definition). As I wrote before, I have the code that fixes the >> issue if you're interested. For simple cases your code works well >> (checked earlier). >> >> milosz >> >> On 1 July 2015 at 10:10, Milosz Wasilewski <milosz.wasilew...@linaro.org> >> wrote: >>> >>> Luis, >>> >>> >>> >>> On 1 July 2015 at 08:21, Luis Araujo <luis.ara...@collabora.co.uk> wrote: >>>> >>>> On 06/30/2015 09:39 PM, Milosz Wasilewski wrote: >>>>> >>>>> On 30 June 2015 at 10:05, Milosz Wasilewski >>>>> <milosz.wasilew...@linaro.org> wrote: >>>>>> >>>>>> On 25 June 2015 at 16:32, Luis Araujo <luis.ara...@collabora.co.uk> >>>>>> wrote: >>>>>>>> >>>>>>>> I checked and it doesn't work (doesn't detect missing results). Here >>>>>>>> is example job: >>>>>>>> https://validation.linaro.org/scheduler/job/382325 (I'm not sure >>>>>>>> it's >>>>>>>> publicly available) >>>>>>> >>>>>>> >>>>>>> I cannot access it (even logged in with my launchpad account). >>>>>>> >>>>>>> Can you send me a link publicly available with this same problem?, I >>>>>>> really >>>>>>> would like to check this out. >>>>>>> >>>>>> I'll try to reproduce it with some other device. >>>>>> >>>>> Luis, >>>>> >>>>> Not exactly the same case, but I'm able to show that the missing >>>>> results are not detected. Here is the job: >>>>> https://validation.linaro.org/scheduler/job/410147 >>>>> >>>>> Here is my output: >>>>> Generating lqa report for job(s): 410147 >>>>> >>>>> Report for job(s) (Tue Jun 30 14:33:11 2015): >>>>> 410147 >>>>> 1 test job(s) ran: 1 complete (0 fully successful, 1 wih failures), 0 >>>>> incomplete >>>>> >>>>> * --- Failed Jobs --- * >>>>> >>>>> (F) Jobs with failed tests: >>>>> >>>>> 410147: >>>>> >>>>> https://ci.linaro.org/jenkins/job/linux-linaro-stable-lsk-v3.18/hwpack=beaglebone,label=linaro-kernel-ci/13/ >>>>> >>>>> >>>>> ==================================================================================================================== >>>>> 319 passed, 17 failed, 27 skipped, 0 unknown >>>>> FAILED | rcutorture:rcutorture-start >>>>> FAILED | rcutorture:lava-test-shell-run >>>>> FAILED | pwrmgmt:cputopology_01 >>>>> FAILED | usb-test-basic:list-all-usb-devices >>>>> FAILED | usb-test-basic:examine-all-usb-devices >>>>> FAILED | usb-test-basic:print-supported-protocols >>>>> FAILED | usb-test-basic:print-supported-speeds >>>>> FAILED | lava:test_kernel_exception_1 >>>>> FAILED | lava:test_kernel_exception_1 >>>>> FAILED | lava:test_kernel_exception_1 >>>>> FAILED | lava:test_kernel_exception_1 >>>>> FAILED | lava:test_kernel_exception_1 >>>>> FAILED | lava:test_kernel_exception_1 >>>>> FAILED | lava:test_kernel_exception_1 >>>>> FAILED | lava:test_kernel_exception_1 >>>>> FAILED | lava:test_kernel_exception_1 >>>>> FAILED | lava:test_kernel_exception_1 >>>>> Job: https://validation.linaro.org/scheduler/job/410147 >>>>> Bundle: >>>>> >>>>> https://validation.linaro.org/dashboard/permalink/bundle/f022ec4a6ce822e154ff600d731025ad978a961b/ >>>>> >>>>> In this case the LTP entries are there in the bundle. However one is >>>>> missing the parameters. Anyway, I'll try to reproduce the original >>>>> problem in some other test (maybe an artificial one, so you). In case >>>>> of the original problem the entry for one LTP shell is missing from >>>>> the bundle. >>>>> >>>>> milosz >>>> >>>> >>>> Right, the method won't report missing results for this job since all >>>> the >>>> testdef's >>>> specified in the yaml are available in the bundle: >>>> >>>> https://validation.linaro.org/scheduler/job/410147/definition >>>> >>>> https://validation.linaro.org/dashboard/streams/anonymous/mwasilew/bundles/f022ec4a6ce822e154ff600d731025ad978a961b/?search=&length=25#table >>>> >>>> The _find_missing_tests method is actually used only to check for those >>>> testdef from >>>> the lava_test_shell command that for some reason were not properly >>>> reported >>>> in the >>>> bundle stream. Now, if these tests have missing results, certainly, the >>>> method won't check >>>> for those, but this could be a later improvement for sure. >>>> >>> As I wrote this isn't exactly the same issue as I reported previously. >>> If you take one of LTP result entries out from the bundle you will get >>> what I wrote about before. So the situation is you ask for 2 LTP test >>> shells with different parameters and result bundle only contains one >>> result for LTP. The other one is entirely missing. >>> >>> If you want to do the missing detection a bit better, take a look here: >>> >>> https://git.linaro.org/qa/qa-reports.git/blob/refs/heads/refactoring:/utils/testminer.py#l172 >>> Eventually it should be fixed on LAVA side, but for the moment this is >>> what we get. >>> >>> milosz > > > _______________________________________________ linaro-validation mailing list linaro-validation@lists.linaro.org https://lists.linaro.org/mailman/listinfo/linaro-validation
