Hi Luis, I totally forgot about that - adding to my 'todo' list.
milosz On 9 September 2015 at 11:27, Luis Araujo <luis.ara...@collabora.co.uk> wrote: > Hello Milosz, > > I wonder if you had the chance to test this patch after all. > > Cheers, > > Luis > > > On 07/07/2015 04:16 PM, Milosz Wasilewski wrote: >> >> Luis, >> >> I got the patch but I can't test it now as we have LAVA >> maintenance/migration happening this week. I'll get back to it when >> migration is done. >> >> milosz >> >> On 3 July 2015 at 11:56, Luis Araujo <luis.ara...@collabora.co.uk> wrote: >>> >>> Hello Milosz, >>> >>> Thanks for going checking this issue that far and pointing me to the >>> cause >>> of it. >>> >>> I glanced over your code and this certainly will involve quite few >>> operations to properly >>> get reported the exact missing test defs from LAVA. I also noticed that >>> one >>> of the >>> problem for lqa not reporting the 'mmtests' is this particular case was >>> about not >>> properly handling duplicated tests names from the bundles. >>> >>> I put a simple solution that should handle the below case better, with >>> the >>> patch from my branch at: >>> >>> >>> https://git.collabora.com/cgit/user/araujo/lqa.git/log/?h=WIP-fix-find-missing-tests-duplicate >>> >>> Even though we don't detect the same exact missing tests, we are mainly >>> interested >>> at the beginning to have an idea of what went wrong with any of the >>> tests. >>> >>> Let me know if you still don't get missing results with that patch, or >>> maybe >>> if >>> you find other case giving such a failure. >>> >>> Cheers, >>> >>> Luis >>> >>> >>> On 07/02/2015 11:18 PM, Milosz Wasilewski wrote: >>>> >>>> Luis, >>>> >>>> I have the exact case that fails for restricted job. Here it is: >>>> ./lqa -c ./examples/lqa.yaml analyse 415150 >>>> Generating lqa report for job(s): 415150 >>>> >>>> Report for job(s) (Thu Jul 02 16:16:09 2015): >>>> 415150 >>>> 1 test job(s) ran: 1 complete (0 fully successful, 1 wih failures), 0 >>>> incomplete >>>> >>>> * --- Failed Jobs --- * >>>> >>>> (F) Jobs with failed tests: >>>> >>>> 415150: >>>> >>>> https://ci.linaro.org/jenkins/job/linux-linaro-stable-lsk-v3.18/hwpack=beaglebone,label=linaro-kernel-ci/12/ >>>> >>>> >>>> ==================================================================================================================== >>>> 116 passed, 4 failed, 0 skipped, 0 unknown >>>> FAILED | lava:test_kernel_exception_1 >>>> FAILED | lava:test_kernel_exception_1 >>>> FAILED | lava:lava_test_shell >>>> FAILED | lava:test_kernel_exception_1 >>>> Job: https://validation.linaro.org/scheduler/job/415150 >>>> Bundle: >>>> >>>> https://validation.linaro.org/dashboard/permalink/bundle/d54379110280ae1d33c9cd45dcfb02fbe608153f/ >>>> >>>> Hope that helps (one instance of mmtest is missing - compare results >>>> with definition). As I wrote before, I have the code that fixes the >>>> issue if you're interested. For simple cases your code works well >>>> (checked earlier). >>>> >>>> milosz >>>> >>>> On 1 July 2015 at 10:10, Milosz Wasilewski >>>> <milosz.wasilew...@linaro.org> >>>> wrote: >>>>> >>>>> Luis, >>>>> >>>>> >>>>> >>>>> On 1 July 2015 at 08:21, Luis Araujo <luis.ara...@collabora.co.uk> >>>>> wrote: >>>>>> >>>>>> On 06/30/2015 09:39 PM, Milosz Wasilewski wrote: >>>>>>> >>>>>>> On 30 June 2015 at 10:05, Milosz Wasilewski >>>>>>> <milosz.wasilew...@linaro.org> wrote: >>>>>>>> >>>>>>>> On 25 June 2015 at 16:32, Luis Araujo <luis.ara...@collabora.co.uk> >>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> I checked and it doesn't work (doesn't detect missing results). >>>>>>>>>> Here >>>>>>>>>> is example job: >>>>>>>>>> https://validation.linaro.org/scheduler/job/382325 (I'm not sure >>>>>>>>>> it's >>>>>>>>>> publicly available) >>>>>>>>> >>>>>>>>> >>>>>>>>> I cannot access it (even logged in with my launchpad account). >>>>>>>>> >>>>>>>>> Can you send me a link publicly available with this same problem?, >>>>>>>>> I >>>>>>>>> really >>>>>>>>> would like to check this out. >>>>>>>>> >>>>>>>> I'll try to reproduce it with some other device. >>>>>>>> >>>>>>> Luis, >>>>>>> >>>>>>> Not exactly the same case, but I'm able to show that the missing >>>>>>> results are not detected. Here is the job: >>>>>>> https://validation.linaro.org/scheduler/job/410147 >>>>>>> >>>>>>> Here is my output: >>>>>>> Generating lqa report for job(s): 410147 >>>>>>> >>>>>>> Report for job(s) (Tue Jun 30 14:33:11 2015): >>>>>>> 410147 >>>>>>> 1 test job(s) ran: 1 complete (0 fully successful, 1 wih failures), 0 >>>>>>> incomplete >>>>>>> >>>>>>> * --- Failed Jobs --- * >>>>>>> >>>>>>> (F) Jobs with failed tests: >>>>>>> >>>>>>> 410147: >>>>>>> >>>>>>> >>>>>>> https://ci.linaro.org/jenkins/job/linux-linaro-stable-lsk-v3.18/hwpack=beaglebone,label=linaro-kernel-ci/13/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> ==================================================================================================================== >>>>>>> 319 passed, 17 failed, 27 skipped, 0 unknown >>>>>>> FAILED | rcutorture:rcutorture-start >>>>>>> FAILED | rcutorture:lava-test-shell-run >>>>>>> FAILED | pwrmgmt:cputopology_01 >>>>>>> FAILED | usb-test-basic:list-all-usb-devices >>>>>>> FAILED | usb-test-basic:examine-all-usb-devices >>>>>>> FAILED | usb-test-basic:print-supported-protocols >>>>>>> FAILED | usb-test-basic:print-supported-speeds >>>>>>> FAILED | lava:test_kernel_exception_1 >>>>>>> FAILED | lava:test_kernel_exception_1 >>>>>>> FAILED | lava:test_kernel_exception_1 >>>>>>> FAILED | lava:test_kernel_exception_1 >>>>>>> FAILED | lava:test_kernel_exception_1 >>>>>>> FAILED | lava:test_kernel_exception_1 >>>>>>> FAILED | lava:test_kernel_exception_1 >>>>>>> FAILED | lava:test_kernel_exception_1 >>>>>>> FAILED | lava:test_kernel_exception_1 >>>>>>> FAILED | lava:test_kernel_exception_1 >>>>>>> Job: https://validation.linaro.org/scheduler/job/410147 >>>>>>> Bundle: >>>>>>> >>>>>>> >>>>>>> https://validation.linaro.org/dashboard/permalink/bundle/f022ec4a6ce822e154ff600d731025ad978a961b/ >>>>>>> >>>>>>> In this case the LTP entries are there in the bundle. However one is >>>>>>> missing the parameters. Anyway, I'll try to reproduce the original >>>>>>> problem in some other test (maybe an artificial one, so you). In case >>>>>>> of the original problem the entry for one LTP shell is missing from >>>>>>> the bundle. >>>>>>> >>>>>>> milosz >>>>>> >>>>>> >>>>>> Right, the method won't report missing results for this job since all >>>>>> the >>>>>> testdef's >>>>>> specified in the yaml are available in the bundle: >>>>>> >>>>>> https://validation.linaro.org/scheduler/job/410147/definition >>>>>> >>>>>> >>>>>> https://validation.linaro.org/dashboard/streams/anonymous/mwasilew/bundles/f022ec4a6ce822e154ff600d731025ad978a961b/?search=&length=25#table >>>>>> >>>>>> The _find_missing_tests method is actually used only to check for >>>>>> those >>>>>> testdef from >>>>>> the lava_test_shell command that for some reason were not properly >>>>>> reported >>>>>> in the >>>>>> bundle stream. Now, if these tests have missing results, certainly, >>>>>> the >>>>>> method won't check >>>>>> for those, but this could be a later improvement for sure. >>>>>> >>>>> As I wrote this isn't exactly the same issue as I reported previously. >>>>> If you take one of LTP result entries out from the bundle you will get >>>>> what I wrote about before. So the situation is you ask for 2 LTP test >>>>> shells with different parameters and result bundle only contains one >>>>> result for LTP. The other one is entirely missing. >>>>> >>>>> If you want to do the missing detection a bit better, take a look here: >>>>> >>>>> >>>>> https://git.linaro.org/qa/qa-reports.git/blob/refs/heads/refactoring:/utils/testminer.py#l172 >>>>> Eventually it should be fixed on LAVA side, but for the moment this is >>>>> what we get. >>>>> >>>>> milosz >>> >>> >>> > > _______________________________________________ linaro-validation mailing list linaro-validation@lists.linaro.org https://lists.linaro.org/mailman/listinfo/linaro-validation
