Hello Milosz,Thanks for going checking this issue that far and pointing me to the cause of it.
I glanced over your code and this certainly will involve quite few operations to properly get reported the exact missing test defs from LAVA. I also noticed that one of the problem for lqa not reporting the 'mmtests' is this particular case was about not
properly handling duplicated tests names from the bundles. I put a simple solution that should handle the below case better, with the patch from my branch at: https://git.collabora.com/cgit/user/araujo/lqa.git/log/?h=WIP-fix-find-missing-tests-duplicateEven though we don't detect the same exact missing tests, we are mainly interested
at the beginning to have an idea of what went wrong with any of the tests.Let me know if you still don't get missing results with that patch, or maybe if
you find other case giving such a failure. Cheers, Luis On 07/02/2015 11:18 PM, Milosz Wasilewski wrote:
Luis, I have the exact case that fails for restricted job. Here it is: ./lqa -c ./examples/lqa.yaml analyse 415150 Generating lqa report for job(s): 415150 Report for job(s) (Thu Jul 02 16:16:09 2015): 415150 1 test job(s) ran: 1 complete (0 fully successful, 1 wih failures), 0 incomplete * --- Failed Jobs --- * (F) Jobs with failed tests: 415150: https://ci.linaro.org/jenkins/job/linux-linaro-stable-lsk-v3.18/hwpack=beaglebone,label=linaro-kernel-ci/12/ ==================================================================================================================== 116 passed, 4 failed, 0 skipped, 0 unknown FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:lava_test_shell FAILED | lava:test_kernel_exception_1 Job: https://validation.linaro.org/scheduler/job/415150 Bundle: https://validation.linaro.org/dashboard/permalink/bundle/d54379110280ae1d33c9cd45dcfb02fbe608153f/ Hope that helps (one instance of mmtest is missing - compare results with definition). As I wrote before, I have the code that fixes the issue if you're interested. For simple cases your code works well (checked earlier). milosz On 1 July 2015 at 10:10, Milosz Wasilewski <milosz.wasilew...@linaro.org> wrote:Luis, On 1 July 2015 at 08:21, Luis Araujo <luis.ara...@collabora.co.uk> wrote:On 06/30/2015 09:39 PM, Milosz Wasilewski wrote:On 30 June 2015 at 10:05, Milosz Wasilewski <milosz.wasilew...@linaro.org> wrote:On 25 June 2015 at 16:32, Luis Araujo <luis.ara...@collabora.co.uk> wrote:I checked and it doesn't work (doesn't detect missing results). Here is example job: https://validation.linaro.org/scheduler/job/382325 (I'm not sure it's publicly available)I cannot access it (even logged in with my launchpad account). Can you send me a link publicly available with this same problem?, I really would like to check this out.I'll try to reproduce it with some other device.Luis, Not exactly the same case, but I'm able to show that the missing results are not detected. Here is the job: https://validation.linaro.org/scheduler/job/410147 Here is my output: Generating lqa report for job(s): 410147 Report for job(s) (Tue Jun 30 14:33:11 2015): 410147 1 test job(s) ran: 1 complete (0 fully successful, 1 wih failures), 0 incomplete * --- Failed Jobs --- * (F) Jobs with failed tests: 410147: https://ci.linaro.org/jenkins/job/linux-linaro-stable-lsk-v3.18/hwpack=beaglebone,label=linaro-kernel-ci/13/ ==================================================================================================================== 319 passed, 17 failed, 27 skipped, 0 unknown FAILED | rcutorture:rcutorture-start FAILED | rcutorture:lava-test-shell-run FAILED | pwrmgmt:cputopology_01 FAILED | usb-test-basic:list-all-usb-devices FAILED | usb-test-basic:examine-all-usb-devices FAILED | usb-test-basic:print-supported-protocols FAILED | usb-test-basic:print-supported-speeds FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 Job: https://validation.linaro.org/scheduler/job/410147 Bundle: https://validation.linaro.org/dashboard/permalink/bundle/f022ec4a6ce822e154ff600d731025ad978a961b/ In this case the LTP entries are there in the bundle. However one is missing the parameters. Anyway, I'll try to reproduce the original problem in some other test (maybe an artificial one, so you). In case of the original problem the entry for one LTP shell is missing from the bundle. miloszRight, the method won't report missing results for this job since all the testdef's specified in the yaml are available in the bundle: https://validation.linaro.org/scheduler/job/410147/definition https://validation.linaro.org/dashboard/streams/anonymous/mwasilew/bundles/f022ec4a6ce822e154ff600d731025ad978a961b/?search=&length=25#table The _find_missing_tests method is actually used only to check for those testdef from the lava_test_shell command that for some reason were not properly reported in the bundle stream. Now, if these tests have missing results, certainly, the method won't check for those, but this could be a later improvement for sure.As I wrote this isn't exactly the same issue as I reported previously. If you take one of LTP result entries out from the bundle you will get what I wrote about before. So the situation is you ask for 2 LTP test shells with different parameters and result bundle only contains one result for LTP. The other one is entirely missing. If you want to do the missing detection a bit better, take a look here: https://git.linaro.org/qa/qa-reports.git/blob/refs/heads/refactoring:/utils/testminer.py#l172 Eventually it should be fixed on LAVA side, but for the moment this is what we get. milosz
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ linaro-validation mailing list linaro-validation@lists.linaro.org https://lists.linaro.org/mailman/listinfo/linaro-validation
