Luis,
I have the exact case that fails for restricted job. Here it is:
./lqa -c ./examples/lqa.yaml analyse 415150
Generating lqa report for job(s): 415150
Report for job(s) (Thu Jul 02 16:16:09 2015):
415150
1 test job(s) ran: 1 complete (0 fully successful, 1 wih failures), 0 incomplete
* --- Failed Jobs --- *
(F) Jobs with failed tests:
415150:
https://ci.linaro.org/jenkins/job/linux-linaro-stable-lsk-v3.18/hwpack=beaglebone,label=linaro-kernel-ci/12/
====================================================================================================================
116 passed, 4 failed, 0 skipped, 0 unknown
FAILED | lava:test_kernel_exception_1
FAILED | lava:test_kernel_exception_1
FAILED | lava:lava_test_shell
FAILED | lava:test_kernel_exception_1
Job: https://validation.linaro.org/scheduler/job/415150
Bundle:
https://validation.linaro.org/dashboard/permalink/bundle/d54379110280ae1d33c9cd45dcfb02fbe608153f/
Hope that helps (one instance of mmtest is missing - compare results
with definition). As I wrote before, I have the code that fixes the
issue if you're interested. For simple cases your code works well
(checked earlier).
milosz
On 1 July 2015 at 10:10, Milosz Wasilewski <milosz.wasilew...@linaro.org> wrote:
> Luis,
>
>
>
> On 1 July 2015 at 08:21, Luis Araujo <luis.ara...@collabora.co.uk> wrote:
>> On 06/30/2015 09:39 PM, Milosz Wasilewski wrote:
>>>
>>> On 30 June 2015 at 10:05, Milosz Wasilewski
>>> <milosz.wasilew...@linaro.org> wrote:
>>>>
>>>> On 25 June 2015 at 16:32, Luis Araujo <luis.ara...@collabora.co.uk>
>>>> wrote:
>>>>>>
>>>>>> I checked and it doesn't work (doesn't detect missing results). Here
>>>>>> is example job:
>>>>>> https://validation.linaro.org/scheduler/job/382325 (I'm not sure it's
>>>>>> publicly available)
>>>>>
>>>>>
>>>>> I cannot access it (even logged in with my launchpad account).
>>>>>
>>>>> Can you send me a link publicly available with this same problem?, I
>>>>> really
>>>>> would like to check this out.
>>>>>
>>>> I'll try to reproduce it with some other device.
>>>>
>>> Luis,
>>>
>>> Not exactly the same case, but I'm able to show that the missing
>>> results are not detected. Here is the job:
>>> https://validation.linaro.org/scheduler/job/410147
>>>
>>> Here is my output:
>>> Generating lqa report for job(s): 410147
>>>
>>> Report for job(s) (Tue Jun 30 14:33:11 2015):
>>> 410147
>>> 1 test job(s) ran: 1 complete (0 fully successful, 1 wih failures), 0
>>> incomplete
>>>
>>> * --- Failed Jobs --- *
>>>
>>> (F) Jobs with failed tests:
>>>
>>> 410147:
>>> https://ci.linaro.org/jenkins/job/linux-linaro-stable-lsk-v3.18/hwpack=beaglebone,label=linaro-kernel-ci/13/
>>>
>>> ====================================================================================================================
>>> 319 passed, 17 failed, 27 skipped, 0 unknown
>>> FAILED | rcutorture:rcutorture-start
>>> FAILED | rcutorture:lava-test-shell-run
>>> FAILED | pwrmgmt:cputopology_01
>>> FAILED | usb-test-basic:list-all-usb-devices
>>> FAILED | usb-test-basic:examine-all-usb-devices
>>> FAILED | usb-test-basic:print-supported-protocols
>>> FAILED | usb-test-basic:print-supported-speeds
>>> FAILED | lava:test_kernel_exception_1
>>> FAILED | lava:test_kernel_exception_1
>>> FAILED | lava:test_kernel_exception_1
>>> FAILED | lava:test_kernel_exception_1
>>> FAILED | lava:test_kernel_exception_1
>>> FAILED | lava:test_kernel_exception_1
>>> FAILED | lava:test_kernel_exception_1
>>> FAILED | lava:test_kernel_exception_1
>>> FAILED | lava:test_kernel_exception_1
>>> FAILED | lava:test_kernel_exception_1
>>> Job: https://validation.linaro.org/scheduler/job/410147
>>> Bundle:
>>> https://validation.linaro.org/dashboard/permalink/bundle/f022ec4a6ce822e154ff600d731025ad978a961b/
>>>
>>> In this case the LTP entries are there in the bundle. However one is
>>> missing the parameters. Anyway, I'll try to reproduce the original
>>> problem in some other test (maybe an artificial one, so you). In case
>>> of the original problem the entry for one LTP shell is missing from
>>> the bundle.
>>>
>>> milosz
>>
>>
>> Right, the method won't report missing results for this job since all the
>> testdef's
>> specified in the yaml are available in the bundle:
>>
>> https://validation.linaro.org/scheduler/job/410147/definition
>> https://validation.linaro.org/dashboard/streams/anonymous/mwasilew/bundles/f022ec4a6ce822e154ff600d731025ad978a961b/?search=&length=25#table
>>
>> The _find_missing_tests method is actually used only to check for those
>> testdef from
>> the lava_test_shell command that for some reason were not properly reported
>> in the
>> bundle stream. Now, if these tests have missing results, certainly, the
>> method won't check
>> for those, but this could be a later improvement for sure.
>>
>
> As I wrote this isn't exactly the same issue as I reported previously.
> If you take one of LTP result entries out from the bundle you will get
> what I wrote about before. So the situation is you ask for 2 LTP test
> shells with different parameters and result bundle only contains one
> result for LTP. The other one is entirely missing.
>
> If you want to do the missing detection a bit better, take a look here:
> https://git.linaro.org/qa/qa-reports.git/blob/refs/heads/refactoring:/utils/testminer.py#l172
> Eventually it should be fixed on LAVA side, but for the moment this is
> what we get.
>
> milosz
_______________________________________________
linaro-validation mailing list
linaro-validation@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/linaro-validation
