On Tue, 27 Nov 2018 at 12:22, Oleksandr Terentiev <otere...@cisco.com>
wrote:
> Thanks for replay.
> Is there any public LAVA instance where I can run the example test? Maybe
> on staging.validation.linaro.org ?
> Can I be registered there and how? Or maybe there is another approach to
> run that.
>
I can run the LAVA job definition, if you share with me.
Cheers,
Anibal
>
> On 27.11.18 19:41, Anibal Limon wrote:
>
>
>
>
> On Tue, 27 Nov 2018 at 08:29, Oleksandr Terentiev <otere...@cisco.com>
> wrote:
>
>> Hi Abibal,
>>
>> In our project we need to analyze a total number of passed and failed
>> tests for each packet. To distinguish packets we use lava-test-set feature.
>> In order to implement that I modified ptest.py and send-to-lava.sh
>> scripts. Could you please look at the patch and express your opinion?
>> Maybe this code can be added to git.linaro.org/qa/test-definitions.git ?
>>
>
> Hi Oleksandr,
>
> The code looks good, can you have an example of the LAVA test run to see
> the actual results?
>
> Regards,
> Anibal
>
>
>> Best regards,
>> Alex
>>
>> automated/linux/ptest: Analyze each test in package tests
>>
>> Currently ptest.py analyze only exit code of each package test
>> to decide if it passed or not. However, ptest-runner can return
>> success code even though some tests failed. So we need to parse
>> test output and analyze it.
>>
>> It also quite useful to see exactly which tests failed. So results are
>> recorded for each particular test, and lava-test-set feature is used
>> to distinguish packages.
>>
>> Signed-off-by: Oleksandr Terentiev <otere...@cisco.com>
>> <otere...@cisco.com>
>>
>> diff --git a/automated/linux/ptest/ptest.py
>> b/automated/linux/ptest/ptest.py
>> index 13feb4d..a28d7f0 100755
>> --- a/automated/linux/ptest/ptest.py
>> +++ b/automated/linux/ptest/ptest.py
>> @@ -84,20 +84,60 @@ def filter_ptests(ptests, requested_ptests, exclude):
>>
>> return filter_ptests
>>
>> +def parse_line(line):
>> + test_status_list = {
>> + 'pass': re.compile("^PASS:(.+)"),
>> + 'fail': re.compile("^FAIL:(.+)"),
>> + 'skip': re.compile("^SKIP:(.+)")
>> + }
>> +
>> + for test_status, status_regex in test_status_list.items():
>> + test_name = status_regex.search(line)
>> + if test_name:
>> + return [test_name.group(1), test_status]
>>
>> -def check_ptest(ptest_dir, ptest_name, output_log):
>> - status = 'pass'
>> + return None
>>
>> - try:
>> - output = subprocess.check_call('ptest-runner -d %s %s' %
>> - (ptest_dir, ptest_name),
>> shell=True,
>> - stderr=subprocess.STDOUT)
>> - except subprocess.CalledProcessError:
>> - status = 'fail'
>> +def parse_ptest(log_file):
>> + result = []
>>
>> - with open(output_log, 'a+') as f:
>> - f.write("%s %s\n" % (ptest_name, status))
>> + with open(log_file, 'r') as f:
>> + for line in f:
>> + result_tuple = parse_line(line)
>> + if not result_tuple:
>> + continue
>> + print(result_tuple)
>> + result.append(result_tuple)
>> + continue
>>
>> + return result
>> +
>> +def run_command(command, log_file):
>> + process = subprocess.Popen(command,
>> + shell=True,
>> + stdout=subprocess.PIPE,
>> + stderr=subprocess.STDOUT)
>> + with open(log_file, 'w') as f:
>> + while True:
>> + output = process.stdout.readline()
>> + if output == '' and process.poll() is not None:
>> + break
>> + if output:
>> + print output.strip()
>> + f.write("%s\n" % output.strip())
>> + rc = process.poll()
>> + return rc
>> +
>> +def check_ptest(ptest_dir, ptest_name, output_log):
>> + log_name = os.path.join(os.getcwd(), '%s.log' % ptest_name)
>> + status = run_command('ptest-runner -d %s %s' % (ptest_dir,
>> ptest_name), log_name)
>> +
>> + with open(output_log, 'a+') as f:
>> + f.write("lava-test-set start %s\n" % ptest_name)
>> + f.write("%s %s\n" % (ptest_name, "pass" if status == 0 else
>> "fail"))
>> + for test, test_status in parse_ptest(log_name):
>> + f.write("%s %s\n" % (re.sub(r'[^\w-]', '', test),
>> test_status))
>> + f.write("lava-test-set stop %s\n" % ptest_name)
>>
>> def main():
>> parser = argparse.ArgumentParser(description="LAVA/OE ptest script",
>> diff --git a/automated/utils/send-to-lava.sh
>> b/automated/utils/send-to-lava.sh
>> index bf2a477..db4442c 100755
>> --- a/automated/utils/send-to-lava.sh
>> +++ b/automated/utils/send-to-lava.sh
>> @@ -4,6 +4,8 @@ RESULT_FILE="$1"
>>
>> which lava-test-case > /dev/null 2>&1
>> lava_test_case="$?"
>> +which lava-test-set > /dev/null 2>&1
>> +lava_test_set="$?"
>>
>> if [ -f "${RESULT_FILE}" ]; then
>> while read -r line; do
>> @@ -31,6 +33,18 @@ if [ -f "${RESULT_FILE}" ]; then
>> else
>> echo "<TEST_CASE_ID=${test} RESULT=${result}
>> MEASUREMENT=${measurement} UNITS=${units}>"
>> fi
>> + elif echo "${line}" | egrep -iq "^lava-test-set.*"; then
>> + test_set_status="$(echo "${line}" | awk '{print $2}')"
>> + test_set_name="$(echo "${line}" | awk '{print $3}')"
>> + if [ "${lava_test_set}" -eq 0 ]; then
>> + lava-test-set "${test_set_status}" "${test_set_name}"
>> + else
>> + if [ "${test_set_status}" == "start" ]; then
>> + echo "<LAVA_SIGNAL_TESTSET START ${test_set_name}>"
>> + else
>> + echo "<LAVA_SIGNAL_TESTSET STOP>"
>> + fi
>> + fi
>> fi
>> done < "${RESULT_FILE}"
>> else
>>
>>
>>
>> On 01.10.18 17:09, Anibal Limon wrote:
>>
>> Hi,
>>
>> I was on vacation, that's the reason for the slow response, comments
>> below,
>>
>> On Fri, 28 Sep 2018 at 03:59, Oleksandr Terentiev <otere...@cisco.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I would like to discuss the following question.
>>> As it was said now we have to analyze pass/fail of every ptest. From my
>>> point of view there are couple options.
>>>
>>> The first we can parse output and mark ptest as failed if there is even
>>> only one failed test found.
>>>
>>
>> Right I will choice this approach changes needs to be done in the ptest
>> lava script [1] to fail when any of the package tests failed like [2].
>>
>>
>>>
>>> The second we can analyze each test within some packet and record
>>> corresponding results.
>>> I see a few issues here. First of all there will be a large number
>>> of test results as each ptest can run lots of tests.
>>>
>>
>> Right but that need to be handled in every OE ptest script, I mean if you
>> want to fail if certain test inside a ptest fails needs to be done in OE.
>>
>>
>>>
>>> Another thing is that we need somehow separate test results between
>>> particular packets.
>>>
>>
>> Currently we use QA reports to see only the package test result, now If
>> you want to be able look at details you need to see the ptest log.
>>
>>
>>> As an option we can use lava-test-set feature for that. So each test wit
>>> hin ptest will be marked as test case and packet name we will see as
>>> test set.
>>>
>> What do you think about that?
>>>
>>
>> May be the lava-test-set is an option.
>>
>> I would go to do the 1st option and then start to review/implement the
>> idea of use lava-test-feature.
>>
>> Regards,
>> Anibal
>>
>> [1]
>> https://git.linaro.org/qa/test-definitions.git/tree/automated/linux/ptest/ptest.py
>> [2]
>> http://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/runtime/cases/ptest.py#n87
>>
>> Regards,
>>> Alex
>>>
>>> On 23.08.18 16:10, Anibal Limon wrote:
>>>
>>>
>>>
>>> On Thu, 23 Aug 2018 at 05:54, Oleksandr Terentiev <otere...@cisco.com>
>>> wrote:
>>>
>>>> Thank you Anibal for the fast response
>>>>
>>>> On 22.08.18 19:50, Anibal Limon wrote:
>>>>
>>>>
>>>>
>>>> On Wed, 22 Aug 2018 at 11:39, Oleksandr Terentiev <otere...@cisco.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I launched util-linux ptest using automated/linux/ptest/ptest.yaml
>>>>> from
>>>>> https://git.linaro.org/qa/test-definitions.git and received the
>>>>> following results:
>>>>> https://pastebin.com/nj9PYQzE
>>>>>
>>>>> As you can see some tests failed. However, case util-linux marked as
>>>>> passed. It looks like ptest.py only analyze return code of
>>>>> ptest-runner
>>>>> -d <ptest_dir> <ptest_name> command. And since ptest-runner finishes
>>>>> correctly exit code is 0. Therefore all tests are always marked as
>>>>> passed, and users never know when some of the tests fail.
>>>>>
>>>>> Maybe it worth to analyze each test?
>>>>>
>>>>
>>>> Talking about each ptest the result comes from the ptest script in the
>>>> OE recipe [1], for convention if the OE ptest returns 0 means pass, so
>>>> needs to be fixed in the OE ptest [2].
>>>>
>>>> I’ve read https://wiki.yoctoproject.org/wiki/Ptest carefully a few
>>>> times more. There are prescriptions about output format. But I didn’t find
>>>> any mention about return code processing or a reference to the convention
>>>> you mentioned in the answer.
>>>>
>>>> I looked through some OE run-ptest scripts. I suspect they don’t verify
>>>> if some of their tests failed, and exit with 0 even if all their tests
>>>> failed.
>>>>
>>>>
>>>> http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/util-linux/util-linux/run-ptest
>>>>
>>>> http://git.openembedded.org/openembedded-core/tree/meta/recipes-support/attr/acl/run-ptest
>>>>
>>>> http://git.openembedded.org/openembedded-core/tree/meta/recipes-support/attr/files/run-ptest
>>>>
>>>> http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/dbus/dbus/run-ptest
>>>>
>>>> http://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/e2fsprogs/e2fsprogs/run-ptest
>>>>
>>>> http://git.openembedded.org/openembedded-core/tree/meta/recipes-extended/gawk/gawk/run-ptest
>>>>
>>>>
>>> Right, looks that the OEQA test case was update since i worked on it
>>> [1], so now it takes into account the pass/fail of every ptest.
>>> So the ptest.py needs to implement the same behavior.
>>>
>>> Regards,
>>> Anibal
>>>
>>> [1]
>>> http://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/runtime/cases/ptest.py#n80
>>>
>>>
>>>>
>>>> Regarding the LAVA ptest.py script, I mark the run as succeed if there
>>>> is no critical error in the ptest-runner and we have a QA-reports tool to
>>>> analyse pass/fails
>>>> in detail for every ptest executed [3].
>>>>
>>>> I heard about QA-reports tool but I’ve never used it before, so maybe I
>>>> missed something.
>>>> From
>>>> https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/1890442/suite/linux-ptest/tests/
>>>> I see all ptests passed. Still, in log
>>>>
>>>> https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/1890442/log
>>>> I found 54 failed tests and wasn’t able to find a report which
>>>> indicates those failures.
>>>>
>>>> Is there such a report? It would be really useful to know that some
>>>> tests failed.
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>> [1]
>>>> http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/util-linux/util-linux/run-ptest
>>>> [2] https://wiki.yoctoproject.org/wiki/Ptest
>>>> [3]
>>>> https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/1890442/
>>>>
>>>> Regards,
>>>> Anibal
>>>>
>>>>>
>>>>> Best regards,
>>>>> Alex
>>>>>
>>>>>
>>>>
>>>
_______________________________________________
linaro-validation mailing list
linaro-validation@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/linaro-validation
