On Tue, 27 Nov 2018 20:22:42 +0200 Oleksandr Terentiev <otere...@cisco.com> wrote:
> Thanks for replay. > Is there any public LAVA instance where I can run the example test? > Maybe on staging.validation.linaro.org > <https://staging.validation.linaro.org/> ? > Can I be registered there and how? Staging (and other instances hosted by Linaro) share the same authentication support as validation.linaro.org, which is described in the LAVA documentation. https://staging.validation.linaro.org/static/docs/v2/first_steps.html#linaro-lab-users If you do not have a Linaro LDAP account, you can register at https://register.linaro.org/. You could use validation.linaro.org for your test. > Or maybe there is another approach > to run that. Once the script is available in a public git repository instead of only as a patch, it can be executed from any suitable LAVA test job. So push the script and associated support to public git somewhere and post the URL here. Someone on this list can put that into a test job and run that. If you have a test job definition ready to go, put that alongside the script and test definition. Any public git repo will be fine, it doesn't have to be the original one on git.linaro.org. > On 27.11.18 19:41, Anibal Limon wrote: > > > > > > > > On Tue, 27 Nov 2018 at 08:29, Oleksandr Terentiev > > <otere...@cisco.com <mailto:otere...@cisco.com>> wrote: > > > > Hi Abibal, > > > > In our project we need to analyze a total number of passed and > > failed tests for each packet. To distinguish packets we use > > lava-test-set feature. > > In order to implement that I modified ptest.py and > > send-to-lava.sh scripts. Could you please look at the patch and > > express your opinion? Maybe this code can be added to > > git.linaro.org/qa/test-definitions.git > > <http://git.linaro.org/qa/test-definitions.git> ? > > > > > > Hi Oleksandr, > > > > The code looks good, can you have an example of the LAVA test run > > to see the actual results? > > > > Regards, > > Anibal > > > > > > Best regards, > > Alex > > > > automated/linux/ptest: Analyze each test in package tests > > > > Currently ptest.py analyze only exit code of each package test > > to decide if it passed or not. However, ptest-runner can return > > success code even though some tests failed. So we need to parse > > test output and analyze it. > > > > It also quite useful to see exactly which tests failed. So > > results are > > recorded for each particular test, and lava-test-set feature is > > used to distinguish packages. > > > > Signed-off-by: Oleksandr Terentiev <otere...@cisco.com> > > <mailto:otere...@cisco.com> > > > > diff --git a/automated/linux/ptest/ptest.py > > b/automated/linux/ptest/ptest.py > > index 13feb4d..a28d7f0 100755 > > --- a/automated/linux/ptest/ptest.py > > +++ b/automated/linux/ptest/ptest.py > > @@ -84,20 +84,60 @@ def filter_ptests(ptests, requested_ptests, > > exclude): > > > > return filter_ptests > > > > +def parse_line(line): > > + test_status_list = { > > + 'pass': re.compile("^PASS:(.+)"), > > + 'fail': re.compile("^FAIL:(.+)"), > > + 'skip': re.compile("^SKIP:(.+)") > > + } > > + > > + for test_status, status_regex in test_status_list.items(): > > + test_name = status_regex.search(line) > > + if test_name: > > + return [test_name.group(1), test_status] > > > > -def check_ptest(ptest_dir, ptest_name, output_log): > > - status = 'pass' > > + return None > > > > - try: > > - output = subprocess.check_call('ptest-runner -d %s %s' > > % > > - (ptest_dir, ptest_name), > > shell=True, > > - stderr=subprocess.STDOUT) > > - except subprocess.CalledProcessError: > > - status = 'fail' > > +def parse_ptest(log_file): > > + result = [] > > > > - with open(output_log, 'a+') as f: > > - f.write("%s %s\n" % (ptest_name, status)) > > + with open(log_file, 'r') as f: > > + for line in f: > > + result_tuple = parse_line(line) > > + if not result_tuple: > > + continue > > + print(result_tuple) > > + result.append(result_tuple) > > + continue > > > > + return result > > + > > +def run_command(command, log_file): > > + process = subprocess.Popen(command, > > + shell=True, > > + stdout=subprocess.PIPE, > > + stderr=subprocess.STDOUT) > > + with open(log_file, 'w') as f: > > + while True: > > + output = process.stdout.readline() > > + if output == '' and process.poll() is not None: > > + break > > + if output: > > + print output.strip() > > + f.write("%s\n" % output.strip()) > > + rc = process.poll() > > + return rc > > + > > +def check_ptest(ptest_dir, ptest_name, output_log): > > + log_name = os.path.join(os.getcwd(), '%s.log' % ptest_name) > > + status = run_command('ptest-runner -d %s %s' % (ptest_dir, > > ptest_name), log_name) > > + > > + with open(output_log, 'a+') as f: > > + f.write("lava-test-set start %s\n" % ptest_name) > > + f.write("%s %s\n" % (ptest_name, "pass" if status == 0 > > else "fail")) > > + for test, test_status in parse_ptest(log_name): > > + f.write("%s %s\n" % (re.sub(r'[^\w-]', '', test), > > test_status)) > > + f.write("lava-test-set stop %s\n" % ptest_name) > > > > def main(): > > parser = argparse.ArgumentParser(description="LAVA/OE ptest > > script", > > diff --git a/automated/utils/send-to-lava.sh > > b/automated/utils/send-to-lava.sh > > index bf2a477..db4442c 100755 > > --- a/automated/utils/send-to-lava.sh > > +++ b/automated/utils/send-to-lava.sh > > @@ -4,6 +4,8 @@ RESULT_FILE="$1" > > > > which lava-test-case > /dev/null 2>&1 > > lava_test_case="$?" > > +which lava-test-set > /dev/null 2>&1 > > +lava_test_set="$?" > > > > if [ -f "${RESULT_FILE}" ]; then > > while read -r line; do > > @@ -31,6 +33,18 @@ if [ -f "${RESULT_FILE}" ]; then > > else > > echo "<TEST_CASE_ID=${test} RESULT=${result} > > MEASUREMENT=${measurement} UNITS=${units}>" > > fi > > + elif echo "${line}" | egrep -iq "^lava-test-set.*"; > > then > > + test_set_status="$(echo "${line}" | awk '{print > > $2}')" > > + test_set_name="$(echo "${line}" | awk '{print > > $3}')" > > + if [ "${lava_test_set}" -eq 0 ]; then > > + lava-test-set "${test_set_status}" > > "${test_set_name}" > > + else > > + if [ "${test_set_status}" == "start" ]; then > > + echo "<LAVA_SIGNAL_TESTSET START > > ${test_set_name}>" > > + else > > + echo "<LAVA_SIGNAL_TESTSET STOP>" > > + fi > > + fi > > fi > > done < "${RESULT_FILE}" > > else > > > > > > > > On 01.10.18 17:09, Anibal Limon wrote: > >> Hi, > >> > >> I was on vacation, that's the reason for the slow response, > >> comments below, > >> > >> On Fri, 28 Sep 2018 at 03:59, Oleksandr Terentiev > >> <otere...@cisco.com <mailto:otere...@cisco.com>> wrote: > >> > >> Hi, > >> > >> I would like to discuss the following question. > >> As it was said now we have to analyze pass/fail of > >> every ptest. From my point of view there are couple > >> options. > >> > >> The first we can parse output and mark ptest as failed if > >> there is even only one failed test found. > >> > >> > >> Right I will choice this approach changes needs to be done in > >> the ptest lava script [1] to fail when any of the package tests > >> failed like [2]. > >> > >> > >> > >> The second we can analyze each test within some packet and > >> record corresponding results. > >> I see a few issues here. First of all there will be a large > >> number of test results as each ptest can run lots of tests. > >> > >> > >> Right but that need to be handled in every OE ptest script, I > >> mean if you want to fail if certain test inside a ptest fails > >> needs to be done in OE. > >> > >> > >> Another thing is that we need somehow separate test results > >> between particular packets. > >> > >> > >> Currently we use QA reports to see only the package test > >> result, now If you want to be able look at details you need to see > >> the ptest log. > >> > >> As an option we can use lava-test-set feature for that. So > >> each test within ptest will be marked as test case and > >> packet name we will see as test set. > >> > >> What do you think about that? > >> > >> > >> May be the lava-test-set is an option. > >> > >> I would go to do the 1st option and then start to > >> review/implement the idea of use lava-test-feature. > >> Regards, > >> Anibal > >> > >> [1] > >> > >> https://git.linaro.org/qa/test-definitions.git/tree/automated/linux/ptest/ptest.py > >> [2] > >> > >> http://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/runtime/cases/ptest.py#n87 > >> > >> Regards, > >> Alex > >> > >> On 23.08.18 16:10, Anibal Limon wrote: > >>> > >>> > >>> On Thu, 23 Aug 2018 at 05:54, Oleksandr Terentiev > >>> <otere...@cisco.com <mailto:otere...@cisco.com>> wrote: > >>> > >>> Thank you Anibal for the fast response > >>> > >>> > >>> On 22.08.18 19:50, Anibal Limon wrote: > >>>> > >>>> > >>>> On Wed, 22 Aug 2018 at 11:39, Oleksandr Terentiev > >>>> <otere...@cisco.com <mailto:otere...@cisco.com>> > >>>> wrote: > >>>> > >>>> Hi, > >>>> > >>>> I launched util-linux ptest using > >>>> automated/linux/ptest/ptest.yaml from > >>>> https://git.linaro.org/qa/test-definitions.git > >>>> and received the > >>>> following results: > >>>> https://pastebin.com/nj9PYQzE > >>>> > >>>> As you can see some tests failed. However, case > >>>> util-linux marked as > >>>> passed. It looks like ptest.py only analyze > >>>> return code of ptest-runner > >>>> -d <ptest_dir> <ptest_name> command. And since > >>>> ptest-runner finishes > >>>> correctly exit code is 0. Therefore all tests are > >>>> always marked as > >>>> passed, and users never know when some of the > >>>> tests fail. > >>>> > >>>> Maybe it worth to analyze each test? > >>>> > >>>> > >>>> Talking about each ptest the result comes from the > >>>> ptest script in the OE recipe [1], for convention if > >>>> the OE ptest returns 0 means pass, so > >>>> needs to be fixed in the OE ptest [2]. > >>> > >>> I’ve read https://wiki.yoctoproject.org/wiki/Ptest > >>> carefully a few times more. There are prescriptions > >>> about output format. But I didn’t find any mention > >>> about return code processing or a reference to the convention > >>> you mentioned in the answer. > >>> > >>> I looked through some OE run-ptest scripts. I suspect > >>> they don’t verify if some of their tests failed, and > >>> exit with 0 even if all their tests failed. > >>> > >>> > >>> http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/util-linux/util-linux/run-ptest > >>> > >>> http://git.openembedded.org/openembedded-core/tree/meta/recipes-support/attr/acl/run-ptest > >>> > >>> http://git.openembedded.org/openembedded-core/tree/meta/recipes-support/attr/files/run-ptest > >>> > >>> http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/dbus/dbus/run-ptest > >>> > >>> http://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/e2fsprogs/e2fsprogs/run-ptest > >>> > >>> http://git.openembedded.org/openembedded-core/tree/meta/recipes-extended/gawk/gawk/run-ptest > >>> > >>> > >>> > >>> Right, looks that the OEQA test case was update since i > >>> worked on it [1], so now it takes into account the > >>> pass/fail of every ptest. > >>> So the ptest.py needs to implement the same behavior. > >>> > >>> Regards, > >>> Anibal > >>> > >>> [1] > >>> > >>> http://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/runtime/cases/ptest.py#n80 > >>> > >>>> > >>>> Regarding the LAVA ptest.py script, I mark the run as > >>>> succeed if there is no critical error in the > >>>> ptest-runner and we have a QA-reports tool to analyse > >>>> pass/fails > >>>> in detail for every ptest executed [3]. > >>> > >>> I heard about QA-reports tool but I’ve never used > >>> it before, so maybe I missed something. > >>> From > >>> > >>> https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/1890442/suite/linux-ptest/tests/ > >>> I see all ptests passed. Still, in log > >>> > >>> https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/1890442/log > >>> I found 54 failed tests and wasn’t able to find a > >>> report which indicates those failures. > >>> > >>> Is there such a report? It would be really useful > >>> to know that some tests failed. > >>> > >>> Thanks > >>> > >>> > >>>> > >>>> [1] > >>>> > >>>> http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/util-linux/util-linux/run-ptest > >>>> [2] https://wiki.yoctoproject.org/wiki/Ptest > >>>> [3] > >>>> > >>>> https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/1890442/ > >>>> > >>>> Regards, > >>>> Anibal > >>>> > >>>> > >>>> Best regards, > >>>> Alex > >>>> > >>> > >> -- Neil Williams h...@codehelp.co.uk
pgpXrG_o6Uwg8.pgp
Description: OpenPGP digital signature
_______________________________________________ linaro-validation mailing list linaro-validation@lists.linaro.org https://lists.linaro.org/mailman/listinfo/linaro-validation
