Re: Where are results of rtems-tester test results archived?
On Wed, Sep 25, 2019 at 7:37 PM Chris Johns wrote: > > On 26/9/19 3:25 am, Gedare Bloom wrote: > > I just meant that the view of the running tests is not useful for > > comparison, > > Yes. It lets you see if something major is wrong with a test run. The very > original implementation listed the failing tests by name but this added little > value and you could not track the state of the run so it was removed. > > > it is exactly this summary (the end result) that helps. > > Great, this is the important bit. > > > If we had regular testing, > > parsing the results and producing a status matrix could help for > > understanding > > the tiers. I'm not saying I know how this would be accomplished, and it > > seems it > > would require coordination among community members who test on different > > bsps. > > A score board? This would be really nice to have. I think something that hooks > into procmail and monitors the emails posted to the bu...@rtems.org list would > work. I would to encourage anyone and everyone to post results for BSPs they > have. A score board can then be used to maintain the tiers. > > Also there is a ticket to have the tester take the console output and generate > the results. If this also posted the results email it would help make the test > results more widely available. > > The tester work so far has been unfunded or GSoC projects and I cannot see > this > changing soon however it is vital to our users, the community and a wider > audience that we have quality current published results. > Speaking of unfunded work, this idea might translate reasonably well into several GCI coding tasks. Just a thought, in case we participate. Gedare ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: Where are results of rtems-tester test results archived?
On 26/9/19 3:25 am, Gedare Bloom wrote: > I just meant that the view of the running tests is not useful for comparison, Yes. It lets you see if something major is wrong with a test run. The very original implementation listed the failing tests by name but this added little value and you could not track the state of the run so it was removed. > it is exactly this summary (the end result) that helps. Great, this is the important bit. > If we had regular testing, > parsing the results and producing a status matrix could help for understanding > the tiers. I'm not saying I know how this would be accomplished, and it seems > it > would require coordination among community members who test on different bsps. A score board? This would be really nice to have. I think something that hooks into procmail and monitors the emails posted to the bu...@rtems.org list would work. I would to encourage anyone and everyone to post results for BSPs they have. A score board can then be used to maintain the tiers. Also there is a ticket to have the tester take the console output and generate the results. If this also posted the results email it would help make the test results more widely available. The tester work so far has been unfunded or GSoC projects and I cannot see this changing soon however it is vital to our users, the community and a wider audience that we have quality current published results. Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: Where are results of rtems-tester test results archived?
On Tue, Sep 24, 2019, 7:37 PM Chris Johns wrote: > On 25/9/19 1:08 am, Gedare Bloom wrote: > > On Sat, Sep 21, 2019 at 9:41 AM Joel Sherrill wrote: > >> On Sat, Sep 21, 2019, 10:18 AM wrote: > On Sep 21, 2019, at 11:03 , Joel Sherrill wrote: > On Sat, Sep 21, 2019, 9:55 AM Peter Dufault wrote: > I’ve searched but can’t find anywhere. I’d like to see the results > of the tests on all architectures to compare to what I see on > PowerPC-beatnik. > There is a build@ mailing list and the archives are at > https://lists.rtems.org/pipermail/build/ > There should be results from at least me for psim. > You are encouraged to subscribe to the list and post results. Many > boards have no results. > >>> That doesn’t look like what I want. I’m looking for something like > the following (a small snippet of my test run in progress) to see what > failures are shared by what board support packages. > >>> > >>> [141/597] p:128 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: telnetd01.exe > >>> [142/597] p:129 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios.exe > >>> [143/597] p:129 f:7 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios01.exe > >>> [144/597] p:129 f:8 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios02.exe > >>> [145/597] p:129 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios03.exe > >>> [146/597] p:130 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios04.exe > >>> [147/597] p:131 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios05.exe > >> > >> Most are for tools builds. I have a script I run periodically which > includes tools and bsps. This is a psim results post: > https://lists.rtems.org/pipermail/build/2019-August/002973.html > >> > >> It shows the failures but still doesn't show the in process view. Does > that help at all? > >> > > Just to jump in, as I recall the "in process" view is not always > > useful, since it does not exactly tell you what has failed (when you > > run several tests in parallel, that is). Collecting the end results > > of could be nice. > > Sorry, I am not following what is being said here. The tester collects the > results for a single run as shown in ... > > https://lists.rtems.org/pipermail/build/2019-August/002970.html > > which is ... > > Summary > === > > Passed:578 > Failed: 2 > User Input: 6 > Expected Fail: 0 > Indeterminate: 0 > Benchmark: 3 > Timeout: 0 > Invalid: 0 > Wrong Version: 0 > Wrong Build: 0 > Wrong Tools: 0 > -- > Total: 589 > > Failures: > dl06.exe > dl09.exe > User Input: > dl10.exe > monitor.exe > termios.exe > top.exe > capture.exe > fileio.exe > Benchmark: > dhrystone.exe > linpack.exe > whetstone.exe > > Are you asking for something across separate runs of the tester? > > > Automating this and creating a colored status matrix > > would be a nice little project for someone. > > Where would this be run and the data held? > > Chris > I just meant that the view of the running tests is not useful for comparison, it is exactly this summary (the end result) that helps. If we had regular testing, parsing the results and producing a status matrix could help for understanding the tiers. I'm not saying I know how this would be accomplished, and it seems it would require coordination among community members who test on different bsps. ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: Where are results of rtems-tester test results archived?
On 25/9/19 1:08 am, Gedare Bloom wrote: > On Sat, Sep 21, 2019 at 9:41 AM Joel Sherrill wrote: >> On Sat, Sep 21, 2019, 10:18 AM wrote: On Sep 21, 2019, at 11:03 , Joel Sherrill wrote: On Sat, Sep 21, 2019, 9:55 AM Peter Dufault wrote: I’ve searched but can’t find anywhere. I’d like to see the results of the tests on all architectures to compare to what I see on PowerPC-beatnik. There is a build@ mailing list and the archives are at https://lists.rtems.org/pipermail/build/ There should be results from at least me for psim. You are encouraged to subscribe to the list and post results. Many boards have no results. >>> That doesn’t look like what I want. I’m looking for something like the >>> following (a small snippet of my test run in progress) to see what failures >>> are shared by what board support packages. >>> >>> [141/597] p:128 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | >>> powerpc/beatnik: telnetd01.exe >>> [142/597] p:129 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | >>> powerpc/beatnik: termios.exe >>> [143/597] p:129 f:7 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >>> powerpc/beatnik: termios01.exe >>> [144/597] p:129 f:8 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >>> powerpc/beatnik: termios02.exe >>> [145/597] p:129 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >>> powerpc/beatnik: termios03.exe >>> [146/597] p:130 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >>> powerpc/beatnik: termios04.exe >>> [147/597] p:131 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >>> powerpc/beatnik: termios05.exe >> >> Most are for tools builds. I have a script I run periodically which includes >> tools and bsps. This is a psim results post: >> https://lists.rtems.org/pipermail/build/2019-August/002973.html >> >> It shows the failures but still doesn't show the in process view. Does that >> help at all? >> > Just to jump in, as I recall the "in process" view is not always > useful, since it does not exactly tell you what has failed (when you > run several tests in parallel, that is). Collecting the end results > of could be nice. Sorry, I am not following what is being said here. The tester collects the results for a single run as shown in ... https://lists.rtems.org/pipermail/build/2019-August/002970.html which is ... Summary === Passed:578 Failed: 2 User Input: 6 Expected Fail: 0 Indeterminate: 0 Benchmark: 3 Timeout: 0 Invalid: 0 Wrong Version: 0 Wrong Build: 0 Wrong Tools: 0 -- Total: 589 Failures: dl06.exe dl09.exe User Input: dl10.exe monitor.exe termios.exe top.exe capture.exe fileio.exe Benchmark: dhrystone.exe linpack.exe whetstone.exe Are you asking for something across separate runs of the tester? > Automating this and creating a colored status matrix > would be a nice little project for someone. Where would this be run and the data held? Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: Where are results of rtems-tester test results archived?
On Sat, Sep 21, 2019 at 9:41 AM Joel Sherrill wrote: > > > > On Sat, Sep 21, 2019, 10:18 AM wrote: >> >> >> > On Sep 21, 2019, at 11:03 , Joel Sherrill wrote: >> > >> > >> > >> > On Sat, Sep 21, 2019, 9:55 AM Peter Dufault wrote: >> > I’ve searched but can’t find anywhere. I’d like to see the results of the >> > tests on all architectures to compare to what I see on PowerPC-beatnik. >> > >> > There is a build@ mailing list and the archives are at >> > https://lists.rtems.org/pipermail/build/ >> > >> > There should be results from at least me for psim. >> > >> > You are encouraged to subscribe to the list and post results. Many boards >> > have no results. >> > >> > >> >> That doesn’t look like what I want. I’m looking for something like the >> following (a small snippet of my test run in progress) to see what failures >> are shared by what board support packages. >> >> [141/597] p:128 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | >> powerpc/beatnik: telnetd01.exe >> [142/597] p:129 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | >> powerpc/beatnik: termios.exe >> [143/597] p:129 f:7 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >> powerpc/beatnik: termios01.exe >> [144/597] p:129 f:8 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >> powerpc/beatnik: termios02.exe >> [145/597] p:129 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >> powerpc/beatnik: termios03.exe >> [146/597] p:130 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >> powerpc/beatnik: termios04.exe >> [147/597] p:131 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >> powerpc/beatnik: termios05.exe > > > Most are for tools builds. I have a script I run periodically which includes > tools and bsps. This is a psim results post: > https://lists.rtems.org/pipermail/build/2019-August/002973.html > > It shows the failures but still doesn't show the in process view. Does that > help at all? > Just to jump in, as I recall the "in process" view is not always useful, since it does not exactly tell you what has failed (when you run several tests in parallel, that is). Collecting the end results of could be nice. Automating this and creating a colored status matrix would be a nice little project for someone. > FWIW I haven't investigated the 15 failures on psim. > > --joel >> >> >> >> >> >> >> Peter >> - >> Peter Dufault >> HD Associates, Inc. Software and System Engineering >> >> This email is delivered through the public internet using protocols subject >> to interception and tampering. >> > ___ > devel mailing list > devel@rtems.org > http://lists.rtems.org/mailman/listinfo/devel ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: Where are results of rtems-tester test results archived?
On Sat, Sep 21, 2019, 6:21 PM Chris Johns wrote: > > > On 22/9/19 7:58 am, dufa...@hda.com wrote: > > > > > >> On Sep 21, 2019, at 17:49 , wrote: > >> > >> > >> > >>> On Sep 21, 2019, at 16:41 , Chris Johns wrote: > >>> > >>> On 22/9/19 1:18 am, dufa...@hda.com wrote: > > On Sep 21, 2019, at 11:03 , Joel Sherrill wrote: > > On Sat, Sep 21, 2019, 9:55 AM Peter Dufault wrote: > > I’ve searched but can’t find anywhere. I’d like to see the results > of the tests on all architectures to compare to what I see on > PowerPC-beatnik. > > > > There is a build@ mailing list and the archives are at > https://lists.rtems.org/pipermail/build/ > > > > There should be results from at least me for psim. > > > > You are encouraged to subscribe to the list and post results. Many > boards have no results. > > > That doesn’t look like what I want. I’m looking for something like > the following (a small snippet of my test run in progress) to see what > failures are shared by what board support packages. > > [141/597] p:128 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: telnetd01.exe > [142/597] p:129 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios.exe > [143/597] p:129 f:7 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios01.exe > [144/597] p:129 f:8 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios02.exe > [145/597] p:129 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios03.exe > [146/597] p:130 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios04.exe > [147/597] p:131 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios05.exe > >>> > >>> The --log option should place all the test results in a log. Is that > what you want? > >>> > >>> There are other options that let you log the console data as well. > This is > >>> disabled by default to make the log more compact. > >>> > >> > >> I want a collection of results for all tested BSPs so that I can > compare what I’m seeing with “beatnik” with others. > >> > >> I thought there was a collection of systems that went through automated > tests, an RTEMS testing lab of sorts. Am I confused? I was ready to try > to get an MVME5500 and whatever else was needed to the lab. > >> > > Beatnik finished: > > [595/597] p:563 f:19 u:6 e:0 I:0 B:3 t:1 i:2 W:0 | > powerpc/beatnik: tmonetoone.exe > > [596/597] p:564 f:19 u:6 e:0 I:0 B:3 t:1 i:2 W:0 | > powerpc/beatnik: tmoverhd.exe > > [597/597] p:565 f:19 u:6 e:0 I:0 B:3 t:1 i:2 W:0 | > powerpc/beatnik: tmtimer01.exe > > Passed:565 > > Failed: 20 > > User Input: 6 > > Expected Fail: 0 > > Indeterminate: 0 > > Benchmark: 3 > > Timeout: 1 > > Invalid: 2 > > Wrong Version: 0 > > Wrong Build: 0 > > Wrong Tools: 0 > > -- > > Total: 597 > > Average test time: 0:00:20.652608 > > Testing time : 3:25:29.607257 > > The results are relative to the BSP. It would be nice to say we can expect > all > tests to pass on all BSPs however the likelihood of that happening is low. > To > manage this so a user of a specific BSP can see if there are regressions > tests > can be tagged with the expected result for a specific BSP. If a test is > known to > fail the test can be set to expected-fail for a BSP and you should see no > failures, invalids or timeouts. I raised a ticket for this .. > > https://devel.rtems.org/ticket/2962 > > ... and while I can do some I cannot do them all. I am wondering if the > lack of > feedback is related to the process needed to tag a specific test for a > BSP. I am > considering a tool to make the process simpler. It also takes time to > examine > the results and to determine if expected-fail is suitable. > Some of the failures I see on mips are very clearly lack of support for something on that architecture. This will be common if we get results on anything which doesn't have TLS, dynamic loading, or debugging support. Addressing this type of issue at a level higher than individual bsps would be helpful. > > The test controls are documented here > > > https://docs.rtems.org/branches/master/user/testing/tests.html#test-controls > > Chris > ___ > devel mailing list > devel@rtems.org > http://lists.rtems.org/mailman/listinfo/devel ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: Where are results of rtems-tester test results archived?
On 22/9/19 7:58 am, dufa...@hda.com wrote: > > >> On Sep 21, 2019, at 17:49 , wrote: >> >> >> >>> On Sep 21, 2019, at 16:41 , Chris Johns wrote: >>> >>> On 22/9/19 1:18 am, dufa...@hda.com wrote: > On Sep 21, 2019, at 11:03 , Joel Sherrill wrote: > On Sat, Sep 21, 2019, 9:55 AM Peter Dufault wrote: > I’ve searched but can’t find anywhere. I’d like to see the results of > the tests on all architectures to compare to what I see on > PowerPC-beatnik. > > There is a build@ mailing list and the archives are at > https://lists.rtems.org/pipermail/build/ > > There should be results from at least me for psim. > > You are encouraged to subscribe to the list and post results. Many boards > have no results. > That doesn’t look like what I want. I’m looking for something like the following (a small snippet of my test run in progress) to see what failures are shared by what board support packages. [141/597] p:128 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | powerpc/beatnik: telnetd01.exe [142/597] p:129 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | powerpc/beatnik: termios.exe [143/597] p:129 f:7 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | powerpc/beatnik: termios01.exe [144/597] p:129 f:8 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | powerpc/beatnik: termios02.exe [145/597] p:129 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | powerpc/beatnik: termios03.exe [146/597] p:130 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | powerpc/beatnik: termios04.exe [147/597] p:131 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | powerpc/beatnik: termios05.exe >>> >>> The --log option should place all the test results in a log. Is that what >>> you want? >>> >>> There are other options that let you log the console data as well. This is >>> disabled by default to make the log more compact. >>> >> >> I want a collection of results for all tested BSPs so that I can compare >> what I’m seeing with “beatnik” with others. >> >> I thought there was a collection of systems that went through automated >> tests, an RTEMS testing lab of sorts. Am I confused? I was ready to try to >> get an MVME5500 and whatever else was needed to the lab. >> > Beatnik finished: > [595/597] p:563 f:19 u:6 e:0 I:0 B:3 t:1 i:2 W:0 | > powerpc/beatnik: tmonetoone.exe > [596/597] p:564 f:19 u:6 e:0 I:0 B:3 t:1 i:2 W:0 | > powerpc/beatnik: tmoverhd.exe > [597/597] p:565 f:19 u:6 e:0 I:0 B:3 t:1 i:2 W:0 | > powerpc/beatnik: tmtimer01.exe > Passed:565 > Failed: 20 > User Input: 6 > Expected Fail: 0 > Indeterminate: 0 > Benchmark: 3 > Timeout: 1 > Invalid: 2 > Wrong Version: 0 > Wrong Build: 0 > Wrong Tools: 0 > -- > Total: 597 > Average test time: 0:00:20.652608 > Testing time : 3:25:29.607257 The results are relative to the BSP. It would be nice to say we can expect all tests to pass on all BSPs however the likelihood of that happening is low. To manage this so a user of a specific BSP can see if there are regressions tests can be tagged with the expected result for a specific BSP. If a test is known to fail the test can be set to expected-fail for a BSP and you should see no failures, invalids or timeouts. I raised a ticket for this .. https://devel.rtems.org/ticket/2962 ... and while I can do some I cannot do them all. I am wondering if the lack of feedback is related to the process needed to tag a specific test for a BSP. I am considering a tool to make the process simpler. It also takes time to examine the results and to determine if expected-fail is suitable. The test controls are documented here https://docs.rtems.org/branches/master/user/testing/tests.html#test-controls Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: Where are results of rtems-tester test results archived?
> On Sep 21, 2019, at 17:49 , wrote: > > > >> On Sep 21, 2019, at 16:41 , Chris Johns wrote: >> >> On 22/9/19 1:18 am, dufa...@hda.com wrote: On Sep 21, 2019, at 11:03 , Joel Sherrill wrote: On Sat, Sep 21, 2019, 9:55 AM Peter Dufault wrote: I’ve searched but can’t find anywhere. I’d like to see the results of the tests on all architectures to compare to what I see on PowerPC-beatnik. There is a build@ mailing list and the archives are at https://lists.rtems.org/pipermail/build/ There should be results from at least me for psim. You are encouraged to subscribe to the list and post results. Many boards have no results. >>> That doesn’t look like what I want. I’m looking for something like the >>> following (a small snippet of my test run in progress) to see what failures >>> are shared by what board support packages. >>> >>> [141/597] p:128 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | >>> powerpc/beatnik: telnetd01.exe >>> [142/597] p:129 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | >>> powerpc/beatnik: termios.exe >>> [143/597] p:129 f:7 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >>> powerpc/beatnik: termios01.exe >>> [144/597] p:129 f:8 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >>> powerpc/beatnik: termios02.exe >>> [145/597] p:129 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >>> powerpc/beatnik: termios03.exe >>> [146/597] p:130 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >>> powerpc/beatnik: termios04.exe >>> [147/597] p:131 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >>> powerpc/beatnik: termios05.exe >> >> The --log option should place all the test results in a log. Is that what >> you want? >> >> There are other options that let you log the console data as well. This is >> disabled by default to make the log more compact. >> > > I want a collection of results for all tested BSPs so that I can compare what > I’m seeing with “beatnik” with others. > > I thought there was a collection of systems that went through automated > tests, an RTEMS testing lab of sorts. Am I confused? I was ready to try to > get an MVME5500 and whatever else was needed to the lab. > Beatnik finished: [595/597] p:563 f:19 u:6 e:0 I:0 B:3 t:1 i:2 W:0 | powerpc/beatnik: tmonetoone.exe [596/597] p:564 f:19 u:6 e:0 I:0 B:3 t:1 i:2 W:0 | powerpc/beatnik: tmoverhd.exe [597/597] p:565 f:19 u:6 e:0 I:0 B:3 t:1 i:2 W:0 | powerpc/beatnik: tmtimer01.exe Passed:565 Failed: 20 User Input: 6 Expected Fail: 0 Indeterminate: 0 Benchmark: 3 Timeout: 1 Invalid: 2 Wrong Version: 0 Wrong Build: 0 Wrong Tools: 0 -- Total: 597 Average test time: 0:00:20.652608 Testing time : 3:25:29.607257 Traceback (most recent call last): File "../rtems-tools/tester/rt/cmd-test.py", line 42, in test.run(sys.argv[1:], command_path = base) File "/home/dufault/development/rtems/rtems-tools/tester/rt/test.py", line 455, in run for report_format in report_formats: TypeError: 'NoneType' object is not iterable Peter - Peter Dufault HD Associates, Inc. Software and System Engineering This email is delivered through the public internet using protocols subject to interception and tampering. ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: Where are results of rtems-tester test results archived?
> On Sep 21, 2019, at 16:41 , Chris Johns wrote: > > On 22/9/19 1:18 am, dufa...@hda.com wrote: >>> On Sep 21, 2019, at 11:03 , Joel Sherrill wrote: >>> On Sat, Sep 21, 2019, 9:55 AM Peter Dufault wrote: >>> I’ve searched but can’t find anywhere. I’d like to see the results of the >>> tests on all architectures to compare to what I see on PowerPC-beatnik. >>> >>> There is a build@ mailing list and the archives are at >>> https://lists.rtems.org/pipermail/build/ >>> >>> There should be results from at least me for psim. >>> >>> You are encouraged to subscribe to the list and post results. Many boards >>> have no results. >>> >> That doesn’t look like what I want. I’m looking for something like the >> following (a small snippet of my test run in progress) to see what failures >> are shared by what board support packages. >> >> [141/597] p:128 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | >> powerpc/beatnik: telnetd01.exe >> [142/597] p:129 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | >> powerpc/beatnik: termios.exe >> [143/597] p:129 f:7 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >> powerpc/beatnik: termios01.exe >> [144/597] p:129 f:8 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >> powerpc/beatnik: termios02.exe >> [145/597] p:129 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >> powerpc/beatnik: termios03.exe >> [146/597] p:130 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >> powerpc/beatnik: termios04.exe >> [147/597] p:131 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | >> powerpc/beatnik: termios05.exe > > The --log option should place all the test results in a log. Is that what you > want? > > There are other options that let you log the console data as well. This is > disabled by default to make the log more compact. > I want a collection of results for all tested BSPs so that I can compare what I’m seeing with “beatnik” with others. I thought there was a collection of systems that went through automated tests, an RTEMS testing lab of sorts. Am I confused? I was ready to try to get an MVME5500 and whatever else was needed to the lab. Peter - Peter Dufault HD Associates, Inc. Software and System Engineering This email is delivered through the public internet using protocols subject to interception and tampering. ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: Where are results of rtems-tester test results archived?
On 22/9/19 1:18 am, dufa...@hda.com wrote: >> On Sep 21, 2019, at 11:03 , Joel Sherrill wrote: >> On Sat, Sep 21, 2019, 9:55 AM Peter Dufault wrote: >> I’ve searched but can’t find anywhere. I’d like to see the results of the >> tests on all architectures to compare to what I see on PowerPC-beatnik. >> >> There is a build@ mailing list and the archives are at >> https://lists.rtems.org/pipermail/build/ >> >> There should be results from at least me for psim. >> >> You are encouraged to subscribe to the list and post results. Many boards >> have no results. >> > That doesn’t look like what I want. I’m looking for something like the > following (a small snippet of my test run in progress) to see what failures > are shared by what board support packages. > > [141/597] p:128 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: telnetd01.exe > [142/597] p:129 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios.exe > [143/597] p:129 f:7 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios01.exe > [144/597] p:129 f:8 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios02.exe > [145/597] p:129 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios03.exe > [146/597] p:130 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios04.exe > [147/597] p:131 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios05.exe The --log option should place all the test results in a log. Is that what you want? There are other options that let you log the console data as well. This is disabled by default to make the log more compact. Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: Where are results of rtems-tester test results archived?
On Sat, Sep 21, 2019, 10:18 AM wrote: > > > On Sep 21, 2019, at 11:03 , Joel Sherrill wrote: > > > > > > > > On Sat, Sep 21, 2019, 9:55 AM Peter Dufault wrote: > > I’ve searched but can’t find anywhere. I’d like to see the results of > the tests on all architectures to compare to what I see on PowerPC-beatnik. > > > > There is a build@ mailing list and the archives are at > https://lists.rtems.org/pipermail/build/ > > > > There should be results from at least me for psim. > > > > You are encouraged to subscribe to the list and post results. Many > boards have no results. > > > > > > That doesn’t look like what I want. I’m looking for something like the > following (a small snippet of my test run in progress) to see what failures > are shared by what board support packages. > > [141/597] p:128 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: telnetd01.exe > [142/597] p:129 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios.exe > [143/597] p:129 f:7 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios01.exe > [144/597] p:129 f:8 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios02.exe > [145/597] p:129 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios03.exe > [146/597] p:130 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios04.exe > [147/597] p:131 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | > powerpc/beatnik: termios05.exe > Most are for tools builds. I have a script I run periodically which includes tools and bsps. This is a psim results post: https://lists.rtems.org/pipermail/build/2019-August/002973.html It shows the failures but still doesn't show the in process view. Does that help at all? FWIW I haven't investigated the 15 failures on psim. --joel > > > > > > Peter > - > Peter Dufault > HD Associates, Inc. Software and System Engineering > > This email is delivered through the public internet using protocols > subject to interception and tampering. > > ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: Where are results of rtems-tester test results archived?
> On Sep 21, 2019, at 11:03 , Joel Sherrill wrote: > > > > On Sat, Sep 21, 2019, 9:55 AM Peter Dufault wrote: > I’ve searched but can’t find anywhere. I’d like to see the results of the > tests on all architectures to compare to what I see on PowerPC-beatnik. > > There is a build@ mailing list and the archives are at > https://lists.rtems.org/pipermail/build/ > > There should be results from at least me for psim. > > You are encouraged to subscribe to the list and post results. Many boards > have no results. > > That doesn’t look like what I want. I’m looking for something like the following (a small snippet of my test run in progress) to see what failures are shared by what board support packages. [141/597] p:128 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | powerpc/beatnik: telnetd01.exe [142/597] p:129 f:7 u:2 e:0 I:0 B:3 t:0 i:0 W:0 | powerpc/beatnik: termios.exe [143/597] p:129 f:7 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | powerpc/beatnik: termios01.exe [144/597] p:129 f:8 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | powerpc/beatnik: termios02.exe [145/597] p:129 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | powerpc/beatnik: termios03.exe [146/597] p:130 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | powerpc/beatnik: termios04.exe [147/597] p:131 f:9 u:3 e:0 I:0 B:3 t:0 i:0 W:0 | powerpc/beatnik: termios05.exe Peter - Peter Dufault HD Associates, Inc. Software and System Engineering This email is delivered through the public internet using protocols subject to interception and tampering. ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: Where are results of rtems-tester test results archived?
On Sat, Sep 21, 2019, 9:55 AM Peter Dufault wrote: > I’ve searched but can’t find anywhere. I’d like to see the results of the > tests on all architectures to compare to what I see on PowerPC-beatnik. > There is a build@ mailing list and the archives are at https://lists.rtems.org/pipermail/build/ There should be results from at least me for psim. You are encouraged to subscribe to the list and post results. Many boards have no results. > Peter > - > Peter Dufault > HD Associates, Inc. Software and System Engineering > > This email is delivered through the public internet using protocols > subject to interception and tampering. > > ___ > devel mailing list > devel@rtems.org > http://lists.rtems.org/mailman/listinfo/devel ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Where are results of rtems-tester test results archived?
I’ve searched but can’t find anywhere. I’d like to see the results of the tests on all architectures to compare to what I see on PowerPC-beatnik. Peter - Peter Dufault HD Associates, Inc. Software and System Engineering This email is delivered through the public internet using protocols subject to interception and tampering. ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
