Re: [tools] tester: Normalize JSON and YAML reports
Hello, I sent a v2 of the patch: https://lists.rtems.org/pipermail/devel/2022-July/072679.html On 01/07/2022 01:12, Chris Johns wrote: From my point of view the JSON/YAML reports should not satisfy every possible consumer directly. How do you determine the list you are satisfying? I can see this argument working in two directions. The report is satisfactory when it includes all information available to the tester and not more. A machine readable report should be without redundancy. If there a technical issue with providing this data? For this grouping we have to assume a certain layout of the test executables. From my point of view the tester should be generalized to support running all sorts of tests and not just tests of the RTEMS testsuites. Does the output provide format metadata to determine the version of the output? I added a 'report-version' attribute in v2. -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [tools] tester: Normalize JSON and YAML reports
On 30/6/2022 4:34 pm, Sebastian Huber wrote:
> On 30/06/2022 07:58, Sebastian Huber wrote:
>> On 29/06/2022 17:54, Kinsey Moore wrote:
>>> On 6/29/2022 04:34, Sebastian Huber wrote:
On 29/06/2022 11:20, Chris Johns wrote:
>
>> On 29 Jun 2022, at 4:42 pm, Sebastian Huber
>> wrote:
>>
>> On 29/06/2022 08:40, Sebastian Huber wrote:
>>> Report the same data in JSON and YAML reports. Do not report redundant
>>> information.
>>> Update 4671.
>>
>> This patch changes the JSON reports. Are there already consumers for the
>> JSON reports so that we have to be backward compatible?
>
> Could compatibility be added back in if this proves to be an issue?
>
> I am wondering if that could be considered if comparability becomes an
> issue.
The JSON report was added by:
commit 5fcaf843bdb491e2da14dc1af61806ac6d20bae5
Author: Kinsey Moore
Date: Wed Aug 21 16:34:12 2019 +
Add JSON log generation
Add log formatter hooks and JSON log formatter to the test
infrastructure
for consumption by automated processes or report generators.
I am not sure if these automated processes or report generators already
exist.
The existing attribute names are quite inconsistent, for example "Command
Line", "passed_count", "wrong-version_count". I would use lower case only
with "-" as a separator. The JSON report should contain all information of
a
test run.
The new report looks like this:
{
"command-line": [
"/opt/rtems/6/bin/rtems-test",
"--rtems-bsp=xilinx_zynq_a9_qemu",
"--report-format=json",
"--report-path=report",
"build/arm/xilinx_zynq_a9_qemu/testsuites/samples/"
],
"end-time": "2022-06-28T14:08:47.595131",
"host": "Linux-5.3.18-150300.59.76-default-x86_64-with-glibc2.3.4
(Linux
lupus 5.3.18-150300.59.76-default #1 SMP Thu Jun 16 04:23:47 UTC 2022
(2cc2ade) x86_64 x86_64)",
"python": "3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]",
"reports": [
{
"arch": "arm",
"bsp": "xilinx_zynq_a9_qemu",
"command-line": "qemu-system-arm -no-reboot -nographic -net
none
-serial null -serial mon:stdio -M xilinx-zynq-a9 -m 256M -kernel
build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
"end-time": "2022-06-28T12:08:48.161691+00:00",
"executable":
"build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
"executable-sha512":
"413bc8a506053c95a0e1d93942cce9fd0d653f7b3cde42623085ce147aea324f4baeff7ce0102008322fb1341bc2b38defd3925a64144e47e13cc5f32185bf78",
"output": [
"qemu-system-arm: warning: nic cadence_gem.0 has no peer",
"qemu-system-arm: warning: nic cadence_gem.1 has no peer",
"",
"",
"*** BEGIN OF TEST CONSTRUCTOR/DESTRUCTOR ***",
"*** TEST VERSION:
6.0.0.3302b72754df5f37214e86dd68522189857772c7",
"*** TEST STATE: EXPECTED_PASS",
"*** TEST BUILD: RTEMS_DEBUG RTEMS_POSIX_API",
"*** TEST TOOLS: 12.1.1 20220622 (RTEMS 6, RSB
bf3f66c29a046ec1c5029e4f919b525947df6245, Newlib e7ffbdb)",
"GLOBAL: Hey I'm in base class constructor number 1 for
0x214474.",
"GLOBAL: Hey I'm in base class constructor number 2 for
0x214480.",
"GLOBAL: Hey I'm in derived class constructor number 3 for
0x214480.",
"LOCAL: Hey I'm in base class constructor number 4 for
0x228864.",
...
"WRoZqtoO3A8BDJkUSmRGNOBaaHW1UwAAoQGQAQwAAABck8Bx+K/znDWWTEcA",
"AKEB2P///wEMVP6TRqQZY+4+srvAAAChAfD///8BDMXOxS0Rhzqx6Old2wAA",
"oQH4AQwAAADJyx1ZCnZ4l+m/XkMAAKEB0P///wEMN3+9YAwMW8gTHIoPAACh",
"Adj///8BDBtxsj3zXZF/+UqzfAAAoQHwAQwwMDMeE7mphT6yu8AAAKEB",
"8P///wEMr2rLCcwzVnf5SrN8AAChAfD///8A",
"*** END OF GCOV INFO BASE64 ***",
""
],
"result": "passed",
"start-time": "2022-06-28T12:08:47.721822+00:00"
}
],
"start-time": "2022-06-28T14:08:47.595131"
}
>>> There are no publicly available report generators for the existing format,
>>> but I do have one I use to generate STR documents for delivery using
>>> mustache
>>> templates and further processing. The existing format was designed to be
>>> easily consumable by that and other simple templating mechanisms by
>>> providing
>>> structure (test subsets) and p
Re: [tools] tester: Normalize JSON and YAML reports
On 6/30/2022 01:34, Sebastian Huber wrote:
On 30/06/2022 07:58, Sebastian Huber wrote:
On 29/06/2022 17:54, Kinsey Moore wrote:
On 6/29/2022 04:34, Sebastian Huber wrote:
On 29/06/2022 11:20, Chris Johns wrote:
On 29 Jun 2022, at 4:42 pm, Sebastian Huber
wrote:
On 29/06/2022 08:40, Sebastian Huber wrote:
Report the same data in JSON and YAML reports. Do not report
redundant
information.
Update 4671.
This patch changes the JSON reports. Are there already consumers
for the JSON reports so that we have to be backward compatible?
Could compatibility be added back in if this proves to be an issue?
I am wondering if that could be considered if comparability
becomes an issue.
The JSON report was added by:
commit 5fcaf843bdb491e2da14dc1af61806ac6d20bae5
Author: Kinsey Moore
Date: Wed Aug 21 16:34:12 2019 +
Add JSON log generation
Add log formatter hooks and JSON log formatter to the test
infrastructure
for consumption by automated processes or report generators.
I am not sure if these automated processes or report generators
already exist.
The existing attribute names are quite inconsistent, for example
"Command Line", "passed_count", "wrong-version_count". I would use
lower case only with "-" as a separator. The JSON report should
contain all information of a test run.
The new report looks like this:
{
"command-line": [
"/opt/rtems/6/bin/rtems-test",
"--rtems-bsp=xilinx_zynq_a9_qemu",
"--report-format=json",
"--report-path=report",
"build/arm/xilinx_zynq_a9_qemu/testsuites/samples/"
],
"end-time": "2022-06-28T14:08:47.595131",
"host":
"Linux-5.3.18-150300.59.76-default-x86_64-with-glibc2.3.4 (Linux
lupus 5.3.18-150300.59.76-default #1 SMP Thu Jun 16 04:23:47 UTC
2022 (2cc2ade) x86_64 x86_64)",
"python": "3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]",
"reports": [
{
"arch": "arm",
"bsp": "xilinx_zynq_a9_qemu",
"command-line": "qemu-system-arm -no-reboot -nographic
-net none -serial null -serial mon:stdio -M xilinx-zynq-a9 -m 256M
-kernel build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
"end-time": "2022-06-28T12:08:48.161691+00:00",
"executable":
"build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
"executable-sha512":
"413bc8a506053c95a0e1d93942cce9fd0d653f7b3cde42623085ce147aea324f4baeff7ce0102008322fb1341bc2b38defd3925a64144e47e13cc5f32185bf78",
"output": [
"qemu-system-arm: warning: nic cadence_gem.0 has no
peer",
"qemu-system-arm: warning: nic cadence_gem.1 has no
peer",
"",
"",
"*** BEGIN OF TEST CONSTRUCTOR/DESTRUCTOR ***",
"*** TEST VERSION:
6.0.0.3302b72754df5f37214e86dd68522189857772c7",
"*** TEST STATE: EXPECTED_PASS",
"*** TEST BUILD: RTEMS_DEBUG RTEMS_POSIX_API",
"*** TEST TOOLS: 12.1.1 20220622 (RTEMS 6, RSB
bf3f66c29a046ec1c5029e4f919b525947df6245, Newlib e7ffbdb)",
"GLOBAL: Hey I'm in base class constructor number 1
for 0x214474.",
"GLOBAL: Hey I'm in base class constructor number 2
for 0x214480.",
"GLOBAL: Hey I'm in derived class constructor
number 3 for 0x214480.",
"LOCAL: Hey I'm in base class constructor number 4
for 0x228864.",
...
"WRoZqtoO3A8BDJkUSmRGNOBaaHW1UwAAoQGQAQwAAABck8Bx+K/znDWWTEcA",
"AKEB2P///wEMVP6TRqQZY+4+srvAAAChAfD///8BDMXOxS0Rhzqx6Old2wAA",
"oQH4AQwAAADJyx1ZCnZ4l+m/XkMAAKEB0P///wEMN3+9YAwMW8gTHIoPAACh",
"Adj///8BDBtxsj3zXZF/+UqzfAAAoQHwAQwwMDMeE7mphT6yu8AAAKEB",
"8P///wEMr2rLCcwzVnf5SrN8AAChAfD///8A",
"*** END OF GCOV INFO BASE64 ***",
""
],
"result": "passed",
"start-time": "2022-06-28T12:08:47.721822+00:00"
}
],
"start-time": "2022-06-28T14:08:47.595131"
}
There are no publicly available report generators for the existing
format, but I do have one I use to generate STR documents for
delivery using mustache templates and further processing. The
existing format was designed to be easily consumable by that and
other simple templating mechanisms by providing structure (test
subsets) and precalculating values that would otherwise be implicit
in the data. Changes to the names of various fields should be easy
to accommodate, but a more significant restructure is going to
present problems continuing with any simple templating mechanism and
would instead require a more comprehensive document generator for
all cases.
I'm not necessarily opposed to this change as it significantly
simplifies the report generation code, but it will definitely
require some rework in my processes. I can't speak for any users of
the YAML repo
Re: [tools] tester: Normalize JSON and YAML reports
On 30/06/2022 07:58, Sebastian Huber wrote:
On 29/06/2022 17:54, Kinsey Moore wrote:
On 6/29/2022 04:34, Sebastian Huber wrote:
On 29/06/2022 11:20, Chris Johns wrote:
On 29 Jun 2022, at 4:42 pm, Sebastian Huber
wrote:
On 29/06/2022 08:40, Sebastian Huber wrote:
Report the same data in JSON and YAML reports. Do not report
redundant
information.
Update 4671.
This patch changes the JSON reports. Are there already consumers
for the JSON reports so that we have to be backward compatible?
Could compatibility be added back in if this proves to be an issue?
I am wondering if that could be considered if comparability becomes
an issue.
The JSON report was added by:
commit 5fcaf843bdb491e2da14dc1af61806ac6d20bae5
Author: Kinsey Moore
Date: Wed Aug 21 16:34:12 2019 +
Add JSON log generation
Add log formatter hooks and JSON log formatter to the test
infrastructure
for consumption by automated processes or report generators.
I am not sure if these automated processes or report generators
already exist.
The existing attribute names are quite inconsistent, for example
"Command Line", "passed_count", "wrong-version_count". I would use
lower case only with "-" as a separator. The JSON report should
contain all information of a test run.
The new report looks like this:
{
"command-line": [
"/opt/rtems/6/bin/rtems-test",
"--rtems-bsp=xilinx_zynq_a9_qemu",
"--report-format=json",
"--report-path=report",
"build/arm/xilinx_zynq_a9_qemu/testsuites/samples/"
],
"end-time": "2022-06-28T14:08:47.595131",
"host": "Linux-5.3.18-150300.59.76-default-x86_64-with-glibc2.3.4
(Linux lupus 5.3.18-150300.59.76-default #1 SMP Thu Jun 16 04:23:47
UTC 2022 (2cc2ade) x86_64 x86_64)",
"python": "3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]",
"reports": [
{
"arch": "arm",
"bsp": "xilinx_zynq_a9_qemu",
"command-line": "qemu-system-arm -no-reboot -nographic
-net none -serial null -serial mon:stdio -M xilinx-zynq-a9 -m 256M
-kernel build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
"end-time": "2022-06-28T12:08:48.161691+00:00",
"executable":
"build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
"executable-sha512":
"413bc8a506053c95a0e1d93942cce9fd0d653f7b3cde42623085ce147aea324f4baeff7ce0102008322fb1341bc2b38defd3925a64144e47e13cc5f32185bf78",
"output": [
"qemu-system-arm: warning: nic cadence_gem.0 has no
peer",
"qemu-system-arm: warning: nic cadence_gem.1 has no
peer",
"",
"",
"*** BEGIN OF TEST CONSTRUCTOR/DESTRUCTOR ***",
"*** TEST VERSION:
6.0.0.3302b72754df5f37214e86dd68522189857772c7",
"*** TEST STATE: EXPECTED_PASS",
"*** TEST BUILD: RTEMS_DEBUG RTEMS_POSIX_API",
"*** TEST TOOLS: 12.1.1 20220622 (RTEMS 6, RSB
bf3f66c29a046ec1c5029e4f919b525947df6245, Newlib e7ffbdb)",
"GLOBAL: Hey I'm in base class constructor number 1
for 0x214474.",
"GLOBAL: Hey I'm in base class constructor number 2
for 0x214480.",
"GLOBAL: Hey I'm in derived class constructor number
3 for 0x214480.",
"LOCAL: Hey I'm in base class constructor number 4
for 0x228864.",
...
"WRoZqtoO3A8BDJkUSmRGNOBaaHW1UwAAoQGQAQwAAABck8Bx+K/znDWWTEcA",
"AKEB2P///wEMVP6TRqQZY+4+srvAAAChAfD///8BDMXOxS0Rhzqx6Old2wAA",
"oQH4AQwAAADJyx1ZCnZ4l+m/XkMAAKEB0P///wEMN3+9YAwMW8gTHIoPAACh",
"Adj///8BDBtxsj3zXZF/+UqzfAAAoQHwAQwwMDMeE7mphT6yu8AAAKEB",
"8P///wEMr2rLCcwzVnf5SrN8AAChAfD///8A",
"*** END OF GCOV INFO BASE64 ***",
""
],
"result": "passed",
"start-time": "2022-06-28T12:08:47.721822+00:00"
}
],
"start-time": "2022-06-28T14:08:47.595131"
}
There are no publicly available report generators for the existing
format, but I do have one I use to generate STR documents for delivery
using mustache templates and further processing. The existing format
was designed to be easily consumable by that and other simple
templating mechanisms by providing structure (test subsets) and
precalculating values that would otherwise be implicit in the data.
Changes to the names of various fields should be easy to accommodate,
but a more significant restructure is going to present problems
continuing with any simple templating mechanism and would instead
require a more comprehensive document generator for all cases.
I'm not necessarily opposed to this change as it significantly
simplifies the report generation code, but it will definitely require
some rework in my processes. I can't speak for any users of the YAML
report format, but it was committed over a year an
Re: [tools] tester: Normalize JSON and YAML reports
On 29/06/2022 17:54, Kinsey Moore wrote:
On 6/29/2022 04:34, Sebastian Huber wrote:
On 29/06/2022 11:20, Chris Johns wrote:
On 29 Jun 2022, at 4:42 pm, Sebastian Huber
wrote:
On 29/06/2022 08:40, Sebastian Huber wrote:
Report the same data in JSON and YAML reports. Do not report
redundant
information.
Update 4671.
This patch changes the JSON reports. Are there already consumers for
the JSON reports so that we have to be backward compatible?
Could compatibility be added back in if this proves to be an issue?
I am wondering if that could be considered if comparability becomes
an issue.
The JSON report was added by:
commit 5fcaf843bdb491e2da14dc1af61806ac6d20bae5
Author: Kinsey Moore
Date: Wed Aug 21 16:34:12 2019 +
Add JSON log generation
Add log formatter hooks and JSON log formatter to the test
infrastructure
for consumption by automated processes or report generators.
I am not sure if these automated processes or report generators
already exist.
The existing attribute names are quite inconsistent, for example
"Command Line", "passed_count", "wrong-version_count". I would use
lower case only with "-" as a separator. The JSON report should
contain all information of a test run.
The new report looks like this:
{
"command-line": [
"/opt/rtems/6/bin/rtems-test",
"--rtems-bsp=xilinx_zynq_a9_qemu",
"--report-format=json",
"--report-path=report",
"build/arm/xilinx_zynq_a9_qemu/testsuites/samples/"
],
"end-time": "2022-06-28T14:08:47.595131",
"host": "Linux-5.3.18-150300.59.76-default-x86_64-with-glibc2.3.4
(Linux lupus 5.3.18-150300.59.76-default #1 SMP Thu Jun 16 04:23:47
UTC 2022 (2cc2ade) x86_64 x86_64)",
"python": "3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]",
"reports": [
{
"arch": "arm",
"bsp": "xilinx_zynq_a9_qemu",
"command-line": "qemu-system-arm -no-reboot -nographic
-net none -serial null -serial mon:stdio -M xilinx-zynq-a9 -m 256M
-kernel build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
"end-time": "2022-06-28T12:08:48.161691+00:00",
"executable":
"build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
"executable-sha512":
"413bc8a506053c95a0e1d93942cce9fd0d653f7b3cde42623085ce147aea324f4baeff7ce0102008322fb1341bc2b38defd3925a64144e47e13cc5f32185bf78",
"output": [
"qemu-system-arm: warning: nic cadence_gem.0 has no
peer",
"qemu-system-arm: warning: nic cadence_gem.1 has no
peer",
"",
"",
"*** BEGIN OF TEST CONSTRUCTOR/DESTRUCTOR ***",
"*** TEST VERSION:
6.0.0.3302b72754df5f37214e86dd68522189857772c7",
"*** TEST STATE: EXPECTED_PASS",
"*** TEST BUILD: RTEMS_DEBUG RTEMS_POSIX_API",
"*** TEST TOOLS: 12.1.1 20220622 (RTEMS 6, RSB
bf3f66c29a046ec1c5029e4f919b525947df6245, Newlib e7ffbdb)",
"GLOBAL: Hey I'm in base class constructor number 1
for 0x214474.",
"GLOBAL: Hey I'm in base class constructor number 2
for 0x214480.",
"GLOBAL: Hey I'm in derived class constructor number 3
for 0x214480.",
"LOCAL: Hey I'm in base class constructor number 4 for
0x228864.",
...
"WRoZqtoO3A8BDJkUSmRGNOBaaHW1UwAAoQGQAQwAAABck8Bx+K/znDWWTEcA",
"AKEB2P///wEMVP6TRqQZY+4+srvAAAChAfD///8BDMXOxS0Rhzqx6Old2wAA",
"oQH4AQwAAADJyx1ZCnZ4l+m/XkMAAKEB0P///wEMN3+9YAwMW8gTHIoPAACh",
"Adj///8BDBtxsj3zXZF/+UqzfAAAoQHwAQwwMDMeE7mphT6yu8AAAKEB",
"8P///wEMr2rLCcwzVnf5SrN8AAChAfD///8A",
"*** END OF GCOV INFO BASE64 ***",
""
],
"result": "passed",
"start-time": "2022-06-28T12:08:47.721822+00:00"
}
],
"start-time": "2022-06-28T14:08:47.595131"
}
There are no publicly available report generators for the existing
format, but I do have one I use to generate STR documents for delivery
using mustache templates and further processing. The existing format was
designed to be easily consumable by that and other simple templating
mechanisms by providing structure (test subsets) and precalculating
values that would otherwise be implicit in the data. Changes to the
names of various fields should be easy to accommodate, but a more
significant restructure is going to present problems continuing with any
simple templating mechanism and would instead require a more
comprehensive document generator for all cases.
I'm not necessarily opposed to this change as it significantly
simplifies the report generation code, but it will definitely require
some rework in my processes. I can't speak for any users of the YAML
report format, but it was committed over a year and a half ago.
Ok, I will bring back the redun
Re: [tools] tester: Normalize JSON and YAML reports
On 6/29/2022 04:34, Sebastian Huber wrote:
On 29/06/2022 11:20, Chris Johns wrote:
On 29 Jun 2022, at 4:42 pm, Sebastian Huber
wrote:
On 29/06/2022 08:40, Sebastian Huber wrote:
Report the same data in JSON and YAML reports. Do not report
redundant
information.
Update 4671.
This patch changes the JSON reports. Are there already consumers for
the JSON reports so that we have to be backward compatible?
Could compatibility be added back in if this proves to be an issue?
I am wondering if that could be considered if comparability becomes
an issue.
The JSON report was added by:
commit 5fcaf843bdb491e2da14dc1af61806ac6d20bae5
Author: Kinsey Moore
Date: Wed Aug 21 16:34:12 2019 +
Add JSON log generation
Add log formatter hooks and JSON log formatter to the test
infrastructure
for consumption by automated processes or report generators.
I am not sure if these automated processes or report generators
already exist.
The existing attribute names are quite inconsistent, for example
"Command Line", "passed_count", "wrong-version_count". I would use
lower case only with "-" as a separator. The JSON report should
contain all information of a test run.
The new report looks like this:
{
"command-line": [
"/opt/rtems/6/bin/rtems-test",
"--rtems-bsp=xilinx_zynq_a9_qemu",
"--report-format=json",
"--report-path=report",
"build/arm/xilinx_zynq_a9_qemu/testsuites/samples/"
],
"end-time": "2022-06-28T14:08:47.595131",
"host": "Linux-5.3.18-150300.59.76-default-x86_64-with-glibc2.3.4
(Linux lupus 5.3.18-150300.59.76-default #1 SMP Thu Jun 16 04:23:47
UTC 2022 (2cc2ade) x86_64 x86_64)",
"python": "3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]",
"reports": [
{
"arch": "arm",
"bsp": "xilinx_zynq_a9_qemu",
"command-line": "qemu-system-arm -no-reboot -nographic
-net none -serial null -serial mon:stdio -M xilinx-zynq-a9 -m 256M
-kernel build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
"end-time": "2022-06-28T12:08:48.161691+00:00",
"executable":
"build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
"executable-sha512":
"413bc8a506053c95a0e1d93942cce9fd0d653f7b3cde42623085ce147aea324f4baeff7ce0102008322fb1341bc2b38defd3925a64144e47e13cc5f32185bf78",
"output": [
"qemu-system-arm: warning: nic cadence_gem.0 has no
peer",
"qemu-system-arm: warning: nic cadence_gem.1 has no
peer",
"",
"",
"*** BEGIN OF TEST CONSTRUCTOR/DESTRUCTOR ***",
"*** TEST VERSION:
6.0.0.3302b72754df5f37214e86dd68522189857772c7",
"*** TEST STATE: EXPECTED_PASS",
"*** TEST BUILD: RTEMS_DEBUG RTEMS_POSIX_API",
"*** TEST TOOLS: 12.1.1 20220622 (RTEMS 6, RSB
bf3f66c29a046ec1c5029e4f919b525947df6245, Newlib e7ffbdb)",
"GLOBAL: Hey I'm in base class constructor number 1
for 0x214474.",
"GLOBAL: Hey I'm in base class constructor number 2
for 0x214480.",
"GLOBAL: Hey I'm in derived class constructor number 3
for 0x214480.",
"LOCAL: Hey I'm in base class constructor number 4 for
0x228864.",
...
"WRoZqtoO3A8BDJkUSmRGNOBaaHW1UwAAoQGQAQwAAABck8Bx+K/znDWWTEcA",
"AKEB2P///wEMVP6TRqQZY+4+srvAAAChAfD///8BDMXOxS0Rhzqx6Old2wAA",
"oQH4AQwAAADJyx1ZCnZ4l+m/XkMAAKEB0P///wEMN3+9YAwMW8gTHIoPAACh",
"Adj///8BDBtxsj3zXZF/+UqzfAAAoQHwAQwwMDMeE7mphT6yu8AAAKEB",
"8P///wEMr2rLCcwzVnf5SrN8AAChAfD///8A",
"*** END OF GCOV INFO BASE64 ***",
""
],
"result": "passed",
"start-time": "2022-06-28T12:08:47.721822+00:00"
}
],
"start-time": "2022-06-28T14:08:47.595131"
}
There are no publicly available report generators for the existing
format, but I do have one I use to generate STR documents for delivery
using mustache templates and further processing. The existing format was
designed to be easily consumable by that and other simple templating
mechanisms by providing structure (test subsets) and precalculating
values that would otherwise be implicit in the data. Changes to the
names of various fields should be easy to accommodate, but a more
significant restructure is going to present problems continuing with any
simple templating mechanism and would instead require a more
comprehensive document generator for all cases.
I'm not necessarily opposed to this change as it significantly
simplifies the report generation code, but it will definitely require
some rework in my processes. I can't speak for any users of the YAML
report format, but it was committed over a year and a half ago.
Kinsey
___
devel mailing list
Re: [tools] tester: Normalize JSON and YAML reports
On 29/06/2022 11:20, Chris Johns wrote:
On 29 Jun 2022, at 4:42 pm, Sebastian Huber
wrote:
On 29/06/2022 08:40, Sebastian Huber wrote:
Report the same data in JSON and YAML reports. Do not report redundant
information.
Update 4671.
This patch changes the JSON reports. Are there already consumers for the JSON
reports so that we have to be backward compatible?
Could compatibility be added back in if this proves to be an issue?
I am wondering if that could be considered if comparability becomes an issue.
The JSON report was added by:
commit 5fcaf843bdb491e2da14dc1af61806ac6d20bae5
Author: Kinsey Moore
Date: Wed Aug 21 16:34:12 2019 +
Add JSON log generation
Add log formatter hooks and JSON log formatter to the test
infrastructure
for consumption by automated processes or report generators.
I am not sure if these automated processes or report generators already
exist.
The existing attribute names are quite inconsistent, for example
"Command Line", "passed_count", "wrong-version_count". I would use lower
case only with "-" as a separator. The JSON report should contain all
information of a test run.
The new report looks like this:
{
"command-line": [
"/opt/rtems/6/bin/rtems-test",
"--rtems-bsp=xilinx_zynq_a9_qemu",
"--report-format=json",
"--report-path=report",
"build/arm/xilinx_zynq_a9_qemu/testsuites/samples/"
],
"end-time": "2022-06-28T14:08:47.595131",
"host": "Linux-5.3.18-150300.59.76-default-x86_64-with-glibc2.3.4
(Linux lupus 5.3.18-150300.59.76-default #1 SMP Thu Jun 16 04:23:47 UTC
2022 (2cc2ade) x86_64 x86_64)",
"python": "3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]",
"reports": [
{
"arch": "arm",
"bsp": "xilinx_zynq_a9_qemu",
"command-line": "qemu-system-arm -no-reboot -nographic -net
none -serial null -serial mon:stdio -M xilinx-zynq-a9 -m 256M -kernel
build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
"end-time": "2022-06-28T12:08:48.161691+00:00",
"executable":
"build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
"executable-sha512":
"413bc8a506053c95a0e1d93942cce9fd0d653f7b3cde42623085ce147aea324f4baeff7ce0102008322fb1341bc2b38defd3925a64144e47e13cc5f32185bf78",
"output": [
"qemu-system-arm: warning: nic cadence_gem.0 has no peer",
"qemu-system-arm: warning: nic cadence_gem.1 has no peer",
"",
"",
"*** BEGIN OF TEST CONSTRUCTOR/DESTRUCTOR ***",
"*** TEST VERSION:
6.0.0.3302b72754df5f37214e86dd68522189857772c7",
"*** TEST STATE: EXPECTED_PASS",
"*** TEST BUILD: RTEMS_DEBUG RTEMS_POSIX_API",
"*** TEST TOOLS: 12.1.1 20220622 (RTEMS 6, RSB
bf3f66c29a046ec1c5029e4f919b525947df6245, Newlib e7ffbdb)",
"GLOBAL: Hey I'm in base class constructor number 1 for
0x214474.",
"GLOBAL: Hey I'm in base class constructor number 2 for
0x214480.",
"GLOBAL: Hey I'm in derived class constructor number 3
for 0x214480.",
"LOCAL: Hey I'm in base class constructor number 4 for
0x228864.",
...
"WRoZqtoO3A8BDJkUSmRGNOBaaHW1UwAAoQGQAQwAAABck8Bx+K/znDWWTEcA",
"AKEB2P///wEMVP6TRqQZY+4+srvAAAChAfD///8BDMXOxS0Rhzqx6Old2wAA",
"oQH4AQwAAADJyx1ZCnZ4l+m/XkMAAKEB0P///wEMN3+9YAwMW8gTHIoPAACh",
"Adj///8BDBtxsj3zXZF/+UqzfAAAoQHwAQwwMDMeE7mphT6yu8AAAKEB",
"8P///wEMr2rLCcwzVnf5SrN8AAChAfD///8A",
"*** END OF GCOV INFO BASE64 ***",
""
],
"result": "passed",
"start-time": "2022-06-28T12:08:47.721822+00:00"
}
],
"start-time": "2022-06-28T14:08:47.595131"
}
--
embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.hu...@embedded-brains.de
phone: +49-89-18 94 741 - 16
fax: +49-89-18 94 741 - 08
Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel
Re: [tools] tester: Normalize JSON and YAML reports
> On 29 Jun 2022, at 4:42 pm, Sebastian Huber > wrote: > > On 29/06/2022 08:40, Sebastian Huber wrote: >> Report the same data in JSON and YAML reports. Do not report redundant >> information. >> Update 4671. > > This patch changes the JSON reports. Are there already consumers for the JSON > reports so that we have to be backward compatible? Could compatibility be added back in if this proves to be an issue? I am wondering if that could be considered if comparability becomes an issue. Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [tools] tester: Normalize JSON and YAML reports
On 29/06/2022 08:40, Sebastian Huber wrote: Report the same data in JSON and YAML reports. Do not report redundant information. Update 4671. This patch changes the JSON reports. Are there already consumers for the JSON reports so that we have to be backward compatible? -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
[tools] tester: Normalize JSON and YAML reports
Report the same data in JSON and YAML reports. Do not report redundant
information.
Update 4671.
---
tester/rt/test.py | 191 ++
1 file changed, 40 insertions(+), 151 deletions(-)
diff --git a/tester/rt/test.py b/tester/rt/test.py
index 113936c..5b86804 100644
--- a/tester/rt/test.py
+++ b/tester/rt/test.py
@@ -218,76 +218,46 @@ def killall(tests):
test.kill()
+def results_to_data(args, reports, start_time, end_time):
+data = {}
+data['command-line'] = args
+data['host'] = host.label(mode='all')
+data['python'] = sys.version.replace('\n', '')
+data['start-time'] = start_time.isoformat()
+data['end-time'] = start_time.isoformat()
+reports_data = []
+
+for name, run in reports.results.items():
+run_data = {}
+result = run['result']
+run_data['result'] = result
+output = []
+for line in run['output']:
+if line.startswith('] '):
+output.append(line[2:])
+elif line.startswith('=> exe:'):
+run_data['command-line'] = line[9:]
+run_data['output'] = output
+run_data['executable'] = name
+run_data['executable-sha512'] = get_hash512(name)
+run_data['start-time'] = run['start'].isoformat()
+run_data['end-time'] = run['end'].isoformat()
+run_data['bsp'] = run['bsp']
+run_data['arch'] = run['bsp_arch']
+reports_data.append(run_data)
+
+data['reports'] = reports_data
+return data
+
+
def generate_json_report(args, reports, start_time, end_time,
- total, json_file):
+ _total, json_file):
import json
-import sys
-json_log = {}
-json_log['Command Line'] = " ".join(args)
-json_log['Python'] = sys.version.replace('\n', '')
-json_log['test_groups'] = []
-json_log['Host'] = host.label(mode='all')
-json_log['summary'] = {}
-json_log['summary']['passed_count'] = reports.passed
-json_log['summary']['failed_count'] = reports.failed
-json_log['summary']['user-input_count'] = reports.user_input
-json_log['summary']['expected-fail_count'] = reports.expected_fail
-json_log['summary']['indeterminate_count'] = reports.indeterminate
-json_log['summary']['benchmark_count'] = reports.benchmark
-json_log['summary']['timeout_count'] = reports.timeouts
-json_log['summary']['test-too-long_count'] = reports.test_too_long
-json_log['summary']['invalid_count'] = reports.invalids
-json_log['summary']['wrong-version_count'] = reports.wrong_version
-json_log['summary']['wrong-build_count'] = reports.wrong_build
-json_log['summary']['wrong-tools_count'] = reports.wrong_tools
-json_log['summary']['invalid-header_count'] = reports.wrong_header
-json_log['summary']['total_count'] = reports.total
-time_delta = end_time - start_time
-json_log['summary']['average_test_time'] = str(time_delta / total)
-json_log['summary']['testing_time'] = str(time_delta)
-
-result_types = [
-'failed', 'user-input', 'expected-fail', 'indeterminate',
-'benchmark', 'timeout', 'test-too-long', 'invalid',
'wrong-version',
-'wrong-build', 'wrong-tools', 'wrong-header'
-]
-json_results = {}
-for result_type in result_types:
-json_log['summary'][result_type] = []
-
-# collate results for JSON log
-for name in reports.results:
-result_type = reports.results[name]['result']
-# map fatal-error on to failed since the report adds both to the
failed count
-if result_type == "fatal-error":
-result_type = "failed"
-test_parts = name.split("/")
-test_category = test_parts[-2]
-test_name = test_parts[-1]
-if result_type != 'passed':
-json_log['summary'][result_type].append(test_name)
-if test_category not in json_results:
-json_results[test_category] = []
-json_result = {}
-# remove the file extension
-json_result["name"] = test_name.split('.')[0]
-json_result["result"] = result_type
-if result_type == "failed" or result_type == "timeout":
-json_result["output"] = reports.results[name]["output"]
-json_results[test_category].append(json_result)
-
-# convert results to a better format for report generation
-sorted_keys = sorted(json_results.keys())
-for i in range(len(sorted_keys)):
-results_log = {}
-results_log["index"] = i + 1
-results_log["name"] = sorted_keys[i]
-results_log["results"] = json_results[sorted_keys[i]]
-json_log["test_groups"].append(results_log)
-
-# write out JSON log
+
+data = results_to_data(args, reports, start_time, end_time);
+
with open(json_file, 'w') as outfile:
-json.dump(json_log, outfile, sort_keys=True, indent=4)
+json.dump(data, outfile, sort_keys=Tru
