On 13/11/19 5:22 pm, Sebastian Huber wrote: > On 13/11/2019 01:39, Chris Johns wrote: >>> > Ok, it seems a test program has exactly one state for a given BSP. >>> >>> Yes, this is driven by the ABI, hardware profile and similar >>> constraints. >>> >>> >>> This is generally the case but we have a few BSPs where the same executable >>> runs on real hardware and multiple simulators. The SPARC BSPs are at the >>> top of this list. The pc386 is another example. >>> >>> We currently have no way to specify that a test could pass on real hardware, >>> fail on simulator 1 and pass on simulator 2. For the SPARC BSPs, we now >>> have real HW, sis, qemu and tsim. >>> >>> The .tcfg files don't begin to address this and I really don't have any good >>> ideas myself. It is just a very real life case that it would be nice to >>> address >>> so we have 100% expected passes on all execution platforms. >> I think it is a difficult problem to try and solve in RTEMS with test >> states. I >> think this is best handled in the eco-system with rtems-test and anything >> that >> processes the results it generates. In the eco-system we have more >> flexibility >> to define a hardware profile and assign a BSP to it. >> >> A hardware test result is of higher importance that a simulator result and a >> simulator test result is of higher importance than no test results. The >> expected-fail state is feedback driven and we cannot set this state without a >> suitable set of results. The tier a BSP resides in defines how you view the >> test >> state for a specific BSP. >> >> Here are some observations ... >> >> 1. By definition an accurate simulator will always match the hardware test >> results > > This depends on the goals of the simulator. In Qemu for example there is by > design no preservation of the timing properties of the simulated hardware. We > see this in the spintrcritical* tests for example.
No doubt however the assertion is still true. Having tests that pass on simulator and fail on hardware then flagging them as valid does not make sense. >> 2. We need to provide a workable way to capture the test results and feedback >> this back into the tiers >> >> 3. The tiers need to be current and visible to be meaningful to our users >> >> 4. We have no expected-fail states in any tests and we should ... >> >> $ grep 'expected-fail' `find . -name \*.tcfg`` | wc -l >> 0 > > One approach to capture the actual expectations on a particular target system > (e.g. real hardware H, simulator A, simulator B) would be to split the state > definitions. The build system would just build the test or not. I am not following why you would want this. > A particular > test runner which knows if the test runs on H, A, or B could use a test runner > specific data set to get the runtime states (pass, expected-fail, user-input, > indeterminate, benchmark). We should always build and run a test that can be executed. We need to know if a test flagged as expected-fail is now passing. Chris _______________________________________________ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel