On Wed, 2012-03-07 at 13:28 +0000, Julius Baxter wrote: > On Tue, Mar 6, 2012 at 9:12 AM, R. Diez <[email protected]> wrote: > > Hi all: > > > > I have been looking at the test cases for the or1ksim simulator and > for the or1200 OpenRISC core (the test suite is in the ORPSoC v2 > project). I was surprised to hear that or1ksim has many more test > cases than or1200. After all, > > one would think that most of the test cases should be shared, that > is, the same test cases for the OpenRISC instruction set should run > against the software simulator and against the Verilog simulation.
Hi R Diez, Spot on. Or1ksim is supposed to be a golden reference for the ISA, against which implementations are compared. > > I'm thinking about unifying those codes across all platforms, as > well as adding some abstraction layer so that, when running on a real > FPGA host, a test case would call the C run-time routine exit() > instead of executing "l.nop NOP_EXIT". You can't assume any library is there. The simplest tests are just bare binary. > > I'm also thinking about providing some way for the test case to > generate a result data file, something like a "l.nop > WRITE_TO_RESULTS_FILE" opcode, so that the test framework can compare > the test results across different platforms. For example, we could > check that some multiplication results produced under Verilator are > the same as the ones produced under or1ksim. Or maybe it's not worth > going that far: can you think of many test cases where that would be > desirable? This is exactly what you need to do. Have you seen Waqas Ahmed's MSc or my paper on GNU tool chain testing, both of which do this. The former is on OpenCores for download (the old download location), the latter on Embecosm's website. > What we get is basically a binary output of pass/fail from the tests > at the moment. I would think that's enough to be able to tell if > there's regressions due to the last checkin. > > To check if particular values are right or not you will probably want > to use something like the expect stuff that dejagnu uses. This allows > you to specify exactly what the expected output should be, and so can > check for values after multiply (which can be dumped via the report > l.nop.) > > My opinion on this is that the dejagnu/expect stuff is a hassle to > setup, and that lots of smaller tests spitting out pass/fail is easier > to develop with and for. Additionally you probably need to have the > whole thing in an autotools setup which is also a hassle to setup and > use. But, the dejagnu stuff is a bit more thorough in terms of what it > can test. The other worry is that you'll get false positives on the > simpler test mechanism, for example if your comparison functionality > isn't working correctly to see if a multiply result is as expected. > However, we can order the tests to ensure that the more fundamental > features work before relying on those things to test higher > functionality. DejaGnu is a hassle, because getting test frameworks right, that work on lots of hosts, and are reliable is very hard. That's why it took so long to develop, and why it is still continuing to develop. It is now a POSIX standard (1003.3), and we should be working within standard frameworks wherever possible. Or1ksim at least gives you a reasonably clean framework on which to build. Remember you need to test both host and target environments. > But probably the autotools/expect/dejagnu setup is the way to go. > Unifying the code base into a single autotools-based project which can > then have some dejagnu control files which know how to drive or1ksim, > ORPSoC RTL models and an FPGA would probably be what we want. You are > then, however, relying on an external project to test everything, and > that detachment can be a pain. OTOH it would help ensure better > testing of everything and help minimise the verification work. Agreed. Good luck with this. It's been on the agenda for several years. Jeremy -- Tel: +44 (1590) 610184 Cell: +44 (7970) 676050 SkypeID: jeremybennett Email: [email protected] Web: www.embecosm.com _______________________________________________ OpenRISC mailing list [email protected] http://lists.openrisc.net/listinfo/openrisc
