On Tue, 2012-03-20 at 21:34 +0100, R. Diez wrote: > Hi all: > > I am trying to run or1ksim's test-code-or1k test suite against > ORPSoCV2's RTL (Verilog) simulation. So far the following tests run > well, so it's looking promising: basic, cache, cbasic, except-test, > exit, flag, functest, int-test, local-global, mem-test, mycompress, mul, > tick.
Hi R Diez, Excellent progress. > I am tempted to suggest dropping DejaGnu for the reasons explained in > this article (which they all seem to apply based on my limited experience): > > DejaGNU > April 19, 2011 at 7:42 am · Filed under Programming > Airs – Ian Lance Taylor > http://www.airs.com/blog/archives/499 Ian is deeply experienced as the originator particularly of BFD. I agree with lots of what he says, and he is backed up by Joel Brobecker, program lead for GDB, but the key phrase is right at the end: "Although I don’t have a good alternative..." There is a POSIX standard for how testing should be done (POSIX 1003.3), but I believe DejaGnu is still the only serious general POSIX compliant open source solution in town. Perl has its own infrastructure (look at TAP::Harness and Test::Simple), but these are even older than DejaGnu, and lack formal specification or POSIX compliance. The reality is that test management is very hard. At the last company I ran, we developed our own test harness. It started out easy, but over the lifetime of the business probably consumed 5 engineer-years of effort. It evolved more and more into something that looked like DejaGnu, but less usable, less flexible, less portable and harder to maintain. For all its faults we would never have spent so much if we had just used DejaGnu. I think our experience is one commonly shared, particularly if you don't want to pay big bucks for commercial software - not that that guarantees anything better. > But, for the time being, I'm trying with the existing stuff. The trouble > is, I have no experience with DejaGnu or with TCL/expect. This may help get you started with DejaGnu: http://www.embecosm.com/download/ean8.html TCL has tutorials online: http://www.tcl.tk/doc/ "man expect" is actually quite good - there is only a fairly small amount it adds to TCL. It is also worth knowing that "runtest --help" gives a different set of information to "man runtest". There is also useful help on GCC testing, which gives further insight: http://gcc.gnu.org/install/test.html One of my personal beefs about DejaGnu is that the official full documentation is only available in a paid for book. I know Rob Savoye put in a lot of effort, but I've always thought that making the full documentation commercial only is rather outside the spirit of free software. > The Verilog simulation runs much more slowly than or1ksim, especially > when using Icarus Verilog, so the default timeout is too short. The > default timeout is set in these 2 places: > > testsuite/lib/libsim.exp:129:# Timeout 3 seconds is plenty as default > testsuite/lib/or1ksim.exp:147:# Timeout 3 seconds is plenty as default > > The TCL code at those places looks like this: > > # Timeout 3 seconds is plenty as default > set timeout 3 > > I increased that value and it does not seem to have any effect. I had to > add an extra "set timeout 3000" statement inside routine run_or1ksim in > file testsuite/lib/or1ksim.exp in order to increase the timeout. > > Can anybody here (hopefully with more experience with expect/TCL) tell > me why the default timeout is not being propagated properly? There are various places where the timeout is hard-coded. As noted in Ian Lance-Taylor's article above, DejaGnu is far from perfect. I have tried to fix these where I find them. If there are any places in the Or1ksim testsuite, then they should certainly be replaced by conditional code checking for a global value first. > By the way, I wonder whether I should set an infinite timeout. After > all, it's hard to tell how long a Verilog simulation is going to take, > especially when running overnight on some loaded server. All Verilog > simulations should end at some point in time, and if one hangs forever, > the developer will quickly realise and fix it. After all, we don't > impose a timeout when compiling the individual toolchain components in > the automated build scripts. Infinite timeout is a bad idea. There are some failure modes where tests loop forever. You really don't want to lose the remaining 2400 test results because test 100 hung. Just choose something appropriate. On the whole functional tests should be short (OK there are some which have to be long), so execution time is not an issue. It would help to get the Or1ksim test suite running in parallel. DejaGnu supports this (e.g. for the GNU tool chain), but I have no experience setting it up. HTH, Jeremy -- Tel: +44 (1590) 610184 Cell: +44 (7970) 676050 SkypeID: jeremybennett Email: [email protected] Web: www.embecosm.com _______________________________________________ OpenRISC mailing list [email protected] http://lists.openrisc.net/listinfo/openrisc
