Hi Milan, We have some precedence here of testsuites residing in the gate (the dtrace testsuite for e.g.) and STEP(http://step.ireland) / OSTEP( http://www.opensolaris.org/os/community/testing/ostep/) can help hide the idiosyncrasies of the framework from the regression tester.
Regarding Roland's concern with ensuring that the testsuite has a dedicated machine with >1 CPU (and many other H/W requirements), we have precedence here also. This may not resolve all the issues you've identified though. Paul Milan Jurik wrote: > Hi Roland, > > I see you added testing expert to the loop, good :-) > > Per your request ksh93 and testing discuss are part of CC now. > > V p?, 17. 07. 2009 v 02:40, Roland Mainz p??e: > >> Milan Jurik wrote: >> [CC:'ing Jim Walker since he knows the test setup&6farms] >> >>> test plan approved for C-team inception. One thing comming from it - >>> look at CTI >>> (http://opensolaris.org/os/community/testing/testsuites/ctifortet/) and >>> evaluate how much work should be needed to wrap ksh93 test suite in CTI >>> (so it could be used by automated tests later), please. >>> >> It depends on it how the CTI stuff really works. I've discussed that >> with April long ago - it's theoretically possible but it becomes >> difficult for three reasons: >> 1. the ksh93 test suite is CPU-load senstive - a too high CPU load will >> cause some tests to fail because they depend on precise timing (e.g. the >> signal and pipe tests are the top canidates for breakdown since they all >> do some kind of interprocess communication) >> 2. the ksh93 tests are I/O latency-sensitive because the tests assume >> that writes are done after a certain time (e.g. 0.2 seconds etc.) and >> that a co-/parent-process can read the data then >> 3. the ksh93 tests are tied to precisely the ksh93 version they are >> testing. That's why the matching testing code sits in OS/Net and not in >> the test gate (and keeping different gates correctly in sync is >> considered to be difficult and would loose the benefits of having the >> test suite in OS/Net (and delivered with the rest of the system (which >> turned out to be a good idea since we caught dozends of system bugs that >> way (e.g. the Solaris/SystemZ port wouldn't be in the current shape >> without it)))) >> >> The suggested workaround for [1] was to move the tests to the "RT" >> (=realtime) class (and bind the tests to n-1 CPUs (where <n> is the >> number of online CPUs in a system (this leaves one CPU free for >> processes in the "TS" class))) ... however that may cause a whole >> testmachine to hang if it only has a) one CPU and b) one or more test >> starts to consume 100% CPU. And I am not sure whether the test suite >> allows tests to run under the neccesary priviledges. >> The alternative for [1] would be to find a way to "enforce" that the >> test suite is the only thing which runs on the matching machine. Jim... >> is that possible ? >> >> For [2] we may simply use "tmpfs" as working directory... >> ... and for [3] we simply use the version from /usr/demo/ksh/tests/ and >> just add a plugin for CTI which lives in the testing package. >> >> > > I understand your points to have test suite in ON gate, but on the other > hand this complicates the regression testing because you are not able to > do automated testing. People should not be forced to learn one more > testing framework and to do testing manually if they are fixing e.g. > libc (which can have impact on ksh93). > > I hope somebody in testing-discuss@ can evaluate possibilities. I plan > to fill RFE that ksh93 testing should be converted/encapsulated to CTI. > I think it should not be your responsibility but it would be nice from > you if you can be technical consultant for it. > > Best regards, > > Milan > > _______________________________________________ > testing-discuss mailing list > testing-discuss at opensolaris.org >
