----- Original Message -----
> From: [email protected]
> To: [email protected]
> Sent: Wednesday, 20 February, 2013 12:55:07 PM
> Subject: [LTP] [RFC] replacement for runltp + ltp-pan
>
> Hi!
> Some time ago I started to think about replacement for the current
> runltp + ltp-pan test execution system, before I start writing any
> code
> here are a few points that I currently have in mind. Feel free to
> comment on them.
>
> Here is what I think that is wrong with the current system:
>
> * Open Posix Testsuite is not executed by runtest + ltp-pan
>
> - technically this should be as easy as translating the return
> codes and generating runtest files as we build the testsuite
Agreed, I think you also mentioned previously, that some processes can
get reparented under init and continue running without limits.
>
> * Current system doesn't support timeouts
>
> - which should be easy to implement but would require change
> to the runtest file format (one timeout for all doesn't
> work as we have tests that runs less than second and tests
> that takes more than ten minutes).
Finding the default for some may be tricky, as they can depend on
drive speed or amount of RAM.
Is there a plan for some transition period where both of these
would work side-by-side?
>
> * The output to stdout is messy
>
> What I would like is:
>
> [004/243] getrusage04 .................................. [FAILED]
> [005/243] getsid01 ..................................... [RUNNING]
>
> * The new tool should be able search runtest files for particular
> test
> names, blacklist some tests if needed, etc. Which is now done by
> grepping runtest files in the hacked together runltp script.
>
> * There are tests that needs a disk partition or loopback image, we
> should handle that and export right environment variables. This
> should
> be designed, documented and fixed.
>
> * The current system doesn't produce usable logs
>
> Well you get list of testcases that succeeded and list of testcases
> that has failed, which is better than nothing.
>
> But what I'm thinking about is some more structured format that
> will include test output, possibly test duration and so. I
Looking at current output, almost all fields look useful.
Well except 'contacts', not sure if I have ever seen this set to non-empty
string.
<<<test_start>>>
tag=abs01 stime=1361370087
cmdline="abs01"
contacts=""
analysis=exit
<<<test_output>>>
abs01 1 TPASS : Test passed
abs01 2 TPASS : Test passed
abs01 3 TPASS : Test passed
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no
cutime=0 cstime=0
<<<test_end>>>
> personally
> think that JSON is good candidate for this job as it's structured,
> easy to read and widely supported as there are libraries for
> python,
> perl, etc.
>
> Then we could create git repo with reference results. Generate html
> pages from the repo with results. Write scripts for result
> comparsion,
> etc.
>
> The JSON log may look like:
>
> "LTP Version": 20130109,
> "System Info": {
> "Kernel Version": "3.7.1",
> "Hardware": {
> "Arch": "x86_64",
> "CPUs": 2,
> "Ram": 4051840,
> (more filelds like this follows),
> },
> },
> "Results": [
> {
> "Test Name": "getrusage04",
> "Test Result": "FAILED",
> "Test Runtime": 0.011532,
> "Test CPU Time": 0.003234,
> "Test Output": [
> "TINFO : Using 1 as multiply factor for max increment",
> "TINFO : utime: 0us; stime 0us",
> "TINFO : utime: 0us; stime 4000us",
> "TFAIL : stime increased > 2000us",
> ],
> },
> (more test results follows),
> ]
I don't have objections to JSON. I do not parse existing logs, I just
search/read them
when there is failure. And it should be easy to just convert it to plain text
if there is
need for it.
Regards,
Jan
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list
