On Sun, Jun 8, 2014 at 8:47 PM, Dylan Baker <[email protected]> wrote: > On Sunday, June 08, 2014 12:54:09 PM Ilia Mirkin wrote: >> On Sun, Jun 8, 2014 at 12:12 PM, Ilia Mirkin <[email protected]> wrote: >> > On Fri, Jun 6, 2014 at 5:02 PM, Dylan Baker <[email protected]> > wrote: >> >> On Friday, June 06, 2014 04:57:28 PM Ilia Mirkin wrote: >> >>> On Fri, Jun 6, 2014 at 4:54 PM, Dylan Baker <[email protected]> > wrote: >> >>> > Ilia, I've tested this with your use case, does seem reasonable? >> >>> >> >>> The general idea seems reasonable, but I haven't had a chance to >> >>> actually review or test myself. Do you have this in a branch somewhere >> >>> so I can play with it? Or if you just want to push, we can fix up any >> >>> issues later... whatever. >> >>> >> >>> > On Thursday, June 05, 2014 04:33:00 PM Dylan Baker wrote: >> >>> >> This patch updates our json to version 1. Changes from version 0 to >> >>> >> version 1 are as follows: >> >>> >> >> >>> >> - renamed 'main' to 'results.json' >> >>> >> - dmesg must be a string (It was stored a list in some version 0 >> >>> >> >> >>> >> results) >> >>> >> >> >>> >> - subtests are never stored as duplicate entries, a single instance >> >>> >> of >> >>> >> >> >>> >> the test is recorded, (In version 0 both are possible) >> >>> >> >> >>> >> - there is no info entry in version 1, err, out, and returncode are >> >>> >> >> >>> >> always split into seperate entries >> >>> >> >> >>> >> This patch adds support to the results module for handling updates to >> >>> >> the results in a sane way. It does this by adding a result_version >> >>> >> attribute to the TestrunResult (which is stored as json), and >> >>> >> implementing the ability to incrementally update results between >> >>> >> versions. >> >>> >> >> >>> >> It does this automatically on load, non-destructively, moving the old >> >>> >> results to results.json.old, but does write the updated results to >> >>> >> disk, >> >>> >> making the cost of this update a one time cost. >> >>> >> >> >>> >> v2: - Handle non-writable directories and files, this also fixes >> >>> >> using >> >>> >> >> >>> >> file-descriptors as inputs >> >>> >> >> >>> >> Signed-off-by: Dylan Baker <[email protected]> >> >>> >> --- >> >>> > >> >>> > [snip] >> >> >> >> doh! I specifically pushed it to a branch and then forgot to put that in >> >> my >> >> cover letter. >> >> https://github.com/dcbaker/piglit versioned-results-v4 >> > >> > Hm, just checked out that branch, this is what I get: >> > >> > $ python ~/src/piglit/piglit-summary-html.py -e pass -e skip >> > nvc0-comparison <(xzcat nvcf-2014-04-20-mupuf.xz) <(xzcat >> > nve7-2014-06-06-mupuf-gs5.xz) <(xzcat gk208-2014-06-07-mupuf-gs5.xz) >> > --overwrite >> > >> > Traceback (most recent call last): >> > File "/home/ilia/src/piglit/piglit-summary-html.py", line 26, in >> > <module> >> > >> > from framework.programs.summary import html >> > >> > File "/home/ilia/src/piglit/framework/programs/summary.py", line 26, >> > >> > in <module> >> > >> > import framework.summary as summary >> > >> > File "/home/ilia/src/piglit/framework/summary.py", line 258, in <module> >> > >> > class Summary: >> > File "/home/ilia/src/piglit/framework/summary.py", line 266, in Summary >> > >> > TEMPLATE_DIR = path.join(os.environ['PIGLIT_SOURCE_DIR'], 'templates') >> > >> > File "/usr/lib64/python2.7/UserDict.py", line 23, in __getitem__ >> > >> > raise KeyError(key) >> > >> > KeyError: 'PIGLIT_SOURCE_DIR' >> > >> > Operator error of some sort? I tried rebuilding piglit in case there >> > was some sort of compile-time-created dependency, but that didn't >> > help. >> >> Ugh, I guess that piglit top-level command is the only way to run >> things now? I thought it was supposed to be optional... Still doesn't >> work though... >> >> $ python ~/src/piglit/piglit summary html -e pass -e skip >> nvc0-comparison <(xzcat nvcf-2014-04-20-mupuf.xz) <(xzcat >> nve7-2014-06-06-mupuf-gs5.xz) <(xzcat gk208-2014-06-07-mupuf-gs5.xz) >> --overwrite >> WARNING: Could not write updated results /dev/fd/63 >> WARNING: Could not write updated results /dev/fd/62 >> WARNING: Could not write updated results /dev/fd/61 >> Traceback (most recent call last): >> File "/home/ilia/src/piglit/piglit", line 93, in <module> >> main() >> File "/home/ilia/src/piglit/piglit", line 88, in main >> returncode = parsed.func(args) >> File "/home/ilia/src/piglit/framework/programs/summary.py", line 98, in >> html output.generate_html(args.summaryDir, args.exclude_details) >> File "/home/ilia/src/piglit/framework/summary.py", line 429, in >> generate_html path.join(destination, "index.css")) >> File "/usr/lib64/python2.7/shutil.py", line 119, in copy >> copyfile(src, dst) >> File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile >> with open(src, 'rb') as fsrc: >> IOError: [Errno 2] No such file or directory: >> '/home/ilia/src/lib/piglit/templates/index.css' >> >> I guess I have to run out of the piglit source dir directly otherwise >> it assumes some sort of installation occurred? OK, running it out of >> the source dir, the generated files seem a little wrong... >> >> It generates a link to >> >> nvcf-2014-04-20-mupuf/spec/EXT_framebuffer_object/fbo-blending-formats/3.htm >> l >> >> but that file doesn't exist. It also seems to still be doubling >> subtests, I suspect the issues are related. You can try it with those >> same exact files yourself, they're in >> http://people.freedesktop.org/~imirkin/piglit/. There's a "correct" >> output for those 3 files available at >> http://people.freedesktop.org/~imirkin/nvc0-comparison/problems.html . >> >> Also it outputs >> >> WARNING: Could not write updated results /dev/fd/63 >> WARNING: Could not write updated results /dev/fd/62 >> WARNING: Could not write updated results /dev/fd/61 >> >> Which is a little annoying... IMHO it shouldn't try to update the >> results unless you ask for it, and with something like /dev/fd/*, it >> doesn't have a chance. >> >> Lastly, the time it takes to generate results has substantially >> increased. With your branch: >> >> $ time python ~/src/piglit/piglit summary html -e pass -e skip >> ~/piglit-results/nvc0-comparison <(xzcat >> ~/piglit-results/nvcf-2014-04-20-mupuf.xz) <(xzcat >> ~/piglit-results/nve7-2014-06-06-mupuf-gs5.xz) <(xzcat >> ~/piglit-results/gk208-2014-06-07-mupuf-gs5.xz) --overwrite >> WARNING: Could not write updated results /dev/fd/63 >> WARNING: Could not write updated results /dev/fd/62 >> WARNING: Could not write updated results /dev/fd/61 >> >> real 0m56.297s >> user 0m55.446s >> sys 0m0.783s >> >> With my local branch which has the fixups to deal with the subtests/info >> things: >> >> $ time python ~/src/piglit/piglit-summary-html.py -e pass -e skip >> nvc0-comparison <(xzcat nvcf-2014-04-20-mupuf.xz) <(xzcat >> nve7-2014-06-06-mupuf-gs5.xz) <(xzcat gk208-2014-06-07-mupuf-gs5.xz) >> --overwrite >> >> real 0m6.800s >> user 0m6.613s >> sys 0m0.213s >> >> IMO your branch isn't quite ready for prime-time. >> >> -ilia > > Well, I designed it with the assumption that the conversion cost would be one > time, so being a little slow wasn't a problem. Honestly, I've never run into > anyone who uses that shell syntax, I had no idea it even existed. > > I think the copy bug is the same as the PIGLIT_SOURCE_DIR bug (which I have a > fix for), and I'll see about enhancing the performance.
The performance hit might actually be due to the doubling that still remains. I forgot to look, but I bet the number of tests was around 100K instead of 10K again for one of those runs. I remember when I was first debugging the doubling issue, things ran very slowly as well. -ilia _______________________________________________ Piglit mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/piglit
