[I messed up the editing in my earlier message; use this in place of the earlier one. --Pm]
On Thu, Jan 22, 2009 at 05:57:08PM +0100, Peter Bowers wrote: > Pm wrote: > > I'd welcome a way for us to have automated tests for PmWiki, > > and I think we can support it on pmwiki.org. However, I > > warn that it's not as trivial as it sounds at first, and > > someone other than me will need to implement a prototype or > > proof-of-concept. > > Perhaps if you share your approach it might get some creative juices > flowing...? In that spirit I'll offer a thought... My specific thoughts are below, but the overall approach I was taking was to create wiki pages containing markup and the desired HTML output, and having some specialized actions that would run the tests and report the results. In the earliest versions I was doing each test as its own wiki page, but that quickly became unwieldy. So then I switched it to allow multiple tests on a page, and grouping tests together by topic. That worked better, but keeping tests up-to-date was a big pain, and it was also a pain to generate summary information. Generating summary information might be a lot less painful now that we have pagelists (which we didn't have at the beginning of the 2.0 series). > ===my very off-the-cuff tho'ts=== > I was thinking through using wget, saving off known-good-html files, and > then looping through with various configurations on various servers and etc. > Something like this [caveat - all theoretical at this point]: [ below excerpted for simplicity ] > for htmlfile in $domain/html.d/* > do > curhtml=$(basename $htmlfile) > wget http://${curdomain}/${curhtml} #what options needed? > if diff -q ${htmlfile} ${curhtml} #don't remember how wget names > [...] > done > > Obviously some more error checking has to be thrown in around wget and etc, > but that should give the basic idea... For purposes of discussion, let's call the files in html.d/ our "testing standard" -- i.e., those are what we expect valid tests to produce. The biggest problem with this approach is that "diff" is much too coarsely grained to help us locate problems. For example, if we simply change a skin feature, HTML header, or even just the amount of whitespace that occurs in certain elements, then the above would report that every test has failed because it's no longer _exactly_ the same as our standards in html.d/ . In other words, the "failures" it reports aren't really failures in PmWiki; each of the pages are still semantically correct -- they're still rendered properly in a browser -- but the diff command falsely reports them as having failed. In order to get around these failures, we'd then have to update all of the files in our "test standard" (in html.d/) with new copies of what we expect. That's too onerous to do manually, and if we do it automatically then we aren't really testing for "correctness", we're just testing if the output is changing at all from one version to the next. Perhaps having a system that tells us if _anything_ changes is useful, but I was aiming for something a bit more fine-tuned than that. > Of course the difficulty is not in coding the test script nor in running the > test script, but in choosing the appropriate test data and checking it by > hand and then saving off the known-good-copies (although this saving off can > be automated as well). As I mention above, "checking [the test standard] by hand" is likely to be impractical with this approach, because it would have to be re-checked for many otherwise insignificant changes to the core code. Pm _______________________________________________ pmwiki-users mailing list [email protected] http://www.pmichaud.com/mailman/listinfo/pmwiki-users _______________________________________________ pmwiki-users mailing list [email protected] http://www.pmichaud.com/mailman/listinfo/pmwiki-users
