This is fantastic, Carlos! Thank you! I'll also pass the info along to our developer who is creating reg-test for pdftohtml. Please announce when your python version is ready, and we'll make a pass at integrating.
--josh On 9/12/11 12:39 PM, "Carlos Garcia Campos" <[email protected]> wrote: >Hi all, > >as you all probably know, our current regression test suite are some >custom shell scripts and a lot of pdf files that Albert has collected >for some years. During the last few days I have been working on >improving the regression test, converting Albert's scripts into a >small python program to allow everybody to run their own tests. We are >not going to upload pdfs to the repo, so we still depend on Albert >to make sure a change doesn't actually introduce regressions. We will >only provide the program so that anybody could use it with their own >pdfs. > >So, how does it work? > >usage: poppler-regtests [options ...] command [command-options ...] tests >... > >Common options: > > --utils-dir: The directory where poppler utils are. The script is > thought to be used from the git repo, so by default it's ../utils. > > -b, --backends: The backends to tests, for now only cairo, splash, > text and postscript backends are supported. You can provide a comma > separated list of backends to test. By default it uses all backends. > >Commands: > > create-refs: This is the command to crate the reference files. > > run-tests: This command runs the tests, comparing with references > previously created with create-refs command > >create-refs command: > >For every pdf file to test a new directory is created in the >references dir with the name of the pdf file, creating also other >directories found in the tests dir, for example, for the given tests >dir: > >./tests/foo.pdf >./tests/annots/annot-file.pdf >./tests/forms/form-file.pdf > >the following dirs will be created in references dir: > >./refs/foo.pdf/ >./refs/annots/annot-file.pdf/ >./refs/forms/form-file.pdf/ > >The references directory is /tmp/refs by default, but any other dir can >be used with --refs-dir command line option. > >Reference files are stored in the references dir of every document >prefixed with the backend-name. The following references file can be >created: > > - Checksum files: A file for each backend containing the md5 sum of > every backend dependent file. > > - Backend dependent files: A png file for every document page for > cairo and splash backends, a single ps file for postscript backend > and a single txt file for text backend. These can be removed to save > disk once checksums files are created with the option > --checksums-only. If you have enough disk it's recommended to keep > backend dependent files, so that they can be used later by run-tests > command to generate diffs for failing tests. > > - Crashed files: A file with the extension .crashed indicating the > test crashed while creating reference images. > > - Failed files: A file with the extension .failed indicating the test > failed to run. It didn't crash, but the command returned an error > status. The file contains the error status returned by the command > used to create the reference files. > > - stderr files: A file with extension .stderr indicating the command > used to create the reference files wrote something to stderr. > >For example: > >$ ls -R refs >refs/forms: >forms-scribus.pdf > >refs/forms/forms-scribus.pdf: >cairo.md5 postscript.md5 splash.md5 text.md5 > >refs/sample.pdf: >cairo.crashed cairo.stderr postscript.crashed postscript.ps >postscript.stderr splash.crashed splash.stderr text.md5 > >Reference files are not regenerated unless --force command line option >is used. You can update your reference files just adding a new file to >the tests dir and running create-refs again. > > >run-tests command: > >This command creates reference files in the output dir and compares >the results with the reference files in references dir. > >A test can produce the following results: > > - PASS: Test passed as expected. > > - FAIL: Test was expected to pass but it failed, differences were > found in checksum files. > > - CRASH: Test was expected to pass but it crashed. > > - DOES NOT CRASH: Test was expected to crash, but it didn't. It's not > possible to know whether test passed or not. > > - DOES NOT FAIL: Test was expected to fail with an exit error status, > but it didn't. It's not possible to know whether test passed or > not. > > - FAIL (status error <status>): Test was expected to pass but it > failed, the test program returned with an exit status error. > > - PASS (Expected crash): Test was expected to crash and it > crashed. Even though the test crashed, it's considered to pass > since it's not a regression. > > - PASS (Expected fail with status error <status>): Test was expected > to fail and it failed. Even though the test failed to run, it's > considered to pass since it's not a regression. > >Result files for passing test are automatically removed from output >dir unless --keep-results command line option is used. Result files of >tests that failed are not removed. If backend specific reference files >were not removed from references dir, --create-diffs command line >option can be used to create diff files for tests that fail. The >format of the diff file depends on the backend, and it's not currently >supported by postscript backend. > >At the end a summary of the test results is printed, something like: > >Total 16 tests >15 tests passed (93.75%) >1 tests failed (6.25%): ./tests/annots/annots_move.PDF (splash) >4 tests have stderr output (25.00%): ./tests/sample.pdf >(splash),./tests/sample.pdf (postscript),./tests/sample.pdf >(text),./tests/sample.pdf (cairo) >Tests run in 10 seconds > >The equivalent to what Albert did with his scripts would be something >like: > >./poppler-regtest --backends=splash,cairo create-refs --refs-dir=./refs >--checksums-only ./tests/ >./poppler-regtest --backends=splash,cairo run-tests --refs-dir=./refs >--out-dir=./out ./tests/ > >I think that's all the important stuff. Sorry for the long mail, but >I think it'll make it easier for you to test and review the patch. For >more information use poppler-regtst --help. I'll add all this info to >the README file once the patch is reviewed. > >There are still things to do, like adding an option to provide documents >to skip, html backend support, and I'm sure there are a lot of bugs >too. So, feel free to comment, or even better provide patches :-) > >-- >Carlos Garcia Campos >PGP key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x523E6462 _______________________________________________ poppler mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/poppler
