Albert Astals Cid <[email protected]> writes: > El Wednesday 30 December 2015, a les 17:04:42, Adam Reichold va escriure: >> Hello again, >> >> as discussed in the code modernization thread, if we are going to make >> performance-orient changes, we need a simple way to track functional and >> performance regressions. >> >> The attached patch tries to extend the existing Python-based regtest >> framework to measure run time and memory usage to spot significant >> performance changes in the sense of relative deviations w.r.t. to these >> two parameters. It also collects the sums of both which might be used as >> "ball park" numbers to compare the performance effect of changes over >> document collections. > > Have you tried it? How stable are the numbers? For example here i get for > rendering the same file (discarding the first time that is loading the file > into memory) numbers that range from 620ms to 676ms, i.e. ~10% variation > without no change at all.
I haven't looked at the patches in detail yet, but I don't think the regtest framework should be the one measuring. I would add a tool for that, pdfperf or something like that, that could measure the internals (parsing, output devs, etc.) in a more detailed way. If we need to get more information from the poppler core we could just add a compile option to provide that info. And the regtest framework should just run the perf command and collect the results. A report command could compare results with refs or previous executions. I also think performance tests should be run with a different command, since we don't want to measure the performance of every single document we have in our test suite (it would take too long). Instead, we could select a set of documents with different kind of pdf features to run perf tests on those only. So, in summary, I would use a dedicated tool (depending on a build option if we need to get more info from the core), and maybe even a dedicated perf framework on top of that tool, since I consider perf tests different from rendering regression tests, the same way unit tests are handled by a different framework too. > Cheers, > Albert > >> >> The patch runs the measured commands repeatedly including warm-up >> iterations and collects statistics from these runs. The measurement >> results are stored as JSON documents with the actual program output of >> e.g. pdftotext or pdftoppm being discarded. >> >> To implement the check for relative deviations, it abuses the checksum >> comparison method and hence checksums are still computed for the JSON >> documents even though they are actually unnecessary. It is also limited >> to Unix-like operating systems (due to the use of the wait3 syscall to >> determine resource usage similar to the time command). > > _______________________________________________ > poppler mailing list > [email protected] > http://lists.freedesktop.org/mailman/listinfo/poppler -- Carlos Garcia Campos PGP key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x523E6462
signature.asc
Description: PGP signature
_______________________________________________ poppler mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/poppler
