On 4 Jul 2014, at 14:05, Petr Slabý <sl...@kadel.cz> wrote: > Hi, > following is a description of what we are doing in our company. > > With our software, we run regression tests after each nightly build and > sometimes it is a tough fight. If there is a regression, it is not so easy to > find which commit caused it, because there are potentially many between the > nightly builds.
Our Jenkins build should run after each commit, so that will simplify things a bit. Sometimes it doesn’t but we also have TravisCI which is always does. > Then, the decision whether the change is wanted and expected is in some cases > also difficult (this part might be easier with PDF where there is the "golden > standard" rendering in Acrobat). Yes, Acrobat is the answer here for PDF. Most of the time the decision should be straightforward. > If the change is expected and the new rendering "better" then one has to > commit the new reference. This means that the files produced on the nightly > build machine must be available somehow - it is almost impossible to produce > them locally as the rendering results are slightly different with different > versions of java and many other reasons. Yes, I really want to get local renderings working if possible. That might include some basic restrictions on which JVMs can be used (my “blessed JVM proposal) but also introducing some fuzziness into the image comparisons, perhaps allowing a small per-pixel error. I’m hoping that once AWT rendering of fonts is removed that we’ll see more consistent rendering across JVMs. > All this has to be done before the next regression test is run to avoid that > new regressions are hidden by earlier ones. Our complete build with all tests > runs several hours… > > To improve this workflow, we now use the following schema in addition: > - there is a smaller set of regression tests which runs relatively fast > - these tests are triggered by each commit in formatting and rendering > related projects > - before running the test itself, the modified project(s) are compiled > locally, w/o publishing the result to maven > - the reference rendering files are stored in SVN > - if a test finds a regression, it immediately stores the new result as a new > reference into SVN. This makes sure that a) the test renderings do not get > lost and b) that each regression exactly points to the commit that has caused > it - the one that triggered the test. The failed test creates a new issue in > JIRA with a pointer to SVN to the before and after rendering and a bitmap of > the differencies. The issue is then processed. If we find the change to be > expected then the issue is simply closed, otherwise we take actions to fix > the problem. The only annoying thing about this scheme is that, after > commiting the correction, the test runs again and reports a regression > because it now compares to the faulty version of the rendering. That sounds fairly similar to my proposal, I like the aspect of pushing the server build’s PNGs to SVN. If we can’t get robust local rendering to work then that sounds like a good way to make the images easily available. Thanks -- John
