https://bugzilla.wikimedia.org/show_bug.cgi?id=62633

            Bug ID: 62633
           Summary: Jenkins: Set up perceptual diffs
           Product: Wikimedia
           Version: wmf-deployment
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: Unprioritized
         Component: Continuous integration
          Assignee: wikibugs-l@lists.wikimedia.org
          Reporter: krinklem...@gmail.com
                CC: has...@free.fr, innocentkil...@gmail.com,
                    krinklem...@gmail.com
       Web browser: ---
   Mobile Platform: ---

This is something I've been experimenting with in my spare time for a while and
think it's time to do this.

The idea is: Render a visual diff for one or more urls (comparing the change to
the latest version of the target branch).

Background:
* https://github.com/uber/image-diff/
* https://github.com/bslatkin/dpxdt/
* Talk (Velocity 2013): https://www.youtube.com/watch?v=1wHr-O6gEfc
* Talk (Google Developers): https://www.youtube.com/watch?v=UMnZiTL0tUc

Behind it all is basically just a ImageMagick compare command[1][2][3][4]
between two PNGs.

The PNGs would be created using PhantomJS' render() API.

Jenkins job macro for this:

* For core and extensions, this would be installing MediaWiki and exposing it
in the local Apache, for client-side libraries like VisualEditor and OOjs UI,
this is just running a build script and exposing their demo/ directory in the
local Apache

* Create PNGs for all the configures urls (relative paths to their webroot,
we'd prefix our apache server + unique subdirectory to this).

* Compare them against the ones from the previous post-merge job (stores
somewhere persistently by repository/branch/url).

* If different, make sure the latest.png/change-after.png/change-diff.png for
that url is kept and stored as build artefacts (we should clear them after some
time as they'll grow quickly, as optimisation we can delete them within the job
if there is no diff).

From the post-merge job:

* Run the main macro.
* Copy the created screenshots to the persistent "latest" directory for
comparison by the next unmerged patch set test.


--------

A few example urls for different repositories that we would want visual
regression testing.


mediawiki-core:
  /index.php?title=Main_Page
  /index.php?title=Main_Page&useskin=monobook
  /index.php?title=Main_Page&action=edit
  /index.php?title=Main_Page&action=history
  /index.php?title=Special:UserLogin
  /index.php?title=Special:UserLogin/signup
  /index.php?title=Special:Search&search=wiki

VisualEditor:
  /demos/ve/#!/src/pages/empty.html
  /demos/ve/#!/src/pages/simple.html
  /demos/ve/#!/src/pages/complex.html

oojs-ui:
  /demos/icons.html
  /demos/widgets.html


[1]
https://github.com/bslatkin/dpxdt/blob/8e76f62e5/dpxdt/client/pdiff_worker.py#L115-L127
[2] https://github.com/uber/image-diff/blob/1.0.1/lib/image-diff.js#L53-L65
[3] http://www.imagemagick.org/Usage/compare/
[4] compare
      -verbose
      -metric RMSE
      -highlight-color RED
      -compose Src
      mytest-latest.png
      mytest-build.png
      mytest-diff.png

-- 
You are receiving this mail because:
You are the assignee for the bug.
You are on the CC list for the bug.
_______________________________________________
Wikibugs-l mailing list
Wikibugs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l

Reply via email to