Hi Robert,
this is a great plan.
The only issue is that we are also facing this kind of pixel differences
- between different releases of Qt: for example a Qt 4.6.2 build has
~300 failing tests on the bot which pass on a 4.6.1 build,
- between different distributions: our buildbot runs Debian Lenny but
developers frequently note that they have 300-500 different results on
other distributions or other versions of the same distributions, and we
also experience this problem on our ARMv5 tester, which might or might
not be related to package version differences too.
I'm strongly in favor of unskipping these tests if they are stable
enougth (IIRC Ossy has a list somewhere), and rebaslineing the results,
but we also have to take in account that if we do not find the cause of
the differences between environments or it is not a bug of Qt (which I
suppose) we have to update a whole lot of test results each time we
switch Qt versions, and the landed results would only be relevant on our
buildbot which would be an awkward thing. These tests would also need
comparing of pixel results to the Mac ones because the DRT dumps are not
always reliable. Pixel testing is a time consuming activity.
Another partly relevant thing is that we were thinking about an tool,
which would automatically run all the tests in the skipped list say once
a week, to check if there are possible changes, but there are a few
blocker tests which result in run-webkit-tests infinite loop. I took a
note somewhere, I'll check and file a bug as a starting point. Fixing
these would make some automation possible and ease the further actions
on the skipped list.
BR,
Andras
2010-04-18 23:30 keltezéssel, Robert Hogan írta:
There is a definite class of tests in the Qt skipped list that fail only
because of unimportant pixel differences.
I propose to open a master bug for these the purpose of which will be to
act as a staging pad for removing them from the Skipped list.
I would see it working like this:
- Identify a group of tests that only fail due to pixel differences and for
which it is clear pixel differences are not material
- Add a patch against LayoutTests/platform/qt/Skipped with the tests
removed.
- Along with the patch add a comment to bugzilla which will act as a a
'manifest' listing each test to be removed and why it is safe to use
platform specific results. It might even be best to keep this manifest as a
file in the platform/qt tree.
- Send a note to the buildbot team requesting them to try the patch.
- Wait for a member of the buildbot team to run the patch against the
buildbot. The buildbot team can then post an updated patch with the changes
to the Skipped list and the platform specific results. They mark it for
review.
- A reviewer comes along, satisifies themselves that platform specific
results are justified for each of the skipped tests and approves.
Keeping all this under a single bugzilla entry might get cumbersome after a
while, when that happens a new one can be created to replace it and linked
from the old one. I think it's important to have one go-to place for this
effort though.
Thoughts? Do the buildbot team this is something that could work?
_______________________________________________
webkit-qt mailing list
[email protected]
http://lists.webkit.org/mailman/listinfo.cgi/webkit-qt
_______________________________________________
webkit-qt mailing list
[email protected]
http://lists.webkit.org/mailman/listinfo.cgi/webkit-qt
