----- "Ryan Harper" <[email protected]> wrote: > > >- guest install wizard using md5sum region matching ... ouch. This > is > > >quite fickle. I've seen different kvms generate different md5sum > for > > >the same region a couple of times. I know distributing screenshots > of > > >certain OSes is a grey area, but it would be nice to plumb through > > >screenshot comparison and make that configurable. FWIW, I'll > probably > > >look at pulling the screenshot comparison bits from kvmtest and > getting > > >that integrated in kvm_runtest_2. > > Creating a step file is not as easy as it seems, exactly for that > reason. > > One has to pick a part of the screenshot that only available when > input is > > expected and that would be consistent. We were thinking of replacing > the > > md5sum with a tiny compressed image of the part of the image that > was > > picked. > > It isn't just that step file creation isn't easy is that even with a > good stepfile with smart region boxes, md5sum can *still* fail > because > KVM renders one pixel in the region differently than the system where > the > original was created; this creates false positives failures.
I'd like to comment on this. I don't doubt that some fuzzy matching algorithm (such as calculating match percentages) would generally be more robust. I do however doubt it would significantly lower the false positive rate in our case (which is fairly low already). False positive failures in step files are typically caused by: - an unexpected popup window covering the test region - a dialog which has a different position every time (and varies by many pixels) - a barrier that passes before the controls get input focus, which causes the following keystrokes to have no effect - in text mode, sometimes a line of text is printed unexpectedly and causes the entire screen to scroll up - addition/modification of a KVM feature which changes the course of the installation I may have left something out. In any case, all these problems are solved by picking better barrier regions, but none can be solved by using a more forgiving comparison method. I have encountered a single installation that rendered a single pixel in an indeterministic fashion, and though this problem was easily solved by correcting the barrier (using a stepfile editor), I do agree we might get a small decrease in the false positive rate if we use a more forgiving algorithm. However, there is also a risk: a more forgiving algorithm may forgive KVM for rendering errors. It may also make it risky to pick barriers that are meant to catch small text; I believe a button with a "Yes" caption and a button with a "No" caption would have a very high match percentage, especially if you have to pick the whole button, or maybe even some of its surroundings (and you often do). I still believe it's a good idea to look into other methods (we're already doing that) and start experimenting with them. Thanks, Michael -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
