Hey Vladimir, 2013/3/13 Michael Meeks <michael.me...@suse.com>: > Hi Vladimir, > > First - great to see you here :-) > > On Wed, 2013-03-13 at 16:17 +0100, Vladimir Benes wrote: >> we were discussing recently how to improve testing of LO's GUI. We found >> out that situation if more than bad.
So I agree here with you but sadly I currently have no good idea how to improve that. One idea I had a bit more than one year ago was to use the uno commands for menu entries and an annotaion in the src files for ui elements but I discarded it for several reasons. However with the ui files it might finally be easily possible to follow this idea with the ui files. But please explore all of the concepts that you can think of. Our codebase is so complex that some may work and others will reveal either problems in our code or in the test concept. I'm happy to help with any problems you have in implementing new test concepts. > >> There are several options how to approach this. The first is LDTP [1] >> and the other one (in my opinion far better) is Dogtail [2]. If LO uses >> standard gtk widgets so they're a11y friendly we can easily access them. >> This works for accessing menu items, dialogs and so on. > > We used to have a huge chunk of code that did this GUI testing. The > general consensus is/was that it is unreliable, and broadly worthless - > breaking on minor UI changes, requiring lots of maintenance, incredibly > slow to run, etc. We removed it all :-) I agree with Michael that the old GUI testing code was horrible and unreliable and my big fear is that our horrible accessibility code will not make it easier to use existing GUI testing frameworks but I'm happy if someone prooves me wrong. I have no idea how IA2 and the new widget dialogs play into that. Michael and Caolan may know better which consequences they have on our accesibility code. > >> For comparing visual things like pictures, font alignment, etc there is >> Sikuli [3] framework actively developed at MIT that can be used for this. > > IMHO it would be far more useful to develop unit tests that are run > during the build; and/or to write load/save/load tests such that we > check that our filters are not loosing data left/right as they load and > save :-) We have a lot of such tests already, we also have some layout > test code that dumps the layout tree in writer to try to check that the > output is the same, and more pieces in that vein. > >> Is there anything where we can start or at least help with? What I would >> expect is at least some priority one functionality that should be >> covered first or so. So a few things on my list of untested functionality that is not related to GUI testing and has higher priority on my list: * Copy & paste * Draw is totally untested * Improved export tests: optional validation of the exported files (ODF/OOXML) * Porting the failing API tests to C++ and re-enable them * Exploring and integrating test documents from different providers: gnumeric, abiword, calligra, OpenOffice * Performance tests > > Caolan prolly has some more good ideas around what is needed here; and > we could use more man-power and build-hardware of course. Markus is an > expert in the unit testing area (and elsewhere) and has a rather nice > tool to run a ton of bug-documents through the code: it'd be great to > get that hosted, running regularly, and the bugs filed / fixed alongside > the tinderboxes. That is mostly blocked on my side. There are a few nasty memory bugs also in my local build that either show another memory corruption in our code/toolchain or an error in my concept. I hope to find some free time soon to look into these reports. > > Then of course, there is running that same (15k document) test suite > regularly under valgrind - that would also be useful ;-) > > So - IIWY I'd do GUI testing as the very last thing in the stack of > other more useful things that can be invested in to improve quality. > I'm a bit more enthusiastic about the idea of GUI testing but there must be someone who explores the different ideas and implements a prototype. It took us around half a year until the propotype for import tests was copied by other developers and found widespread use in Libreoffice. So in my opinion the first step is to explore all of the ideas and implement a reliably working prototype that can be used to promote the idea. I know that it takes time to implement and promote a new testing idea and I hope that GUI testing will help to have a testing concept for more of our fixed bugs. So if you want to spend a bit of your time on exploring the different concepts you mentioned you have my support and I'll try to help wherever I can. Regards, Markus _______________________________________________ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
