Hi all, If you ever worked on tests based on our robocop harness, you probably noticed how unpleasant it is to work on them. Our current UI testing code suffers from two major problems:
- Inconsistency/verbosity: The tests and their underlying API are very ad-hoc. We're duplicating a lot of code, the code is very verbose, and the current structure with BaseTest subclasses doesn't scale well and almost inevitably leads to bloated classes with tons of unrelated APIs. - Unreliability: The tests themselves are unreliable in many ways. Each test pokes the UI in a different way to check the same things. I've been doing some explorations around a new API for UI testing in Fennec. Last week, some of us (from the front-end team) met to do some brainstorming and outline how we want the new API to look like. The goals for the new API are pretty obvious. We want UI tests to be: 1. Simple to write and extend 2. Consistently written 3. Reliable I believe the very first step towards these goals is to add structure to our existing UI testing code. I'd like to avoid big rewrites from day one. So our plan is to build the new API *on top* of the current robocop harness and *in parallel* to existing UI tests. This way can gradually build the new API and port existing UI tests without causing any big disruptions. The new structure won't necessarily help in terms of reliability, but the new API should indirectly help us in this front by providing a more consistent code base to work on. The general principles of the new API are: - It should be split into self-contained components covering different parts of the UI. - It should provide clear ways to extend the framework to cover new UI components and/or behaviours. - Each component should contain API to interact with and assert expected behaviour on specific parts of the UI. - Components should be modular enough that we can easily use multiple components in a single test. With that mind, I've just posted a patch that bootstraps the new API (see bug 910859). A few implementation details: - Note how I haven't changed any of the existing tests or harness utils classes. Remember, we're building the new API in parallel. We can probably replace or rewrite the harness code later if we need. But that's not necessary at this point. - Tests using the new API should inherit from UITest, which is completely independent from the BaseTest-based stuff. - All assertions are done using FEST-Assert and FEST-Android. They provide a fluent API for assertions and a well defined framework to create our custom assertions when necessary. - For now, I only bootstrapped enough of the TOOLBAR and ABOUTHOME components to give a more concrete idea of what the API should look like. - UI testing components have a fluent/chainable API and implement their own FEST assertions. - Reusable testing bits were consolidated into helper classes. For now, I've only added ViewHelper, GestureHelper, WaitHelper, and UITestCommon. We'll probably create more as we go. - I wrote two sample tests (testNewTabHistory and testAboutHomeVisibility) just to give a clearer sense of how the API would be used in tests. They don't need to be pushed to repo just yet. - Reflection code is now done using FEST-Reflect. It provides a fluent API for reflection. Much cleaner. Knows issues and open questions: - Using FEST means we won't be doing assertions using FennecMochitestAssert or FennecTalosAssert. I noticed that these classes do a whole bunch of logging (i.e. one log message per assertion). Not sure how important this is. I'd like to understand the logging requirements a bit better here to adapt the new API accordingly. - A whole bunch of things that I haven't considered yet based on your feedback :-) The patch in bug 910859 only implements the bare minimum API to get things started. We'll have to extend it as we try to port existing tests to the new structure. Chenxia filed meta bug 910791 to track our work. Once my patch in bug 910859 lands, we can start working in different components and tests in parallel. The patch is not yet ready for review but definitely ready for feedback. Drive-by reviews are very welcome. Cheers! --lucasr _______________________________________________ mobile-firefox-dev mailing list [email protected] https://mail.mozilla.org/listinfo/mobile-firefox-dev
