This is a great summary, and reflects a ton of hard work over the past year to improve our mobile testing story and reduce our CI spend. Thanks gbrown and everyone else who helped make it happen!
On Fri, Oct 18, 2019 at 11:52 AM Geoffrey Brown <gbr...@mozilla.com> wrote: > The Android test environments used for continuous integration have been > through many changes over the last year or two; here's a review of what we > have today. [1] > > Most of our Android tests run on emulators. Some run on hardware: real > phones. > > Our Android hardware tests run on physical devices -- Motorola G5 and Pixel > 2 phones currently -- and those phones are physically managed by bitbar, a > device farm provider. These test platforms appear on treeherder as "Android > 7.0 MotoG5" and "Android 8.0 Pixel2". Running tests on hardware is > relatively expensive so we make deliberate choices about which tests run on > hardware. All of our performance (raptor) tests, architecture-sensitive > tests like jittest and jsreftest, and select tests requiring special > capabilities run on hardware. > > All other tests -- web-platform tests, most mochitests, reftests, xpcshell > tests, etc -- run on emulators. The emulator test platform appears on > treeherder as "Android 7.0 x86-64". These tests run in the Android x86_64 > emulator on a Linux host. Unlike previous generations of our emulator test > environment, today's emulator tests are fast: Thanks to hardware > acceleration, tests run at about the same rate as they do on our desktop > platforms. > > The Android tests running on trunk today are testing geckoview apps (Fennec > tests continue to run on the esr68 branch). Most raptor tests run in the > geckoview_example app; additional raptor tests run in Fenix and the > Reference Browser; most other tests run in the geckoview test app. > > Both emulator and hardware tests have a fixed pool of instances: Regardless > of load, we can only run N emulator tasks, or M hardware tasks at a time. > Release Engineering Operations monitors backlog for both pools, but > temporary backlogs are expected and tolerated. > > Since our hardware testing capacity is particularly limited, to run Android > hardware tests on try, you must use the --full option with 'mach try fuzzy' > [2]. For instance, you can see the available tests with 'mach try fuzzy > --no-push --full --query "android-hw"' and you could run android-hw > mochitest-media tests with 'mach try fuzzy --full --query "android-hw > mochitest-media"'. > > There are no similar restrictions on try runs for emulator tests -- but > please use responsibly! > > > [1] > > https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=android%2Ctest > [2] https://firefox-source-docs.mozilla.org/tools/try/selectors/fuzzy.html > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
