> That being said, I have a > few concerns I'd like to voice over the handful of test harnesses that have > been implemented in Rust (I see reftest, contenttest.. are there more?).
reftest and contenttest are the automated ones we have currently aside from unit tests. Only the latter currently runs as a part of continuous integration. reftest requires a few crash-on-exit bugs be fixed before it will get switched on. > 1) Test automation needs to be as rock solid as possible. This will be very > hard to do with a language that hasn't even been released yet. I'm not > knocking Rust, but it is a language that is currently in constant flux. With > test automation the goal is to eliminate as many variables as possible, and > you definitely don't want the language the harness is written in to be a > significant one. Agreed that it needs to be solid. Currently the test harnesses have not yet run into any sporadic failures, and they use the Rust library's test framework so they receive quite a lot of attention. Failing tests block commits landing on Rust and Servo master. The harnesses seem pretty solid so far, and within Rust, the test framework gets quite a lot of use (ie, it's tested regularly on Windows, Mac, Linux, and Android. Not to mention we get parallelism and test sharding for free. > 2) The chicken and the egg. I might be a little off base with this one, but > my understanding is that Servo is like a proof of concept for Rust. In this > sense, Servo is almost one giant test for Rust itself. By using Rust to > implement the harnesses you are in a way using the system under test to test > itself. This seems a bit of a stretch. I mean, lots of our tests and harnesses are in JavaScript too or using features that we are also testing. This seems to also be the case in Firefox. > 3) Long term test automation. Right now Servo is a relatively small project, > so it's ok to take shortcuts that enable you to be efficient in the short > term. But eventually you'll want tests running on official infrastructure, > reporting to something like tbpl. I don't know when this will be, but I do > know that if the Servo team has its own ways of doing things, implements > harnesses in Rust, and generally doesn't take the time to understand how the > existing test infrastructure works, it will take us (the tools and > automation team) and releng a massive amount of extra time to get things > into production (think on the order of quarters if not years). You might not be aware, but we already build and run tests on Mozilla infrastructure set up and maintained by the release engineering team. I think one of the original tasks of that work was to output to tbpl, but since all we needed were build logs at the time, they decided to just open up the buildbot logs to the web. > What does this mean for now? Not much. Go about your day as usual. But if > you do go to implement a new test harness, please consider doing so in > Python (2.7). It's what all our other harnesses, infrastructure, monitoring > tools, slaves and reporting depend on. If you aren't very familiar with > Python and need a harness implemented yesterday, by all means use Rust. Just > be aware it could cause you some problems down the road. If you have > concerns about using Python over Rust, I'd love to hear why. We're pretty open to input on how best to test Servo. Perhaps it would be a good idea to have a meeting to chat about what your team needs and would like to see in a future where you're testing Servo, and about what our testing roadmap looks like. I'd love to figure out how to start collaborating with your team. We're really happy with our CI setup and the releng team's help with that, and if we can leverage more Mozilla resources on other teams, we'll be even happier. jack. _______________________________________________ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
