(tl;dr is at the bottom) Some *background*: our (auth-server) remote tests have always run in such a way that each file full of remote tests starts up several child processes to run an auth-server and other server helper stubs. The tests in the file would make requests against this new flock of children, and then shut them down, before moving on to the next test file.
Some of the *problems *we would encounter because of this setup, by my own rating of importance: - Since the tests took a long time to run (around 4 minutes), I'd find myself trying as much as possible to not run them. Our CI would run them, so they weren't ever forgotten, but a developer being passively discouraged to run tests is not a great situation. - We would get sporadic test failures and time outs due to child process startup woes, and not being able to reflect exceptions in the child processes back to the test runner. It's like screaming children in the line at Disney. - Quite often, the child processes wouldn't be killed when the parent test runner process quit, and then we'd have odd errors when consecutive runs tried to start up processes to listen on an already used port. It's a hard knock life, indeed. - Running the tests was slow, even on CI, so waiting for the green checkmarks meant you needed to go do something else and remember to come back and merge later. (Unfortunately, a lot of the time is spent in `npm install`, so it still takes longer than I'd like.) - Even though we hoped different processes would isolate the tests, we polluted the process `ENV` to customize the children, which had an affect of causing test order to affect whether tests passed in some cases. *Pull Request #1760 <https://github.com/mozilla/fxa-auth-server/pull/1760>* is shutting down the orphanage. It includes some adjustments to the `TestServer` type used in our remote tests. It no longer spawns a bunch of children, but instead just starts up all the servers in the same process. This fixes (or greatly reduces) each of the problems listed above. *We now have the full `npm test` run executing in under 1 minute!* And there are no more dangling orphans left behind at fire stations. Fun little bonus, this even discovered a couple of exceptions that were just silently ignored, due to how Hapi reports them and how we ignored that log output. Thankfully, mostly harmless. *tl;dr* - Running the auth-server tests now starts all needed servers in the same process, improving correctness, reducing coupling between tests, and improving test run speed by around 4x!
_______________________________________________ Dev-fxacct mailing list [email protected] https://mail.mozilla.org/listinfo/dev-fxacct
