On Thu, Sep 22, 2016 at 2:49 PM, Nicholas Alexander <nalexan...@mozilla.com>
> On Thu, Sep 22, 2016 at 1:20 PM, Gregory Szorc <g...@mozilla.com> wrote:
>> For years it has been a common pattern for local builds and Firefox
>> automation to copy test files to the object directory (or some other
>> staging area) and run them from there. This creates several inefficiencies:
>> * The build system must symlink or copy thousands of test related files
>> during the build
>> * Running tests requires checking if test files in the objdir are up to
>> date (can add significant latency to the edit-test loop)
>> * Builds in automation must create and upload archives containing
>> thousands of test files
>> * This makes it harder to skip build jobs if all that has changed is a
>> test file
>> * Test jobs in automation must download and extract archives containing
>> test files
>> * Test jobs in automation typically don't have access to a full source
>> checkout and can't easily piggyback on existing infrastructure (such as
>> mach commands, vendored Python packages, etc). This leads to wheel
>> reinvention and inconsistency between local dev environments and automation.
>> The historical reasons for doing things this way were valid. But with
>> moz.build, other improvements to the build system, automation scheduling
>> and configuration living in-tree, and better scaling of Mercurial in
>> automation, these historical reasons are largely no longer valid.
>> Going forward, I'm requesting we change our policy regarding tests and
>> automation to be source checkout first. This means:
>> * Tests should be designed to run from a source checkout with minimal
>> "build" actions required (no file copying/symlinking, no preprocessing into
>> a new file, etc)
>> * Automation should run everything from a source checkout (as opposed to
>> downloading archives containing files derived from source checkouts)
>> Does anyone have any concerns or objections to this?
> This only makes sense for a subset of our harnesses. Tests that need to
> be compiled (gtest?, Android unit and Robocop tests) can never achieve
> this. So as long as we're clear that we want to remove preprocessing and
> special mozharness-foo to work around packaging tests but we're not pushing
> *everything* to source-only, I'm good.
The goal is to reduce the amount of work needed to run tests. You are
correct that compiled tests inherently need a build step to run them and we
can't simply run them from a source checkout.
That being said, it isn't out of the question to defer the compilation of
these tests to test jobs: the build job delays execution of nearly all
tests and making the build job faster by moving work out of that job could
make end-to-end times faster, even if we're doing extra work in individual
test jobs (that have to compile tests). Historically, we didn't have a
compiler available on test machines. But with TaskCluster+Docker and
tooltool, it is much easier to put needed files on testers. So compiling
tests on test machines is something we should consider.
dev-builds mailing list