> On Sep 22, 2016, at 17:27, Aki Sasaki <asas...@mozilla.com> wrote:
> If we compile locally before testing in automation, we're no longer testing 
> the same bits that we're going to be shipping.  That's a major change in 
> direction, and one that may have significant consequences.

We would only compile tests in test jobs: the binaries we ship to users would 
be downloaded from a build job.

> An additional historical reason for keeping development tools off test 
> machines: we once shipped a new runtime dependency that we didn't detect 
> because the test machines all had development environments.  Users on that 
> platform were unable to run Firefox without installing a 3rd party library.

It would be easy to add a test that verifies shared library dependencies 
against a whitelist. I'm, uh, kinda surprised we don't have that already.

> I very much agree with the goal of simplifying and streamlining how we run 
> tests.  I think we do have to keep the above in mind when we go about that 
> simplification.
>> On Thu, Sep 22, 2016 at 3:54 PM, Gregory Szorc <g...@mozilla.com> wrote:
>> On Thu, Sep 22, 2016 at 2:49 PM, Nicholas Alexander <nalexan...@mozilla.com>
>> wrote:
>> >
>> >
>> > On Thu, Sep 22, 2016 at 1:20 PM, Gregory Szorc <g...@mozilla.com> wrote:
>> >
>> >> For years it has been a common pattern for local builds and Firefox
>> >> automation to copy test files to the object directory (or some other
>> >> staging area) and run them from there. This creates several 
>> >> inefficiencies:
>> >>
>> >> * The build system must symlink or copy thousands of test related files
>> >> during the build
>> >> * Running tests requires checking if test files in the objdir are up to
>> >> date (can add significant latency to the edit-test loop)
>> >> * Builds in automation must create and upload archives containing
>> >> thousands of test files
>> >>   * This makes it harder to skip build jobs if all that has changed is a
>> >> test file
>> >> * Test jobs in automation must download and extract archives containing
>> >> test files
>> >> * Test jobs in automation typically don't have access to a full source
>> >> checkout and can't easily piggyback on existing infrastructure (such as
>> >> mach commands, vendored Python packages, etc). This leads to wheel
>> >> reinvention and inconsistency between local dev environments and 
>> >> automation.
>> >>
>> >> The historical reasons for doing things this way were valid. But with
>> >> moz.build, other improvements to the build system, automation scheduling
>> >> and configuration living in-tree, and better scaling of Mercurial in
>> >> automation, these historical reasons are largely no longer valid.
>> >>
>> >> Going forward, I'm requesting we change our policy regarding tests and
>> >> automation to be source checkout first. This means:
>> >>
>> >> * Tests should be designed to run from a source checkout with minimal
>> >> "build" actions required (no file copying/symlinking, no preprocessing 
>> >> into
>> >> a new file, etc)
>> >> * Automation should run everything from a source checkout (as opposed to
>> >> downloading archives containing files derived from source checkouts)
>> >>
>> >> Does anyone have any concerns or objections to this?
>> >>
>> >
>> > This only makes sense for a subset of our harnesses.  Tests that need to
>> > be compiled (gtest?, Android unit and Robocop tests) can never achieve
>> > this.  So as long as we're clear that we want to remove preprocessing and
>> > special mozharness-foo to work around packaging tests but we're not pushing
>> > *everything* to source-only, I'm good.
>> >
>> The goal is to reduce the amount of work needed to run tests. You are
>> correct that compiled tests inherently need a build step to run them and we
>> can't simply run them from a source checkout.
>> That being said, it isn't out of the question to defer the compilation of
>> these tests to test jobs: the build job delays execution of nearly all
>> tests and making the build job faster by moving work out of that job could
>> make end-to-end times faster, even if we're doing extra work in individual
>> test jobs (that have to compile tests). Historically, we didn't have a
>> compiler available on test machines. But with TaskCluster+Docker and
>> tooltool, it is much easier to put needed files on testers. So compiling
>> tests on test machines is something we should consider.
>> _______________________________________________
>> tools mailing list
>> to...@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/tools
dev-builds mailing list

Reply via email to