Re: To what extent is sccache's distributed compilation usable?

[Christopher Manchester]

As announced in this week's project call, sccache-dist schedulers have been
deployed to mozilla offices, and those with hardware available can enroll
servers based on the documentation here
.
That document also contains instructions for building as a client after
servers have been enrolled.

We're still actively working on improvements, but early testing has proved
the current setup to be usable and provide compelling build time
improvements.
Visit https://github.com/mozilla/sccache/issues for open issues or to file
as they come up. Issues specific to integration with Mozilla's build system
are tracked by bug 1439584
.

Please join #sccache on Slack for questions or discussion.

Chris



On Fri, Jun 7, 2019 at 10:13 AM Chris M.  wrote:

> I've been working through issues with distributed sccache, but I've
> recently sent instructions to people currently running icecream in the SF
> office on how to switch to running sccache-dist with IT managed hardware.
> We're going to use that as a test bed before rolling out more widely, and
> if all goes well there this should happen shortly (likely shortly after
> returning from Whistler).
>
> Regarding that bug, that's tracking IT work for the most part so probably
> needs to stay restricted. Most of the work I've been doing is tracked in
> sccache's github repo and can be followed there, or tracked by
> https://bugzilla.mozilla.org/show_bug.cgi?id=1499145.
>
> Chris
>
> On Wed, Jun 5, 2019 at 1:52 AM Miko Mynttinen 
> wrote:
>
>> Has there been any progress with this? Bug 1524533 is still restricted.
>>
>> Miko
>>
>> > On Apr 2, 2019, at 10:39 PM, Jeff Muizelaar 
>> wrote:
>> >
>> > That bug is restricted. Does it need to be?
>> >
>> > -Jeff
>> >
>> > On Tue, Apr 2, 2019 at 4:22 PM Kim Moir  wrote:
>> >>
>> >> The tracking bug for sccache being deployed to offices is here
>> >>
>> >> https://bugzilla.mozilla.org/show_bug.cgi?id=1524533
>> >>
>> >> Kim
>> >>
>> >> On Tue, Apr 2, 2019 at 3:22 AM Frederik Braun 
>> wrote:
>> >>
>> >>> Am 01.04.19 um 22:16 schrieb Chris M.:
>>  Hi Emilio, if you're interested you're encouraged to try it out,
>> however
>>  we're shipping machines to offices now to run the schedulers, the
>> first
>> >>> of
>>  which I'll be testing in the SF office this week, so we're planning
>> an
>>  officially supported setup in the near future.
>> 
>> >>>
>> >>> That's great news. Can we follow progress or detailed plans somewhere?
>> >>> ___
>> >>> dev-platform mailing list
>> >>> dev-platform@lists.mozilla.org
>> >>> https://lists.mozilla.org/listinfo/dev-platform
>> >>>
>> >> ___
>> >> dev-platform mailing list
>> >> dev-platform@lists.mozilla.org
>> >> https://lists.mozilla.org/listinfo/dev-platform
>> > ___
>> > dev-platform mailing list
>> > dev-platform@lists.mozilla.org
>> > https://lists.mozilla.org/listinfo/dev-platform
>>
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Proposal: build-id generation in developer builds

[Christopher Manchester]

Our build-id is currently compiled in to libxul, and is re-generated with
every build invocation. To avoid re-linking libxul with every build
invocation we have some Makefile hacks in place to hide the build
dependency between libxul and the buildid.

As part of improving the build system we'd like to fix this, and the best
proposal we have so far would involve no longer re-generating the build-id
with each build invocation for developer builds. Automation builds would
still generate a new build-id with every build.

If you think this would interfere with your workflow, I'd like to hear from
you. Please reply here or in
https://bugzilla.mozilla.org/show_bug.cgi?id=1298328 .

Thanks,

Chris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Build System Project - Update from the last 2 weeks

[Christopher Manchester]

Hi Armen,

Thanks for noticing that, I think you found an issue with how we're
submitting data caused by something I landed last week. I filed
https://bugzilla.mozilla.org/show_bug.cgi?id=1266183 to track a fix.

Chris

On Wed, Apr 20, 2016 at 12:01 PM, Armen Zambrano G. 
wrote:

> Do we know what happen with the data points of the end of the graph?
>
>
> https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000=%5Bmozilla-inbound,04b9f1fd5577b40a555696555084e68a4ed2c28f,1%5D=%5Bmozilla-inbound,65e0ddb3dc085864cbee77ab034dead6323a1ce6,1%5D=%5Bmozilla-inbound,c0018285639940579da345da71bb7131d372c41e,1%5D=1460493349005.865,1461176501000,0,1
>
> If I read it correctly it feels that we're having this week less pgo
> builds (2/day versus 9-10/day)
>
> This link without opt makes it more obvious:
>
> https://treeherder.mozilla.org/perf.html#/graphs?series=%5Bmozilla-inbound,c0018285639940579da345da71bb7131d372c41e,1%5D=%5Bmozilla-inbound,04b9f1fd5577b40a555696555084e68a4ed2c28f,0%5D=%5Bmozilla-inbound,65e0ddb3dc085864cbee77ab034dead6323a1ce6,1%5D
>
> regards,
> Armen
>
>
> On 16-04-20 11:44 AM, Gregory Szorc wrote:
>
>>
>>
>> On Apr 20, 2016, at 08:16, Nicolas B. Pierron <
>>> nicolas.b.pier...@mozilla.com> wrote:
>>>
>>> Unrelated, Do we have news for clang blockers on Windows?  In
>>> particular, I am thinking about the various Sanitizers.
>>>
>>
>> We haven't really talked about Clang on Windows in our build
>> meeting/plannings. That's not to say someone else hasn't been working on
>> it. But I haven't seen much bug traffic indicating that's the case.
>>
>> If you make a case for Clang on Windows improving developer productivity
>> or improving stability, that's how you get something prioritized. Can you
>> start a thread on dev-builds listing the benefits?
>>
>>
>>> On 04/20/2016 11:00 AM, David Burns wrote:
 We have also started looking at how we can use a global compiler cache
 on
 local builds and not just in automation. This will allow artifact-like
 builds for those who are doing C++ development.

>>>
>>> I am particularly scared about this topic for multiple reasons,
>>> including that my system does not have a /lib directory.  Is there a
>>> location where I can learn more and contribute back to this?
>>>
>>> --
>>> Nicolas B. Pierron
>>> ___
>>> dev-platform mailing list
>>> dev-platform@lists.mozilla.org
>>> https://lists.mozilla.org/listinfo/dev-platform
>>>
>>
>
> --
> Zambrano Gasparnian, Armen
> Automation & Tools Engineer
> http://armenzg.blogspot.ca
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: PSA: Requiring annotations for test support files required by multiple test directories

[Christopher Manchester]

This isn't an explicit goal for now, although where this is possible it can
and should be done. When the patches in the bug I mentioned land, support
files appearing alongside individual tests rather than in the DEFAULT
section will result in only the support files for requested tests being
installed.

For now, the main benefit will be going from installing/syncing every test
and support file when any test is run (tens of thousands of files) to doing
so a directory at a time in the typical case that support files are listed
in the DEFAULT section of the manifest.

Chris

On Tue, Mar 29, 2016 at 1:12 PM, Jared Wein <j...@mozilla.com> wrote:

> Is the goal moving forward to have "support-files" annotations alongside
> the test that requires them instead of global to the manifest file?
>
> Will this allow a further optimization of only installing the support-file
> for tests are being requested?
>
> Thanks,
> Jared
>
> On Tue, Mar 29, 2016 at 12:55 PM, Christopher Manchester <
> chmanches...@gmail.com> wrote:
>
>> As you may know, manifestparser test manifests (mochitest.ini,
>> browser.ini and co.) contain a "support-files" field informing the build
>> system certain files in the tree or generated at build time are required by
>> tests. These files are currently installed to the objdir for some test
>> flavors and incorporated into test archives.
>>
>> Currently, every support and test file is installed whenever tests are
>> run, and many tests rely on support files in far away corners of the tree.
>> We're working to improve our granularity here, so going forward we need to
>> annotate a support file required by a test if that support file is in a
>> different directory.
>>
>> I've prepared a patch to annotate existing dependencies, which I intend
>> to land in bug 1242051. For details on the syntax used, please see that bug
>> and the corresponding update to our in-tree documentation.
>>
>> Chris
>>
>> ___
>> firefox-dev mailing list
>> firefox-...@mozilla.org
>> https://mail.mozilla.org/listinfo/firefox-dev
>>
>>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

PSA: Requiring annotations for test support files required by multiple test directories

[Christopher Manchester]

As you may know, manifestparser test manifests (mochitest.ini, browser.ini
and co.) contain a "support-files" field informing the build system certain
files in the tree or generated at build time are required by tests. These
files are currently installed to the objdir for some test flavors and
incorporated into test archives.

Currently, every support and test file is installed whenever tests are run,
and many tests rely on support files in far away corners of the tree. We're
working to improve our granularity here, so going forward we need to
annotate a support file required by a test if that support file is in a
different directory.

I've prepared a patch to annotate existing dependencies, which I intend to
land in bug 1242051. For details on the syntax used, please see that bug
and the corresponding update to our in-tree documentation.

Chris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

GTests are now running on test machines

[Christopher Manchester]

GTests are now run from their own job on Treeherder instead of during the
build in "make check'. They can be found under the "GTest" symbol on
treeherder on all integration branches, and can be added to try pushes with
"-u gtest" (they will run with "-u all").

As an effect of this change we're now dumping symbols for the GTest libxul,
so crash stacks should have symbols (symbols for assertion stacks
forthcoming).

Chris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Prioritizing tests based on file changes

[Christopher Manchester]

Some tests are more likely than others to fail in response to a given
change, but our automation does little to optimize based on this. As a
result, test automation frequently runs more than necessary for a given
push, leading to excess load on automation and time spent waiting for the
results of tests unlikely to be impacted by a given push.

Work is ongoing on an approach to select/prioritize tests based on the
files that a push changes. The idea is to use build system metadata to map
changed files from source control to impacted tests. This will be based on
the test files included in manifests mentioned in moz.build files along the
path to a given file, and a syntax to specify other relationships between
source files and tests. The syntax isn’t finalized, but the proposal is
that an annotation such as:

with Files(‘testing/mozbase/**’):

   IMPACTED_TESTS += [

   ‘testing/mochitest/tests/**’,

   ]

in a moz.build file will suggest that any file change under
‘testing/mozbase’ is potentially relevant to the mochitest sanity tests.
Specifying tests by tag or test flavor (so you can run all reftests when
parts of layout change, for instance) will be supported in addition to
wildcard file matching.

Tools will use this data to run a subset of tests either before or in place
of a full set of tests. Initially this will be an option for |./mach try|
and |./mach test|, and upcoming work will make it possible to take
advantage of this on integration branches and in coordination with
autoland. There’s also potential to refine and automate this process
further with auxiliary sources of data (such as code coverage), and longer
term I’d like to see annotations become a supplement to a more automatic
prioritization algorithm.

This isn’t intended to be a complete solution, or to replace the current
pipeline: there will always be interactions between code and tests that
can’t reasonably be anticipated. It is, however, an optimization that has
the potential to shorten feedback cycles on Try, help improve integration
tree availability by shortening time to detect regressions, and reduce
overall infrastructure load.

If you’d like to follow this or give feedback on the proposed syntax and
heuristics, initial work is tracked in bug 1184405.


Chris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: It is now possible to apply arbitrary tags to tests/manifests and run all tests with a given tag

[Christopher Manchester]

This landed as the mach try command. For more details run |./mach help try|
(excerpted below) and see the discussion in bug 1149670. The highlights
here are partly automated try syntax generation for testing changes
relevant to tests in certain directories, and pushing try syntax to the try
server without the benefit of mq.

Chris



Pushes the specified tests to try. The simplest way to specify tests is
by using the -u argument, which will behave as usual for try syntax.
This command also provides a mechanism to select test jobs and tests
within a job by path based on tests present in the tree under that
path. Mochitests, xpcshell tests, and reftests are eligible for
selection by this mechanism. Selected tests will be run in a single
chunk of the relevant suite, at this time in chunk 1.

Specifying platforms is still required with the -p argument (a default
is taken from the AUTOTRY_PLATFORM_HINT environment variable if set).

Tests may be further filtered by passing one or more --tag to the
command.

To run suites in addition to those determined from the tree, they
can be passed to the --extra arguent.

The command requires either its own mercurial extension (push-to-try,
installable from mach mercurial-setup) or a git repo using git-cinnabar
(available at https://github.com/glandium/git-cinnabar).

Global Arguments:
  -v, --verbose Print verbose output.
  -l FILENAME, --log-file FILENAME
Filename to write log data to.
  --log-intervalPrefix log line with interval from last message
rather
than relative time. Note that this is NOT execution
time if there are parallel operations.
  --log-no-timesDo not prefix log lines with times. By default, mach
will prefix each output line with the time since
command start.
  -h, --helpShow this help message.
  --debug-command   Start a Python debugger when command is dispatched.

Command Arguments:
  pathsPaths to search for tests to run on try.
  -n   Print detailed information about the resulting test
   selection and commands performed.
  -p PLATFORMS Platforms to run. (required if not found in the
   environment)
  -u TESTS Test jobs to run. These will be used in place of
suites
   determined by test paths, if any.
  --extra EXTRA_TESTS  Additional tests to run. These will be added to
suites
   determined by test paths, if any.
  -b BUILDSBuild types to run (d for debug, o for optimized)
  --tag TAGS   Restrict tests to the given tag (may be specified
   multiple times)
  --no-pushDo not push to try as a result of running this
command
   (if specified this command will only print calculated
   try syntax and selection info).


On Tue, Mar 31, 2015 at 10:57 AM, Christopher Manchester 
chmanches...@gmail.com wrote:

 I filed bug 1149670 for the mach try feature.

 On Tue, Mar 31, 2015 at 10:39 AM, Andrew Halberstadt 
 ahalberst...@mozilla.com wrote:

 It's technically already possible by modifying the in-tree mozharness
 configs here:
 https://dxr.mozilla.org/mozilla-central/source/testing/config/mozharness

 However it's not easy to figure out what needs to be modified to get the
 desired results. Some sort of |mach try| like command is going to be worked
 on in Q2 to make pushing various test configurations to try easier, I'm not
 100% sure how it will be implemented yet though.

 In the meantime, if you get stuck trying to modify the mozharness
 configs, let me know and I can come up with an example patch.

 -Andrew


 On 31/03/15 01:06 PM, Bobby Holley wrote:

 This sounds awesome! Is there an estimate of when we'll be able to use
 it for try pushes?

 On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt 
 ahalberst...@mozilla.com mailto:ahalberst...@mozilla.com wrote:

 As of bug 987360, you can now run all tests with a given tag for
 mochitest (and variants), xpcshell and marionette based harnesses.
 Tags can be applied to either individual tests, or the DEFAULT
 section in manifests. Tests can have multiple tags, in which case
 they should be comma delimited. To run all tests with a given tag,
 pass in --tag tag name to the mach command.

 For example, let's say we want to group all mochitest-plain tests
 related to canvas together. First we'd add a 'canvas' tag to the
 DEFAULT section in
 https://dxr.mozilla.org/mozilla-central/source/dom/
 canvas/test/mochitest.ini

 [DEFAULT]
 tags = canvas

 We notice there is also a canvas related test under dom/media,
 namely:
 https://dxr.mozilla.org/mozilla-central/source/dom/
 media/test/mochitest.ini#541

 Let's pretend it is already tagged with the 'media' tag, but
 that's ok

Re: It is now possible to apply arbitrary tags to tests/manifests and run all tests with a given tag

[Christopher Manchester]

It looks like I inadvertently landed this change on an the wrong branch, so
we aren't pointing to it in production (although we were for a short time
when I tested last week). I'll straighten this out in the morning. Sorry
for the inconvenience.

Chris

On Mon, May 4, 2015 at 4:43 PM, Christopher Manchester 
chmanches...@gmail.com wrote:

 That looks like a valid use of the feature assuming the manifest was
 annotated correctly. I'm not sure what went wrong, I'll reply here when I
 figure out what went wrong or file a bug and investigate further.

 Chris

 On Mon, May 4, 2015 at 4:23 PM, kgu...@mozilla.com wrote:

 On Thursday, April 30, 2015 at 7:22:26 PM UTC-4, Christopher Manchester
 wrote:
  You can now add --tag arguments to try syntax and they will get
 passed to
  test harnesses in your try push. Details of the implementation are in
 bug
  978846, but if you're interested in passing other arguments from try
 syntax
  to a test harness, this can be done by adding those arguments to
  testing/config/mozharness/try_arguments.py. Note this is still rather
  coarse in the sense that arguments are forwarded without regard for
 whether
  a harness supports a particular argument, but I can imagine it being
 useful
  in a number of cases (for instance, when testing the feature with
 xpcshell
  and --tag devtools, I was able to get feedback in about ten minutes
  whether things were working rather than waiting for every xpcshell test
 to
  run).

 Can you provide a sample complete try syntax that uses this? I just did a
 try run with |try: -b do -p linux,macosx64,win32,win64 -u all -t none --tag
 apz| (after adding the apz tag to a mochitest.ini file) and it still ran
 all the mochitests instead of just the ones in the mochitest.ini file. Does
 it only work in some scenarios?

 Cheers,
 kats
 ___
 dev-platform mailing list
 dev-platform@lists.mozilla.org
 https://lists.mozilla.org/listinfo/dev-platform



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: It is now possible to apply arbitrary tags to tests/manifests and run all tests with a given tag

[Christopher Manchester]

That looks like a valid use of the feature assuming the manifest was
annotated correctly. I'm not sure what went wrong, I'll reply here when I
figure out what went wrong or file a bug and investigate further.

Chris

On Mon, May 4, 2015 at 4:23 PM, kgu...@mozilla.com wrote:

 On Thursday, April 30, 2015 at 7:22:26 PM UTC-4, Christopher Manchester
 wrote:
  You can now add --tag arguments to try syntax and they will get passed
 to
  test harnesses in your try push. Details of the implementation are in bug
  978846, but if you're interested in passing other arguments from try
 syntax
  to a test harness, this can be done by adding those arguments to
  testing/config/mozharness/try_arguments.py. Note this is still rather
  coarse in the sense that arguments are forwarded without regard for
 whether
  a harness supports a particular argument, but I can imagine it being
 useful
  in a number of cases (for instance, when testing the feature with
 xpcshell
  and --tag devtools, I was able to get feedback in about ten minutes
  whether things were working rather than waiting for every xpcshell test
 to
  run).

 Can you provide a sample complete try syntax that uses this? I just did a
 try run with |try: -b do -p linux,macosx64,win32,win64 -u all -t none --tag
 apz| (after adding the apz tag to a mochitest.ini file) and it still ran
 all the mochitests instead of just the ones in the mochitest.ini file. Does
 it only work in some scenarios?

 Cheers,
 kats
 ___
 dev-platform mailing list
 dev-platform@lists.mozilla.org
 https://lists.mozilla.org/listinfo/dev-platform

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: It is now possible to apply arbitrary tags to tests/manifests and run all tests with a given tag

[Christopher Manchester]

You can now add --tag arguments to try syntax and they will get passed to
test harnesses in your try push. Details of the implementation are in bug
978846, but if you're interested in passing other arguments from try syntax
to a test harness, this can be done by adding those arguments to
testing/config/mozharness/try_arguments.py. Note this is still rather
coarse in the sense that arguments are forwarded without regard for whether
a harness supports a particular argument, but I can imagine it being useful
in a number of cases (for instance, when testing the feature with xpcshell
and --tag devtools, I was able to get feedback in about ten minutes
whether things were working rather than waiting for every xpcshell test to
run).

Chris

On Thu, Apr 2, 2015 at 2:22 PM, Andrew Halberstadt ahalberst...@mozilla.com
 wrote:

 Minor update. It was pointed out that other list-like manifestparser
 attributes (like head and support-files) are whitespace delimited instead
 of comma delimited. To be consistent I switched tags to whitespace
 delimitation as well.

 E.g both these forms are ok:

 [test_foo.html]
 tags = foo bar baz

 [test_bar.html]
 tags =
 foo
 bar
 baz

 -Andrew


 On 31/03/15 12:30 PM, Andrew Halberstadt wrote:

 As of bug 987360, you can now run all tests with a given tag for
 mochitest (and variants), xpcshell and marionette based harnesses. Tags
 can be applied to either individual tests, or the DEFAULT section in
 manifests. Tests can have multiple tags, in which case they should be
 comma delimited. To run all tests with a given tag, pass in --tag tag
 name to the mach command.

 For example, let's say we want to group all mochitest-plain tests
 related to canvas together. First we'd add a 'canvas' tag to the DEFAULT
 section in

 https://dxr.mozilla.org/mozilla-central/source/dom/canvas/test/mochitest.ini


 [DEFAULT]
 tags = canvas

 We notice there is also a canvas related test under dom/media, namely:

 https://dxr.mozilla.org/mozilla-central/source/dom/media/test/mochitest.ini#541


 Let's pretend it is already tagged with the 'media' tag, but that's ok,
 we can add a second tag no problem:

 [test_video_to_canvas.html]
 tags = media,canvas

 Repeat above for any other tests or manifests scattered in the tree that
 are related to canvas. Now we can run all mochitest-plain tests with:

 ./mach mochitest-plain --tag canvas

 You can also run the union of two tags by specifying --tag more than
 once (though the intersection of two tags is not supported):

 ./mach mochitest-plain --tag canvas --tag media

 So far the xpcshell (./mach xpcshell-test --tag name) and marionette
 (./mach marionette-test --tag name) commands are also supported. Reftest
 is not supported as it has its own special manifest format.

 Applying tags to tests will not affect automation or other people's
 tags. So each organization or team should feel free to use tags in
 whatever creative ways they see fit. Eventually, we'll start using tags
 as a foundation for some more advanced features and analysis. For
 example, we may implement a way to run all tests with a given tag across
 multiple different suites.

 If you have any questions or things aren't working, please let me know!

 Cheers,
 Andrew


 ___
 dev-platform mailing list
 dev-platform@lists.mozilla.org
 https://lists.mozilla.org/listinfo/dev-platform

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Retiring the monolithic test package

[Christopher Manchester]

This message is about bug 917999 - Split test package into per-suite
packages, more can be found there.

I'm working on a set of patches to do this, but the assumption that a build
comes with a tests.zip that has all of the tests to run against that
build seems likely to be a part of more tools than I'm aware of. If you
maintain such a tool, please file a bug blocking 917999 and adjust your
tool based on the information below and/or contact me directly. I'll do
what I can to help.

If you have a tool that looks for a tests.zip in order to run tests, that
tool may need to be modified to keep working. We're planning to do this in
mozharness by uploading a manifest that will map from test harness names
('mochitest', 'cppunittests', etc) to a list of file names that are the
test packages required for that harness. In most cases there is a common
zip and a zip specifically including the harness' support files and tests
(in cases a harness' name does not appear in the manifest, that harness has
not been split, and everything required to run that harness will be in the
common zip).

I'm planning to have this done and landed by the end of the quarter, and
likely by the end of May.

Chris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: It is now possible to apply arbitrary tags to tests/manifests and run all tests with a given tag

[Christopher Manchester]

I filed bug 1149670 for the mach try feature.

On Tue, Mar 31, 2015 at 10:39 AM, Andrew Halberstadt 
ahalberst...@mozilla.com wrote:

 It's technically already possible by modifying the in-tree mozharness
 configs here:
 https://dxr.mozilla.org/mozilla-central/source/testing/config/mozharness

 However it's not easy to figure out what needs to be modified to get the
 desired results. Some sort of |mach try| like command is going to be worked
 on in Q2 to make pushing various test configurations to try easier, I'm not
 100% sure how it will be implemented yet though.

 In the meantime, if you get stuck trying to modify the mozharness configs,
 let me know and I can come up with an example patch.

 -Andrew


 On 31/03/15 01:06 PM, Bobby Holley wrote:

 This sounds awesome! Is there an estimate of when we'll be able to use it
 for try pushes?

 On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt 
 ahalberst...@mozilla.com mailto:ahalberst...@mozilla.com wrote:

 As of bug 987360, you can now run all tests with a given tag for
 mochitest (and variants), xpcshell and marionette based harnesses.
 Tags can be applied to either individual tests, or the DEFAULT
 section in manifests. Tests can have multiple tags, in which case
 they should be comma delimited. To run all tests with a given tag,
 pass in --tag tag name to the mach command.

 For example, let's say we want to group all mochitest-plain tests
 related to canvas together. First we'd add a 'canvas' tag to the
 DEFAULT section in
 https://dxr.mozilla.org/mozilla-central/source/dom/
 canvas/test/mochitest.ini

 [DEFAULT]
 tags = canvas

 We notice there is also a canvas related test under dom/media, namely:
 https://dxr.mozilla.org/mozilla-central/source/dom/
 media/test/mochitest.ini#541

 Let's pretend it is already tagged with the 'media' tag, but
 that's ok, we can add a second tag no problem:

 [test_video_to_canvas.html]
 tags = media,canvas

 Repeat above for any other tests or manifests scattered in the
 tree that are related to canvas. Now we can run all
 mochitest-plain tests with:

 ./mach mochitest-plain --tag canvas

 You can also run the union of two tags by specifying --tag more
 than once (though the intersection of two tags is not supported):

 ./mach mochitest-plain --tag canvas --tag media

 So far the xpcshell (./mach xpcshell-test --tag name) and
 marionette (./mach marionette-test --tag name) commands are also
 supported. Reftest is not supported as it has its own special
 manifest format.

 Applying tags to tests will not affect automation or other
 people's tags. So each organization or team should feel free to
 use tags in whatever creative ways they see fit. Eventually, we'll
 start using tags as a foundation for some more advanced features
 and analysis. For example, we may implement a way to run all tests
 with a given tag across multiple different suites.

 If you have any questions or things aren't working, please let me
 know!

 Cheers,
 Andrew
 ___
 dev-platform mailing list
 dev-platform@lists.mozilla.org mailto:dev-platform@lists.mozilla.org
 
 https://lists.mozilla.org/listinfo/dev-platform



 ___
 dev-platform mailing list
 dev-platform@lists.mozilla.org
 https://lists.mozilla.org/listinfo/dev-platform

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Structured Logging update

[Christopher Manchester]

Myself and others on Automation and Tools have undertaken an effort to
convert our test harnesses to use structured logging. We are making this
conversion to allow our testing infrastructure to move away from regex
based log parsing and to expose more useful log data (with a focus on test
results as JSON) to a wider variety of consumers. Ahmed Kachkach, an intern
with the A*Team this summer posted earlier with an update about Mochitests.
I'm following up here with pointers to more information.

Documentation for the python logging library we're building on (mozlog) is
available on readthedocs[1] and includes some examples if you're curious
about how this works. I've written up a short wiki page [2] including some
background on the project, and we’ve been using  bug 916295 [3] to track
progress.

We're intending to provide a foundation for future tools and infrastructure
improvements without disrupting familiar workflows or breaking
compatibility with existing tools, so the changes will be largely
unobtrusive at this stage. If you have trouble running tests locally or
observe undesired changes to test harness output, this could be an
unintended side effect of this work that needs to be addressed.

Ping me (chmanchester) in #ateam if you have any questions. Bugs/features
for mozlog should be filed in Testing :: Mozbase.

[1] http://mozbase.readthedocs.org/en/latest/mozlog_structured.html
[2] https://wiki.mozilla.org/Auto-tools/Projects/Structured_Logging
[3] https://bugzilla.mozilla.org/show_bug.cgi?id=916295
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform