Re: reducing high try macosx pending counts

[Andrew Halberstadt]

Using the new |mach try fuzzy| selector will also make it a lot easier to
only schedule exactly what you need. To run what the above try syntax uses,
do:

$ ./mach try fuzzy
!osx 'web-platform 

That will run every task that doesn't contain the string 'osx', and does
contain the string 'web-platform'.

On Thu, Aug 3, 2017 at 3:51 AM James Graham  wrote:

> On 02/08/17 22:30, Kim Moir wrote:
> > You may have noticed that the time to wait for macosx test results on try
> > has been very long (>1day) this week.
> >
> > We have taken the following steps to address this problem
> [...]
>
> That sounds great! Thanks.
>
> For everyone else:
>
> It looks like the queues are still pretty long, and I imagine there are
> lots of pending jobs people scheduled that aren't really necessary any
> more (e.g. because you already found issues with a patch on other
> platforms, or landed in spite of the missing results).
>
> If you have a try push which requested OSX jobs, but you don't need them
> now, it would help if you go back and cancel the remaining jobs for that
> push from treeherder (look for the grey circle with a cross inside in
> the top right of the push to cancel all unfinished jobs, and the similar
> icon in the bottom panel for individual jobs). Also if you are making
> new try pushes and don't specifically need OSX testing, it's possible to
> limit tests to certain platforms with try syntax like (for running tests
> on linux64 and linux64-stylo only):
>
> try: -b do -p all -u web-platform-tests[x64,linux64-stylo]
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: reducing high try macosx pending counts

[James Graham]

On 02/08/17 22:30, Kim Moir wrote:

You may have noticed that the time to wait for macosx test results on try
has been very long (>1day) this week.

We have taken the following steps to address this problem

[...]

That sounds great! Thanks.

For everyone else:

It looks like the queues are still pretty long, and I imagine there are 
lots of pending jobs people scheduled that aren't really necessary any 
more (e.g. because you already found issues with a patch on other 
platforms, or landed in spite of the missing results).


If you have a try push which requested OSX jobs, but you don't need them 
now, it would help if you go back and cancel the remaining jobs for that 
push from treeherder (look for the grey circle with a cross inside in 
the top right of the push to cancel all unfinished jobs, and the similar 
icon in the bottom panel for individual jobs). Also if you are making 
new try pushes and don't specifically need OSX testing, it's possible to 
limit tests to certain platforms with try syntax like (for running tests 
on linux64 and linux64-stylo only):


try: -b do -p all -u web-platform-tests[x64,linux64-stylo]

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: reducing high try macosx pending counts

[Bobby Holley]

Thanks to everyone involved for accommodating the extra load from the stylo
side - it's a necessary part of our ramp-up to shipping. Luckily, we should
only be running the two configurations side-by-side for a month or two,
after which point load should drop back down to normal levels.

Thanks Kim for staying on top of this and coming up with ways to keep it
manageable!
bholley

On Wed, Aug 2, 2017 at 2:30 PM, Kim Moir  wrote:

> You may have noticed that the time to wait for macosx test results on try
> has been very long (>1day) this week.
>
> [tracking] macosx test load is unsustainable
> https://bugzilla.mozilla.org/show_bug.cgi?id=1386625
>
> There are several factors that contributed to this situation.
> 1) Additional macosx stylo tests were enabled on trunk which added
> significant test load to a system already running at capacity
> 2) These tests were enabled before some optimizations were in place to
> reduce the number of tests run per push
> 3) Since jobs on try have a lower priority than jobs on trunk, the trunk
> jobs continue to consume the the mac test machine capacity before it can be
> used by try jobs
>
> We have taken the following steps to address this problem
> 1) I have disabled the mac stylo tests on m-i to reduce load (
> https://bugzilla.mozilla.org/show_bug.cgi?id=1386625)
> 2)  The optimizations have been enabled for autoland and m-i branches so
> all the tests are not run on every push for the new stylo tests (
> https://bugzilla.mozilla.org/show_bug.cgi?id=1386405)
> 3) Patches are in progress to disable some non-e10s tests to further reduce
> load (https://bugzilla.mozilla.org/show_bug.cgi?id=1386689)
>
> We will continue to monitor the situation and implement additional measures
> to allow macosx test runs on try to complete in a more reasonable interval.
>
> Please ping us in #releng or follow the tracking bug if you have questions
> https://bugzilla.mozilla.org/show_bug.cgi?id=1386625
>
> Kim
> ___
> dev-planning mailing list
> dev-plann...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

reducing high try macosx pending counts

[Kim Moir]

You may have noticed that the time to wait for macosx test results on try
has been very long (>1day) this week.

[tracking] macosx test load is unsustainable
https://bugzilla.mozilla.org/show_bug.cgi?id=1386625

There are several factors that contributed to this situation.
1) Additional macosx stylo tests were enabled on trunk which added
significant test load to a system already running at capacity
2) These tests were enabled before some optimizations were in place to
reduce the number of tests run per push
3) Since jobs on try have a lower priority than jobs on trunk, the trunk
jobs continue to consume the the mac test machine capacity before it can be
used by try jobs

We have taken the following steps to address this problem
1) I have disabled the mac stylo tests on m-i to reduce load (
https://bugzilla.mozilla.org/show_bug.cgi?id=1386625)
2)  The optimizations have been enabled for autoland and m-i branches so
all the tests are not run on every push for the new stylo tests (
https://bugzilla.mozilla.org/show_bug.cgi?id=1386405)
3) Patches are in progress to disable some non-e10s tests to further reduce
load (https://bugzilla.mozilla.org/show_bug.cgi?id=1386689)

We will continue to monitor the situation and implement additional measures
to allow macosx test runs on try to complete in a more reasonable interval.

Please ping us in #releng or follow the tracking bug if you have questions
https://bugzilla.mozilla.org/show_bug.cgi?id=1386625

Kim
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform