Faster Windows builds everywhere!

[Chris AtLee]

A few weeks ago I posted about switching our Windows builds on Try over to
EC2, resulting in a 30 minute speed improvement.

Last week we made the same change to the rest of the Windows build
infrastructure. All our Windows builds are now running in AWS. We're seeing
good performance gains there too. On mozilla-inbound, we've reduced opt
build times by at least 45 minutes, and nearly two hours (!!) off of our
PGO build times.

Big thanks again to Rob Thijssen (:grenade), Mark Cornmesser (:markco) and
the rest of our Release Engineering and Operations team for getting this
done. Please send your kudos and thanks to them on #releng, or in person at
Orlando next week!

Cheers,
Chris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Faster Windows builds everywhere!

[Justin Dolske]

On 12/1/15 12:41 PM, Chris AtLee wrote:


Last week we made the same change to the rest of the Windows build
infrastructure. All our Windows builds are now running in AWS. We're seeing
good performance gains there too. On mozilla-inbound, we've reduced opt
build times by at least 45 minutes, and nearly two hours (!!) off of our
PGO build times.


Nice!

What builds/tests have _not_ moved to the cloud? AIUI the two biggies 
are OS X (can't move because OS licensing), and perf tests... How close 
are we to transitioning everything else off MoCo metal?


Justin
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Faster Windows builds everywhere!

[Chris AtLee]

Right now we've got debug OSX builds in the cloud on Try in parallel with
the regular builds. There's a bunch more work to be done there to be able
to switch over, but we're definitely making progress.

All Windows / OSX unit tests are currently done on our own infra. Q Fortier
is working on getting Windows unittests stood up in AWS, and the results
are very promising. I'm not sure when we'll be able to switch over yet
though. There are no obvious solutions for OSX test infrastructure other
than maintaining our own racks of minis.

Perf tests on all platforms will stay on hardware for now. Some people have
done experiments on EC2 to see how talos performs, but I don't think we
know enough about the impact of this to decide if we can move these off
bare metal or not.

We also run some tests for mobile on panda boards, but those are going away
eventually.

On Tue, Dec 1, 2015 at 4:52 PM, Justin Dolske  wrote:

> On 12/1/15 12:41 PM, Chris AtLee wrote:
>
> Last week we made the same change to the rest of the Windows build
>> infrastructure. All our Windows builds are now running in AWS. We're
>> seeing
>> good performance gains there too. On mozilla-inbound, we've reduced opt
>> build times by at least 45 minutes, and nearly two hours (!!) off of our
>> PGO build times.
>>
>
> Nice!
>
> What builds/tests have _not_ moved to the cloud? AIUI the two biggies are
> OS X (can't move because OS licensing), and perf tests... How close are we
> to transitioning everything else off MoCo metal?
>
> Justin
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Faster Windows builds everywhere!

[Chris AtLee]

On Tue, Dec 1, 2015 at 5:27 PM, Gregory Szorc  wrote:

>
> On Tue, Dec 1, 2015 at 2:21 PM, Chris AtLee  wrote:
>
>> Right now we've got debug OSX builds in the cloud on Try in parallel with
>> the regular builds. There's a bunch more work to be done there to be able
>> to switch over, but we're definitely making progress.
>>
>> All Windows / OSX unit tests are currently done on our own infra. Q
>> Fortier
>> is working on getting Windows unittests stood up in AWS, and the results
>> are very promising. I'm not sure when we'll be able to switch over yet
>> though. There are no obvious solutions for OSX test infrastructure other
>> than maintaining our own racks of minis.
>>
>> Perf tests on all platforms will stay on hardware for now. Some people
>> have
>> done experiments on EC2 to see how talos performs, but I don't think we
>> know enough about the impact of this to decide if we can move these off
>> bare metal or not.
>>
>
> Amazon now supports dedicated instances, which means you fully control
> what runs on the machine and other random tenants aren't fighting you for
> CPU and I/O. Assuming the performance variance from other tenants is what
> was preventing us from moving Talos to AWS, that blocker may no longer
> exist.
>

I think you probably want Dedicated Hosts rather than dedicated instances,
otherwise multiple of your own workloads could end up on the same physical
box I think. https://aws.amazon.com/ec2/dedicated-hosts/

I have two main concerns with dedicated infra on AWS:
* You're still running under a hypervisor of some kind; will it introduce
too much noise into the results? Seems like a worthwhile experiment!

* Dedicated host pricing is quite a bit more expensive than what we're
paying now for test infrastructure.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Faster Windows builds everywhere!

[Gregory Szorc]

On Tue, Dec 1, 2015 at 2:21 PM, Chris AtLee  wrote:

> Right now we've got debug OSX builds in the cloud on Try in parallel with
> the regular builds. There's a bunch more work to be done there to be able
> to switch over, but we're definitely making progress.
>
> All Windows / OSX unit tests are currently done on our own infra. Q Fortier
> is working on getting Windows unittests stood up in AWS, and the results
> are very promising. I'm not sure when we'll be able to switch over yet
> though. There are no obvious solutions for OSX test infrastructure other
> than maintaining our own racks of minis.
>
> Perf tests on all platforms will stay on hardware for now. Some people have
> done experiments on EC2 to see how talos performs, but I don't think we
> know enough about the impact of this to decide if we can move these off
> bare metal or not.
>

Amazon now supports dedicated instances, which means you fully control what
runs on the machine and other random tenants aren't fighting you for CPU
and I/O. Assuming the performance variance from other tenants is what was
preventing us from moving Talos to AWS, that blocker may no longer exist.


>
> We also run some tests for mobile on panda boards, but those are going away
> eventually.
>
> On Tue, Dec 1, 2015 at 4:52 PM, Justin Dolske  wrote:
>
> > On 12/1/15 12:41 PM, Chris AtLee wrote:
> >
> > Last week we made the same change to the rest of the Windows build
> >> infrastructure. All our Windows builds are now running in AWS. We're
> >> seeing
> >> good performance gains there too. On mozilla-inbound, we've reduced opt
> >> build times by at least 45 minutes, and nearly two hours (!!) off of our
> >> PGO build times.
> >>
> >
> > Nice!
> >
> > What builds/tests have _not_ moved to the cloud? AIUI the two biggies are
> > OS X (can't move because OS licensing), and perf tests... How close are
> we
> > to transitioning everything else off MoCo metal?
> >
> > Justin
> >
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform