Hi Dan, Thanks a lot for your detailed answers, it is really helpful. The machine i used for benchmarking build time has indeed less than ideal mass storage configuration. I will do more benchmarking on another machine that has nvme.
I as well started to look into RBE (see my other email). Thanks again, Chris On Tuesday, 7 June 2022 at 07:44:11 UTC+12 Dan Willemsen wrote: > This is probably one of two things: > > 1. Disk Speed. Once you saturate your CPUs (and have the associated memory > required -- somewhere around 2GB/core is usually decent), the next thing > you'll often hit is disk bandwidth. This is where ccache tends to hurt more > than it helps, unless the ccache is on a completely separate [also fast] > disk (full cache hits may get better, but writing the cache ends up being > 2x the disk bandwidth). Remote cache/execution plus Bazel's Build without > the bytes <https://blog.bazel.build/2019/05/07/builds-without-bytes.html> > concept > is a more ideal caching solution for this case, but that's not available > for Android at this point. > > 2. That -j isn't a great limiter. It worked "okay" when every build action > was single threaded, but more and more tools are multithreaded these days. > So if we have 64 cpu threads, and use -j64, if you get really unlucky with > a tool that spawns a process thread for every CPU, we could end up with > 4096 (64*64) process threads running, well over the 64 specified by `-j`. > In a more limited case, this may mean that on average you're still using a > good portion of your CPU, even with smaller -j values. > > Anyone knows how to disable some of the build tasks? For example i've >> noticed that there is a lot of java source processing and wonder if they >> are all needed. I'm pretty sure that some documentation is generated, in >> our case we don't need that. >> > > You're probably seeing metalava/etc. While that is used to generate > documentation, it is also used to generate the API stubs used later in the > build, so it can't just be turned off. > > If you haven't seen it yet, Soong's Performance doc > <https://android.googlesource.com/platform/build/soong/+/master/docs/perf.md> > is a good overview on how some performance issues can be diagnosed. > > Hopefully you're doing many more incremental builds than full builds, but > I know that gets really complicated with some use cases. > > --- > > For reference, the newer desktops at Google end up being 2x 18-core Intel > Xeon Gold 6154 (so 72 threads) or the Ryzen 3995WX (128 threads), at least > with NVME, but often extended with SSDs. We also run builds on a variety of > GCE-based machines, from 32 to 128 vCPUs, and the difference there between > pd-standard and pd-ssd can be substantial. > > On my Intel desktop (w/NVME), without ccache, I'm seeing about a 10% > improvement from -j36 -> -j74, so a little better than your comparison, but > not a ton: > > `lunch aosp_arm64-userdebug; m` (effectively -j74) takes 32m46s > `lunch aosp_arm64-userdebug; m -j54` takes 34m00s > `lunch aosp_arm64-userdebug; m -j36` takes 36m19s > > With RBE (Google's implementation of > https://bazel.build/community/remote-execution-services): > > `lunch aosp_arm64-userdebug; USE_RBE=true m` takes 18m46s > > - Dan > > On Wed, Jun 1, 2022 at 4:21 PM Christian Gagneraud <[email protected]> > wrote: > >> Hi all, >> >> We're looking into reducing our build time. >> The first benchmark we did was to measure build time of aosp_arm64 versus >> number of parallel jobs. >> We have a machine with 28 cores (56 threads), so i ran a script to build >> aosp from 4 jobs up to 56 jobs with an increment of 4. >> >> The results surprised me, in a nutshell, with ccache enabled and 100% hit >> rate >> 4 cores: 1h >> 8 cores: 42m >> 12 cores: 35m >> 16+ cores: 32m >> >> Throwing more than 16 cores doesn't influence build time anymore, the >> curve is flat! >> >> I just can't explain what is going on here. Does anyone know why the >> build process doesn't scale? >> >> I'm currently running the same benchmark with ccache disabled. Will post >> an update once i have the data. >> >> Anyone knows how to disable some of the build tasks? For example i've >> noticed that there is a lot of java source processing and wonder if they >> are all needed. I'm pretty sure that some documentation is generated, in >> our case we don't need that. >> >> As well, any tips or feedback around reducing build times would be more >> than welcome. >> >> Thanks, >> Chris >> >> -- >> -- >> You received this message because you are subscribed to the "Android >> Building" mailing list. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/android-building?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "Android Building" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/android-building/a5b97997-311e-4913-a829-111d0433d227n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/android-building/a5b97997-311e-4913-a829-111d0433d227n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- -- You received this message because you are subscribed to the "Android Building" mailing list. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/android-building?hl=en --- You received this message because you are subscribed to the Google Groups "Android Building" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/android-building/69ed413b-0c79-41bd-a8a7-512c64fc7ecbn%40googlegroups.com.
