Thank you for explaining the build process. I totally understand it's a long and complicated migration process nobody wants to deal with it until it's completely out of control causing significant losses. Currently Soong is the only step so heavy on RAM usage. Did anyone look for a cheap patch that can be applied to the first step in order to avoid the full scan of the modules? As I understood from your answer and the talk, the parsing process is linear in nature. Can we look at the files' modification times and replace the appropriate ninja entries in the manifest, rewriting them incrementally?
On Monday, October 27th, 2025 at 5:12 PM, 'chris simmonds' via Android Building <[email protected]> wrote: > Hi > > On Mon, 20 Oct 2025 at 17:59, 'asquator' via Android Building > <[email protected]> wrote: > >>>The analysis results are persisted to disk, so that we only need to rerun >>>them when one of it's inputs changes (an Android.bp/mk, glob in an >>>Android.bp, the build system itself, etc). >> >>>This means that most iterations when you're iterating on your code is fairly >>>fast, only going through the execution phase. But when you do touch an >>>Android.bp it'll take a few minutes (on a sufficient machine) to re-analyze >>>before switching to running the compilers/etc. >> >> Yes, I've noticed that, but isn't there a way to make the analysis stage >> incremental too? In case a single module file was modified while everything >> else is unchanged, we want to re-analyze the module itself and all of its >> recursive dependencies. As we have a cached graph already, it looks like we >> can rely on it to find all the dependents. Is there something I'm missing >> that requires rebuilding the graph from scratch? >> >>> When we were running under make in 2015, it would take a couple minutes to >>> start every single build, and the builds have gotten significantly more >>> complex and much larger since then. >> >> Exactly, so being able to load the entire graph into RAM is a bold >> assumption, isn't it? As the system will continue growing, more modules will >> be added and the memory footprint will constantly increase, until we need >> 128GB for a comfortable build. Every build system strives to avoid touching >> the files not being changed, is there an inherent technological limitation >> in Android that prevents us from doing that? > > This is a limitation in the Android build system, which as we all know is a > mess. There are three stages: > > 1. Soong parses *all* Android.bp files and creates a ninja manifest file, > out/soong/build.ninja. This file is typically 4 to 8 Gb > 2. Kati parses all the Android.mk files and other mk fragments and creates > more ninja manifests > 3. Ninja takes the manifest files and the target device name, calculates the > dependency tree and then schedules the jobs to generate the final target > > The problem is that the dependency tree is not known until stage 3. Also, > touching any Android,.bp file will trigger stage 1. > > More details in my talk: https://www.youtube.com/watch?v=hQZz2PRNdxI > > Could this be fixed? Yes, but only by making large changes throughout the > build system . The Android team started to do this by migrating the whole > thing to Bazel, but the project was cancelled, presumably because there is no > economic reason for them to do so. Another option that I have written about > int he past is to migrate to a mature build system such as BitBake: > https://www.linkedin.com/pulse/using-yocto-build-aosp-chris-simmonds-ymhof/ > > So, it's a problem, there are solutions but it needs someone with sufficient > resouces to fix > > HTH > > Chris Simmonds > >> -- >> -- >> You received this message because you are subscribed to the "Android >> Building" mailing list. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [[email protected]](mailto:android-building%[email protected]) >> For more options, visit this group at >> http://groups.google.com/group/android-building?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "Android Building" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to >> [[email protected]](mailto:android-building%[email protected]). >> To view this discussion visit >> https://groups.google.com/d/msgid/android-building/Ry5PgIwBCqANURKWMzqxWYD96bnwijw2owLLT9NowcPdL8bW_Z4mviaQEU37LK6WiAhJCybS2dFWpSJV-fdWEF5gEz8TyUgWe1dOcF7Jvqs%3D%40proton.me. > > -- > -- > You received this message because you are subscribed to the "Android > Building" mailing list. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/android-building?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "Android Building" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion visit > https://groups.google.com/d/msgid/android-building/CAN%2BjWE9Z05EZwKwZOhtO8dJskk1Rnc7qDmVmsU%3DpyD5%2BqV%2BkTg%40mail.gmail.com. -- -- You received this message because you are subscribed to the "Android Building" mailing list. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/android-building?hl=en --- You received this message because you are subscribed to the Google Groups "Android Building" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/android-building/M-U17hZ-KigdOa0lU3-Sp3C2weO0IiJXBWCwQFzuqiG1ELyMBHi6ikYxeVxvfamXkIkW5-ire_K-n5kE2zHTvwWAdcx79z89wmvU7qzIe5I%3D%40proton.me.
