ayokunle321 wrote: Hi @rnk @ChuanqiXu9 @jansvoboda11 @hahnjo, a gentle ping to update you on a revision of this patch.
The earlier version did the dedup on the reader side. When loading modules, it would figure out which input files were duplicates and reuse their SLoc allocations. The problem was that, to know what could be reused and how much space to allocate, I had to scan every loaded module up front. That meant materializing SLocEntries that otherwise would have stayed lazy, which goes against the design and increased RSS. The new version moves the dedup to the writer side and is inspired by #86216 and #176288, i.e., instead of searching later, resolve the identity from information that's already available while writing. I also ended up looking at some of @ilya-biryukov's work and this patch extends the module-map dedup in #112015 and #116374 to the source headers those module maps bring in. So now, if a file already exists in an imported module, we don't serialize another copy. Locations that would have pointed into the duplicate are redirected to the existing copy instead. The nice part is that the reader doesn't need to do any funky remapping anymore. The extra work happens when writing the module, and consumers just load fewer SLoc entries. I've been testing it on ROOT, since it's one of the only open codebases with a partially modularized build. It ships its own Clang for Cling, and generates both dictionaries and .pcm files for its libraries through rootcling. So building the dictionaries gives a fairly direct way to exercise the module-writing path. Numbers: SLoc address space: 13.65 MB less, from 75.33 MB to 61.02 MB, about 19%, all from duplicated source headers. ROOT startup: about 5.4 MB less memory, since the deduplicated PCMs contain fewer source-location entries. rootcling: the import-heavy TMVA dictionary/PCM also saves about 5.4 MB. A smaller case with a single import, Thread, is about 0.7 MB worse, where the writer-side overhead is larger than the saving. Compile time: no regression that I can see. Startup is identical, and rootcling differences are under 0.5%, which looks like noise. I'd appreciate your thoughts on whether you think this is the right direction and if you think it's getting close to something we could merge. If not, I'd like to know what you think is still missing or what you'd want me to work on next. Also, if you have a codebase where this would be easy to try, I'd appreciate it if you could test it there. https://github.com/llvm/llvm-project/pull/209795 _______________________________________________ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
