llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT--> @llvm/pr-subscribers-clang-modules Author: Ayokunle Amodu (ayokunle321) <details> <summary>Changes</summary> ### Summary Stage 1 of the sloc-dedup work. Adds detection and measurement of duplicate file entries in the loaded SLoc address space. There is no reuse yet and no behavior change in default builds. ### Motivation When multiple modules import the same header, each one reserves its own offset space for that file. Before doing anything invasive, we want to confirm the duplication is real and measure exactly how much is reclaimable. ### Implementation SourceManager gains a DenseMap<const FileEntry*, UIntTy> of first-seen offsets plus NumDuplicateLoadedFiles and DuplicateLoadedBytes counters, reported in -print-stats. ASTReader runs a small eager scan in the SOURCE_LOCATION_OFFSETS case and feeds each SM_SLOC_FILE_ENTRY's file and size into SourceManager. Detection lives at the eager allocation site because that's where offset space is actually reserved. A lazy hook in ReadSLocEntry was tried first but did not fire for most entries. Gated behind CLANG_SLOC_DEDUP=1 so default builds and lit are unaffected. ### Results Using the two-module reproducer in `dedup-test/`, where both modules import `shared.h`, the one-module baseline loaded 43,764 bytes of SLoc data and reported no duplicates. Adding the second module increased the loaded SLoc space to 87,528 bytes and identified 2 duplicate file allocations, corresponding to 20,694 bytes that could be reused (about 24% of the loaded space). The duplicates were `shared.h` itself and the module map. The one-module baseline reports zero duplicates, suggesting the detection is not producing false positives. Stage 2 will focus on reclaiming those 20,694 bytes. --- Patch is 27.10 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/203154.diff 10 Files Affected: - (modified) .gitignore (+1) - (modified) clang/include/clang/Basic/SourceManager.h (+25) - (modified) clang/lib/Basic/SourceManager.cpp (+5) - (modified) clang/lib/Serialization/ASTReader.cpp (+44) - (added) dedup-test/a.h (+5) - (added) dedup-test/b.h (+5) - (added) dedup-test/module.modulemap (+2) - (added) dedup-test/shared.h (+403) - (added) dedup-test/use.cpp (+3) - (added) dedup-test/use_err.cpp (+7) ``````````diff diff --git a/.gitignore b/.gitignore index 9d4e86ab10caa..67f2c8a0a6f95 100644 --- a/.gitignore +++ b/.gitignore @@ -28,6 +28,7 @@ # Nested build directory /build* +/test* #==============================================================================# # Explicit files to ignore (only matches one). diff --git a/clang/include/clang/Basic/SourceManager.h b/clang/include/clang/Basic/SourceManager.h index 4217b8683da1e..733574d4118ec 100644 --- a/clang/include/clang/Basic/SourceManager.h +++ b/clang/include/clang/Basic/SourceManager.h @@ -749,6 +749,31 @@ class SourceManager : public RefCountedBase<SourceManager> { static const SourceLocation::UIntTy MaxLoadedOffset = 1ULL << (8 * sizeof(SourceLocation::UIntTy) - 1); + /// The first global offset at which each file's SLoc entry was loaded. + /// Lets us recognize when a later module re-loads a file already present in + /// the loaded address space (e.g. a header included by several modules). + llvm::DenseMap<const FileEntry *, SourceLocation::UIntTy> LoadedFileFirstOffset; + /// Number of loaded file SLoc entries that duplicated an already-loaded file. + unsigned NumDuplicateLoadedFiles = 0; + /// SLoc address-space bytes occupied by those duplicate entries (the prize + /// that reuse could reclaim). + uint64_t DuplicateLoadedBytes = 0; + +public: + /// Record that a loaded file SLoc entry was created. If the same file was + /// already loaded into the address space, it is counted as a duplicate. + void noteLoadedFileSLocEntry(const FileEntry *FE, + SourceLocation::UIntTy Offset, uint64_t Size) { + if (!FE) + return; + auto Res = LoadedFileFirstOffset.try_emplace(FE, Offset); + if (!Res.second) { + ++NumDuplicateLoadedFiles; + DuplicateLoadedBytes += Size; + } + } + +private: /// A bitmap that indicates whether the entries of LoadedSLocEntryTable /// have already been loaded from the external source. /// diff --git a/clang/lib/Basic/SourceManager.cpp b/clang/lib/Basic/SourceManager.cpp index b6cc6ec9365f5..70597f2c56457 100644 --- a/clang/lib/Basic/SourceManager.cpp +++ b/clang/lib/Basic/SourceManager.cpp @@ -2125,6 +2125,11 @@ void SourceManager::PrintStats() const { << " bytes of capacity), " << MaxLoadedOffset - CurrentLoadedOffset << "B of SLoc address space used.\n"; + if (NumDuplicateLoadedFiles) + llvm::errs() << NumDuplicateLoadedFiles + << " duplicate loaded file SLocEntries detected (" + << DuplicateLoadedBytes + << "B of SLoc address space reusable).\n"; unsigned NumLineNumsComputed = 0; unsigned NumFileBytesMapped = 0; diff --git a/clang/lib/Serialization/ASTReader.cpp b/clang/lib/Serialization/ASTReader.cpp index f8a6a38bb9b5c..bb99376c8e31f 100644 --- a/clang/lib/Serialization/ASTReader.cpp +++ b/clang/lib/Serialization/ASTReader.cpp @@ -4250,6 +4250,50 @@ llvm::Error ASTReader::ReadASTBlock(ModuleFile &F, - SLocSpaceSize,&F)); TotalNumSLocEntries += F.LocalNumSLocEntries; + + // Eagerly scan this module's SLoc file + // entries to detect ones that duplicate a file already loaded by an + // earlier module, and measure the reusable address space. + // + // Gated by an env var so normal builds and the test suite + // are unaffected. + static const bool DedupDetect = ::getenv("CLANG_SLOC_DEDUP") != nullptr; + if (DedupDetect) { + unsigned N = F.LocalNumSLocEntries; + SavedStreamPosition SavedPos(F.SLocEntryCursor); + SmallVector<uint32_t, 64> Offs(N, 0); + SmallVector<const FileEntry *, 64> Files(N, nullptr); + bool ScanOK = true; + for (unsigned i = 0; i != N && ScanOK; ++i) { + if (llvm::Error Err = F.SLocEntryCursor.JumpToBit( + F.SLocEntryOffsetsBase + F.SLocEntryOffsets[i])) { + consumeError(std::move(Err)); + ScanOK = false; + break; + } + Expected<llvm::BitstreamEntry> E = F.SLocEntryCursor.advance(); + if (!E) { consumeError(E.takeError()); ScanOK = false; break; } + if (E->Kind != llvm::BitstreamEntry::Record) { ScanOK = false; break; } + RecordData R; + StringRef B; + Expected<unsigned> C = F.SLocEntryCursor.readRecord(E->ID, R, &B); + if (!C) { consumeError(C.takeError()); ScanOK = false; break; } + Offs[i] = (uint32_t)R[0]; + if (C.get() == SM_SLOC_FILE_ENTRY) + if (OptionalFileEntryRef File = getInputFile(F, R[4]).getFile()) + Files[i] = &File->getFileEntry(); + } + if (ScanOK) { + for (unsigned i = 0; i != N; ++i) { + if (!Files[i]) + continue; + uint64_t Size = + (uint64_t)((i + 1 < N ? Offs[i + 1] : SLocSpaceSize) - Offs[i]); + SourceMgr.noteLoadedFileSLocEntry( + Files[i], F.SLocEntryBaseOffset + Offs[i], Size); + } + } + } break; } diff --git a/dedup-test/a.h b/dedup-test/a.h new file mode 100644 index 0000000000000..1fe7fc27db549 --- /dev/null +++ b/dedup-test/a.h @@ -0,0 +1,5 @@ +#ifndef A_H +#define A_H +#include "shared.h" +inline int a_entry(int x) { return shared_fn_0(x); } +#endif diff --git a/dedup-test/b.h b/dedup-test/b.h new file mode 100644 index 0000000000000..85839261fc602 --- /dev/null +++ b/dedup-test/b.h @@ -0,0 +1,5 @@ +#ifndef B_H +#define B_H +#include "shared.h" +inline int b_entry(int x) { return shared_fn_1(x); } +#endif diff --git a/dedup-test/module.modulemap b/dedup-test/module.modulemap new file mode 100644 index 0000000000000..2da1c03a22211 --- /dev/null +++ b/dedup-test/module.modulemap @@ -0,0 +1,2 @@ +module A { header "a.h" export * } +module B { header "b.h" export * } diff --git a/dedup-test/shared.h b/dedup-test/shared.h new file mode 100644 index 0000000000000..3ac57a45355f9 --- /dev/null +++ b/dedup-test/shared.h @@ -0,0 +1,403 @@ +#ifndef SHARED_H +#define SHARED_H +inline int shared_fn_0(int x) { return x + 0; } +inline int shared_fn_1(int x) { return x + 1; } +inline int shared_fn_2(int x) { return x + 2; } +inline int shared_fn_3(int x) { return x + 3; } +inline int shared_fn_4(int x) { return x + 4; } +inline int shared_fn_5(int x) { return x + 5; } +inline int shared_fn_6(int x) { return x + 6; } +inline int shared_fn_7(int x) { return x + 7; } +inline int shared_fn_8(int x) { return x + 8; } +inline int shared_fn_9(int x) { return x + 9; } +inline int shared_fn_10(int x) { return x + 10; } +inline int shared_fn_11(int x) { return x + 11; } +inline int shared_fn_12(int x) { return x + 12; } +inline int shared_fn_13(int x) { return x + 13; } +inline int shared_fn_14(int x) { return x + 14; } +inline int shared_fn_15(int x) { return x + 15; } +inline int shared_fn_16(int x) { return x + 16; } +inline int shared_fn_17(int x) { return x + 17; } +inline int shared_fn_18(int x) { return x + 18; } +inline int shared_fn_19(int x) { return x + 19; } +inline int shared_fn_20(int x) { return x + 20; } +inline int shared_fn_21(int x) { return x + 21; } +inline int shared_fn_22(int x) { return x + 22; } +inline int shared_fn_23(int x) { return x + 23; } +inline int shared_fn_24(int x) { return x + 24; } +inline int shared_fn_25(int x) { return x + 25; } +inline int shared_fn_26(int x) { return x + 26; } +inline int shared_fn_27(int x) { return x + 27; } +inline int shared_fn_28(int x) { return x + 28; } +inline int shared_fn_29(int x) { return x + 29; } +inline int shared_fn_30(int x) { return x + 30; } +inline int shared_fn_31(int x) { return x + 31; } +inline int shared_fn_32(int x) { return x + 32; } +inline int shared_fn_33(int x) { return x + 33; } +inline int shared_fn_34(int x) { return x + 34; } +inline int shared_fn_35(int x) { return x + 35; } +inline int shared_fn_36(int x) { return x + 36; } +inline int shared_fn_37(int x) { return x + 37; } +inline int shared_fn_38(int x) { return x + 38; } +inline int shared_fn_39(int x) { return x + 39; } +inline int shared_fn_40(int x) { return x + 40; } +inline int shared_fn_41(int x) { return x + 41; } +inline int shared_fn_42(int x) { return x + 42; } +inline int shared_fn_43(int x) { return x + 43; } +inline int shared_fn_44(int x) { return x + 44; } +inline int shared_fn_45(int x) { return x + 45; } +inline int shared_fn_46(int x) { return x + 46; } +inline int shared_fn_47(int x) { return x + 47; } +inline int shared_fn_48(int x) { return x + 48; } +inline int shared_fn_49(int x) { return x + 49; } +inline int shared_fn_50(int x) { return x + 50; } +inline int shared_fn_51(int x) { return x + 51; } +inline int shared_fn_52(int x) { return x + 52; } +inline int shared_fn_53(int x) { return x + 53; } +inline int shared_fn_54(int x) { return x + 54; } +inline int shared_fn_55(int x) { return x + 55; } +inline int shared_fn_56(int x) { return x + 56; } +inline int shared_fn_57(int x) { return x + 57; } +inline int shared_fn_58(int x) { return x + 58; } +inline int shared_fn_59(int x) { return x + 59; } +inline int shared_fn_60(int x) { return x + 60; } +inline int shared_fn_61(int x) { return x + 61; } +inline int shared_fn_62(int x) { return x + 62; } +inline int shared_fn_63(int x) { return x + 63; } +inline int shared_fn_64(int x) { return x + 64; } +inline int shared_fn_65(int x) { return x + 65; } +inline int shared_fn_66(int x) { return x + 66; } +inline int shared_fn_67(int x) { return x + 67; } +inline int shared_fn_68(int x) { return x + 68; } +inline int shared_fn_69(int x) { return x + 69; } +inline int shared_fn_70(int x) { return x + 70; } +inline int shared_fn_71(int x) { return x + 71; } +inline int shared_fn_72(int x) { return x + 72; } +inline int shared_fn_73(int x) { return x + 73; } +inline int shared_fn_74(int x) { return x + 74; } +inline int shared_fn_75(int x) { return x + 75; } +inline int shared_fn_76(int x) { return x + 76; } +inline int shared_fn_77(int x) { return x + 77; } +inline int shared_fn_78(int x) { return x + 78; } +inline int shared_fn_79(int x) { return x + 79; } +inline int shared_fn_80(int x) { return x + 80; } +inline int shared_fn_81(int x) { return x + 81; } +inline int shared_fn_82(int x) { return x + 82; } +inline int shared_fn_83(int x) { return x + 83; } +inline int shared_fn_84(int x) { return x + 84; } +inline int shared_fn_85(int x) { return x + 85; } +inline int shared_fn_86(int x) { return x + 86; } +inline int shared_fn_87(int x) { return x + 87; } +inline int shared_fn_88(int x) { return x + 88; } +inline int shared_fn_89(int x) { return x + 89; } +inline int shared_fn_90(int x) { return x + 90; } +inline int shared_fn_91(int x) { return x + 91; } +inline int shared_fn_92(int x) { return x + 92; } +inline int shared_fn_93(int x) { return x + 93; } +inline int shared_fn_94(int x) { return x + 94; } +inline int shared_fn_95(int x) { return x + 95; } +inline int shared_fn_96(int x) { return x + 96; } +inline int shared_fn_97(int x) { return x + 97; } +inline int shared_fn_98(int x) { return x + 98; } +inline int shared_fn_99(int x) { return x + 99; } +inline int shared_fn_100(int x) { return x + 100; } +inline int shared_fn_101(int x) { return x + 101; } +inline int shared_fn_102(int x) { return x + 102; } +inline int shared_fn_103(int x) { return x + 103; } +inline int shared_fn_104(int x) { return x + 104; } +inline int shared_fn_105(int x) { return x + 105; } +inline int shared_fn_106(int x) { return x + 106; } +inline int shared_fn_107(int x) { return x + 107; } +inline int shared_fn_108(int x) { return x + 108; } +inline int shared_fn_109(int x) { return x + 109; } +inline int shared_fn_110(int x) { return x + 110; } +inline int shared_fn_111(int x) { return x + 111; } +inline int shared_fn_112(int x) { return x + 112; } +inline int shared_fn_113(int x) { return x + 113; } +inline int shared_fn_114(int x) { return x + 114; } +inline int shared_fn_115(int x) { return x + 115; } +inline int shared_fn_116(int x) { return x + 116; } +inline int shared_fn_117(int x) { return x + 117; } +inline int shared_fn_118(int x) { return x + 118; } +inline int shared_fn_119(int x) { return x + 119; } +inline int shared_fn_120(int x) { return x + 120; } +inline int shared_fn_121(int x) { return x + 121; } +inline int shared_fn_122(int x) { return x + 122; } +inline int shared_fn_123(int x) { return x + 123; } +inline int shared_fn_124(int x) { return x + 124; } +inline int shared_fn_125(int x) { return x + 125; } +inline int shared_fn_126(int x) { return x + 126; } +inline int shared_fn_127(int x) { return x + 127; } +inline int shared_fn_128(int x) { return x + 128; } +inline int shared_fn_129(int x) { return x + 129; } +inline int shared_fn_130(int x) { return x + 130; } +inline int shared_fn_131(int x) { return x + 131; } +inline int shared_fn_132(int x) { return x + 132; } +inline int shared_fn_133(int x) { return x + 133; } +inline int shared_fn_134(int x) { return x + 134; } +inline int shared_fn_135(int x) { return x + 135; } +inline int shared_fn_136(int x) { return x + 136; } +inline int shared_fn_137(int x) { return x + 137; } +inline int shared_fn_138(int x) { return x + 138; } +inline int shared_fn_139(int x) { return x + 139; } +inline int shared_fn_140(int x) { return x + 140; } +inline int shared_fn_141(int x) { return x + 141; } +inline int shared_fn_142(int x) { return x + 142; } +inline int shared_fn_143(int x) { return x + 143; } +inline int shared_fn_144(int x) { return x + 144; } +inline int shared_fn_145(int x) { return x + 145; } +inline int shared_fn_146(int x) { return x + 146; } +inline int shared_fn_147(int x) { return x + 147; } +inline int shared_fn_148(int x) { return x + 148; } +inline int shared_fn_149(int x) { return x + 149; } +inline int shared_fn_150(int x) { return x + 150; } +inline int shared_fn_151(int x) { return x + 151; } +inline int shared_fn_152(int x) { return x + 152; } +inline int shared_fn_153(int x) { return x + 153; } +inline int shared_fn_154(int x) { return x + 154; } +inline int shared_fn_155(int x) { return x + 155; } +inline int shared_fn_156(int x) { return x + 156; } +inline int shared_fn_157(int x) { return x + 157; } +inline int shared_fn_158(int x) { return x + 158; } +inline int shared_fn_159(int x) { return x + 159; } +inline int shared_fn_160(int x) { return x + 160; } +inline int shared_fn_161(int x) { return x + 161; } +inline int shared_fn_162(int x) { return x + 162; } +inline int shared_fn_163(int x) { return x + 163; } +inline int shared_fn_164(int x) { return x + 164; } +inline int shared_fn_165(int x) { return x + 165; } +inline int shared_fn_166(int x) { return x + 166; } +inline int shared_fn_167(int x) { return x + 167; } +inline int shared_fn_168(int x) { return x + 168; } +inline int shared_fn_169(int x) { return x + 169; } +inline int shared_fn_170(int x) { return x + 170; } +inline int shared_fn_171(int x) { return x + 171; } +inline int shared_fn_172(int x) { return x + 172; } +inline int shared_fn_173(int x) { return x + 173; } +inline int shared_fn_174(int x) { return x + 174; } +inline int shared_fn_175(int x) { return x + 175; } +inline int shared_fn_176(int x) { return x + 176; } +inline int shared_fn_177(int x) { return x + 177; } +inline int shared_fn_178(int x) { return x + 178; } +inline int shared_fn_179(int x) { return x + 179; } +inline int shared_fn_180(int x) { return x + 180; } +inline int shared_fn_181(int x) { return x + 181; } +inline int shared_fn_182(int x) { return x + 182; } +inline int shared_fn_183(int x) { return x + 183; } +inline int shared_fn_184(int x) { return x + 184; } +inline int shared_fn_185(int x) { return x + 185; } +inline int shared_fn_186(int x) { return x + 186; } +inline int shared_fn_187(int x) { return x + 187; } +inline int shared_fn_188(int x) { return x + 188; } +inline int shared_fn_189(int x) { return x + 189; } +inline int shared_fn_190(int x) { return x + 190; } +inline int shared_fn_191(int x) { return x + 191; } +inline int shared_fn_192(int x) { return x + 192; } +inline int shared_fn_193(int x) { return x + 193; } +inline int shared_fn_194(int x) { return x + 194; } +inline int shared_fn_195(int x) { return x + 195; } +inline int shared_fn_196(int x) { return x + 196; } +inline int shared_fn_197(int x) { return x + 197; } +inline int shared_fn_198(int x) { return x + 198; } +inline int shared_fn_199(int x) { return x + 199; } +inline int shared_fn_200(int x) { return x + 200; } +inline int shared_fn_201(int x) { return x + 201; } +inline int shared_fn_202(int x) { return x + 202; } +inline int shared_fn_203(int x) { return x + 203; } +inline int shared_fn_204(int x) { return x + 204; } +inline int shared_fn_205(int x) { return x + 205; } +inline int shared_fn_206(int x) { return x + 206; } +inline int shared_fn_207(int x) { return x + 207; } +inline int shared_fn_208(int x) { return x + 208; } +inline int shared_fn_209(int x) { return x + 209; } +inline int shared_fn_210(int x) { return x + 210; } +inline int shared_fn_211(int x) { return x + 211; } +inline int shared_fn_212(int x) { return x + 212; } +inline int shared_fn_213(int x) { return x + 213; } +inline int shared_fn_214(int x) { return x + 214; } +inline int shared_fn_215(int x) { return x + 215; } +inline int shared_fn_216(int x) { return x + 216; } +inline int shared_fn_217(int x) { return x + 217; } +inline int shared_fn_218(int x) { return x + 218; } +inline int shared_fn_219(int x) { return x + 219; } +inline int shared_fn_220(int x) { return x + 220; } +inline int shared_fn_221(int x) { return x + 221; } +inline int shared_fn_222(int x) { return x + 222; } +inline int shared_fn_223(int x) { return x + 223; } +inline int shared_fn_224(int x) { return x + 224; } +inline int shared_fn_225(int x) { return x + 225; } +inline int shared_fn_226(int x) { return x + 226; } +inline int shared_fn_227(int x) { return x + 227; } +inline int shared_fn_228(int x) { return x + 228; } +inline int shared_fn_229(int x) { return x + 229; } +inline int shared_fn_230(int x) { return x + 230; } +inline int shared_fn_231(int x) { return x + 231; } +inline int shared_fn_232(int x) { return x + 232; } +inline int shared_fn_233(int x) { return x + 233; } +inline int shared_fn_234(int x) { return x + 234; } +inline int shared_fn_235(int x) { return x + 235; } +inline int shared_fn_236(int x) { return x + 236; } +inline int shared_fn_237(int x) { return x + 237; } +inline int shared_fn_238(int x) { return x + 238; } +inline int shared_fn_239(int x) { return x + 239; } +inline int shared_fn_240(int x) { return x + 240; } +inline int shared_fn_241(int x) { return x + 241; } +inline int shared_fn_242(int x) { return x + 242; } +inline int shared_fn_243(int x) { return x + 243; } +inline int shared_fn_244(int x) { return x + 244; } +inline int shared_fn_245(int x) { return x + 245; } +inline int shared_fn_246(int x) { return x + 246; } +inline int shared_fn_247(int x) { return x + 247; } +inline int shared_fn_248(int x) { return x + 248; } +inline int shared_fn_249(int x) { return x + 249; } +inline int shared_fn_250(int x) { return x + 250; } +inline int shared_fn_251(int x) { return x + 251; } +inline int shared_fn_252(int x) { return x + 252; } +inline int shared_fn_253(int x) { return x + 253; } +inline int shared_fn_254(int x) { return x + 254; } +inline int shared_fn_255(int x) { return x + 255; } +inline int shared_fn_256(int x) { return x + 256; } +inline int shared_fn_257(int x) { return x + 257; } +inline int shared_fn_258(int x) { return x + 258; } +inline int shared_fn_259(int x) { return x + 259; } +inline int shared_fn_260(int x) { return x + 260; } +inline int shared_fn_261(int x) { return x + 261; } +inline int shared_fn_262(int x) { return x + 262; } +inline int shared_fn_263(int x) { return x + 263; } +inline int shared_fn_264(int x) { return x + 264; } +inline int shared_fn_265(int x) { return x + 265; } +inline int shared_fn_266(int x) { return x + 266; } +inline int shared_fn_267(int x) { return x + 267; } +inline int shared_fn_268(int x) { return x + 268; } +inline int shared_fn_269(int x) { return x + 269; } +inline int shared_fn_270(int x) { return x + 270; } +inline int shared_fn_271(int x) { return x + 271; } +inline int shared_fn_272(int x) { return x + 272; } +inline int shared_fn_273(int x) { return x + 273; } +inline int shared_fn_274(int x) { return x + 274; } +inline in... [truncated] `````````` </details> https://github.com/llvm/llvm-project/pull/203154 _______________________________________________ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
