Re: r245779 - [modules] Rearrange how redeclaration chains are loaded, to remove a walk over
Richard Smith via cfe-commits cfe-commits@lists.llvm.org writes: Author: rsmith Date: Fri Aug 21 20:47:18 2015 New Revision: 245779 URL: http://llvm.org/viewvc/llvm-project?rev=245779view=rev Log: [modules] Rearrange how redeclaration chains are loaded, to remove a walk over all modules and reduce the number of declarations we load when loading a redeclaration chain. It looks like ASTDeclWriter::AddFirstDeclFromEachModule is called with a null `this` since this change, which has been causing ubsan failures on 76 tests since it went in: ASTWriterDecl.cpp:170:18: runtime error: member call on null pointer of type 'clang::ASTReader' Logs and other failures here: http://lab.llvm.org:8080/green/job/clang-stage2-cmake-RgSan_check/146/ I guess you must've missed the failure email. Could you please take a look? The new approach is: * when loading the first declaration of an entity within a module file, we first load all declarations of the entity that were imported into that module file, and then load all the other declarations of that entity from that module file and build a suitable decl chain from them * when loading any other declaration of an entity, we first load the first declaration from the same module file As before, we complete redecl chains through name lookup where necessary. To make this work, I also had to change the way that template specializations are stored -- it no longer suffices to track only canonical specializations; we now emit all first local declarations when emitting a list of specializations for a template. On one testcase with several thousand imported module files, this reduces the total runtime by 72%. Modified: cfe/trunk/include/clang/Serialization/ASTWriter.h cfe/trunk/lib/Serialization/ASTReader.cpp cfe/trunk/lib/Serialization/ASTReaderDecl.cpp cfe/trunk/lib/Serialization/ASTWriter.cpp cfe/trunk/lib/Serialization/ASTWriterDecl.cpp cfe/trunk/test/Modules/cxx-templates.cpp Modified: cfe/trunk/include/clang/Serialization/ASTWriter.h URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Serialization/ASTWriter.h?rev=245779r1=245778r2=245779view=diff == --- cfe/trunk/include/clang/Serialization/ASTWriter.h (original) +++ cfe/trunk/include/clang/Serialization/ASTWriter.h Fri Aug 21 20:47:18 2015 @@ -405,6 +405,10 @@ private: /// \brief The set of declarations that may have redeclaration chains that /// need to be serialized. llvm::SmallVectorconst Decl *, 16 Redeclarations; + + /// \brief A cache of the first local declaration for interesting + /// redeclaration chains. + llvm::DenseMapconst Decl *, const Decl * FirstLocalDeclCache; /// \brief Statements that we've encountered while serializing a /// declaration or type. @@ -676,6 +680,10 @@ public: const ASTTemplateArgumentListInfo *ASTTemplArgList, RecordDataImpl Record); + /// \brief Find the first local declaration of a given local redeclarable + /// decl. + const Decl *getFirstLocalDecl(const Decl *D); + /// \brief Emit a reference to a declaration. void AddDeclRef(const Decl *D, RecordDataImpl Record); @@ -857,12 +865,6 @@ public: void CompletedTagDefinition(const TagDecl *D) override; void AddedVisibleDecl(const DeclContext *DC, const Decl *D) override; void AddedCXXImplicitMember(const CXXRecordDecl *RD, const Decl *D) override; - void AddedCXXTemplateSpecialization(const ClassTemplateDecl *TD, - const ClassTemplateSpecializationDecl *D) override; - void AddedCXXTemplateSpecialization(const VarTemplateDecl *TD, - const VarTemplateSpecializationDecl *D) override; - void AddedCXXTemplateSpecialization(const FunctionTemplateDecl *TD, - const FunctionDecl *D) override; void ResolvedExceptionSpec(const FunctionDecl *FD) override; void DeducedReturnType(const FunctionDecl *FD, QualType ReturnType) override; void ResolvedOperatorDelete(const CXXDestructorDecl *DD, Modified: cfe/trunk/lib/Serialization/ASTReader.cpp URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Serialization/ASTReader.cpp?rev=245779r1=245778r2=245779view=diff == --- cfe/trunk/lib/Serialization/ASTReader.cpp (original) +++ cfe/trunk/lib/Serialization/ASTReader.cpp Fri Aug 21 20:47:18 2015 @@ -8123,11 +8123,8 @@ void ASTReader::finishPendingActions() { PendingIncompleteDeclChains.clear(); // Load pending declaration chains. -for (unsigned I = 0; I != PendingDeclChains.size(); ++I) { - PendingDeclChainsKnown.erase(PendingDeclChains[I]); +for (unsigned I = 0; I != PendingDeclChains.size();
Re: r245779 - [modules] Rearrange how redeclaration chains are loaded, to remove a walk over
On Thu, Aug 27, 2015 at 10:47 AM, Justin Bogner m...@justinbogner.com wrote: Richard Smith via cfe-commits cfe-commits@lists.llvm.org writes: Author: rsmith Date: Fri Aug 21 20:47:18 2015 New Revision: 245779 URL: http://llvm.org/viewvc/llvm-project?rev=245779view=rev Log: [modules] Rearrange how redeclaration chains are loaded, to remove a walk over all modules and reduce the number of declarations we load when loading a redeclaration chain. It looks like ASTDeclWriter::AddFirstDeclFromEachModule is called with a null `this` since this change, which has been causing ubsan failures on 76 tests since it went in: ASTWriterDecl.cpp:170:18: runtime error: member call on null pointer of type 'clang::ASTReader' Logs and other failures here: http://lab.llvm.org:8080/green/job/clang-stage2-cmake-RgSan_check/146/ I guess you must've missed the failure email. Could you please take a look? Thanks, fixed in r246215. The new approach is: * when loading the first declaration of an entity within a module file, we first load all declarations of the entity that were imported into that module file, and then load all the other declarations of that entity from that module file and build a suitable decl chain from them * when loading any other declaration of an entity, we first load the first declaration from the same module file As before, we complete redecl chains through name lookup where necessary. To make this work, I also had to change the way that template specializations are stored -- it no longer suffices to track only canonical specializations; we now emit all first local declarations when emitting a list of specializations for a template. On one testcase with several thousand imported module files, this reduces the total runtime by 72%. Modified: cfe/trunk/include/clang/Serialization/ASTWriter.h cfe/trunk/lib/Serialization/ASTReader.cpp cfe/trunk/lib/Serialization/ASTReaderDecl.cpp cfe/trunk/lib/Serialization/ASTWriter.cpp cfe/trunk/lib/Serialization/ASTWriterDecl.cpp cfe/trunk/test/Modules/cxx-templates.cpp Modified: cfe/trunk/include/clang/Serialization/ASTWriter.h URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Serialization/ASTWriter.h?rev=245779r1=245778r2=245779view=diff == --- cfe/trunk/include/clang/Serialization/ASTWriter.h (original) +++ cfe/trunk/include/clang/Serialization/ASTWriter.h Fri Aug 21 20:47:18 2015 @@ -405,6 +405,10 @@ private: /// \brief The set of declarations that may have redeclaration chains that /// need to be serialized. llvm::SmallVectorconst Decl *, 16 Redeclarations; + + /// \brief A cache of the first local declaration for interesting + /// redeclaration chains. + llvm::DenseMapconst Decl *, const Decl * FirstLocalDeclCache; /// \brief Statements that we've encountered while serializing a /// declaration or type. @@ -676,6 +680,10 @@ public: const ASTTemplateArgumentListInfo *ASTTemplArgList, RecordDataImpl Record); + /// \brief Find the first local declaration of a given local redeclarable + /// decl. + const Decl *getFirstLocalDecl(const Decl *D); + /// \brief Emit a reference to a declaration. void AddDeclRef(const Decl *D, RecordDataImpl Record); @@ -857,12 +865,6 @@ public: void CompletedTagDefinition(const TagDecl *D) override; void AddedVisibleDecl(const DeclContext *DC, const Decl *D) override; void AddedCXXImplicitMember(const CXXRecordDecl *RD, const Decl *D) override; - void AddedCXXTemplateSpecialization(const ClassTemplateDecl *TD, - const ClassTemplateSpecializationDecl *D) override; - void AddedCXXTemplateSpecialization(const VarTemplateDecl *TD, - const VarTemplateSpecializationDecl *D) override; - void AddedCXXTemplateSpecialization(const FunctionTemplateDecl *TD, - const FunctionDecl *D) override; void ResolvedExceptionSpec(const FunctionDecl *FD) override; void DeducedReturnType(const FunctionDecl *FD, QualType ReturnType) override; void ResolvedOperatorDelete(const CXXDestructorDecl *DD, Modified: cfe/trunk/lib/Serialization/ASTReader.cpp URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Serialization/ASTReader.cpp?rev=245779r1=245778r2=245779view=diff == --- cfe/trunk/lib/Serialization/ASTReader.cpp (original) +++ cfe/trunk/lib/Serialization/ASTReader.cpp Fri Aug 21 20:47:18 2015 @@ -8123,11 +8123,8 @@ void ASTReader::finishPendingActions() { PendingIncompleteDeclChains.clear(); // Load pending declaration chains. -for (unsigned I =
Re: r245779 - [modules] Rearrange how redeclaration chains are loaded, to remove a walk over
On Sat, Aug 22, 2015 at 1:16 AM, Vassil Vassilev vvasi...@cern.ch wrote: On 22/08/15 03:47, Richard Smith via cfe-commits wrote: Author: rsmith Date: Fri Aug 21 20:47:18 2015 New Revision: 245779 URL: http://llvm.org/viewvc/llvm-project?rev=245779view=rev Log: [modules] Rearrange how redeclaration chains are loaded, to remove a walk over all modules and reduce the number of declarations we load when loading a redeclaration chain. The new approach is: * when loading the first declaration of an entity within a module file, we first load all declarations of the entity that were imported into that module file, and then load all the other declarations of that entity from that module file and build a suitable decl chain from them * when loading any other declaration of an entity, we first load the first declaration from the same module file As before, we complete redecl chains through name lookup where necessary. To make this work, I also had to change the way that template specializations are stored -- it no longer suffices to track only canonical specializations; we now emit all first local declarations when emitting a list of specializations for a template. On one testcase with several thousand imported module files, this reduces the total runtime by 72%. Very nice! Does it reduce the depth of the redecl chains when merging? I.e. does this mean memory footprint reduction too? I wouldn't expect any difference there, and in any case, I think this would only affect the stack depth, which is typically bounded anyway since we normally build modules on a separate thread. Modified: cfe/trunk/include/clang/Serialization/ASTWriter.h cfe/trunk/lib/Serialization/ASTReader.cpp cfe/trunk/lib/Serialization/ASTReaderDecl.cpp cfe/trunk/lib/Serialization/ASTWriter.cpp cfe/trunk/lib/Serialization/ASTWriterDecl.cpp cfe/trunk/test/Modules/cxx-templates.cpp Modified: cfe/trunk/include/clang/Serialization/ASTWriter.h URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Serialization/ASTWriter.h?rev=245779r1=245778r2=245779view=diff == --- cfe/trunk/include/clang/Serialization/ASTWriter.h (original) +++ cfe/trunk/include/clang/Serialization/ASTWriter.h Fri Aug 21 20:47:18 2015 @@ -405,6 +405,10 @@ private: /// \brief The set of declarations that may have redeclaration chains that /// need to be serialized. llvm::SmallVectorconst Decl *, 16 Redeclarations; + + /// \brief A cache of the first local declaration for interesting + /// redeclaration chains. + llvm::DenseMapconst Decl *, const Decl * FirstLocalDeclCache; /// \brief Statements that we've encountered while serializing a /// declaration or type. @@ -676,6 +680,10 @@ public: const ASTTemplateArgumentListInfo *ASTTemplArgList, RecordDataImpl Record); + /// \brief Find the first local declaration of a given local redeclarable + /// decl. + const Decl *getFirstLocalDecl(const Decl *D); + /// \brief Emit a reference to a declaration. void AddDeclRef(const Decl *D, RecordDataImpl Record); @@ -857,12 +865,6 @@ public: void CompletedTagDefinition(const TagDecl *D) override; void AddedVisibleDecl(const DeclContext *DC, const Decl *D) override; void AddedCXXImplicitMember(const CXXRecordDecl *RD, const Decl *D) override; - void AddedCXXTemplateSpecialization(const ClassTemplateDecl *TD, - const ClassTemplateSpecializationDecl *D) override; - void AddedCXXTemplateSpecialization(const VarTemplateDecl *TD, - const VarTemplateSpecializationDecl *D) override; - void AddedCXXTemplateSpecialization(const FunctionTemplateDecl *TD, - const FunctionDecl *D) override; void ResolvedExceptionSpec(const FunctionDecl *FD) override; void DeducedReturnType(const FunctionDecl *FD, QualType ReturnType) override; void ResolvedOperatorDelete(const CXXDestructorDecl *DD, Modified: cfe/trunk/lib/Serialization/ASTReader.cpp URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Serialization/ASTReader.cpp?rev=245779r1=245778r2=245779view=diff == --- cfe/trunk/lib/Serialization/ASTReader.cpp (original) +++ cfe/trunk/lib/Serialization/ASTReader.cpp Fri Aug 21 20:47:18 2015 @@ -8123,11 +8123,8 @@ void ASTReader::finishPendingActions() { PendingIncompleteDeclChains.clear(); // Load pending declaration chains. -for (unsigned I = 0; I != PendingDeclChains.size(); ++I) { - PendingDeclChainsKnown.erase(PendingDeclChains[I]); +for (unsigned I = 0; I != PendingDeclChains.size(); ++I)
Re: r245779 - [modules] Rearrange how redeclaration chains are loaded, to remove a walk over
On 22/08/15 03:47, Richard Smith via cfe-commits wrote: Author: rsmith Date: Fri Aug 21 20:47:18 2015 New Revision: 245779 URL: http://llvm.org/viewvc/llvm-project?rev=245779view=rev Log: [modules] Rearrange how redeclaration chains are loaded, to remove a walk over all modules and reduce the number of declarations we load when loading a redeclaration chain. The new approach is: * when loading the first declaration of an entity within a module file, we first load all declarations of the entity that were imported into that module file, and then load all the other declarations of that entity from that module file and build a suitable decl chain from them * when loading any other declaration of an entity, we first load the first declaration from the same module file As before, we complete redecl chains through name lookup where necessary. To make this work, I also had to change the way that template specializations are stored -- it no longer suffices to track only canonical specializations; we now emit all first local declarations when emitting a list of specializations for a template. On one testcase with several thousand imported module files, this reduces the total runtime by 72%. Very nice! Does it reduce the depth of the redecl chains when merging? I.e. does this mean memory footprint reduction too? Modified: cfe/trunk/include/clang/Serialization/ASTWriter.h cfe/trunk/lib/Serialization/ASTReader.cpp cfe/trunk/lib/Serialization/ASTReaderDecl.cpp cfe/trunk/lib/Serialization/ASTWriter.cpp cfe/trunk/lib/Serialization/ASTWriterDecl.cpp cfe/trunk/test/Modules/cxx-templates.cpp Modified: cfe/trunk/include/clang/Serialization/ASTWriter.h URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Serialization/ASTWriter.h?rev=245779r1=245778r2=245779view=diff == --- cfe/trunk/include/clang/Serialization/ASTWriter.h (original) +++ cfe/trunk/include/clang/Serialization/ASTWriter.h Fri Aug 21 20:47:18 2015 @@ -405,6 +405,10 @@ private: /// \brief The set of declarations that may have redeclaration chains that /// need to be serialized. llvm::SmallVectorconst Decl *, 16 Redeclarations; + + /// \brief A cache of the first local declaration for interesting + /// redeclaration chains. + llvm::DenseMapconst Decl *, const Decl * FirstLocalDeclCache; /// \brief Statements that we've encountered while serializing a /// declaration or type. @@ -676,6 +680,10 @@ public: const ASTTemplateArgumentListInfo *ASTTemplArgList, RecordDataImpl Record); + /// \brief Find the first local declaration of a given local redeclarable + /// decl. + const Decl *getFirstLocalDecl(const Decl *D); + /// \brief Emit a reference to a declaration. void AddDeclRef(const Decl *D, RecordDataImpl Record); @@ -857,12 +865,6 @@ public: void CompletedTagDefinition(const TagDecl *D) override; void AddedVisibleDecl(const DeclContext *DC, const Decl *D) override; void AddedCXXImplicitMember(const CXXRecordDecl *RD, const Decl *D) override; - void AddedCXXTemplateSpecialization(const ClassTemplateDecl *TD, - const ClassTemplateSpecializationDecl *D) override; - void AddedCXXTemplateSpecialization(const VarTemplateDecl *TD, - const VarTemplateSpecializationDecl *D) override; - void AddedCXXTemplateSpecialization(const FunctionTemplateDecl *TD, - const FunctionDecl *D) override; void ResolvedExceptionSpec(const FunctionDecl *FD) override; void DeducedReturnType(const FunctionDecl *FD, QualType ReturnType) override; void ResolvedOperatorDelete(const CXXDestructorDecl *DD, Modified: cfe/trunk/lib/Serialization/ASTReader.cpp URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Serialization/ASTReader.cpp?rev=245779r1=245778r2=245779view=diff == --- cfe/trunk/lib/Serialization/ASTReader.cpp (original) +++ cfe/trunk/lib/Serialization/ASTReader.cpp Fri Aug 21 20:47:18 2015 @@ -8123,11 +8123,8 @@ void ASTReader::finishPendingActions() { PendingIncompleteDeclChains.clear(); // Load pending declaration chains. -for (unsigned I = 0; I != PendingDeclChains.size(); ++I) { - PendingDeclChainsKnown.erase(PendingDeclChains[I]); +for (unsigned I = 0; I != PendingDeclChains.size(); ++I) loadPendingDeclChain(PendingDeclChains[I]); -} -assert(PendingDeclChainsKnown.empty()); PendingDeclChains.clear(); assert(RedeclsDeserialized.empty() some redecls not wired up); Modified: cfe/trunk/lib/Serialization/ASTReaderDecl.cpp URL:
r245779 - [modules] Rearrange how redeclaration chains are loaded, to remove a walk over
Author: rsmith Date: Fri Aug 21 20:47:18 2015 New Revision: 245779 URL: http://llvm.org/viewvc/llvm-project?rev=245779view=rev Log: [modules] Rearrange how redeclaration chains are loaded, to remove a walk over all modules and reduce the number of declarations we load when loading a redeclaration chain. The new approach is: * when loading the first declaration of an entity within a module file, we first load all declarations of the entity that were imported into that module file, and then load all the other declarations of that entity from that module file and build a suitable decl chain from them * when loading any other declaration of an entity, we first load the first declaration from the same module file As before, we complete redecl chains through name lookup where necessary. To make this work, I also had to change the way that template specializations are stored -- it no longer suffices to track only canonical specializations; we now emit all first local declarations when emitting a list of specializations for a template. On one testcase with several thousand imported module files, this reduces the total runtime by 72%. Modified: cfe/trunk/include/clang/Serialization/ASTWriter.h cfe/trunk/lib/Serialization/ASTReader.cpp cfe/trunk/lib/Serialization/ASTReaderDecl.cpp cfe/trunk/lib/Serialization/ASTWriter.cpp cfe/trunk/lib/Serialization/ASTWriterDecl.cpp cfe/trunk/test/Modules/cxx-templates.cpp Modified: cfe/trunk/include/clang/Serialization/ASTWriter.h URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Serialization/ASTWriter.h?rev=245779r1=245778r2=245779view=diff == --- cfe/trunk/include/clang/Serialization/ASTWriter.h (original) +++ cfe/trunk/include/clang/Serialization/ASTWriter.h Fri Aug 21 20:47:18 2015 @@ -405,6 +405,10 @@ private: /// \brief The set of declarations that may have redeclaration chains that /// need to be serialized. llvm::SmallVectorconst Decl *, 16 Redeclarations; + + /// \brief A cache of the first local declaration for interesting + /// redeclaration chains. + llvm::DenseMapconst Decl *, const Decl * FirstLocalDeclCache; /// \brief Statements that we've encountered while serializing a /// declaration or type. @@ -676,6 +680,10 @@ public: const ASTTemplateArgumentListInfo *ASTTemplArgList, RecordDataImpl Record); + /// \brief Find the first local declaration of a given local redeclarable + /// decl. + const Decl *getFirstLocalDecl(const Decl *D); + /// \brief Emit a reference to a declaration. void AddDeclRef(const Decl *D, RecordDataImpl Record); @@ -857,12 +865,6 @@ public: void CompletedTagDefinition(const TagDecl *D) override; void AddedVisibleDecl(const DeclContext *DC, const Decl *D) override; void AddedCXXImplicitMember(const CXXRecordDecl *RD, const Decl *D) override; - void AddedCXXTemplateSpecialization(const ClassTemplateDecl *TD, - const ClassTemplateSpecializationDecl *D) override; - void AddedCXXTemplateSpecialization(const VarTemplateDecl *TD, - const VarTemplateSpecializationDecl *D) override; - void AddedCXXTemplateSpecialization(const FunctionTemplateDecl *TD, - const FunctionDecl *D) override; void ResolvedExceptionSpec(const FunctionDecl *FD) override; void DeducedReturnType(const FunctionDecl *FD, QualType ReturnType) override; void ResolvedOperatorDelete(const CXXDestructorDecl *DD, Modified: cfe/trunk/lib/Serialization/ASTReader.cpp URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Serialization/ASTReader.cpp?rev=245779r1=245778r2=245779view=diff == --- cfe/trunk/lib/Serialization/ASTReader.cpp (original) +++ cfe/trunk/lib/Serialization/ASTReader.cpp Fri Aug 21 20:47:18 2015 @@ -8123,11 +8123,8 @@ void ASTReader::finishPendingActions() { PendingIncompleteDeclChains.clear(); // Load pending declaration chains. -for (unsigned I = 0; I != PendingDeclChains.size(); ++I) { - PendingDeclChainsKnown.erase(PendingDeclChains[I]); +for (unsigned I = 0; I != PendingDeclChains.size(); ++I) loadPendingDeclChain(PendingDeclChains[I]); -} -assert(PendingDeclChainsKnown.empty()); PendingDeclChains.clear(); assert(RedeclsDeserialized.empty() some redecls not wired up); Modified: cfe/trunk/lib/Serialization/ASTReaderDecl.cpp URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Serialization/ASTReaderDecl.cpp?rev=245779r1=245778r2=245779view=diff == --- cfe/trunk/lib/Serialization/ASTReaderDecl.cpp (original) +++ cfe/trunk/lib/Serialization/ASTReaderDecl.cpp Fri Aug 21 20:47:18 2015 @@