https://github.com/benlangmuir created https://github.com/llvm/llvm-project/pull/229219
In some paths through LexDependencyDirectiveToken we were not calling startToken, so an uninitialized token could cause an assert if we called setLength before setKind. Match Lex() by calling startToken() at the start of the function consistently. The unit test is a bit subtle in order to setup an annotated token that will reach the assert in setLength at the right point. The actual failure was seen in the uninitialized token in CheckEndOfDirective, which is hard to mimic, and the real one only triggered it very rarely. rdar://189061928 >From 7ea0181bfee5f3af9740d4d9c4ecb122b61e21ac Mon Sep 17 00:00:00 2001 From: Ben Langmuir <[email protected]> Date: Mon, 5 Oct 2026 13:44:55 -0700 Subject: [PATCH] [clang][lex] Fix assertion in dependency directives lexing In some paths through LexDependencyDirectiveToken we were not calling startToken, so an uninitialized token could cause an assert if we called setLength before setKind. Match Lex() by calling startToken() at the start of the function consistently. The unit test is a bit subtle in order to setup an annotated token that will reach the assert in setLength at the right point. The actual failure was seen in the uninitialized token in CheckEndOfDirective, which is hard to mimic, and the real one only triggered it very rarely. rdar://189061928 --- clang/lib/Lex/Lexer.cpp | 2 + .../Lex/PPDependencyDirectivesTest.cpp | 61 +++++++++++++++++++ 2 files changed, 63 insertions(+) diff --git a/clang/lib/Lex/Lexer.cpp b/clang/lib/Lex/Lexer.cpp index 4762b38e67c00af..50162b11360de2e 100644 --- a/clang/lib/Lex/Lexer.cpp +++ b/clang/lib/Lex/Lexer.cpp @@ -4679,6 +4679,8 @@ const char *Lexer::convertDependencyDirectiveToken( bool Lexer::LexDependencyDirectiveToken(Token &Result) { assert(isDependencyDirectivesLexer()); + Result.startToken(); + using namespace dependency_directives_scan; if (BufferPtr == BufferEnd) diff --git a/clang/unittests/Lex/PPDependencyDirectivesTest.cpp b/clang/unittests/Lex/PPDependencyDirectivesTest.cpp index 7b11fc163f6eb81..1bd4c7721f8d96c 100644 --- a/clang/unittests/Lex/PPDependencyDirectivesTest.cpp +++ b/clang/unittests/Lex/PPDependencyDirectivesTest.cpp @@ -305,4 +305,65 @@ TEST_F(PPDependencyDirectivesTest, Embed) { EXPECT_EQ(EmbeddedFiles, ExpectedEmbeds); } +TEST_F(PPDependencyDirectivesTest, LexIntoUninitializedToken) { + // The dependency directives lexer must not depend on the token it is given, + // which may be uninitialized. The following checks a token that happens to + // get annotation kind must not assert in setLength(). + auto VFS = new llvm::vfs::InMemoryFileSystem(); + VFS->addFile("header.h", 0, llvm::MemoryBuffer::getMemBuffer("")); + VFS->addFile("main.c", 0, + llvm::MemoryBuffer::getMemBuffer( + "#if __has_include(\"header.h\")")); + FileMgr.setVirtualFileSystem(VFS); + + OptionalFileEntryRef FE; + ASSERT_THAT_ERROR(FileMgr.getFileRef("main.c").moveInto(FE), + llvm::Succeeded()); + SourceMgr.setMainFileID( + SourceMgr.createFileID(*FE, SourceLocation(), SrcMgr::C_User)); + + TestDependencyDirectivesGetter GetDependencyDirectives(FileMgr); + + PreprocessorOptions PPOpts; + HeaderSearchOptions HSOpts; + TrivialModuleLoader ModLoader; + HeaderSearch HeaderInfo(HSOpts, SourceMgr, Diags, LangOpts, Target.get()); + Preprocessor PP(PPOpts, Diags, LangOpts, SourceMgr, HeaderInfo, ModLoader, + /*IILookup =*/nullptr, + /*OwnsHeaderSearch =*/false); + PP.Initialize(*Target); + PP.setDependencyDirectivesGetter(GetDependencyDirectives); + + // Once __has_include has been evaluated we are still inside the directive + // and at the end of the buffer; lex the 'eod' into a token we control. + class HasIncludeCatcher : public PPCallbacks { + Preprocessor &PP; + + public: + bool Lexed = false; + HasIncludeCatcher(Preprocessor &PP) : PP(PP) {} + + void HasInclude(SourceLocation Loc, StringRef FileName, bool IsAngled, + OptionalFileEntryRef File, + SrcMgr::CharacteristicKind FileType) override { + Token Tok; + Tok.startToken(); + Tok.setKind(tok::annot_module_include); + PP.LexUnexpandedToken(Tok); + EXPECT_EQ(Tok.getKind(), tok::eod); + // Lexing the 'eod' ended the directive; restore the state so the + // directive being evaluated can lex its own 'eod'. + PP.getCurrentLexer()->setParsingPreprocessorDirective(true); + Lexed = true; + } + }; + auto Owned = std::make_unique<HasIncludeCatcher>(PP); + HasIncludeCatcher *Callbacks = Owned.get(); + PP.addPPCallbacks(std::move(Owned)); + + PP.EnterMainSourceFile(); + PP.LexTokensUntilEOF(); + EXPECT_TRUE(Callbacks->Lexed); +} + } // anonymous namespace _______________________________________________ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
