https://github.com/hongtaihu created https://github.com/llvm/llvm-project/pull/212242
### Summary `TokenCollector` receives `tok::eod` from the preprocessor for an end-of-directive marker, but that token has no one-to-one raw spelling. Recording it as an expanded token can therefore make the expanded-to-spelled mapping fail in clangd. Skip `tok::eod` alongside non-module annotation tokens. Keep the existing `annot_module_name` reconstruction path, as it intentionally supplies a spelling. Add a regression test for: ```cpp } #pragma clang __debug dump ``` ### Test Plan - `tools/clang/unittests/AllClangUnitTests --gtest_filter=TokenCollectorTest.DebugPragmaAtEndOfFile` - Replayed issue #197652 through clangd `textDocument/hover`; clangd returned the hover response and no longer hit the expanded-to-spelled mapping assertion. >From f00c6734cc3b015166cb80309ad6cc975a2de2da Mon Sep 17 00:00:00 2001 From: hongtaihu <[email protected]> Date: Mon, 27 Jul 2026 20:58:49 +0800 Subject: [PATCH] [clang][Tooling] Ignore end-of-directive tokens in TokenCollector --- clang/lib/Tooling/Syntax/Tokens.cpp | 9 ++++++--- clang/unittests/Tooling/Syntax/TokensTest.cpp | 9 +++++++++ 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/clang/lib/Tooling/Syntax/Tokens.cpp b/clang/lib/Tooling/Syntax/Tokens.cpp index 927333fda18bb..6b75c17cab67e 100644 --- a/clang/lib/Tooling/Syntax/Tokens.cpp +++ b/clang/lib/Tooling/Syntax/Tokens.cpp @@ -688,11 +688,14 @@ TokenCollector::TokenCollector(Preprocessor &PP) : PP(PP) { this->PP.getLangOpts()); Expanded.push_back( syntax::Token(T.getLocation(), Text.size(), tok::annot_module_name)); - } else if (T.isAnnotation()) { return; - } else { - Expanded.push_back(syntax::Token(T)); } + + // These tokens do not have a one-to-one raw spelling. + if (T.isAnnotation() || T.is(tok::eod)) + return; + + Expanded.push_back(syntax::Token(T)); DEBUG_WITH_TYPE("collect-tokens", llvm::dbgs() << "Token: " << syntax::Token(T).dumpForTests( diff --git a/clang/unittests/Tooling/Syntax/TokensTest.cpp b/clang/unittests/Tooling/Syntax/TokensTest.cpp index 6418cc8f87d9d..ebc640b8d0923 100644 --- a/clang/unittests/Tooling/Syntax/TokensTest.cpp +++ b/clang/unittests/Tooling/Syntax/TokensTest.cpp @@ -1169,6 +1169,15 @@ TEST_F(TokenCollectorTest, Pragmas) { )cpp"); } +TEST_F(TokenCollectorTest, DebugPragmaAtEndOfFile) { + AllowErrors = true; + recordTokens("}\n#pragma clang __debug dump\n"); + + // The end-of-directive token has no spelling and must not be collected. + EXPECT_THAT(Buffer.expandedTokens(), + ElementsAre(Kind(tok::r_brace), Kind(tok::eof))); +} + TEST_F(TokenBufferTest, EofTokenOnBracketDepthLimit) { // Force parser to bail out due to exceeding the bracket depth limit. recordTokens("((;", {"-fbracket-depth=1"}); _______________________________________________ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
