[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
ilya-biryukov marked an inline comment as done. ilya-biryukov added inline comments. Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:270 + for (const syntax::Token : spelledTokensTouching(Loc, Tokens)) +if (Tok.kind() == tok::identifier) + return sammccall wrote: > ilya-biryukov wrote: > > kbobyrev wrote: > > > ilya-biryukov wrote: > > > > sammccall wrote: > > > > > ilya-biryukov wrote: > > > > > > NIT: add braces around `if` statement > > > > > Is there some reference for preferred LLVM style for this, or > > > > > personal preference? (Real question, as this comes up a bunch) > > > > > > > > > > I ask because that's my *least*-preferred option - no braces > braces > > > > > on for > braces on both > braces on (only) if. > > > > > > > > > > Added braces to the `for` (maybe that's what you meant?) > > > > Not sure if it's in LLVM style guide, but files inside Syntax folder > > > > definitely use this style: put braces everywhere until you reach the > > > > last level of nesting for non-leaf statements (i.e. having other > > > > statements as children, e.g. loops,. if statements, etc) > > > > > > > > > > > > It's my personal preference, happy to discuss whether having this makes > > > > sense. > > > I guess it's a personal preference (also for me), but I don't think there > > > is a strict guideline on that. Interestingly enough, I think there is a > > > piece of code in the styleguide that looks exactly like the code you had: > > > https://llvm.org/docs/CodingStandards.html#turn-predicate-loops-into-predicate-functions > > > > > > Some Clang subprojects tend to put braces everywhere though. > > > > > > That being said, I guess no braces at all would be the best option here. > > Yeah, so LLVM style is unclear, but has examples with no braces. > > Could be the argument to get rid of those. > > > > I'd still stick to my personal preferences where I can, but if everyone > > thinks no braces is better, happy to go with this option. > I think we're good here - I'm fine with the braces on `for`, and it sounds > like that's what you wanted. > (I guess I misread "braces around if statement" as "braces around if body") Ah, great, we're on the same page then! Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D71356/new/ https://reviews.llvm.org/D71356 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
This revision was automatically updated to reflect the committed changes.
sammccall marked 2 inline comments as done.
Closed by commit rG3f8da5d09107: [Tooling/Syntax] Helpers to find spelled
tokens touching a location. (authored by sammccall).
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D71356/new/
https://reviews.llvm.org/D71356
Files:
clang/include/clang/Basic/SourceLocation.h
clang/include/clang/Tooling/Syntax/Tokens.h
clang/lib/Tooling/Syntax/Tokens.cpp
clang/unittests/Tooling/Syntax/TokensTest.cpp
Index: clang/unittests/Tooling/Syntax/TokensTest.cpp
===
--- clang/unittests/Tooling/Syntax/TokensTest.cpp
+++ clang/unittests/Tooling/Syntax/TokensTest.cpp
@@ -793,4 +793,45 @@
ActualMacroRanges.push_back(Expansion->range(SM));
EXPECT_EQ(ExpectedMacroRanges, ActualMacroRanges);
}
+
+TEST_F(TokenBufferTest, Touching) {
+ llvm::Annotations Code("^i^nt^ ^a^b^=^1;^");
+ recordTokens(Code.code());
+
+ auto Touching = [&](int Index) {
+SourceLocation Loc = SourceMgr->getComposedLoc(SourceMgr->getMainFileID(),
+ Code.points()[Index]);
+return spelledTokensTouching(Loc, Buffer);
+ };
+ auto Identifier = [&](int Index) {
+SourceLocation Loc = SourceMgr->getComposedLoc(SourceMgr->getMainFileID(),
+ Code.points()[Index]);
+const syntax::Token *Tok = spelledIdentifierTouching(Loc, Buffer);
+return Tok ? Tok->text(*SourceMgr) : "";
+ };
+
+ EXPECT_THAT(Touching(0), SameRange(findSpelled("int")));
+ EXPECT_EQ(Identifier(0), "");
+ EXPECT_THAT(Touching(1), SameRange(findSpelled("int")));
+ EXPECT_EQ(Identifier(1), "");
+ EXPECT_THAT(Touching(2), SameRange(findSpelled("int")));
+ EXPECT_EQ(Identifier(2), "");
+
+ EXPECT_THAT(Touching(3), SameRange(findSpelled("ab")));
+ EXPECT_EQ(Identifier(3), "ab");
+ EXPECT_THAT(Touching(4), SameRange(findSpelled("ab")));
+ EXPECT_EQ(Identifier(4), "ab");
+
+ EXPECT_THAT(Touching(5), SameRange(findSpelled("ab =")));
+ EXPECT_EQ(Identifier(5), "ab");
+
+ EXPECT_THAT(Touching(6), SameRange(findSpelled("= 1")));
+ EXPECT_EQ(Identifier(6), "");
+
+ EXPECT_THAT(Touching(7), SameRange(findSpelled(";")));
+ EXPECT_EQ(Identifier(7), "");
+
+ ASSERT_EQ(Code.points().size(), 8u);
+}
+
} // namespace
Index: clang/lib/Tooling/Syntax/Tokens.cpp
===
--- clang/lib/Tooling/Syntax/Tokens.cpp
+++ clang/lib/Tooling/Syntax/Tokens.cpp
@@ -248,6 +248,31 @@
return E;
}
+llvm::ArrayRef
+syntax::spelledTokensTouching(SourceLocation Loc,
+ const syntax::TokenBuffer ) {
+ assert(Loc.isFileID());
+ llvm::ArrayRef All =
+ Tokens.spelledTokens(Tokens.sourceManager().getFileID(Loc));
+ // Comparing SourceLocations is well-defined within a FileID.
+ auto *Right = llvm::partition_point(
+ All, [&](const syntax::Token ) { return Tok.location() < Loc; });
+ bool AcceptRight = Right != All.end() && Right->location() <= Loc;
+ bool AcceptLeft = Right != All.begin() && (Right - 1)->endLocation() >= Loc;
+ return llvm::makeArrayRef(Right - (AcceptLeft ? 1 : 0),
+Right + (AcceptRight ? 1 : 0));
+}
+
+const syntax::Token *
+syntax::spelledIdentifierTouching(SourceLocation Loc,
+ const syntax::TokenBuffer ) {
+ for (const syntax::Token : spelledTokensTouching(Loc, Tokens)) {
+if (Tok.kind() == tok::identifier)
+ return
+ }
+ return nullptr;
+}
+
std::vector
TokenBuffer::macroExpansions(FileID FID) const {
auto FileIt = Files.find(FID);
Index: clang/include/clang/Tooling/Syntax/Tokens.h
===
--- clang/include/clang/Tooling/Syntax/Tokens.h
+++ clang/include/clang/Tooling/Syntax/Tokens.h
@@ -309,6 +309,17 @@
const SourceManager *SourceMgr;
};
+/// The spelled tokens that overlap or touch a spelling location Loc.
+/// This always returns 0-2 tokens.
+llvm::ArrayRef
+spelledTokensTouching(SourceLocation Loc, const syntax::TokenBuffer );
+
+/// The identifier token that overlaps or touches a spelling location Loc.
+/// If there is none, returns nullptr.
+const syntax::Token *
+spelledIdentifierTouching(SourceLocation Loc,
+ const syntax::TokenBuffer );
+
/// Lex the text buffer, corresponding to \p FID, in raw mode and record the
/// resulting spelled tokens. Does minimal post-processing on raw identifiers,
/// setting the appropriate token kind (instead of the raw_identifier reported
Index: clang/include/clang/Basic/SourceLocation.h
===
--- clang/include/clang/Basic/SourceLocation.h
+++ clang/include/clang/Basic/SourceLocation.h
@@ -188,9 +188,20 @@
return !(LHS == RHS);
}
+// Ordering is meaningful
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
sammccall marked 6 inline comments as done. sammccall added inline comments. Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:261 + bool AcceptRight = Right != All.end() && !(Loc < Right->location()); + bool AcceptLeft = Right != All.begin() && !((Right - 1)->endLocation() < Loc); + return llvm::makeArrayRef(Right - (AcceptLeft ? 1 : 0), ilya-biryukov wrote: > sammccall wrote: > > ilya-biryukov wrote: > > > Maybe compare offsets instead of locations? > > > It might be more code, but it would read much cleaner. > > > > > > It always feels weird to compare source locations for anything other than > > > storing them in `std::map` > > I find the `SM.getFileOffset(...)` everywhere pretty hard to read and > > obscures the actual locations. > > But agree about the operators, I've added the rest. Having `operator<` is > > pretty evil, but I don't think the others are particularly incrementally > > evil. > I find it to be the only way to write code, which unambiguously says what it > wants to do with source locations. > > Not saying it's pretty, but I the source location API was designed to be used > this way. Arguably, using something else in special corner cases does more > harm than good (hard-to-read, non-conventional code). > > But up to you. > But up to you. I would rather use the operators, with the comment for the unusual usage, if that's OK. (I agree it doesn't generalize and about the intent of the APIs, but think the APIs hurt readability to the extent that there's less value in consistency than in clear intent at individual callsites). Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:270 + for (const syntax::Token : spelledTokensTouching(Loc, Tokens)) +if (Tok.kind() == tok::identifier) + return ilya-biryukov wrote: > kbobyrev wrote: > > ilya-biryukov wrote: > > > sammccall wrote: > > > > ilya-biryukov wrote: > > > > > NIT: add braces around `if` statement > > > > Is there some reference for preferred LLVM style for this, or personal > > > > preference? (Real question, as this comes up a bunch) > > > > > > > > I ask because that's my *least*-preferred option - no braces > braces > > > > on for > braces on both > braces on (only) if. > > > > > > > > Added braces to the `for` (maybe that's what you meant?) > > > Not sure if it's in LLVM style guide, but files inside Syntax folder > > > definitely use this style: put braces everywhere until you reach the last > > > level of nesting for non-leaf statements (i.e. having other statements as > > > children, e.g. loops,. if statements, etc) > > > > > > > > > It's my personal preference, happy to discuss whether having this makes > > > sense. > > I guess it's a personal preference (also for me), but I don't think there > > is a strict guideline on that. Interestingly enough, I think there is a > > piece of code in the styleguide that looks exactly like the code you had: > > https://llvm.org/docs/CodingStandards.html#turn-predicate-loops-into-predicate-functions > > > > Some Clang subprojects tend to put braces everywhere though. > > > > That being said, I guess no braces at all would be the best option here. > Yeah, so LLVM style is unclear, but has examples with no braces. > Could be the argument to get rid of those. > > I'd still stick to my personal preferences where I can, but if everyone > thinks no braces is better, happy to go with this option. I think we're good here - I'm fine with the braces on `for`, and it sounds like that's what you wanted. (I guess I misread "braces around if statement" as "braces around if body") Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D71356/new/ https://reviews.llvm.org/D71356 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
merge_guards_bot added a comment.
{icon check-circle color=green} Unit tests: pass. 60699 tests passed, 0 failed
and 726 were skipped.
{icon check-circle color=green} clang-format: pass.
Build artifacts
:
console-log.txt
,
CMakeCache.txt
,
test-results.xml
,
diff.json
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D71356/new/
https://reviews.llvm.org/D71356
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
ilya-biryukov added inline comments. Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:270 + for (const syntax::Token : spelledTokensTouching(Loc, Tokens)) +if (Tok.kind() == tok::identifier) + return kbobyrev wrote: > ilya-biryukov wrote: > > sammccall wrote: > > > ilya-biryukov wrote: > > > > NIT: add braces around `if` statement > > > Is there some reference for preferred LLVM style for this, or personal > > > preference? (Real question, as this comes up a bunch) > > > > > > I ask because that's my *least*-preferred option - no braces > braces on > > > for > braces on both > braces on (only) if. > > > > > > Added braces to the `for` (maybe that's what you meant?) > > Not sure if it's in LLVM style guide, but files inside Syntax folder > > definitely use this style: put braces everywhere until you reach the last > > level of nesting for non-leaf statements (i.e. having other statements as > > children, e.g. loops,. if statements, etc) > > > > > > It's my personal preference, happy to discuss whether having this makes > > sense. > I guess it's a personal preference (also for me), but I don't think there is > a strict guideline on that. Interestingly enough, I think there is a piece of > code in the styleguide that looks exactly like the code you had: > https://llvm.org/docs/CodingStandards.html#turn-predicate-loops-into-predicate-functions > > Some Clang subprojects tend to put braces everywhere though. > > That being said, I guess no braces at all would be the best option here. Yeah, so LLVM style is unclear, but has examples with no braces. Could be the argument to get rid of those. I'd still stick to my personal preferences where I can, but if everyone thinks no braces is better, happy to go with this option. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D71356/new/ https://reviews.llvm.org/D71356 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
ilya-biryukov marked an inline comment as done. ilya-biryukov added inline comments. Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:261 + bool AcceptRight = Right != All.end() && !(Loc < Right->location()); + bool AcceptLeft = Right != All.begin() && !((Right - 1)->endLocation() < Loc); + return llvm::makeArrayRef(Right - (AcceptLeft ? 1 : 0), sammccall wrote: > ilya-biryukov wrote: > > Maybe compare offsets instead of locations? > > It might be more code, but it would read much cleaner. > > > > It always feels weird to compare source locations for anything other than > > storing them in `std::map` > I find the `SM.getFileOffset(...)` everywhere pretty hard to read and > obscures the actual locations. > But agree about the operators, I've added the rest. Having `operator<` is > pretty evil, but I don't think the others are particularly incrementally evil. I find it to be the only way to write code, which unambiguously says what it wants to do with source locations. Not saying it's pretty, but I the source location API was designed to be used this way. Arguably, using something else in special corner cases does more harm than good (hard-to-read, non-conventional code). But up to you. Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:270 + for (const syntax::Token : spelledTokensTouching(Loc, Tokens)) +if (Tok.kind() == tok::identifier) + return sammccall wrote: > ilya-biryukov wrote: > > NIT: add braces around `if` statement > Is there some reference for preferred LLVM style for this, or personal > preference? (Real question, as this comes up a bunch) > > I ask because that's my *least*-preferred option - no braces > braces on for > > braces on both > braces on (only) if. > > Added braces to the `for` (maybe that's what you meant?) Not sure if it's in LLVM style guide, but files inside Syntax folder definitely use this style: put braces everywhere until you reach the last level of nesting for non-leaf statements (i.e. having other statements as children, e.g. loops,. if statements, etc) It's my personal preference, happy to discuss whether having this makes sense. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D71356/new/ https://reviews.llvm.org/D71356 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
kbobyrev added inline comments. Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:270 + for (const syntax::Token : spelledTokensTouching(Loc, Tokens)) +if (Tok.kind() == tok::identifier) + return ilya-biryukov wrote: > sammccall wrote: > > ilya-biryukov wrote: > > > NIT: add braces around `if` statement > > Is there some reference for preferred LLVM style for this, or personal > > preference? (Real question, as this comes up a bunch) > > > > I ask because that's my *least*-preferred option - no braces > braces on > > for > braces on both > braces on (only) if. > > > > Added braces to the `for` (maybe that's what you meant?) > Not sure if it's in LLVM style guide, but files inside Syntax folder > definitely use this style: put braces everywhere until you reach the last > level of nesting for non-leaf statements (i.e. having other statements as > children, e.g. loops,. if statements, etc) > > > It's my personal preference, happy to discuss whether having this makes sense. I guess it's a personal preference (also for me), but I don't think there is a strict guideline on that. Interestingly enough, I think there is a piece of code in the styleguide that looks exactly like the code you had: https://llvm.org/docs/CodingStandards.html#turn-predicate-loops-into-predicate-functions Some Clang subprojects tend to put braces everywhere though. That being said, I guess no braces at all would be the best option here. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D71356/new/ https://reviews.llvm.org/D71356 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
merge_guards_bot added a comment.
{icon check-circle color=green} Unit tests: pass. 60699 tests passed, 0 failed
and 726 were skipped.
{icon check-circle color=green} clang-format: pass.
Build artifacts
:
console-log.txt
,
CMakeCache.txt
,
test-results.xml
,
diff.json
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D71356/new/
https://reviews.llvm.org/D71356
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
sammccall added inline comments. Comment at: clang/include/clang/Tooling/Syntax/Tokens.h:320 +const syntax::Token * +spelledIdentifierTouching(SourceLocation Loc, + const syntax::TokenBuffer ); kbobyrev wrote: > sammccall wrote: > > kbobyrev wrote: > > > Maybe `llvm::Optional<>` here? > > Pointers are the idiomatic way to express "optional, by-reference" in LLVM. > > > > By contrast, optional is illegal, and optional is > > unidiomatic > Ah, I see, I thought a copy is quite cheap in this case. Makes sense then. Ah, it is, so Optional would be an option. But the address is meaningful - spelled tokens have contiguous storage that indicates their sequence. So returning a pointer is actually more useful. Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:261 + bool AcceptRight = Right != All.end() && !(Loc < Right->location()); + bool AcceptLeft = Right != All.begin() && !((Right - 1)->endLocation() < Loc); + return llvm::makeArrayRef(Right - (AcceptLeft ? 1 : 0), ilya-biryukov wrote: > Maybe compare offsets instead of locations? > It might be more code, but it would read much cleaner. > > It always feels weird to compare source locations for anything other than > storing them in `std::map` I find the `SM.getFileOffset(...)` everywhere pretty hard to read and obscures the actual locations. But agree about the operators, I've added the rest. Having `operator<` is pretty evil, but I don't think the others are particularly incrementally evil. Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:270 + for (const syntax::Token : spelledTokensTouching(Loc, Tokens)) +if (Tok.kind() == tok::identifier) + return ilya-biryukov wrote: > NIT: add braces around `if` statement Is there some reference for preferred LLVM style for this, or personal preference? (Real question, as this comes up a bunch) I ask because that's my *least*-preferred option - no braces > braces on for > braces on both > braces on (only) if. Added braces to the `for` (maybe that's what you meant?) Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D71356/new/ https://reviews.llvm.org/D71356 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
sammccall updated this revision to Diff 233557.
sammccall marked 10 inline comments as done.
sammccall added a comment.
braces
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D71356/new/
https://reviews.llvm.org/D71356
Files:
clang/include/clang/Basic/SourceLocation.h
clang/include/clang/Tooling/Syntax/Tokens.h
clang/lib/Tooling/Syntax/Tokens.cpp
clang/unittests/Tooling/Syntax/TokensTest.cpp
Index: clang/unittests/Tooling/Syntax/TokensTest.cpp
===
--- clang/unittests/Tooling/Syntax/TokensTest.cpp
+++ clang/unittests/Tooling/Syntax/TokensTest.cpp
@@ -793,4 +793,45 @@
ActualMacroRanges.push_back(Expansion->range(SM));
EXPECT_EQ(ExpectedMacroRanges, ActualMacroRanges);
}
+
+TEST_F(TokenBufferTest, Touching) {
+ llvm::Annotations Code("^i^nt^ ^a^b^=^1;^");
+ recordTokens(Code.code());
+
+ auto Touching = [&](int Index) {
+SourceLocation Loc = SourceMgr->getComposedLoc(SourceMgr->getMainFileID(),
+ Code.points()[Index]);
+return spelledTokensTouching(Loc, Buffer);
+ };
+ auto Identifier = [&](int Index) {
+SourceLocation Loc = SourceMgr->getComposedLoc(SourceMgr->getMainFileID(),
+ Code.points()[Index]);
+const syntax::Token *Tok = spelledIdentifierTouching(Loc, Buffer);
+return Tok ? Tok->text(*SourceMgr) : "";
+ };
+
+ EXPECT_THAT(Touching(0), SameRange(findSpelled("int")));
+ EXPECT_EQ(Identifier(0), "");
+ EXPECT_THAT(Touching(1), SameRange(findSpelled("int")));
+ EXPECT_EQ(Identifier(1), "");
+ EXPECT_THAT(Touching(2), SameRange(findSpelled("int")));
+ EXPECT_EQ(Identifier(2), "");
+
+ EXPECT_THAT(Touching(3), SameRange(findSpelled("ab")));
+ EXPECT_EQ(Identifier(3), "ab");
+ EXPECT_THAT(Touching(4), SameRange(findSpelled("ab")));
+ EXPECT_EQ(Identifier(4), "ab");
+
+ EXPECT_THAT(Touching(5), SameRange(findSpelled("ab =")));
+ EXPECT_EQ(Identifier(5), "ab");
+
+ EXPECT_THAT(Touching(6), SameRange(findSpelled("= 1")));
+ EXPECT_EQ(Identifier(6), "");
+
+ EXPECT_THAT(Touching(7), SameRange(findSpelled(";")));
+ EXPECT_EQ(Identifier(7), "");
+
+ ASSERT_EQ(Code.points().size(), 8u);
+}
+
} // namespace
Index: clang/lib/Tooling/Syntax/Tokens.cpp
===
--- clang/lib/Tooling/Syntax/Tokens.cpp
+++ clang/lib/Tooling/Syntax/Tokens.cpp
@@ -248,6 +248,31 @@
return E;
}
+llvm::ArrayRef
+syntax::spelledTokensTouching(SourceLocation Loc,
+ const syntax::TokenBuffer ) {
+ assert(Loc.isFileID());
+ llvm::ArrayRef All =
+ Tokens.spelledTokens(Tokens.sourceManager().getFileID(Loc));
+ // Comparing SourceLocations is well-defined within a FileID.
+ auto *Right = llvm::partition_point(
+ All, [&](const syntax::Token ) { return Tok.location() < Loc; });
+ bool AcceptRight = Right != All.end() && Right->location() <= Loc;
+ bool AcceptLeft = Right != All.begin() && (Right - 1)->endLocation() >= Loc;
+ return llvm::makeArrayRef(Right - (AcceptLeft ? 1 : 0),
+Right + (AcceptRight ? 1 : 0));
+}
+
+const syntax::Token *
+syntax::spelledIdentifierTouching(SourceLocation Loc,
+ const syntax::TokenBuffer ) {
+ for (const syntax::Token : spelledTokensTouching(Loc, Tokens)) {
+if (Tok.kind() == tok::identifier)
+ return
+ }
+ return nullptr;
+}
+
std::vector
TokenBuffer::macroExpansions(FileID FID) const {
auto FileIt = Files.find(FID);
Index: clang/include/clang/Tooling/Syntax/Tokens.h
===
--- clang/include/clang/Tooling/Syntax/Tokens.h
+++ clang/include/clang/Tooling/Syntax/Tokens.h
@@ -309,6 +309,17 @@
const SourceManager *SourceMgr;
};
+/// The spelled tokens that overlap or touch a spelling location Loc.
+/// This always returns 0-2 tokens.
+llvm::ArrayRef
+spelledTokensTouching(SourceLocation Loc, const syntax::TokenBuffer );
+
+/// The identifier token that overlaps or touches a spelling location Loc.
+/// If there is none, returns nullptr.
+const syntax::Token *
+spelledIdentifierTouching(SourceLocation Loc,
+ const syntax::TokenBuffer );
+
/// Lex the text buffer, corresponding to \p FID, in raw mode and record the
/// resulting spelled tokens. Does minimal post-processing on raw identifiers,
/// setting the appropriate token kind (instead of the raw_identifier reported
Index: clang/include/clang/Basic/SourceLocation.h
===
--- clang/include/clang/Basic/SourceLocation.h
+++ clang/include/clang/Basic/SourceLocation.h
@@ -188,9 +188,20 @@
return !(LHS == RHS);
}
+// Ordering is meaningful only if LHS and RHS have the same FileID!
+// Otherwise use SourceManager::isBeforeInTranslationUnit().
inline bool
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
sammccall updated this revision to Diff 233550.
sammccall added a comment.
Add overloads for SourceLocation comparison, and document when they're
meaningful.
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D71356/new/
https://reviews.llvm.org/D71356
Files:
clang/include/clang/Basic/SourceLocation.h
clang/include/clang/Tooling/Syntax/Tokens.h
clang/lib/Tooling/Syntax/Tokens.cpp
clang/unittests/Tooling/Syntax/TokensTest.cpp
Index: clang/unittests/Tooling/Syntax/TokensTest.cpp
===
--- clang/unittests/Tooling/Syntax/TokensTest.cpp
+++ clang/unittests/Tooling/Syntax/TokensTest.cpp
@@ -793,4 +793,45 @@
ActualMacroRanges.push_back(Expansion->range(SM));
EXPECT_EQ(ExpectedMacroRanges, ActualMacroRanges);
}
+
+TEST_F(TokenBufferTest, Touching) {
+ llvm::Annotations Code("^i^nt^ ^a^b^=^1;^");
+ recordTokens(Code.code());
+
+ auto Touching = [&](int Index) {
+SourceLocation Loc = SourceMgr->getComposedLoc(SourceMgr->getMainFileID(),
+ Code.points()[Index]);
+return spelledTokensTouching(Loc, Buffer);
+ };
+ auto Identifier = [&](int Index) {
+SourceLocation Loc = SourceMgr->getComposedLoc(SourceMgr->getMainFileID(),
+ Code.points()[Index]);
+const syntax::Token *Tok = spelledIdentifierTouching(Loc, Buffer);
+return Tok ? Tok->text(*SourceMgr) : "";
+ };
+
+ EXPECT_THAT(Touching(0), SameRange(findSpelled("int")));
+ EXPECT_EQ(Identifier(0), "");
+ EXPECT_THAT(Touching(1), SameRange(findSpelled("int")));
+ EXPECT_EQ(Identifier(1), "");
+ EXPECT_THAT(Touching(2), SameRange(findSpelled("int")));
+ EXPECT_EQ(Identifier(2), "");
+
+ EXPECT_THAT(Touching(3), SameRange(findSpelled("ab")));
+ EXPECT_EQ(Identifier(3), "ab");
+ EXPECT_THAT(Touching(4), SameRange(findSpelled("ab")));
+ EXPECT_EQ(Identifier(4), "ab");
+
+ EXPECT_THAT(Touching(5), SameRange(findSpelled("ab =")));
+ EXPECT_EQ(Identifier(5), "ab");
+
+ EXPECT_THAT(Touching(6), SameRange(findSpelled("= 1")));
+ EXPECT_EQ(Identifier(6), "");
+
+ EXPECT_THAT(Touching(7), SameRange(findSpelled(";")));
+ EXPECT_EQ(Identifier(7), "");
+
+ ASSERT_EQ(Code.points().size(), 8u);
+}
+
} // namespace
Index: clang/lib/Tooling/Syntax/Tokens.cpp
===
--- clang/lib/Tooling/Syntax/Tokens.cpp
+++ clang/lib/Tooling/Syntax/Tokens.cpp
@@ -248,6 +248,30 @@
return E;
}
+llvm::ArrayRef
+syntax::spelledTokensTouching(SourceLocation Loc,
+ const syntax::TokenBuffer ) {
+ assert(Loc.isFileID());
+ llvm::ArrayRef All =
+ Tokens.spelledTokens(Tokens.sourceManager().getFileID(Loc));
+ // Comparing SourceLocations is well-defined within a FileID.
+ auto *Right = llvm::partition_point(
+ All, [&](const syntax::Token ) { return Tok.location() < Loc; });
+ bool AcceptRight = Right != All.end() && Right->location() <= Loc;
+ bool AcceptLeft = Right != All.begin() && (Right - 1)->endLocation() >= Loc;
+ return llvm::makeArrayRef(Right - (AcceptLeft ? 1 : 0),
+Right + (AcceptRight ? 1 : 0));
+}
+
+const syntax::Token *
+syntax::spelledIdentifierTouching(SourceLocation Loc,
+ const syntax::TokenBuffer ) {
+ for (const syntax::Token : spelledTokensTouching(Loc, Tokens))
+if (Tok.kind() == tok::identifier)
+ return
+ return nullptr;
+}
+
std::vector
TokenBuffer::macroExpansions(FileID FID) const {
auto FileIt = Files.find(FID);
Index: clang/include/clang/Tooling/Syntax/Tokens.h
===
--- clang/include/clang/Tooling/Syntax/Tokens.h
+++ clang/include/clang/Tooling/Syntax/Tokens.h
@@ -309,6 +309,17 @@
const SourceManager *SourceMgr;
};
+/// The spelled tokens that overlap or touch a spelling location Loc.
+/// This always returns 0-2 tokens.
+llvm::ArrayRef
+spelledTokensTouching(SourceLocation Loc, const syntax::TokenBuffer );
+
+/// The identifier token that overlaps or touches a spelling location Loc.
+/// If there is none, returns nullptr.
+const syntax::Token *
+spelledIdentifierTouching(SourceLocation Loc,
+ const syntax::TokenBuffer );
+
/// Lex the text buffer, corresponding to \p FID, in raw mode and record the
/// resulting spelled tokens. Does minimal post-processing on raw identifiers,
/// setting the appropriate token kind (instead of the raw_identifier reported
Index: clang/include/clang/Basic/SourceLocation.h
===
--- clang/include/clang/Basic/SourceLocation.h
+++ clang/include/clang/Basic/SourceLocation.h
@@ -188,9 +188,20 @@
return !(LHS == RHS);
}
+// Ordering is meaningful only if LHS and RHS have the same FileID!
+// Otherwise use
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
kbobyrev added inline comments.
Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:260
+ All, [&](const syntax::Token ) { return Tok.location() < Loc; });
+ bool AcceptRight = Right != All.end() && !(Loc < Right->location());
+ bool AcceptLeft = Right != All.begin() && !((Right - 1)->endLocation() <
Loc);
ilya-biryukov wrote:
> NIT: why not `Right->location() <= Loc` instead of `!(Loc < Right->location)`?
> Do we only overload `operator <` for `SourceLocations`?
Yes, we only have operator< :(
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D71356/new/
https://reviews.llvm.org/D71356
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
ilya-biryukov added inline comments.
Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:260
+ All, [&](const syntax::Token ) { return Tok.location() < Loc; });
+ bool AcceptRight = Right != All.end() && !(Loc < Right->location());
+ bool AcceptLeft = Right != All.begin() && !((Right - 1)->endLocation() <
Loc);
NIT: why not `Right->location() <= Loc` instead of `!(Loc < Right->location)`?
Do we only overload `operator <` for `SourceLocations`?
Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:261
+ bool AcceptRight = Right != All.end() && !(Loc < Right->location());
+ bool AcceptLeft = Right != All.begin() && !((Right - 1)->endLocation() <
Loc);
+ return llvm::makeArrayRef(Right - (AcceptLeft ? 1 : 0),
Maybe compare offsets instead of locations?
It might be more code, but it would read much cleaner.
It always feels weird to compare source locations for anything other than
storing them in `std::map`
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D71356/new/
https://reviews.llvm.org/D71356
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
ilya-biryukov added inline comments. Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:270 + for (const syntax::Token : spelledTokensTouching(Loc, Tokens)) +if (Tok.kind() == tok::identifier) + return NIT: add braces around `if` statement Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D71356/new/ https://reviews.llvm.org/D71356 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
kbobyrev marked an inline comment as done.
kbobyrev added inline comments.
Comment at: clang/include/clang/Tooling/Syntax/Tokens.h:320
+const syntax::Token *
+spelledIdentifierTouching(SourceLocation Loc,
+ const syntax::TokenBuffer );
sammccall wrote:
> kbobyrev wrote:
> > Maybe `llvm::Optional<>` here?
> Pointers are the idiomatic way to express "optional, by-reference" in LLVM.
>
> By contrast, optional is illegal, and optional is unidiomatic
Ah, I see, I thought a copy is quite cheap in this case. Makes sense then.
Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:254
+ const syntax::TokenBuffer ) {
+ assert(Loc.isFileID());
+ llvm::ArrayRef All =
sammccall wrote:
> kbobyrev wrote:
> > Nit: maybe mention this requirement in the header comment?
> It is mentioned - Loc must be a spelling location.
Ah, okay, I didn't know that spelling location implies exactly this property,
my bad.
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D71356/new/
https://reviews.llvm.org/D71356
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
merge_guards_bot added a comment.
{icon check-circle color=green} Unit tests: pass. 60699 tests passed, 0 failed
and 726 were skipped.
{icon check-circle color=green} clang-format: pass.
Build artifacts
:
console-log.txt
,
CMakeCache.txt
,
test-results.xml
,
diff.json
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D71356/new/
https://reviews.llvm.org/D71356
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
sammccall updated this revision to Diff 233415.
sammccall marked 6 inline comments as done.
sammccall added a comment.
comments
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D71356/new/
https://reviews.llvm.org/D71356
Files:
clang/include/clang/Tooling/Syntax/Tokens.h
clang/lib/Tooling/Syntax/Tokens.cpp
clang/unittests/Tooling/Syntax/TokensTest.cpp
Index: clang/unittests/Tooling/Syntax/TokensTest.cpp
===
--- clang/unittests/Tooling/Syntax/TokensTest.cpp
+++ clang/unittests/Tooling/Syntax/TokensTest.cpp
@@ -793,4 +793,45 @@
ActualMacroRanges.push_back(Expansion->range(SM));
EXPECT_EQ(ExpectedMacroRanges, ActualMacroRanges);
}
+
+TEST_F(TokenBufferTest, Touching) {
+ llvm::Annotations Code("^i^nt^ ^a^b^=^1;^");
+ recordTokens(Code.code());
+
+ auto Touching = [&](int Index) {
+SourceLocation Loc = SourceMgr->getComposedLoc(SourceMgr->getMainFileID(),
+ Code.points()[Index]);
+return spelledTokensTouching(Loc, Buffer);
+ };
+ auto Identifier = [&](int Index) {
+SourceLocation Loc = SourceMgr->getComposedLoc(SourceMgr->getMainFileID(),
+ Code.points()[Index]);
+const syntax::Token *Tok = spelledIdentifierTouching(Loc, Buffer);
+return Tok ? Tok->text(*SourceMgr) : "";
+ };
+
+ EXPECT_THAT(Touching(0), SameRange(findSpelled("int")));
+ EXPECT_EQ(Identifier(0), "");
+ EXPECT_THAT(Touching(1), SameRange(findSpelled("int")));
+ EXPECT_EQ(Identifier(1), "");
+ EXPECT_THAT(Touching(2), SameRange(findSpelled("int")));
+ EXPECT_EQ(Identifier(2), "");
+
+ EXPECT_THAT(Touching(3), SameRange(findSpelled("ab")));
+ EXPECT_EQ(Identifier(3), "ab");
+ EXPECT_THAT(Touching(4), SameRange(findSpelled("ab")));
+ EXPECT_EQ(Identifier(4), "ab");
+
+ EXPECT_THAT(Touching(5), SameRange(findSpelled("ab =")));
+ EXPECT_EQ(Identifier(5), "ab");
+
+ EXPECT_THAT(Touching(6), SameRange(findSpelled("= 1")));
+ EXPECT_EQ(Identifier(6), "");
+
+ EXPECT_THAT(Touching(7), SameRange(findSpelled(";")));
+ EXPECT_EQ(Identifier(7), "");
+
+ ASSERT_EQ(Code.points().size(), 8u);
+}
+
} // namespace
Index: clang/lib/Tooling/Syntax/Tokens.cpp
===
--- clang/lib/Tooling/Syntax/Tokens.cpp
+++ clang/lib/Tooling/Syntax/Tokens.cpp
@@ -248,6 +248,30 @@
return E;
}
+llvm::ArrayRef
+syntax::spelledTokensTouching(SourceLocation Loc,
+ const syntax::TokenBuffer ) {
+ assert(Loc.isFileID());
+ llvm::ArrayRef All =
+ Tokens.spelledTokens(Tokens.sourceManager().getFileID(Loc));
+ // SourceLocation < SourceLocation is well-defined within a FileID.
+ auto *Right = llvm::partition_point(
+ All, [&](const syntax::Token ) { return Tok.location() < Loc; });
+ bool AcceptRight = Right != All.end() && !(Loc < Right->location());
+ bool AcceptLeft = Right != All.begin() && !((Right - 1)->endLocation() < Loc);
+ return llvm::makeArrayRef(Right - (AcceptLeft ? 1 : 0),
+Right + (AcceptRight ? 1 : 0));
+}
+
+const syntax::Token *
+syntax::spelledIdentifierTouching(SourceLocation Loc,
+ const syntax::TokenBuffer ) {
+ for (const syntax::Token : spelledTokensTouching(Loc, Tokens))
+if (Tok.kind() == tok::identifier)
+ return
+ return nullptr;
+}
+
std::vector
TokenBuffer::macroExpansions(FileID FID) const {
auto FileIt = Files.find(FID);
Index: clang/include/clang/Tooling/Syntax/Tokens.h
===
--- clang/include/clang/Tooling/Syntax/Tokens.h
+++ clang/include/clang/Tooling/Syntax/Tokens.h
@@ -309,6 +309,17 @@
const SourceManager *SourceMgr;
};
+/// The spelled tokens that overlap or touch a spelling location Loc.
+/// This always returns 0-2 tokens.
+llvm::ArrayRef
+spelledTokensTouching(SourceLocation Loc, const syntax::TokenBuffer );
+
+/// The identifier token that overlaps or touches a spelling location Loc.
+/// If there is none, returns nullptr.
+const syntax::Token *
+spelledIdentifierTouching(SourceLocation Loc,
+ const syntax::TokenBuffer );
+
/// Lex the text buffer, corresponding to \p FID, in raw mode and record the
/// resulting spelled tokens. Does minimal post-processing on raw identifiers,
/// setting the appropriate token kind (instead of the raw_identifier reported
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
sammccall added inline comments.
Comment at: clang/include/clang/Tooling/Syntax/Tokens.h:320
+const syntax::Token *
+spelledIdentifierTouching(SourceLocation Loc,
+ const syntax::TokenBuffer );
kbobyrev wrote:
> Maybe `llvm::Optional<>` here?
Pointers are the idiomatic way to express "optional, by-reference" in LLVM.
By contrast, optional is illegal, and optional is unidiomatic
Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:254
+ const syntax::TokenBuffer ) {
+ assert(Loc.isFileID());
+ llvm::ArrayRef All =
kbobyrev wrote:
> Nit: maybe mention this requirement in the header comment?
It is mentioned - Loc must be a spelling location.
Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:257
+ Tokens.spelledTokens(Tokens.sourceManager().getFileID(Loc));
+ // SourceLocation < SourceLocation is OK for one file ID.
+ auto *Right = llvm::partition_point(
kbobyrev wrote:
> Nit: "Comparing SourceLocations within one file using operator less/< is a
> valid operation"? Can't come up with something better, but this is slightly
> hard to understand.
done (I think). FileID rather than file here, because it's not true (or at
least not clear) about files.
Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:262
+ bool AcceptLeft = Right != All.begin() && !((Right - 1)->endLocation() <
Loc);
+ return llvm::makeArrayRef(Right - int(AcceptLeft), Right + int(AcceptRight));
+}
kbobyrev wrote:
> Nit: static casts here or in variable declarations?
Replaced with an explicit ternary instead, which is (maybe) clearer?
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D71356/new/
https://reviews.llvm.org/D71356
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
kbobyrev added inline comments.
Comment at: clang/include/clang/Tooling/Syntax/Tokens.h:320
+const syntax::Token *
+spelledIdentifierTouching(SourceLocation Loc,
+ const syntax::TokenBuffer );
Maybe `llvm::Optional<>` here?
Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:254
+ const syntax::TokenBuffer ) {
+ assert(Loc.isFileID());
+ llvm::ArrayRef All =
Nit: maybe mention this requirement in the header comment?
Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:257
+ Tokens.spelledTokens(Tokens.sourceManager().getFileID(Loc));
+ // SourceLocation < SourceLocation is OK for one file ID.
+ auto *Right = llvm::partition_point(
Nit: "Comparing SourceLocations within one file using operator less/< is a
valid operation"? Can't come up with something better, but this is slightly
hard to understand.
Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:262
+ bool AcceptLeft = Right != All.begin() && !((Right - 1)->endLocation() <
Loc);
+ return llvm::makeArrayRef(Right - int(AcceptLeft), Right + int(AcceptRight));
+}
Nit: static casts here or in variable declarations?
Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:267
+syntax::spelledIdentifierTouching(SourceLocation Loc,
+ const syntax::TokenBuffer ) {
+ for (const syntax::Token : spelledTokensTouching(Loc, Tokens))
The formatting looks weird here, does it come from Clang-Format?
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D71356/new/
https://reviews.llvm.org/D71356
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
merge_guards_bot added a comment.
{icon check-circle color=green} Unit tests: pass. 60699 tests passed, 0 failed
and 726 were skipped.
{icon times-circle color=red} clang-format: fail. Please format your changes
with clang-format by running `git-clang-format HEAD^` or apply this patch
.
Build artifacts
:
console-log.txt
,
CMakeCache.txt
,
clang-format.patch
,
test-results.xml
,
diff.json
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D71356/new/
https://reviews.llvm.org/D71356
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D71356: [Tooling/Syntax] Helpers to find spelled tokens touching a location.
sammccall created this revision.
sammccall added reviewers: ilya-biryukov, kbobyrev.
Herald added a project: clang.
Herald added a subscriber: cfe-commits.
Useful when positions are used to target nodes, with before/after ambiguity.
Repository:
rG LLVM Github Monorepo
https://reviews.llvm.org/D71356
Files:
clang/include/clang/Tooling/Syntax/Tokens.h
clang/lib/Tooling/Syntax/Tokens.cpp
clang/unittests/Tooling/Syntax/TokensTest.cpp
Index: clang/unittests/Tooling/Syntax/TokensTest.cpp
===
--- clang/unittests/Tooling/Syntax/TokensTest.cpp
+++ clang/unittests/Tooling/Syntax/TokensTest.cpp
@@ -793,4 +793,45 @@
ActualMacroRanges.push_back(Expansion->range(SM));
EXPECT_EQ(ExpectedMacroRanges, ActualMacroRanges);
}
+
+TEST_F(TokenBufferTest, Touching) {
+ llvm::Annotations Code("^i^nt^ ^a^b^=^1;^");
+ recordTokens(Code.code());
+
+ auto Touching = [&](int Index) {
+SourceLocation Loc = SourceMgr->getComposedLoc(SourceMgr->getMainFileID(),
+ Code.points()[Index]);
+return spelledTokensTouching(Loc, Buffer);
+ };
+ auto Identifier = [&](int Index) {
+SourceLocation Loc = SourceMgr->getComposedLoc(SourceMgr->getMainFileID(),
+ Code.points()[Index]);
+const syntax::Token *Tok = spelledIdentifierTouching(Loc, Buffer);
+return Tok ? Tok->text(*SourceMgr) : "";
+ };
+
+ EXPECT_THAT(Touching(0), SameRange(findSpelled("int")));
+ EXPECT_EQ(Identifier(0), "");
+ EXPECT_THAT(Touching(1), SameRange(findSpelled("int")));
+ EXPECT_EQ(Identifier(1), "");
+ EXPECT_THAT(Touching(2), SameRange(findSpelled("int")));
+ EXPECT_EQ(Identifier(2), "");
+
+ EXPECT_THAT(Touching(3), SameRange(findSpelled("ab")));
+ EXPECT_EQ(Identifier(3), "ab");
+ EXPECT_THAT(Touching(4), SameRange(findSpelled("ab")));
+ EXPECT_EQ(Identifier(4), "ab");
+
+ EXPECT_THAT(Touching(5), SameRange(findSpelled("ab =")));
+ EXPECT_EQ(Identifier(5), "ab");
+
+ EXPECT_THAT(Touching(6), SameRange(findSpelled("= 1")));
+ EXPECT_EQ(Identifier(6), "");
+
+ EXPECT_THAT(Touching(7), SameRange(findSpelled(";")));
+ EXPECT_EQ(Identifier(7), "");
+
+ ASSERT_EQ(Code.points().size(), 8u);
+}
+
} // namespace
Index: clang/lib/Tooling/Syntax/Tokens.cpp
===
--- clang/lib/Tooling/Syntax/Tokens.cpp
+++ clang/lib/Tooling/Syntax/Tokens.cpp
@@ -248,6 +248,29 @@
return E;
}
+llvm::ArrayRef
+syntax::spelledTokensTouching(SourceLocation Loc,
+ const syntax::TokenBuffer ) {
+ assert(Loc.isFileID());
+ llvm::ArrayRef All =
+ Tokens.spelledTokens(Tokens.sourceManager().getFileID(Loc));
+ // SourceLocation < SourceLocation is OK for one file ID.
+ auto *Right = llvm::partition_point(
+ All, [&](const syntax::Token ) { return Tok.location() < Loc; });
+ bool AcceptRight = Right != All.end() && !(Loc < Right->location());
+ bool AcceptLeft = Right != All.begin() && !((Right - 1)->endLocation() < Loc);
+ return llvm::makeArrayRef(Right - int(AcceptLeft), Right + int(AcceptRight));
+}
+
+const syntax::Token *
+syntax::spelledIdentifierTouching(SourceLocation Loc,
+ const syntax::TokenBuffer ) {
+ for (const syntax::Token : spelledTokensTouching(Loc, Tokens))
+if (Tok.kind() == tok::identifier)
+ return
+ return nullptr;
+}
+
std::vector
TokenBuffer::macroExpansions(FileID FID) const {
auto FileIt = Files.find(FID);
Index: clang/include/clang/Tooling/Syntax/Tokens.h
===
--- clang/include/clang/Tooling/Syntax/Tokens.h
+++ clang/include/clang/Tooling/Syntax/Tokens.h
@@ -309,6 +309,17 @@
const SourceManager *SourceMgr;
};
+/// The spelled tokens that overlap or touch a spelling location Loc.
+/// This always returns 0-2 tokens.
+llvm::ArrayRef
+spelledTokensTouching(SourceLocation Loc, const syntax::TokenBuffer );
+
+/// The identifier token that overlaps or touches a spelling location Loc.
+/// If there is none, returns nullptr.
+const syntax::Token *
+spelledIdentifierTouching(SourceLocation Loc,
+ const syntax::TokenBuffer );
+
/// Lex the text buffer, corresponding to \p FID, in raw mode and record the
/// resulting spelled tokens. Does minimal post-processing on raw identifiers,
/// setting the appropriate token kind (instead of the raw_identifier reported
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
