================
@@ -573,6 +573,22 @@ Module *Preprocessor::getCurrentModuleImplementation() {
return getHeaderSearchInfo().lookupModule(getLangOpts().ModuleName);
}
+bool Preprocessor::LexTokensInString(SmallVectorImpl<Token> &Tokens,
+ StringRef Code, SourceLocation Loc) {
+ std::unique_ptr<Lexer> L = Lexer::CreateScratchLexer(Code, Loc, Loc, *this);
+ L->ProduceEOFWhenDone = true; // Stop lexing once 'L' is done.
+ EnterSourceFileWithLexer(std::move(L), nullptr);
+ for (;;) {
+ Token Tok;
+ Lex(Tok);
+ if (Tok.is(tok::unknown))
+ return true;
+ if (Tok.is(tok::eof))
+ return false;
+ Tokens.push_back(Tok);
+ }
+}
+
----------------
cor3ntin wrote:
I am not a fan of the EOF thing.
First of, if, anything, I think the handling for EOF should be in LexEndOfFile.
But, more to the point `Lex` can fail, you should check the return value.
Did we take a decision for macro expansion?
Given we are going to call LexToken immediately after, why at this stage not
perform a raw lexing? (that would naturally produce EOF)
https://github.com/llvm/llvm-project/pull/208877
_______________________________________________
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
