================ Comment at: lib/ASTMatchers/Dynamic/Parser.cpp:97-98 @@ +96,4 @@ + size_t TokenLength = 0; + while (TokenLength < Code.size() && isAlphanumeric(Code[TokenLength])) + ++TokenLength; + Result.Kind = TokenInfo::TK_IDENT; ---------------- Dmitri Gribenko wrote: > What if this loop executes 0 times? This should be an error. Fixed. Added an 'invalid' token kind.
================ Comment at: lib/ASTMatchers/Dynamic/Parser.cpp:242 @@ +241,3 @@ +/// parse function. +bool parseToken(const TokenInfo &Token, CodeTokenizer *Tokenizer, + Parser::TokenProcessor *Processor, VariantValue *Value, ---------------- Dmitri Gribenko wrote: > Samuel Benzaquen wrote: > > Dmitri Gribenko wrote: > > > Why doesn't lexer do this? Every token could have an associated value > > > (optional, of course). > > It could do the rest of the parsing, except for the matchers. I would have > > to pass more state to it, though. > It would be best to associate values with single-token literals in the lexer. > Recursive-descent parsing will be simplified because you will need to parse > only high-level constructs in the parser. Done. Literals are handled by the lexer, and changed TK_STRING to TK_LITERAL that will include other literals too. http://llvm-reviews.chandlerc.com/D714 BRANCH svn ARCANIST PROJECT clang _______________________________________________ cfe-commits mailing list [email protected] http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits
