I'm not sure this is a good idea. At the least, I think it should be behind a routine like PeekNextTokenKind(). The file format should be kept separate from the internal implementation, and this is spreading the parsing of the file format to new places.
In addition, what does this save? In the common case won't we end up discarding 0 tokens? If the PTHLexer always had the next token available, it seems to me that this means that in the common case we would never be doing any unnecessary work. But I don't really grok this code, so... - Daniel On Wed, Dec 17, 2008 at 3:52 PM, Ted Kremenek <[email protected]> wrote: > Author: kremenek > Date: Wed Dec 17 17:52:11 2008 > New Revision: 61172 > > URL: http://llvm.org/viewvc/llvm-project?rev=61172&view=rev > Log: > Rewrite PTHLexer::DiscardToEndOfLine() to not use GetToken and instead only > read the bytes needed to determine if a token is not at the start of the line. > > Modified: > cfe/trunk/lib/Lex/PTHLexer.cpp > > Modified: cfe/trunk/lib/Lex/PTHLexer.cpp > URL: > http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Lex/PTHLexer.cpp?rev=61172&r1=61171&r2=61172&view=diff > > ============================================================================== > --- cfe/trunk/lib/Lex/PTHLexer.cpp (original) > +++ cfe/trunk/lib/Lex/PTHLexer.cpp Wed Dec 17 17:52:11 2008 > @@ -131,15 +131,24 @@ > assert(ParsingPreprocessorDirective && ParsingFilename == false && > "Must be in a preprocessing directive!"); > > - // Already at end-of-file? > - if (AtLastToken()) > - return; > - > - // Find the first token that is not the start of the *current* line. > - Token T; > - for (Lex(T); !AtLastToken(); Lex(T)) > - if (GetToken().isAtStartOfLine()) > - return; > + // Skip tokens by only peeking at their token kind and the flags. > + // We don't need to actually reconstruct full tokens from the token buffer. > + // This saves some copies and it also reduces IdentifierInfo* lookup. > + const char* p = CurPtr; > + while (1) { > + // Read the token kind. Are we at the end of the file? > + tok::TokenKind x = (tok::TokenKind) (uint8_t) *p; > + if (x == tok::eof) break; > + > + // Read the token flags. Are we at the start of the next line? > + Token::TokenFlags y = (Token::TokenFlags) (uint8_t) p[1]; > + if (y == Token::StartOfLine) break; > + > + // Skip to the next token. > + p += DISK_TOKEN_SIZE; > + } > + > + CurPtr = p; > } > > > //===----------------------------------------------------------------------===// > > > _______________________________________________ > cfe-commits mailing list > [email protected] > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits > _______________________________________________ cfe-commits mailing list [email protected] http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits
