ilya-biryukov marked an inline comment as not done. ilya-biryukov added inline comments.
================ Comment at: clang/lib/Lex/Preprocessor.cpp:956-957 --LexLevel; + if (OnToken) + OnToken(Result, Source); } ---------------- ilya-biryukov wrote: > ilya-biryukov wrote: > > rsmith wrote: > > > This seems like it's going to be extremely hard to use. If you just want > > > the expanded token stream, then removing all the other calls to `OnToken` > > > and only calling it here when `LexLevel == 0` will give you that. > > I've tried `LexLevel == 0 && IsNewToken` and it **almost** works. > > The only trouble left is dealing with the delayed parsing. E.g. I'm getting > > the callbacks for the same tokens multiple times in cases like parsing of > > method bodies. > > > > Any ideas on how to best tackle this? > > > > The example is: > > ``` > > struct X { > > int method() { return 10; } > > int a = 10; > > }; > > ``` > > > > Seeing multiple callbacks for `{ return 10; }` and `= 10;`, want to see > > only the first one. > It almost works now, see the update revision. > Almost no tokens get reported by the callback twice now, except the following > case. > Input: > ``` > struct X { > int method() { return method(); } > }; > ``` > After the body of the class, we get a an extra callback for `)` token from > one of the functions under `ParsedLexedMethodDef`. > This seems to be happening only for parenthesis. More precisely, this happens from a following sequence of calls: ``` Lex() {LexLevel=0} CachingLex() {LexLevel=1} // which does not have a token in `CachedTokens`, so it recurses into ... Lex() {LexLevel=1} // which results in a call to ... CurTokenLexer->Lex() ``` At this point `CurTokenLexer` returns a token of a pre-lexed method definition, but the current implementation forgets that this was the case by the time the token is processed at the the top of the callstack (`Lex() {LexLevel=0}`). So the token ends up being reported by a callback. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D59885/new/ https://reviews.llvm.org/D59885 _______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits