Hi,
Sorry for the delay. Thanks for the comments, Doug!
The attached, simple, patch enables clang to compile incrementally given
input.
Okay to commit?
Vassil
-------- Original Message --------
Subject: Re: [cfe-dev] Incremental parsing/compilation question
Date: Fri, 2 Dec 2011 07:43:55 -0800
From: Douglas Gregor <[email protected]>
To: Vassil Vassilev <[email protected]>
CC: [email protected] <[email protected]>
On Dec 2, 2011, at 7:02 AM, Vassil Vassilev wrote:
Hi,
In cling we have line-by-line input that comes from a terminal-like prompt.
We do incremental compilation of the input. The input lines come as
llvm::MemoryBuffers
We compile each memory buffer by passing them to clang. However, when clang
parses a buffer it sees EOF in the end and destroys it's current lexer
and what not.
For example cling can have:
[cling$] extern "C" int printf(const char* fmt, ...);
[cling$] int i = 12;
[cling$] printf("%d\n", i);
Every line comes in memory buffer containing \0 in the end. Clang
considers that as
an EOF. Ideally I want to tell the parser that the parsing of the
translation unit is not
yet done, but that it should be suspended until next user's input.
Correct me if I am wrong, but the best way of doing that would be to
implement a
'suspend' token. When the lexer and parser see that 'suspend' token
they would stop
as if it was EOF token but without deleting/cleaning anything, so that
the parsing could
be restarted later with the same state.
'suspend' should probably just be a special handling of the 'eof' token, so
that the parser/lexer doesn't tear everything down (but otherwise acts exactly
the same). We don't want the parser or preprocessor to have to check 'is this
eof or suspend?' every time it currently checks for eof.
If this is the right approach what would be the best way to represent
the 'suspend'
token? i.e. which ascii char that would trigger the suspension? (I'd
really like to have
"$" but it is already part of an extension).
I suggest looking at how the code-completion token is created. It uses \0 + a
file offset to distinguish between a \0 at the end of the buffer and an
embedded \0 that is the code completion point.
- Doug
Index: tools/clang/include/clang/Lex/Preprocessor.h
===================================================================
--- tools/clang/include/clang/Lex/Preprocessor.h (revision 152030)
+++ tools/clang/include/clang/Lex/Preprocessor.h (working copy)
@@ -148,6 +148,10 @@
/// with this preprocessor.
std::vector<CommentHandler *> CommentHandlers;
+ /// \brief True if we want to ignore EOF token and continue later on (thus
+ /// avoid tearing the Lexer and etc. down).
+ bool IncrementalProcessing;
+
/// \brief The code-completion handler.
CodeCompletionHandler *CodeComplete;
@@ -344,7 +348,8 @@
ModuleLoader &TheModuleLoader,
IdentifierInfoLookup *IILookup = 0,
bool OwnsHeaderSearch = false,
- bool DelayInitialization = false);
+ bool DelayInitialization = false,
+ bool IncrProcessing = false);
~Preprocessor();
@@ -691,6 +696,14 @@
/// \brief Recompute the current lexer kind based on the
CurLexer/CurPTHLexer/
/// CurTokenLexer pointers.
void recomputeCurLexerKind();
+
+ /// \brief Returns true if incremental processing is enabled
+ bool isIncrementalProcessingEnabled() const { return IncrementalProcessing; }
+
+ /// \brief Enables the incremental processing
+ void enableIncrementalProcessing(bool value = true) {
+ IncrementalProcessing = value;
+ }
/// \brief Specify the point at which code-completion will be performed.
///
Index: tools/clang/lib/Lex/PPLexerChange.cpp
===================================================================
--- tools/clang/lib/Lex/PPLexerChange.cpp (revision 152030)
+++ tools/clang/lib/Lex/PPLexerChange.cpp (working copy)
@@ -328,16 +328,18 @@
CurLexer->BufferPtr = EndPos;
CurLexer->FormTokenWithChars(Result, EndPos, tok::eof);
- // We're done with the #included file.
- CurLexer.reset();
+ if (!isIncrementalProcessingEnabled())
+ // We're done with lexing.
+ CurLexer.reset();
} else {
assert(CurPTHLexer && "Got EOF but no current lexer set!");
CurPTHLexer->getEOF(Result);
CurPTHLexer.reset();
}
+
+ if (!isIncrementalProcessingEnabled())
+ CurPPLexer = 0;
- CurPPLexer = 0;
-
// This is the end of the top-level file. 'WarnUnusedMacroLocs' has collected
// all macro locations that we need to warn because they are not used.
for (WarnUnusedMacroLocsTy::iterator
Index: tools/clang/lib/Lex/Preprocessor.cpp
===================================================================
--- tools/clang/lib/Lex/Preprocessor.cpp (revision 152030)
+++ tools/clang/lib/Lex/Preprocessor.cpp (working copy)
@@ -54,18 +54,19 @@
HeaderSearch &Headers, ModuleLoader
&TheModuleLoader,
IdentifierInfoLookup* IILookup,
bool OwnsHeaders,
- bool DelayInitialization)
+ bool DelayInitialization,
+ bool IncrProcessing)
: Diags(&diags), Features(opts),
Target(target),FileMgr(Headers.getFileMgr()),
SourceMgr(SM), HeaderInfo(Headers), TheModuleLoader(TheModuleLoader),
- ExternalSource(0),
- Identifiers(opts, IILookup), CodeComplete(0),
+ ExternalSource(0), Identifiers(opts, IILookup),
+ IncrementalProcessing(IncrProcessing), CodeComplete(0),
CodeCompletionFile(0), CodeCompletionOffset(0), CodeCompletionReached(0),
SkipMainFilePreamble(0, true), CurPPLexer(0),
CurDirLookup(0), CurLexerKind(CLK_Lexer), Callbacks(0), MacroArgCache(0),
Record(0), MIChainHead(0), MICache(0)
{
OwnsHeaderSearch = OwnsHeaders;
-
+
if (!DelayInitialization) {
assert(Target && "Must provide target information for PP initialization");
Initialize(*Target);
Index: tools/clang/lib/Parse/Parser.cpp
===================================================================
--- tools/clang/lib/Parse/Parser.cpp (revision 152030)
+++ tools/clang/lib/Parse/Parser.cpp (working copy)
@@ -475,6 +475,11 @@
bool Parser::ParseTopLevelDecl(DeclGroupPtrTy &Result) {
DelayedCleanupPoint CleanupRAII(TopLevelDeclCleanupPool);
+ // Skip over the EOF token, flagging end of previous input for incremental
+ // processing
+ if (PP.isIncrementalProcessingEnabled() && Tok.is(tok::eof))
+ ConsumeToken();
+
while (Tok.is(tok::annot_pragma_unused))
HandlePragmaUnused();
@@ -483,15 +488,17 @@
// Late template parsing can begin.
if (getLang().DelayedTemplateParsing)
Actions.SetLateTemplateParser(LateTemplateParserCallback, this);
+ if (!PP.isIncrementalProcessingEnabled())
+ Actions.ActOnEndOfTranslationUnit();
+ //else don't tell Sema that we ended parsing: more input might come.
- Actions.ActOnEndOfTranslationUnit();
return true;
}
ParsedAttributesWithRange attrs(AttrFactory);
MaybeParseCXX0XAttributes(attrs);
MaybeParseMicrosoftAttributes(attrs);
-
+
Result = ParseExternalDeclaration(attrs);
return false;
}
_______________________________________________
cfe-commits mailing list
[email protected]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits
