[il-antlr-interest: 32079] Re: [antlr-interest] Q: how to incorporate a preprocessor in the flow?
I tried that approach when I first started with ANTLR but had difficulty handling arbitrary token rearrangement. Early on I couldn't figure out how to backtrack in the token stream in order to detect identifier construction using macros. Something like the following requires that 'prefix' be lexed again after macro substitution in order to detect if the string from suffix and 'prefix' will be merged into one identifier. define suffix(name) name prefix`suffix We use this often in RTL for bus port lists. Even though the spec seems to explicitly disallow this, Modelsim and DC will accept it. Lexing twice solves this case easily but now the tokens point to a non-existent source. On Mon, Apr 4, 2011 at 8:59 PM, Martin d'Anjou poin...@magma.ca wrote: Hi, Thanks to both of you for sharing your approaches. Right now I am pondering how to alter the sequence of tokens before they hit the parser. Intuitively I want to have three processing units (lexer, pre-processor, parser) connected together through io pipes of tokens (e.g. token fifos), but this is not how ANTLR was architected (it's how I would have done it in hardware though!). Martin On 11-04-04 09:25 AM, Sam Harwell wrote: I used a hand-crafted implementation of TokenSource between the lexer and parser. In the preprocessor, whenever I manipulated a token I used a new token class derived from CommonToken (call it SubstitutedToken) which contained a linked list leading from the effective position in the stream (stored in CommonToken) all the way back to the original location (file and position) of the token definition. When a CommonToken substitution occurs, the linked list has one node containing the original source position where defined. Whenever a SubstitutedToken substitution occurs, a new node for the token's previous effective position is added to the linked list and that new head pointer is stored in the new token. `define x 3 `define y `x `y In this case, token `y is eventually replaced with a SubstitutedToken which appears at (line 2, column 1, length 1, text 3) containing the following linked list: Line 3, column 1, length 2 (list head, the location where `y was substituted with `x) Line 2, column 11, length 2 (the location where `x was substituted with '3') Line 1, column 11, length 1 (the actual source location where the token '3' is defined) This list allows true relative ordering of all tokens in the processed source: when two tokens appear to be at the same location in the preprocessed stream, you simply compare the positions of the first node in the position list. Sam -Original Message- From: antlr-interest-boun...@antlr.org [mailto:antlr-interest-boun...@antlr.org] On Behalf Of A Z Sent: Monday, April 04, 2011 12:13 AM To: Martin d'Anjou Cc: antlr-interest@antlr.org Subject: Re: [antlr-interest] Q: how to incorporate a preprocessor in the flow? Hi Martin, I just completed an SV preprocessor which can parse UVM 1.0 successfully. After 2 revisions I settled on a completely separate preprocessor(lexer and parser). As you saw, you need to tokenize the macro_text in order to easily support macros with arguments and detect the three escaped tokens `, `\` and ``. I'm not sure how well a lexer only approach could handle cases where a macro substitution can merge text with a previously lexed token. The separate approach still has flaws, such as good error reporting. Of course I could be missing an obvious easy solution. On Sun, Apr 3, 2011 at 9:51 PM, Martin d'Anjoupoin...@magma.ca wrote: Hello, I am trying to find a way to incorporate a preprocessor in the ANTLR flow. I thought of doing this before the lexer, but I need to tokenize the incoming char stream for macro substitution to be easy. I thought of doing it between the lexer and the parser, and replace the preprocessor tokens with their expansion before feeding the token stream to the parser, so I guess I would end up using something like the TokenRewriteStream??? Can someone steer me in the right direction please? Or should I be using lexer rule actions? In which case, any example on how to access the token stream of the replacement token list of an identifier? Too many questions sorry. The language I am hoping to tokenize is SystemVerilog and has C-like preprocessor macros (`include, `ifdef, `define NAME(params,...), token concatenation, etc.). Regards, Martin List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups
[il-antlr-interest: 32084] [antlr-interest] antlr v4 wish list
List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups il-antlr-interest group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 32085] Re: [antlr-interest] Examples of semantic predicates with ...
Hi James,
In this case, I'm dealing with semantic predicates of the form {...}? and
gated semantic predicates of the form {...}?=. The ones you're using in
this case are syntactic predicates, which aren't processed by the Boolean
logic algorithm I've been working on recently.
Sam
-Original Message-
From: antlr-interest-boun...@antlr.org
[mailto:antlr-interest-boun...@antlr.org] On Behalf Of James Ladd
Sent: Tuesday, April 05, 2011 12:09 AM
To: antlr-interest@antlr.org
Subject: [antlr-interest] Examples of semantic predicates with ...
Hi Sam,
When I dealt with semantic predicates I put the unique prefix part of the
rule into a predicate.
I put the predicate on those rules that ANTLR said were ambiguous.
For example:
| (binarySelector) = binarySelector variableName (MINUS MINUS) =
| specialBinarySelector variableName
I found out most of the information on predicates by googling ANTLR
Semantic Predicates
Rgs, James.
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
il-antlr-interest group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 32087] [antlr-interest] Which to use StringTempates version 3 or 4?
I have the lexing, parsing and tree rewriting completed and will now be doing a proof of concept for the next phase to see if StringTemplates will work as expected. This will be my first use of StringTemplates. Since ST4 is so new and is not documented in the books, should I start with ST3 or jump into ST4? Thanks, Eric List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups il-antlr-interest group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 32088] Re: [antlr-interest] Which to use StringTempates version 3 or 4?
ST4 is *tremendously* faster than ST3 and includes the most useful debugger for any template engine I've seen to date. For a new application, it's not even a question IMO - definitely ST4. Sam -Original Message- From: antlr-interest-boun...@antlr.org [mailto:antlr-interest-boun...@antlr.org] On Behalf Of The Researcher Sent: Tuesday, April 05, 2011 9:26 AM To: antlr-interest@antlr.org Subject: [antlr-interest] Which to use StringTempates version 3 or 4? I have the lexing, parsing and tree rewriting completed and will now be doing a proof of concept for the next phase to see if StringTemplates will work as expected. This will be my first use of StringTemplates. Since ST4 is so new and is not documented in the books, should I start with ST3 or jump into ST4? Thanks, Eric List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups il-antlr-interest group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 32089] Re: [antlr-interest] Which to use StringTempates version 3 or 4?
On Tue, Apr 5, 2011 at 11:01 AM, Sam Harwell sharw...@pixelminegames.comwrote: ST4 is *tremendously* faster than ST3 and includes the most useful debugger for any template engine I've seen to date. For a new application, it's not even a question IMO - definitely ST4. Sam Thanks, Eric List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups il-antlr-interest group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 32090] [antlr-interest] Grammar not detecting stray syntax after certain valid blocks
Hello list,
I currently have a grammar that looks something like this:
//this is the root
public prog
: globalStmt+
;
globalStmt: stateDef;
stateDef
: 'state' ID stateBlock;
stateBlock
: '{' stateBlockContent* '}'
;
stateBlockContent
: NEWLINE!
;
This is enough to demonstrate the problem. When I create a program such as:
state hello
{
}
}
The trailing close bracket is not detected as an error. In fact, the grammar
only seems to trigger an error when the trailing content could be part of a
subrule farther down the chain such as an ID. What am I doing wrong and what
is the best way to resolve this?
Under antlrworks when the debugger gets to the errant bracket the
debug cusor moves to the semi at the end of prog and the closing bracket
stays greyed out like it's never consumed or evaluated.
Thank you for your time.
--
David Daeschler
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
il-antlr-interest group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 32091] Re: [antlr-interest] Grammar not detecting stray syntax after certain valid blocks
This is enough to demonstrate the problem. When I create a program such as:
state hello
{
}
}
The trailing close bracket is not detected as an error.
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
Try to anchor it by adding an EOF at the end of your entry-rule:
public prog
: globalStmt+ EOF
;
That way, you force the parser to go through the entire token stream, and
not stop prematurely.
Regards,
Bart.
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
il-antlr-interest group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 32092] Re: [antlr-interest] Grammar not detecting stray syntax after certain valid blocks
Thank you Bart!
That of course did the trick. I dont know why I didnt think of that.
I appreciate the quick response, have a nice week!
--
David Daeschler
On Tue, Apr 5, 2011 at 2:31 PM, Bart Kiers bki...@gmail.com wrote:
This is enough to demonstrate the problem. When I create a program such as:
state hello
{
}
}
The trailing close bracket is not detected as an error.
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
Try to anchor it by adding an EOF at the end of your entry-rule:
public prog
: globalStmt+ EOF
;
That way, you force the parser to go through the entire token stream, and
not stop prematurely.
Regards,
Bart.
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
il-antlr-interest group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 32096] Re: [antlr-interest] AST cardinality from called rules
The AST operators are the most efficient manner: localVariableDeclaration : LOCAL^ variableDeclarationList ; variableDeclarationList : variableDcl ( ','! variableDcl )* ; Or use rewrites: localVariableDeclaration : LOCAL variableDeclarationList - ^(LOCAL variableDeclarationList) ; variableDeclarationList : variableDcl ( ',' variableDcl )* - variableDcl+ ; -Original Message- From: antlr-interest-boun...@antlr.org [mailto:antlr-interest-boun...@antlr.org] On Behalf Of Graham Mer Sent: Tuesday, April 05, 2011 4:40 PM To: antlr-interest@antlr.org Subject: [antlr-interest] AST cardinality from called rules Hello list, The following rule generates the AST that I want: localVariableDeclaration : LOCAL variableDcl ( ',' variableDcl )* - ^(LOCAL variableDcl)+ ; But how do I generate an equivalent AST if the variableDcl list is not in the localVariableDeclaration rule, as in the following fragment? localVariableDeclaration : LOCAL variableDeclarationList ; variableDeclarationList : variableDcl ( ',' variableDcl )* ; I want to produce a set of ^(LOCAL variableDcl) subtrees, but I want to reuse the variableDeclarationList from other rules, so I'd rather not inline it. Thanks! List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups il-antlr-interest group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 32097] Re: [antlr-interest] AST cardinality from called rules
Hi Graham
Does this give you what you want ?
localVariableDeclaration
: LOCAL! variableDeclarationList
;
variableDeclarationList
: variableDcl ( ',' variableDcl )* - ^(LOCAL variableDcl)+
;
If you also use the variableDeclarationList rule for non-local
declarations you could give it a parameter...
localVariableDeclaration
: LOCAL! variableDeclarationList[LOCAL]
;
variableDeclarationList[int type]
: variableDcl ( ',' variableDcl )*
- {type == LOCAL}? ^(LOCAL variableDcl)+
- ^(OTHER variableDcl)+
;
Michael
On 6 April 2011 07:39, Graham Mer gd.an...@gmail.com wrote:
Hello list,
The following rule generates the AST that I want:
localVariableDeclaration
: LOCAL variableDcl ( ',' variableDcl )* - ^(LOCAL variableDcl)+
;
But how do I generate an equivalent AST if the variableDcl list is not
in the localVariableDeclaration rule, as in the following fragment?
localVariableDeclaration
: LOCAL variableDeclarationList
;
variableDeclarationList
: variableDcl ( ',' variableDcl )*
;
I want to produce a set of ^(LOCAL variableDcl) subtrees, but I want
to reuse the variableDeclarationList from other rules, so I'd rather
not inline it.
Thanks!
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
il-antlr-interest group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 32098] Re: [antlr-interest] AST cardinality from called rules
Problem solved. Thanks for the replies, the parameterized version does
exactly what I need:
localVariableDeclaration:
: LOCAL! variableDeclarationList[LOCAL]
;
variableDeclarationList[int type]
: variableDcl ( ',' variableDcl )*
- {type == LOCAL}? ^(LOCAL variableDcl)+
- // others
;
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
il-antlr-interest group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 32099] Re: [antlr-interest] to stop walking into a subtree
Hello,
I personally prefer a manual tree-visitor (which walks the AST), that
way you can just evaluate a node and based on the result take
appropriate action, it also allows more control (e.g. multi-phases if
needed). The main reason I do it that way is I don't want any of my
grammars getting cluttered with hard to maintain code in between (Java
e.g..) Disadvantage is, if your grammar is large, your visitor may get
complex. So it both has its advantages and disadvantages.
Cheers!
Martijn
On Wed, Apr 6, 2011 at 3:13 AM, Gary Miller miller.ga...@gmail.com wrote:
Hi Porosh,
One way to do this is to skip the walk of the second operand (op2) is
the tree grammar and call some java code to continue the walk if
necessary. See example (this is sample code and is not expected to
compile as is) below.
Question: Does anyone know and a way this can be done without needing
to create a new TreeNodeStream? Maybe using a predicate to acess the
op2 without changing the position in the TreeNodeStream?
Regards
Gary
Sample code.
@members {
boolean continueBooleanexpression(TreeParser walker, Tree op) throws
RecognitionException {
// Save the current TreeNodeStream
TreeNodeStream tns = walker.getTreeNodeStream();
boolean result =
try {
CommonTreeNodeStream nodes = new CommonTreeNodeStream(op);
nodes.setTokenStream(walker.getTreeNodeStream().getTokenStream());
// Use the same walker so state (scope, member etc.) is preserved.
walker.setTreeNodeStream(nodes);
// Continue the walk
// Probably doesn't return a boolean,
// might need to be something like walker.booleanexpression().result;
result = walker.booleanexpression();
} finally {
// Resort original TreeNodeStream
walker.setTreeNodeStream(tns);
}
return result;
}
}
booleanexpression returns[boolean result]
: ^(AND op1=booleanexpression op2=.)
{
if( $op1 ) {
boolean op2Result = continueBooleanexpression(this, $op2);
if( op2Result ) {
result = true;
} else {
result =false;
}
} else {
result = false;
}
}
;
booleanexpression returns[boolean result]
:* ^(AND op1 = booleanexpression op2 = booleanexpression) {if(op1op2)
result = * true; else result =false;}
;
My question:
I want to stop walking once I found op1 as false. In that case, I don't need
to
evaluate op2 anymore, the result is false anyway.
Is there any way to implement this?
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
il-antlr-interest group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
