[il-antlr-interest: 28745] [antlr-interest] antlr doubt
Hello, I'm totally new to AntLR. I created a simple grammar for a lexer in ANTLRWorks and generated it via the Generate menu. However, when I'm going to compile the generated Java code I get an error. I'm compiling it using javac mygrammarLexer.java. I'm running Windows XP and installed ANTLR via apt-get, the package manager of Ubuntu. This is the error I get when I try to compile: $ javac locoTokensLexer.java locoTokensLexer.java:3: package org.antlr.runtime does not exist import org.antlr.runtime.*; ^ locoTokensLexer.java:8: cannot find symbol symbol: class Lexer public class locoTokensLexer extends Lexer { ^ Then I get a lot of other errors. Please some one help me out. Regards, Nishanth. List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups il-antlr-interest group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 28747] Re: [antlr-interest] antlr doubt
On Thu, May 6, 2010 at 3:44 PM, Nishanth singh singh.nishan...@gmail.comwrote: ... This is the error I get when I try to compile: $ javac locoTokensLexer.java locoTokensLexer.java:3: package org.antlr.runtime does not exist import org.antlr.runtime.*; ^ locoTokensLexer.java:8: cannot find symbol symbol: class Lexer public class locoTokensLexer extends Lexer { ^ Then I get a lot of other errors. Please some one help me out. Regards, Nishanth. Nishanth, don't take this the wrong way, but this mailing list is not the best place to ask your question. It seems you're struggling with the very basics of Java (the classpath specifically). When compiling Java source files from your command, you either must have all classes your source file(s) use already in your classpath, or must provide all of them through the command line parameter when invoking javac. I highly recommend doing some basic tutorials: http://java.sun.com/docs/books/tutorial/getStarted/index.html and/or Google a bit about classpath issues. If you don't get past it, I suggest you ask your question on a less specific forum (Stackoverflow.com is hot these days), and post back here if you have specific ANTLR questions. Best of luck! Regards, Bart. List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups il-antlr-interest group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 28749] Re: [antlr-interest] Custom data in ANTLR3_BASE_TREE in C runtime
Dear Elvis, I think that I have changed this so it is NULLed in the current development version, but you should be able to do this: 1) Set it to NULL yourself at an appropriate place. 2) In the ANTLR source code, you will find: #ifndef ANTLR3_MALLOC /// Default definition of ANTLR3_MALLOC. You can override this before including /// antlr3.h if you wish to use your own implementation. /// #define ANTLR3_MALLOC(request) malloc ((size_t)(request)) #endif Change that to #ifndef ANTLR3_MALLOC /// Default definition of ANTLR3_MALLOC. You can override this before including /// antlr3.h if you wish to use your own implementation. /// #define ANTLR3_MALLOC(request) calloc (1, (size_t)(request)) #endif And rebuild the runtime and you will sacrifice a little performance for nulled space. 3) Find the code that creates new nodes from a tree factory (the function newPooltree in antlr3commontree.c) and before the return statement, add: tree-baseTree.u = NULL; This is essentially the fix in the current development branch. Jim -Original Message- From: antlr-interest-boun...@antlr.org [mailto:antlr-interest- boun...@antlr.org] On Behalf Of Elvis Presley Sent: Thursday, May 06, 2010 10:12 AM To: antlr-interest@antlr.org Subject: [antlr-interest] Custom data in ANTLR3_BASE_TREE in C runtime I'm using the ANTLR3 C runtime and I'd like to be able to add custom data to my AST. I see there is a u field in the ANTLR3_BASE_TREE, which I would like to use to point to my own struct that contains the data I need for that node in the tree. My problem is that it doesn't look like u is ever initialized to NULL so I don't have a way to determine if I need to allocate new memory for my struct or not. I can't quite figure out by looking at the code if there is a way to override the behavior in ANTLR3_BASE_TREE or in the adaptor so that I can always guarantee that u is always NULL or points to one of my custom structs. Any help here would be appreciated. Thanks --Bryce List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your- email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups il-antlr-interest group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 28750] Re: [antlr-interest] Error building from source: The scm url cannot be null
Thanks. Seems like a fairly old problem. I tried the solution proposed by one poster, adding the scm tag to the top-level pom. On Apr 13, 2010, at 15:27:57, Jim Idle wrote: Please use: antlr.markmail.org Where you will find this explained many times. Thanks, Jim -Original Message- From: antlr-interest-boun...@antlr.org [mailto:antlr-interest- boun...@antlr.org] On Behalf Of Rick Mann Sent: Tuesday, April 13, 2010 3:18 PM To: antlr-interest@antlr.org Subject: [antlr-interest] Error building from source: The scm url cannot be null Hi all, I just downloaded antlr-3.2, and installed Maven 2.2.1. After downloading a bunch of jars, it ran into an error. The maven invocation and error are below. Did I do something wrong? $ mvn -Dmaven.test.skip=true ... [INFO] [buildnumber:create {execution: default}] [INFO] Storing buildNumber: Apr 13, 2010 15:13:46 at timestamp: 1271196826772 [INFO] [ERROR] FATAL ERROR [INFO] [INFO] The scm url cannot be null. [INFO] [INFO] Trace java.lang.NullPointerException: The scm url cannot be null. at org.apache.maven.scm.manager.AbstractScmManager.makeScmRepository(Abstr actScmManager.java:181) at org.codehaus.mojo.build.CreateMojo.getScmRepository(CreateMojo.java:722 ) at org.codehaus.mojo.build.CreateMojo.getScmBranch(CreateMojo.java:593) at org.codehaus.mojo.build.CreateMojo.execute(CreateMojo.java:452) at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginM anager.java:490) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(Defaul tLifecycleExecutor.java:694) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalWithLife cycle(DefaultLifecycleExecutor.java:556) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(Default LifecycleExecutor.java:535) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandl eFailures(DefaultLifecycleExecutor.java:387) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments (DefaultLifecycleExecutor.java:348) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLife cycleExecutor.java:180) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138) at org.apache.maven.cli.MavenCli.main(MavenCli.java:362) at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja va:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso rImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315) at org.codehaus.classworlds.Launcher.launch(Launcher.java:255) at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430) at org.codehaus.classworlds.Launcher.main(Launcher.java:375) [INFO] [INFO] Total time: 1 minute 41 seconds [INFO] Finished at: Tue Apr 13 15:13:46 PDT 2010 [INFO] Final Memory: 20M/79M [INFO] List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your- email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups il-antlr-interest group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 28751] Re: [antlr-interest] Error building from source: The scm url cannot be null
Sorry, ignore that last email from me. Meant to delete the draft. List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups il-antlr-interest group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 28752] [antlr-interest] TreeRewriter with Python
Hi, Reading 'Language Implementation Patterns', enjoying it very much. I wanted to try the rewriting e.g.s using Python. I don't see a 3.2 version available, so wrote a quick and dirty rewriter to work with version 3.1.3, it's attached. Tested with the Reduce grammar from the book, it works. - had to write a new Visitor to do the child replacement - the Rewriter is not the same as java because 'failed' is not tracked in the state object for 3.1.3 - I just catch Backtrackingfailed and ignore it. The changed Reduce grammar from the walking chapter is below. (the backtracking option doesn't work in Python generation; the generated code doesn't compile. Because applyOnce sets backtracking, I assume it is ok not set the backtracking option in the grammar). Simple Test below it Anybody tried Tree rewriting with Python? Does anybody see issues with the attached Rewriter? I would like to use it in my Project, but wanted to get feedback on this before diving deeper; especially if there is a major issue with this approach. Thanks for your help. Regards, Harish treerewriter.py The Grammar: tree grammar Reduce; options { language=Python; output=AST; ASTLabelType=CommonTree; tokenVocab=VecMath; superClass=TreeRewriter; //backtrack=true; // allow backtracking if it's needed } @header { from antlr3.treerewriter import TreeRewriter } bottomup: xPlusx | multBy2 | combineShifts ; xPlusx: ^('+' i=INT j=INT {int($i.text) == int($j.text)}?) - ^(MULT[*] INT[2] $j); multBy2 : ^('*' x=INT {int($x.text) == 2}? y=.) - ^(SHIFT[] $y INT[1]) | ^('*' a=. b=INT {int($b.text)==2}?) - ^(SHIFT[] $a INT[1]) ; combineShifts : ^(SHIFT ^(SHIFT e=. n=INT) m=INT) - ^(SHIFT[] $e INT[str(int($n.text) + int($m.text))]) ; Test Code: --- from antlr3 import * from antlr3.tree import * from VecMathLexer import VecMathLexer from VecMathParser import VecMathParser from Reduce import Reduce cStream = StringStream(print (3 + 3) * 2 * 2) lexer = VecMathLexer(cStream) tStream = CommonTokenStream(lexer) parser = VecMathParser(tStream) t = parser.prog().tree print Original tree: + t.toStringTree() nodes = CommonTreeNodeStream(t) red = Reduce(nodes); t = red.downup(t, False) print Simplified tree: + t.toStringTree() List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address treerewriter.py Description: treerewriter.py -- You received this message because you are subscribed to the Google Groups il-antlr-interest group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 28753] Re: [antlr-interest] Is parser control over the lexer possible?
Hey all, OK, let me try a related but far less involved question: ANTLR tokenizes all input into an internal list before parsing anything in that list. (Right?) Hence, it runs out of memory trying to read my 6.2-million-line input file, because that list is huge. What's the ANTLR way to handle such large input streams? Thanks, -Chris On Thu, Apr 29, 2010 at 4:33 PM, Chris verBurg cheetomons...@gmail.comwrote: Hey guys, A question was posted a few days ago about dealing with an infinite input stream, and the suggestion was to subclass TokenStream so that it didn't read in all of the input upfront. I'm running into a similar problem, but before I go run off and subclass things I thought I'd see if there's a best practice for my situation. It also overlaps with the how do I use keywords as identifiershttp://www.antlr.org/wiki/pages/viewpage.action?pageId=1741 FAQ. I have a data-file grammar that recognizes strings, numbers, and a ton of keywords. Pretending VERSION and LIMIT are two keywords, here's (part of) the .g file: data_file: 'VERSION' STRING ';' | 'LIMIT' NUMBER ';' ; NUMBER: ('-'|'+')? ('0'..'9')+ | ('-'|'+')? ('0'..'9')* '.' ('0'..'9')* ; STRING: ('a'..'z' | 'A'..'Z' | '_' | '.' | '0'..'9')+ ; Problem input #1: VERSION 1.2 ; The 1.2 is lexed as a number instead of a string, so I get a parse error. Problem input #2: VERSION LIMIT ; The LIMIT is lexed as a keyword instead of a string, so I get a parse error. I saw the FAQ about keywords-as-identifiers, but I don't think it's helpful for me. For the NUMBER-that-should-be-a-STRING problem, there's no exact string I could pass to input.LT(1).getText().equals(), because it requires a regex to match a NUMBER. The other solution was to make an identifier rule to match all possibilities -- is the best solution here really to change the rule to 'VERSION' (STRING | NUMBER) ';'? For the keyword-that-should-be-a-STRING problem, I'm hesitant to use either of those solutions because of the sheer number of keywords in this grammar. Ideally what I'd like to do is what I did in Flex and Bison (which I'm porting this grammar from). What I did there was have the parser control how the lexer interpreted subsequent tokens. I embedded a rule in the parser, immediately after the 'VERSION' token, to tell Flex to enter a force-the-next-token-to-be-a-STRING-no-matter-what start state. It worked beautifully. I got most of the way through implementing that in my ANTLR grammar when I found out that ANTLRFileStream reads all the tokens in before the parser even starts up -- which means the parser can't give the lexer any direction over token interpretation. Thoughts, suggestions, outrageous flames? Is there a good way to do this, or maybe is there a completely different approach I should take? Thanks! -Chris List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups il-antlr-interest group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 28754] Re: [antlr-interest] Is parser control over the lexer possible?
Hi Chris, Yes, antlr reads the whole file into memory. I don't know how to stop it from doing that. Cheers ./m Chris verBurg wrote: Hey all, OK, let me try a related but far less involved question: ANTLR tokenizes all input into an internal list before parsing anything in that list. (Right?) Hence, it runs out of memory trying to read my 6.2-million-line input file, because that list is huge. What's the ANTLR way to handle such large input streams? Thanks, -Chris On Thu, Apr 29, 2010 at 4:33 PM, Chris verBurg cheetomons...@gmail.comwrote: Hey guys, A question was posted a few days ago about dealing with an infinite input stream, and the suggestion was to subclass TokenStream so that it didn't read in all of the input upfront. I'm running into a similar problem, but before I go run off and subclass things I thought I'd see if there's a best practice for my situation. It also overlaps with the how do I use keywords as identifiershttp://www.antlr.org/wiki/pages/viewpage.action?pageId=1741 FAQ. I have a data-file grammar that recognizes strings, numbers, and a ton of keywords. Pretending VERSION and LIMIT are two keywords, here's (part of) the .g file: data_file: 'VERSION' STRING ';' | 'LIMIT' NUMBER ';' ; NUMBER: ('-'|'+')? ('0'..'9')+ | ('-'|'+')? ('0'..'9')* '.' ('0'..'9')* ; STRING: ('a'..'z' | 'A'..'Z' | '_' | '.' | '0'..'9')+ ; Problem input #1: VERSION 1.2 ; The 1.2 is lexed as a number instead of a string, so I get a parse error. Problem input #2: VERSION LIMIT ; The LIMIT is lexed as a keyword instead of a string, so I get a parse error. I saw the FAQ about keywords-as-identifiers, but I don't think it's helpful for me. For the NUMBER-that-should-be-a-STRING problem, there's no exact string I could pass to input.LT(1).getText().equals(), because it requires a regex to match a NUMBER. The other solution was to make an identifier rule to match all possibilities -- is the best solution here really to change the rule to 'VERSION' (STRING | NUMBER) ';'? For the keyword-that-should-be-a-STRING problem, I'm hesitant to use either of those solutions because of the sheer number of keywords in this grammar. Ideally what I'd like to do is what I did in Flex and Bison (which I'm porting this grammar from). What I did there was have the parser control how the lexer interpreted subsequent tokens. I embedded a rule in the parser, immediately after the 'VERSION' token, to tell Flex to enter a force-the-next-token-to-be-a-STRING-no-matter-what start state. It worked beautifully. I got most of the way through implementing that in my ANTLR grammar when I found out that ANTLRFileStream reads all the tokens in before the parser even starts up -- which means the parser can't give the lexer any direction over token interpretation. Thoughts, suggestions, outrageous flames? Is there a good way to do this, or maybe is there a completely different approach I should take? Thanks! -Chris List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately. List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups il-antlr-interest group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 28755] Re: [antlr-interest] Is parser control over the lexer possible?
I too would be interested in changing that behavior. I use ANTLR as a command parser, so instantiating everything for every new command is a lot of overhead. I think ANTLR needs a mode in where it only fetches one line at a time, by calling a GetInput routine that we supply -Brian -Original Message- From: antlr-interest-boun...@antlr.org [mailto:antlr-interest-boun...@antlr.org] On Behalf Of Mike Matera Sent: Friday, May 07, 2010 10:13 To: Chris verBurg Cc: antlr-interest@antlr.org Subject: Re: [antlr-interest] Is parser control over the lexer possible? Hi Chris, Yes, antlr reads the whole file into memory. I don't know how to stop it from doing that. Cheers ./m Chris verBurg wrote: Hey all, OK, let me try a related but far less involved question: ANTLR tokenizes all input into an internal list before parsing anything in that list. (Right?) Hence, it runs out of memory trying to read my 6.2-million-line input file, because that list is huge. What's the ANTLR way to handle such large input streams? Thanks, -Chris On Thu, Apr 29, 2010 at 4:33 PM, Chris verBurg cheetomons...@gmail.comwrote: Hey guys, A question was posted a few days ago about dealing with an infinite input stream, and the suggestion was to subclass TokenStream so that it didn't read in all of the input upfront. I'm running into a similar problem, but before I go run off and subclass things I thought I'd see if there's a best practice for my situation. It also overlaps with the how do I use keywords as identifiershttp://www.antlr.org/wiki/pages/viewpage.action?pageId=1741 FAQ. I have a data-file grammar that recognizes strings, numbers, and a ton of keywords. Pretending VERSION and LIMIT are two keywords, here's (part of) the .g file: data_file: 'VERSION' STRING ';' | 'LIMIT' NUMBER ';' ; NUMBER: ('-'|'+')? ('0'..'9')+ | ('-'|'+')? ('0'..'9')* '.' ('0'..'9')* ; STRING: ('a'..'z' | 'A'..'Z' | '_' | '.' | '0'..'9')+ ; Problem input #1: VERSION 1.2 ; The 1.2 is lexed as a number instead of a string, so I get a parse error. Problem input #2: VERSION LIMIT ; The LIMIT is lexed as a keyword instead of a string, so I get a parse error. I saw the FAQ about keywords-as-identifiers, but I don't think it's helpful for me. For the NUMBER-that-should-be-a-STRING problem, there's no exact string I could pass to input.LT(1).getText().equals(), because it requires a regex to match a NUMBER. The other solution was to make an identifier rule to match all possibilities -- is the best solution here really to change the rule to 'VERSION' (STRING | NUMBER) ';'? For the keyword-that-should-be-a-STRING problem, I'm hesitant to use either of those solutions because of the sheer number of keywords in this grammar. Ideally what I'd like to do is what I did in Flex and Bison (which I'm porting this grammar from). What I did there was have the parser control how the lexer interpreted subsequent tokens. I embedded a rule in the parser, immediately after the 'VERSION' token, to tell Flex to enter a force-the-next-token-to-be-a-STRING-no-matter-what start state. It worked beautifully. I got most of the way through implementing that in my ANTLR grammar when I found out that ANTLRFileStream reads all the tokens in before the parser even starts up -- which means the parser can't give the lexer any direction over token interpretation. Thoughts, suggestions, outrageous flames? Is there a good way to do this, or maybe is there a completely different approach I should take? Thanks! -Chris List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately. List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups il-antlr-interest group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.