Hi Gordon, I won't object that Shaoting Cai certainly knew what he was doing, but Antlr4 is in some respects greatly different from 2 and 3. I bet if Antlr4 had been available at the time Falcon was initiate, he would have used that. Not only does it seem to allow much simpler grammars, it also generates an interface for each grammar in which "enterXYZ" and "exitXYZ" callbacks are defined for every rule in the grammar. I can hereby create an implementation of that interface and construct the FalconAST in here directly. Usually you parse the input and get an AntlrAST and use a treewalker in conjunction with that Interface implementation, but you can also hook that up as parse-listener and it's used by the initial parsing. This way I think I could remove several tree traversals.
I also got closer on the JBurg part. I was actually wrong with the runtime and generator part. It's actually all just a Generator. I therefore have create a jburg-maven-plugin that I can hook up into the build. The cool thing about that ist hat every plugin has it's own classloader and therefore JBurg can use whatever it likes without interfering with the rest. I am having one really strange classloading error though that I still need to investigage and as soon as that's finished I promissed tom to donate the plugin to JBurg. I also had a small discussion with tom and he will be willing to update JBurg to update JBurg to Antlr4 and Stringtemplate (The external stringtemplate, not the Antlr internal version). After that I agreed to make JBurg build with Maven ... so I guess we are having some progress on this :-) Chris -----Ursprüngliche Nachricht----- Von: Gordon Smith [mailto:gsmit...@hotmail.com] Gesendet: Freitag, 25. Juli 2014 04:10 An: dev@flex.apache.org Betreff: RE: AW: AW: AW: AW: AW: Falcon and Antlr4 > I am sort of confused what JBurg is actually working on ... Does it directly > work on FalconAST or is there some sort of "ReducedFalconAST"? The input to the BURM (Bottom-Up Rewrite Machine) that is produced by JBurg (Java Bottom-Up Rewrite Generator) is the FalconAST. For example, a BURM pattern like Pattern addExpr Op_AddID(expression l, expression r); in CMCPatterns.jbg, representing a '+' expression, applies to each subtree headed by a BinaryOperatorPlusNode, because the getNodeId() method of this node class returns Op_AddId. > ABC code (I guess this is the native Flash bytecode?) Right. ABC stands for ActionScript Byte Code, which is the byte code esecuted by the ActionScript Virtual Machine (AVM) in the Flash Player and AIR. > CSS... I would like to simplify this to: > CSS --[CSSLexer]--> TokenStream --[CSSParser]--> [FalconAST] > --[JBurg]--> SWF I won't object, however the CSS implementation was done for Falcon by a former Adobe engineer (Shaoting Cai) who was well versed in Antlr, since he had been a student of the professor who developed Antlr. (The AS lexer and parser are much older, inherited from Flash Builder, and written by engineers who were probably not as expert in Antlr, and were using older version of Antlr that didn't have tree parsers.) So I tend to think that the "Antlr way" may be to let Antlr create an AntlrAST automatically, and then transform it into a more custom FalconAST with a tree parser. On the other hand, if this is slower than having the parser produce the FalconAST directly from the token stream, which seems likely, then I vote for going for speed. > AS... Here I would like to remove the RawASTTokenizer and replace that > with an Antlr4 Lexer AS --[ASLexer]--> TokenStream --[ASParser]--> > FalconAST --[JBurg]--> SWF This is what I had in mind, although again I'm not sure this would be considered Antlr best practice. (As before, if Shaoting had done the AS implementation, he probably would have produced an AntlrAST and then transformed it into a FalconAST with a tree parser.) > Do we even need a dependency to JBurg itself to execute the code generated by > JBurg? Yes. As you discovered, there is a runtime part, which determines which sequence of reductions has the lowest 'cost' and therefore gets used for code generation. > I will contact the JBurg project and check if it was possible to switch to > the new Stringtemplate project (and if it was possible to split up JBurg into > two modules "generator", and "runtime"). If Tom doesn't want to decouple JBurg from Antlr, one option would be for us to fork JBurg and remove its dependency on the Antlr Stringtemplate. Is another option for JBurg to continue using whatever old Antlr it has been using and the AS and CSS lexer/parser to use the latest Antlr? - Gordon > From: christofer.d...@c-ware.de > To: dev@flex.apache.org > Subject: AW: AW: AW: AW: AW: Falcon and Antlr4 > Date: Thu, 24 Jul 2014 09:08:38 +0000 > > Ok ... so I created a JBurg Maven Plugin (Will contact the JBurg > project lead for donating that) > > But (unfortunateley there's a but): > - JBurg seems to consist of a Generator and a Runtime, but both are > located in the same Jar (Even if this wouldn't really be a blocker) > - JBurgs runtime part doesn't directly rely on Antlr AST classes, but > unfortunateley Antlr Stringtemplate which is only bundled in Antlr3 > distributions because it seems to have been created a project spinoff > containing only the stringtemplate part > (http://www.stringtemplate.org/) > > I will contact the JBurg project and check if it was possible to switch to > the new Stringtemplate project (and if it was possible to split up JBurg into > two modules "generator", and "runtime"). > > Chris > > > ________________________________________ > Von: Christofer Dutz <christofer.d...@c-ware.de> > Gesendet: Donnerstag, 24. Juli 2014 09:57 > An: dev@flex.apache.org > Betreff: AW: AW: AW: AW: AW: Falcon and Antlr4 > > So let's just define some words that might help avoid confusion. > > A typical Antlr parser geneates a parse-tree consisting of Antlr objects ... > let's call that AntlrAST. > Falcon has it's own internal representation using its own objects ... let's > call that FalconAST. > Then I am sort of confused that JBurg is actually working on ... Does it > directly work on FalconAST or is there some sort of "ReducedFalconAST"? > > Because I started with the CSS Parser, I saw that we use Antlr3 to generate a > parser for CSS files. That outputs an AntlrAST which is fed into a second > Antlr generated parser "CSSTree" which converts this AntlrAST into a > FalconAST. After that I think (i'm not 100% sure) the FalconAST is passed in > to the CSSReducer which sort of directly generates ABC code (I guess this is > the native Flash bytecode?). > > ////////// > // Css > ////////// > > CSS --[CSSLexer]--> TokenStream --[CSSParser]--> AntlrAST > --[CSSTreeParser]--> FalconAST --[CSSReducer]--> [ReducedFalconAST] > --[JBurg]--> SWF (Everything from the FalconAST still seems a little > mystical to me though) > > I would like to simplyfy this to: > > CSS --[CSSLexer]--> TokenStream --[CSSParser]--> [FalconAST] > --[JBurg]--> SWF > > ////////// > // ActionScript > ////////// > > For AS the procedure is different (As far as I got it together with your > mail). We are using Antlr2 here: > > AS --[RawASTTokenizer]--> TokenStream --[ASParser]--> FalconAST > --[JBurg]--> SWF > > Here I would like to remove the RawASTTokenizer and replace that with > an Antlr4 Lexer > > AS --[ASLexer]--> TokenStream --[ASParser]--> FalconAST --[JBurg]--> > SWF > > ////////// > // JBurg > ////////// > > In the meanwhile I think I had some progress. As far as I understand now > JBurg is built by Antlr and uses Antlr to parse and generate output. The > output however no longer has a reference to Antlr. Do we even need a > dependency to JBurg itself to execute the code generated by JBurg? If not I > would whip up a jburg-maven-plugin which wraps JBurgs execution. Then I think > we shouldn't have any problems with Antlr versions. > > Currently a I want to do is do an Antlr4 POC for the CSS compilation. As soon > as that's done, I would like to compare the performance to that of the > existing one and only if that is at least as fast as the current solution I > would proceed to the next part (If its not slower than the current solution I > would prefer the simpler process). > > Chris > > > ________________________________________ > Von: Gordon Smith <gsmit...@hotmail.com> > Gesendet: Mittwoch, 23. Juli 2014 18:05 > An: dev@flex.apache.org > Betreff: RE: AW: AW: AW: AW: Falcon and Antlr4 > > CSSTree.g isn't involved in compiling AS files. It's involved only in > compiling CSS files. > > For AS compilation: We use JFlex to create RawASTokenizer.java from > RawASTokenizer.lex. This is a lexer which takes an AS file as input and > creates a sequence of ASToken objects as output. We use Antlr 3 to create > ASParser.java from ASParser.g. This is a parser which takes a sequence of > ASToken objects as output and creates an AST consisting of NodeBase objects. > This is a custom AST, not the kind of generic Antlr AST that Antlr can create > automatically. > > If Antlr 4 can tokenize as fast as JFlex can, then I think it would be great > to have a unified Antlr 4 lexer/parser. However, I strongly recommend keeping > Falcon's custom AST classes. > > I'm not an expert on JBurg and wasn't aware that it had a dependency on > Antlr. Can you please show me where this depencency is? The "JBurg guys" are > basically one person, Tom Harwood, who used to work at Adobe but no longer > does. I don't know whether he is still supporting JBurg or not. > > As for eliminating JBurg... This is certainly doable, but would have serious > implications. The main reason that Falcon does code generation using JBurg is > in order to perform various code optimizations such as constant propagation. > These code optimizations would be much harder to do without JBurg. Ideally, > these optimizations would be done in the ActionScript Virtual Machine rather > than in the compiler (and this was the plan for AS4, which Adobe killed) but > since they are NOT done in the current AVM, if the compiler doesn't do them > then the code will execute slower. However, if cross-compilation to > JavaScript is the main goal for the future, then perhaps slower AVM execution > is tolerable. > > - Gordon > > > > From: christofer.d...@c-ware.de > > To: dev@flex.apache.org > > Subject: AW: AW: AW: AW: Falcon and Antlr4 > > Date: Wed, 23 Jul 2014 12:54:15 +0000 > > > > Hi Eric, > > > > The question is definitely not silly. > > > > I'm not planning on changing anything here. The package tangling > > seems to be related with JBurg needing access to some objects that > > are not part of the interface but part of the internal model. I > > would untangle this by extending the Interface to provide anything > > JBurg needs so there shouldn't be any change in the Falcon AST. But > > after all that's just an optimization. Currently I'm digging in the > > inner workings of JBurg ... gee I sort of forgot why I looked in the > > inner workings of Falcon in the first place ;-) > > > > I would however like to remove as many processing phases and model > > conversions as possible as each one costs time and memory and makes things > > more compilcated to understand. > > > > Chris > > > > ________________________________________ > > Von: Erik de Bruin <e...@ixsoftware.nl> > > Gesendet: Mittwoch, 23. Juli 2014 14:47 > > An: dev@flex.apache.org > > Betreff: Re: AW: AW: AW: Falcon and Antlr4 > > > > I'm not at all familiar with the inner workings of Falcon, but I > > know that FalconJX relies heavily on the current Falcon AST. Will > > the new AST be compatible with the old (that may be a silly > > question, but it's the only one I know to ask)? > > > > EdB > > > > > > > > On Wed, Jul 23, 2014 at 2:36 PM, Christofer Dutz > > <christofer.d...@c-ware.de> > > wrote: > > > > > Ok ... so I used some idle time setting up a clean falcon-antlr4 > > > module starting with the CSS parsing. > > > > > > As far as I can see it, Falcon is currently using Antlr3 to > > > generate the lexer and a parser that generates an Antlr AST from > > > that Then we generate a second parser CSSTree.g to convert that > > > AST into the falcon-internal object structure, which is then > > > passed to JBurg to generate the output. > > > > > > As you mentioned the ability to manipulate the AST. Are you > > > talking about the first (antlr) AST or the one the Falcon AST? > > > I would like to elminiate the second parsing step and generate the > > > internal model directly from the first parser using Antlr4s > > > Listeners. Are there any objections against this? I doubt that > > > anyone is manipulating the Antlr AST directly, much more would I > > > expect operations to be performed on the Falcon AST. > > > > > > I managed to get the CSS Lexer and Parser finished. The grammar is > > > way more readable now and the general structure is much more > > > straight forward :-) Now I would like to Build the internal Falcon > > > object model for that. > > > Guess I would have to contact the JBurg guys for continuing from > > > then as I definitely need Antlr4 support but I would bet that > > > either JBurg is dead or they are working or at least thinking > > > about that. > > > > > > I did detect some ugly cyclic dependencies in the falcon classes > > > though ... for example we have interfaces for defining the > > > internal models interface and have implementations of these > > > interfaces in the internal packages, but unfortunately the > > > interfaces reference classes in the internal model ... I think > > > that's bad design and would like to clean that up. > > > > > > Any objections? (Keep in mind ... it's all in a dedicated branch, > > > but I don't want to make it hard to merge back as soon as I'm > > > finished) > > > > > > There is no DOM or SAX involved in Falcon ... I was using a > > > metaphor to explain what I was thinking about by using an example > > > from the XML processing world. Here you can load an XML document > > > to memory (DOM), transform that into a second memory > > > representation and so on in contrast to SAX processing where > > > parser generates a stream of events which can be processed > > > allowing you to process petabyte big documents with as much ram as > > > a calculator ;-) > > > > > > > > > Chris > > > > > > > > > ________________________________________ > > > Von: Alex Harui <aha...@adobe.com> > > > Gesendet: Mittwoch, 23. Juli 2014 08:48 > > > An: dev@flex.apache.org > > > Betreff: Re: AW: AW: AW: Falcon and Antlr4 > > > > > > On 7/22/14 12:07 PM, "Christofer Dutz" <christofer.d...@c-ware.de> wrote: > > > > > > >The way I understood JBurg is that it seems to have been designed > > > >to manipulate ASTs produced by Antlr2 and Antlr3. > > > >Therefore it has a compile and runtime dependency to Antlr (I > > > >also sort of dislike them bundling the generator together with > > > >the runtime stuff) but nevertheless ... I can't really use JBurg > > > >with Antlr4 in my project not without splitting the module up but I > > > >think that's rather ugly. > > > I'm definitely not an expert on the subject. I'm surprised the > > > AST has any Antlr-isms in it. Maybe Gordon can confirm. When I > > > look at the AST, I don't see anything I find surprising. > > > > > > > > > > >I too was thinking about making JBurg obsolete too ... the way I > > > >understand Antlr4 it looked as if it provided a lot which could > > > >make life a lot easier. For example currently the ASParser > > > >creates the AST objects and JBurg then performs it's magic on > > > >that AST tree. In Antlr4 I could create a Parser-Listener that > > > >produces the AST object (Probably needed for IDE suppoer) but > > > >could probably create a second listener that directly outputs > > > >flash bytecode. I think this approach should be maximum > > > >performant, maximum simple and use only a fragment of the memory > > > >the current solutions need ... Sort of XML Dom processing (old AST > > > >parsing) and SAX (Antlr4 direct Bytecode output). > > > I have not played with SAX much nor Antlr, so I have no idea about > > > parser/listener. I will say that there are times folks want to > > > create the AST and then manipulate it before generating the > > > output, so unless there is a huge performance win, making sure > > > there is a way to do that would be preferred. > > > > > > > > > > >But I also think that this would be quite an effort. Perhaps I > > > >should split up my falcon-antlr4 branch even more and sort of > > > >start with the CSS parser and as soon as that's up and running get the > > > >output running ... > > > >then we could compare the performance of both and see if it's > > > >worth going down that path. Don't want to waste time I could be > > > >working on Flexmojos, or the new Flex-Maven-Plugin for a less > > > >performant solution. But if it was faster (I would expect it to be) it > > > >could be quite breakthough ... > > > >after all I have never seen several datastructure conversions be > > > >faster than direct output. > > > Up to you since you are doing the work. I just want to make sure > > > you don't feel like you have to keep all of these subsystems like > > > Jburg and Jflex around. > > > > > > -Alex > > > > > > > > > > > > -- > > Ix Multimedia Software > > > > Jan Luykenstraat 27 > > 3521 VB Utrecht > > > > T. 06-51952295 > > I. www.ixsoftware.nl
