Re: [gwt-contrib] Are there any docs describing internals of gwtc?
2011/3/15 Scott Blum sco...@google.com: That *may* be true but you'd have to try it and find out. I know we use TypeOracle for a couple of other purposes in dev mode, but I don't *think* we use it for anything else in web mode.. It seems to work fine in simple cases in web mode. Is there one class that is dealing specifically with emulating object orientation or it's scattered around many classes? I'm asking because toString() call get's translated to toString_2() javascript call and this method is not being defined anywhere in js source code. I'm trying to pinpoint location where this stuff is handled to understand why I get a call to non-existing place. Hmm, I'm not completely sure what you're asking. GenerateJavaScriptAST is the place to look for how precisely the Java AST is transformed into JavaScript. Yes, after taking more closer look at GenerateJavaScriptAST code I found the logic I was interested in. The problem I described above (broken polymorphic calls) was caused by superClass field being equal to null for nodes representing interfaces. That would lead to broken overrides calculations for an interface containing method called toString (that overrides one from java.lang.Object) and thus to broken JS output. That was fairly non-trivial to find because you need toString being called through an interface to trigger that behaviour. Now, I'm wondering if it wouldn't make sense to have an assertion in some place checking that whether interfaces have java.lang.Object as their superClass as it seems to be invariant for GWT. I've got another question. Do you have any tool for checking correctness of GWT's ASTs or a tool that allows one to dump ASTs and compare them? The scenario I have in mind is like this: 1. I have minimal jribble example that exhibits some obscure problem like with toString described above and I fail to find a bug but I suspect that GWT ASTs I get out of Jribble are broken. 2. I prepare corresponding Java code that imitate code from jribble. This should work (or I found a bug in gwtc itself but this is highly unlikely). Now I'd like to compare those two ASTs. Such comparison should immediately reveal the source of my problems. You can always call toSource() on any GWT AST node to get the source representation. GwtAstBuilderTest utilizes this to verify that the new GwtAstBuilder is mostly source-compatible with the old translator. We also have a bunch of unit tests derived from JJSTestBase that all generally verify that a certain input source + certain transformations turns into expected output. Other that that, we use JUnit / GWTTestCase to actually execute the compiled output and verify correctness JUnit style. Thanks for pointer to JJSTestBase that will be useful for me, too. -- Grzegorz Kossakowski -- http://groups.google.com/group/Google-Web-Toolkit-Contributors
Re: [gwt-contrib] Are there any docs describing internals of gwtc?
On Mon, Mar 21, 2011 at 10:10 AM, Grzegorz Kossakowski grzegorz.kossakow...@gmail.com wrote: The problem I described above (broken polymorphic calls) was caused by superClass field being equal to null for nodes representing interfaces. That would lead to broken overrides calculations for an interface containing method called toString (that overrides one from java.lang.Object) and thus to broken JS output. That was fairly non-trivial to find because you need toString being called through an interface to trigger that behaviour. Now, I'm wondering if it wouldn't make sense to have an assertion in some place checking that whether interfaces have java.lang.Object as their superClass as it seems to be invariant for GWT. I seem to recall at one point we actually put some thought into not having super interfaces have a superClass field at all, and just always have the compiler implicitly assume Object. Not sure where that idea went, or why we ended up not doing that. Scott -- http://groups.google.com/group/Google-Web-Toolkit-Contributors
Re: [gwt-contrib] Are there any docs describing internals of gwtc?
2011/3/21 Scott Blum sco...@google.com: I seem to recall at one point we actually put some thought into not having super interfaces have a superClass field at all, and just always have the compiler implicitly assume Object. Not sure where that idea went, or why we ended up not doing that. That was exactly the behaviour I was expecting from GWT. It's not something I'll be pushing any further but if that idea brought again you've got another reason to implement it. As for now I'll remember to always check my assumptions using automated tests. -- Grzegorz Kossakowski -- http://groups.google.com/group/Google-Web-Toolkit-Contributors
Re: [gwt-contrib] Are there any docs describing internals of gwtc?
Hi Scott: I'm taking the opportunity to somewhat hijack this thread to ask a question regarding this topic: Is there an opportunity to use these internals to augment GWT testing strategies? Specifically, I'm wondering if the data structure generated during compilation can be used to help baseline testing. Can we take the information and create a network that represents a series of decisions taken in a particular way? This network would be used during the test phase. Cheers, jec -- http://groups.google.com/group/Google-Web-Toolkit-Contributors
[gwt-contrib] Are there any docs describing internals of gwtc?
Hi, More and more often I find myself lacking the big picture of gwtc's internals. I'd like to find answers (or hints where to look for answers) for questions like: 1. What's the difference between TypeMap and TypeOracle and why they seem to have overlapping functionality? 2. Why TypeOracleMediator operates on bytecode whereas gwtc is supposed to work with java source, or jdt's asts? 3. How emulation of object orientation is implemented in gwt? 4. What are main phases in gwtc's execution. What are dependencies? I'd like to know where TypeMpa is being built and why it's done twice. Also, TypeOracle seems to be highly mutable data structure. When it gets updated and why? Do you guys have anything (slides, blog posts, discussions, etc.) that would help me to better understand those matters? -- Grzegorz Kossakowski -- http://groups.google.com/group/Google-Web-Toolkit-Contributors
Re: [gwt-contrib] Are there any docs describing internals of gwtc?
On Mon, Mar 14, 2011 at 3:03 PM, Grzegorz Kossakowski grzegorz.kossakow...@gmail.com wrote: 1. What's the difference between TypeMap and TypeOracle and why they seem to have overlapping functionality? In GWT, there are two major pieces of infrastructure that deal with representing Java language input. 1) A reflection of the user's type model for the purposes of running generators. Approximately equivalent to the kind of information you can get via Java reflection. Models types, fields, and methods, but not method bodies. The entire machinery around this can be loosely referred to as TypeOracle, which is the primary interface for accessing this type model. Generators use this type model to generate code. This infrastructure is used in both dev mode and web mode, as Generators run in both contexts. 2) The GWT AST, which is essentially a source-level representation of the user's Java code, including method bodies, which is optimized and translated into JavaScript. May be referred to as GWTC, the compiler, the web-mode compiler, etc. Mostly lives in com.google.gwt.dev.jjs (which stands for 'Java to JavaScript'). TypeMap is a small implementation detail that's only useful for the JDT AST - GWT AST translation. 2. Why TypeOracleMediator operates on bytecode whereas gwtc is supposed to work with java source, or jdt's asts? Historically, we've run JDT twice. Once to build TypeOracle, and once to build the GWT AST. Efforts have been made to transition TypeOracle to be built from bytecode, which would eliminate one of these JDT runs in favor of using external (presumably IDE-built) class files. 3. How emulation of object orientation is implemented in gwt? Using JavaScript prototype chains, and mangling method signatures such that any given override of a method has a globally unique name in the compiled output. Try compiling with -style PRETTY or DETAILED and inspect the output. 4. What are main phases in gwtc's execution. What are dependencies? JavaToJavaScriptCompiler .precompile() is what I would consider the main high-level algorithm. But here's a text overview. 1) Build TypeOracle from the initial set of available source units (CompilationStateBuilder). 2) Begin running the web mode compile (WebModeCompilerFrontEnd). 3) As the web mode compile runs, we encounter GWT.create() calls (FindDeferredBindingSitesVisitor). 4) Use the rebind infrastructure to resolve rebind requests. This may entail running Generators, which generate new code. 5) Newly-generated types get added to CompilationState + TypeOracle in a bit of a loop-back process. 6) Newly-generated types may themselves contain GWT.create() calls, looping back through step 4. 7) Eventually, all our code is generated. Get rid of TypeOracle, we don't need it anymore. 8) GenerateJavaAST turns the JDT AST into the GWT AST. 9) Normalize certain difficult constructs. 10) Optimize. 11) Post-optimization normalizes to turn high-level Java constructs into JS-specific implementation details (but still Java AST). 12) GenerateJavaScriptAST 13) Optimize JS AST 14) Produce JS source text. I'd like to know where TypeMap is being built and why it's done twice. It should only get built once. Also, TypeOracle seems to be highly mutable data structure. When it gets updated and why? From the outside, it should be seen as append only. Existing state should be effectively immutable. It gets updated because Generators produce new types as the compile proceeds. Do you guys have anything (slides, blog posts, discussions, etc.) that would help me to better understand those matters? Here's wiki entries for TypeOracle / CompilationState. The first is outdated by useful to understand the second one. http://code.google.com/p/google-web-toolkit/wiki/CompilationUnit_1_5 http://code.google.com/p/google-web-toolkit/wiki/CompilationUnit -- http://groups.google.com/group/Google-Web-Toolkit-Contributors
Re: [gwt-contrib] Are there any docs describing internals of gwtc?
Hi Scott! Thanks for speedy and seriously awesome response! :-) Some questions inline. 2011/3/14 Scott Blum sco...@google.com: On Mon, Mar 14, 2011 at 3:03 PM, Grzegorz Kossakowski grzegorz.kossakow...@gmail.com wrote: 1. What's the difference between TypeMap and TypeOracle and why they seem to have overlapping functionality? In GWT, there are two major pieces of infrastructure that deal with representing Java language input. 1) A reflection of the user's type model for the purposes of running generators. Approximately equivalent to the kind of information you can get via Java reflection. Models types, fields, and methods, but not method bodies. The entire machinery around this can be loosely referred to as TypeOracle, which is the primary interface for accessing this type model. Generators use this type model to generate code. This infrastructure is used in both dev mode and web mode, as Generators run in both contexts. So, if my GWT app doesn't have any generators exact contents of TypeOracle doesn't matter? I'm asking because I'm thinking of creating some stub data structures for jribble units and just move on to fixing some other issues and come back only once I want to support Generators written in Scala or Generators referring Scala. I'm asking because I don't want to create more problems than I solve at the time. 2) The GWT AST, which is essentially a source-level representation of the user's Java code, including method bodies, which is optimized and translated into JavaScript. May be referred to as GWTC, the compiler, the web-mode compiler, etc. Mostly lives in com.google.gwt.dev.jjs (which stands for 'Java to JavaScript'). TypeMap is a small implementation detail that's only useful for the JDT AST - GWT AST translation. I see. It happens that TypeMap is quite interesting from my point of view because I'm translating Jribble AST to GWT AST by following JDT AST - GWT AST translation (more or less). It's good to know that it's not being used anywhere else, though. 2. Why TypeOracleMediator operates on bytecode whereas gwtc is supposed to work with java source, or jdt's asts? Historically, we've run JDT twice. Once to build TypeOracle, and once to build the GWT AST. Efforts have been made to transition TypeOracle to be built from bytecode, which would eliminate one of these JDT runs in favor of using external (presumably IDE-built) class files. I see. Thanks for clarification. 3. How emulation of object orientation is implemented in gwt? Using JavaScript prototype chains, and mangling method signatures such that any given override of a method has a globally unique name in the compiled output. Try compiling with -style PRETTY or DETAILED and inspect the output. Is there one class that is dealing specifically with emulating object orientation or it's scattered around many classes? I'm asking because toString() call get's translated to toString_2() javascript call and this method is not being defined anywhere in js source code. I'm trying to pinpoint location where this stuff is handled to understand why I get a call to non-existing place. 4. What are main phases in gwtc's execution. What are dependencies? JavaToJavaScriptCompiler .precompile() is what I would consider the main high-level algorithm. But here's a text overview. 1) Build TypeOracle from the initial set of available source units (CompilationStateBuilder). 2) Begin running the web mode compile (WebModeCompilerFrontEnd). 3) As the web mode compile runs, we encounter GWT.create() calls (FindDeferredBindingSitesVisitor). 4) Use the rebind infrastructure to resolve rebind requests. This may entail running Generators, which generate new code. 5) Newly-generated types get added to CompilationState + TypeOracle in a bit of a loop-back process. 6) Newly-generated types may themselves contain GWT.create() calls, looping back through step 4. 7) Eventually, all our code is generated. Get rid of TypeOracle, we don't need it anymore. 8) GenerateJavaAST turns the JDT AST into the GWT AST. 9) Normalize certain difficult constructs. 10) Optimize. 11) Post-optimization normalizes to turn high-level Java constructs into JS-specific implementation details (but still Java AST). 12) GenerateJavaScriptAST 13) Optimize JS AST 14) Produce JS source text. That looks great. Thanks! Do you guys have anything (slides, blog posts, discussions, etc.) that would help me to better understand those matters? Here's wiki entries for TypeOracle / CompilationState. The first is outdated by useful to understand the second one. http://code.google.com/p/google-web-toolkit/wiki/CompilationUnit_1_5 http://code.google.com/p/google-web-toolkit/wiki/CompilationUnit Thanks. This helps to understand another bit of GWT's architecture. I've got another question. Do you have any tool for checking correctness of GWT's ASTs or a tool that allows one to dump ASTs and compare them? The
Re: [gwt-contrib] Are there any docs describing internals of gwtc?
On Mon, Mar 14, 2011 at 4:41 PM, Grzegorz Kossakowski grzegorz.kossakow...@gmail.com wrote: So, if my GWT app doesn't have any generators exact contents of TypeOracle doesn't matter? I'm asking because I'm thinking of creating some stub data structures for jribble units and just move on to fixing some other issues and come back only once I want to support Generators written in Scala or Generators referring Scala. That *may* be true but you'd have to try it and find out. I know we use TypeOracle for a couple of other purposes in dev mode, but I don't *think* we use it for anything else in web mode.. 2) The GWT AST, which is essentially a source-level representation of the user's Java code, including method bodies, which is optimized and translated into JavaScript. May be referred to as GWTC, the compiler, the web-mode compiler, etc. Mostly lives in com.google.gwt.dev.jjs (which stands for 'Java to JavaScript'). TypeMap is a small implementation detail that's only useful for the JDT AST - GWT AST translation. I see. It happens that TypeMap is quite interesting from my point of view because I'm translating Jribble AST to GWT AST by following JDT AST - GWT AST translation (more or less). Eek... to be honest, JDT AST - GWT AST is some of the nastiest code around, mostly due to JDT. If you're going to follow a pattern, you might look at the new GwtAstBuilder, which is at least marginally cleaner than BuildTypeMap/GenerateJavaAST. Is there one class that is dealing specifically with emulating object orientation or it's scattered around many classes? I'm asking because toString() call get's translated to toString_2() javascript call and this method is not being defined anywhere in js source code. I'm trying to pinpoint location where this stuff is handled to understand why I get a call to non-existing place. Hmm, I'm not completely sure what you're asking. GenerateJavaScriptAST is the place to look for how precisely the Java AST is transformed into JavaScript. I've got another question. Do you have any tool for checking correctness of GWT's ASTs or a tool that allows one to dump ASTs and compare them? The scenario I have in mind is like this: 1. I have minimal jribble example that exhibits some obscure problem like with toString described above and I fail to find a bug but I suspect that GWT ASTs I get out of Jribble are broken. 2. I prepare corresponding Java code that imitate code from jribble. This should work (or I found a bug in gwtc itself but this is highly unlikely). Now I'd like to compare those two ASTs. Such comparison should immediately reveal the source of my problems. You can always call toSource() on any GWT AST node to get the source representation. GwtAstBuilderTest utilizes this to verify that the new GwtAstBuilder is mostly source-compatible with the old translator. We also have a bunch of unit tests derived from JJSTestBase that all generally verify that a certain input source + certain transformations turns into expected output. Other that that, we use JUnit / GWTTestCase to actually execute the compiled output and verify correctness JUnit style. Scott Do you have anything handy that would help me with this or do you have some other idea how to solve my problem? -- Best regards, Grzegorz Kossakowski -- http://groups.google.com/group/Google-Web-Toolkit-Contributors
