Revision: 6849
Author: [email protected]
Date: Wed Nov 11 15:07:51 2009
Log: Update CompilationUnit wiki doc for GWT 2.0.
http://code.google.com/p/google-web-toolkit/source/detail?r=6849
Added:
/wiki/CompilationUnit_1_5.wiki
Modified:
/wiki/CompilationUnit.wiki
=======================================
--- /dev/null
+++ /wiki/CompilationUnit_1_5.wiki Wed Nov 11 15:07:51 2009
@@ -0,0 +1,113 @@
+#summary The new design for compilation units as of GWT 1.5
+#labels Phase-Design
+_NOTE: This design document is out of date. For the new design as of GWT
2.0, please see CompilationUnit._
+
+= How Compilation Units are Managed in the GWT Infrastructure =
+== Overview ==
+In GWT, both running hosted mode and compiling for web mode share
infrastructure for building {{{TypeOracle}}} and managing Generators. Up
to now we have not had a good name or well-defined boundaries for this
viscera, but hopefully this design doc will shed some light on things
related to the {{{CompilationUnit}}}. All doc is circa GWT 1.5 RC1, when a
big refactoring effort over all this infrastructure occurred, code
named "Flaming Sword of Death".
+
+Update: _*GRAVEYARD* is now implemented in Gwt trunk (r5048)._
+
+== Goals ==
+ * Read Java source files off the classpath based on module source
inclusions
+ * Compile included Java source using JDT
+ * Perform custom error checking for GWT-specific constructs
+ * Log any compilation errors
+ * Build and maintain a {{{TypeOracle}}}, which reflects the full body of
compilation units
+ * Assimilate new compilation units from Generators
+ * Produce Java class files for use in hosted mode
+ * Efficiently refresh from disk (important for hosted mode refreshes)
+ * Provide compiled class files for hosted mode execution without
additional compilation
+
+== Glossary of Moving Parts ==
+==={{{CompilationState}}}===
+ Represents the total set of {{{CompilationUnits}}} for a module; a
member of a GWT module. The {{{CompilationState}}} is reponsible for
managing the set of {{{CompilationUnits}}} and ensuring orderly state
transitions. The {{{CompilationState}}} also manages the module's
{{{TypeOracle}}}.
+
+==={{{CompilationUnit}}}===
+ The central piece of the whole puzzle. A {{{CompilationUnit}}}
represents a single Java source file, but includes all accumulated state
and output as compilation and validation proceeds. Subparts include:
+ * the parsed JDT AST for the {{{CompilationUnit}}}
+ * a set of {{{CompiledClasses}}}, which contain class-specific
artifacts
+
+==={{{CompilationUnit.State}}}===
+ Any given {{{CompilationUnit}}} can be in exactly one of several
discreet states. Here are the possible states and associated invariants:
+ * *FRESH*: All internal state is cleared; the unit's source has not yet
been compiled by JDT.
+ * *COMPILED*: In this intermediate state, the unit's source has been
compiled by JDT. The unit will contain a set of {{{CompiledClasses}}}.
+ * *ERROR*: In this final state, the unit was compiled, but contained one
or more errors. Those errors are cached inside the unit, but all other
internal state is cleared.
+ * *CHECKED*: In this final state, the unit has been compiled and is
error free. Additionally, all other units this unit depends on
(transitively) are also error free. The unit contains a set of checked
{{{CompiledClasses}}}. The unit and each contained {{{CompiledClass}}}
releases all references to the JDT AST. Each class contains a reference to
a valid JRealClassType, which has been added to the module's
{{{TypeOracle}}}, as well as byte code, JSNI methods, and all other final
state.
+ * *GRAVEYARD*: A *CHECKED* generated unit enters this state at the start
of a refresh. If a generator generates the same unit with identical
source, the unit bypasses costly compilation, validation, and
{{{TypeOracle}}} building. Note that a *GRAVEYARD* unit is invalidated in
any of the following cases:
+ * if its source changes
+ * if the state of any units that it depends on has become *FRESH* or
*ERROR*
+ * if a source unit with the same main type is found
+
+==={{{CompiledClass}}}===
+ Represents a single compiled Java class; a member of a
{{{CompilationUnit}}}. Subparts include:
+ * the parsed JDT AST for the class
+ * a reference to a JRealClassType in the module's {{{TypeOracle}}}
+ * the actual byte code of the class file
+ * a set of JSNI methods associated with the class
+
+==={{{TypeOracleMediator}}}===
+ Helper class to handle the details of reflecting the JDT AST nodes
provided by the {{{CompilationState}}} as a {{{TypeOracle}}}.
+
+==={{{JdtCompiler}}}===
+ Helper class to handle the details of transforming Java source files
into both JDT AST nodes and compiled Java byte code.
+
+==={{{JavaSourceOracle}}}===
+ An abstraction that provides a single unified view of all source files
available to the current module. Implemented on top of a
{{{ResourceOracle}}}.
+
+==={{{ResourceOracle}}}===
+ A classpath abstraction that provides a unified view of all resources
available on the Java classpath given a set of included packages and
optional filters. This is used for both source files and public files, and
the set of included packages is determined by the module's source and
public declarations.
+
+== Class Connection Diagram ==
+
+http://google-web-toolkit.googlecode.com/svn/wiki/TypeOracleCompilationDesign.png
+
+== Algorithm Overview for Various Use Cases ==
+===#1: {{{ModuleDef}}}.getTypeOracle() called the first time===
+ # {{{CompilationState}}} compiles
+ # {{{JdtCompiler}}} compiles all *FRESH* and *ERROR* units; those units
become *COMPILED*
+ # All valid type names for classes with corresponding source code are
recorded for validation
+ # Any units with JDT errors transition to *ERROR* state; all errors are
logged
+ # Custom validation is performed over *COMPILED* units, recording new
errors
+ # JSO restrictions checked using JDT AST
+ # Primitive long access from JSNI methods checked using JDT AST
+ # References to binary types checked using set of valid source types
and JDT AST
+ # Any units with custom errors transition to *ERROR* state; all errors
are logged
+ # Any units (transitively) depending on *ERROR* units are invalidated
and become *FRESH*
+ # Each *COMPILED* unit lazily computes its direct references to other
units
+ # All units depending on *FRESH* or *ERROR* units are transitively
invalidated and become *FRESH*
+ # Each unit that is invalidated generates an error log message
+ # JSNI methods are collected from each unit that remains *COMPILED*
+ # {{{TypeOracleMediator}}} builds a JRealClassType for each class that
belongs to a *COMPILED* unit, and places that JRealClassType into
{{{TypeOracle}}}
+ # All *COMPILED* units become *CHECKED*
+ # Unneeded internal state (such as references to JDT's AST) are now
cleared
+
+===#2: {{{ModuleDef}}} wants to refresh===
+ # {{{ResourceOracle}}} for source path refreshes against file system
+ # {{{JavaSourceOracle}}} refreshes against {{{ResourceOracle}}} if
{{{ResourceOracle}}} state changed
+ # {{{CompilationState}}} refreshes against {{{JavaSourceOracle}}}
+ # All *GRAVEYARD* units are removed
+ # All generated units become *GRAVEYARD*
+ # Any contained JRealClassTypes are removed from {{{TypeOracle}}}
+ # Any source files whose corresponding file was changed in
{{{JavaSourceOracle}}} are invalidated and become *FRESH*
+ # The graveyard units are checked against new source units
+ # All units whose corresponding file was removed from
{{{JavaSourceOracle}}} are removed
+ # Any new source files in {{{JavaSourceOracle}}} have new
{{{CompilationUnits}}} created
+ # All units depending on *FRESH* units are transitively invalidated
and become *FRESH*
+ # The sequence of events in use case #1 occurs
+
+===#3: Newly generated units need to be assimilated===
+ # A set of *FRESH* {{{CompilationUnits}}} contained generated source are
added to the {{{CompilationState}}}
+ # For each new unit, if a *GRAVEYARD* unit with the same type name
already exists, the source of the new unit is compared to the source of the
existing unit.
+ # If the new unit's source is identical to the old unit's source, the
new unit is discarded
+ # All JRealClassTypes in the contained {{{CompiledClasses}}} are
re-added to {{{TypeOracle}}}
+ # If the new unit's source differs from the old unit's source, the old
unit is discarded and the new unit is added
+ # If any *FRESH* units remain, the sequence of events in use case #1
occurs
+
+===#4: A {{{CompilationUnit}}} becomes invalid===
+ # The reference to the JDT cud is removed
+ # The set of cached references to other units is cleared
+ # For each {{{CompiledClass}}}
+ # The contained JRealClassType is removed from {{{TypeOracle}}}
+ # All internal state is cleared
+ # The set of {{{CompiledClasses}}} is cleared
=======================================
--- /wiki/CompilationUnit.wiki Thu Mar 19 15:24:58 2009
+++ /wiki/CompilationUnit.wiki Wed Nov 11 15:07:51 2009
@@ -1,112 +1,34 @@
-#summary The new design for compilation units as of GWT 1.5
+#summary The new design for building CompilationState as of GWT 2.0
#labels Phase-Design
-
-= How Compilation Units are Managed in the GWT Infrastructure =
-== Overview ==
-In GWT, both running hosted mode and compiling for web mode share
infrastructure for building {{{TypeOracle}}} and managing Generators. Up
to now we have not had a good name or well-defined boundaries for this
viscera, but hopefully this design doc will shed some light on things
related to the {{{CompilationUnit}}}. All doc is circa GWT 1.5 RC1, when a
big refactoring effort over all this infrastructure occurred, code
named "Flaming Sword of Death".
-
-Update: _*GRAVEYARD* is now implemented in Gwt trunk (r5048)._
-
-== Goals ==
- * Read Java source files off the classpath based on module source
inclusions
- * Compile included Java source using JDT
- * Perform custom error checking for GWT-specific constructs
- * Log any compilation errors
- * Build and maintain a {{{TypeOracle}}}, which reflects the full body of
compilation units
- * Assimilate new compilation units from Generators
- * Produce Java class files for use in hosted mode
- * Efficiently refresh from disk (important for hosted mode refreshes)
- * Provide compiled class files for hosted mode execution without
additional compilation
-
-== Glossary of Moving Parts ==
-==={{{CompilationState}}}===
- Represents the total set of {{{CompilationUnits}}} for a module; a
member of a GWT module. The {{{CompilationState}}} is reponsible for
managing the set of {{{CompilationUnits}}} and ensuring orderly state
transitions. The {{{CompilationState}}} also manages the module's
{{{TypeOracle}}}.
-
-==={{{CompilationUnit}}}===
- The central piece of the whole puzzle. A {{{CompilationUnit}}}
represents a single Java source file, but includes all accumulated state
and output as compilation and validation proceeds. Subparts include:
- * the parsed JDT AST for the {{{CompilationUnit}}}
- * a set of {{{CompiledClasses}}}, which contain class-specific
artifacts
-
-==={{{CompilationUnit.State}}}===
- Any given {{{CompilationUnit}}} can be in exactly one of several
discreet states. Here are the possible states and associated invariants:
- * *FRESH*: All internal state is cleared; the unit's source has not yet
been compiled by JDT.
- * *COMPILED*: In this intermediate state, the unit's source has been
compiled by JDT. The unit will contain a set of {{{CompiledClasses}}}.
- * *ERROR*: In this final state, the unit was compiled, but contained one
or more errors. Those errors are cached inside the unit, but all other
internal state is cleared.
- * *CHECKED*: In this final state, the unit has been compiled and is
error free. Additionally, all other units this unit depends on
(transitively) are also error free. The unit contains a set of checked
{{{CompiledClasses}}}. The unit and each contained {{{CompiledClass}}}
releases all references to the JDT AST. Each class contains a reference to
a valid JRealClassType, which has been added to the module's
{{{TypeOracle}}}, as well as byte code, JSNI methods, and all other final
state.
- * *GRAVEYARD*: A *CHECKED* generated unit enters this state at the start
of a refresh. If a generator generates the same unit with identical
source, the unit bypasses costly compilation, validation, and
{{{TypeOracle}}} building. Note that a *GRAVEYARD* unit is invalidated in
any of the following cases:
- * if its source changes
- * if the state of any units that it depends on has become *FRESH* or
*ERROR*
- * if a source unit with the same main type is found
-
-==={{{CompiledClass}}}===
- Represents a single compiled Java class; a member of a
{{{CompilationUnit}}}. Subparts include:
- * the parsed JDT AST for the class
- * a reference to a JRealClassType in the module's {{{TypeOracle}}}
- * the actual byte code of the class file
- * a set of JSNI methods associated with the class
-
-==={{{TypeOracleMediator}}}===
- Helper class to handle the details of reflecting the JDT AST nodes
provided by the {{{CompilationState}}} as a {{{TypeOracle}}}.
-
-==={{{JdtCompiler}}}===
- Helper class to handle the details of transforming Java source files
into both JDT AST nodes and compiled Java byte code.
-
-==={{{JavaSourceOracle}}}===
- An abstraction that provides a single unified view of all source files
available to the current module. Implemented on top of a
{{{ResourceOracle}}}.
-
-==={{{ResourceOracle}}}===
- A classpath abstraction that provides a unified view of all resources
available on the Java classpath given a set of included packages and
optional filters. This is used for both source files and public files, and
the set of included packages is determined by the module's source and
public declarations.
-
-== Class Connection Diagram ==
-
-http://google-web-toolkit.googlecode.com/svn/wiki/TypeOracleCompilationDesign.png
-
-== Algorithm Overview for Various Use Cases ==
-===#1: {{{ModuleDef}}}.getTypeOracle() called the first time===
- # {{{CompilationState}}} compiles
- # {{{JdtCompiler}}} compiles all *FRESH* and *ERROR* units; those units
become *COMPILED*
- # All valid type names for classes with corresponding source code are
recorded for validation
- # Any units with JDT errors transition to *ERROR* state; all errors are
logged
- # Custom validation is performed over *COMPILED* units, recording new
errors
- # JSO restrictions checked using JDT AST
- # Primitive long access from JSNI methods checked using JDT AST
- # References to binary types checked using set of valid source types
and JDT AST
- # Any units with custom errors transition to *ERROR* state; all errors
are logged
- # Any units (transitively) depending on *ERROR* units are invalidated
and become *FRESH*
- # Each *COMPILED* unit lazily computes its direct references to other
units
- # All units depending on *FRESH* or *ERROR* units are transitively
invalidated and become *FRESH*
- # Each unit that is invalidated generates an error log message
- # JSNI methods are collected from each unit that remains *COMPILED*
- # {{{TypeOracleMediator}}} builds a JRealClassType for each class that
belongs to a *COMPILED* unit, and places that JRealClassType into
{{{TypeOracle}}}
- # All *COMPILED* units become *CHECKED*
- # Unneeded internal state (such as references to JDT's AST) are now
cleared
-
-===#2: {{{ModuleDef}}} wants to refresh===
- # {{{ResourceOracle}}} for source path refreshes against file system
- # {{{JavaSourceOracle}}} refreshes against {{{ResourceOracle}}} if
{{{ResourceOracle}}} state changed
- # {{{CompilationState}}} refreshes against {{{JavaSourceOracle}}}
- # All *GRAVEYARD* units are removed
- # All generated units become *GRAVEYARD*
- # Any contained JRealClassTypes are removed from {{{TypeOracle}}}
- # Any source files whose corresponding file was changed in
{{{JavaSourceOracle}}} are invalidated and become *FRESH*
- # The graveyard units are checked against new source units
- # All units whose corresponding file was removed from
{{{JavaSourceOracle}}} are removed
- # Any new source files in {{{JavaSourceOracle}}} have new
{{{CompilationUnits}}} created
- # All units depending on *FRESH* units are transitively invalidated
and become *FRESH*
- # The sequence of events in use case #1 occurs
-
-===#3: Newly generated units need to be assimilated===
- # A set of *FRESH* {{{CompilationUnits}}} contained generated source are
added to the {{{CompilationState}}}
- # For each new unit, if a *GRAVEYARD* unit with the same type name
already exists, the source of the new unit is compared to the source of the
existing unit.
- # If the new unit's source is identical to the old unit's source, the
new unit is discarded
- # All JRealClassTypes in the contained {{{CompiledClasses}}} are
re-added to {{{TypeOracle}}}
- # If the new unit's source differs from the old unit's source, the old
unit is discarded and the new unit is added
- # If any *FRESH* units remain, the sequence of events in use case #1
occurs
-
-===#4: A {{{CompilationUnit}}} becomes invalid===
- # The reference to the JDT cud is removed
- # The set of cached references to other units is cleared
- # For each {{{CompiledClass}}}
- # The contained JRealClassType is removed from {{{TypeOracle}}}
- # All internal state is cleared
- # The set of {{{CompiledClasses}}} is cleared
+_NOTE: For the older, GWT 1.5 design please see CompilationUnit_1_5._
+
+= Building {{{CompilationState}}} and {{{TypeOracle}}} =
+== Problem and Background ==
+Before GWT 2.0, {{{CompilationState}}}, and {{{TypeOracle}}} were
per-module singletons. Unfortunately, they were also currently quite
stateful. In hosted mode in particular, types were added to
{{{TypeOracle}}} and {{{CompilationState}}} over time as Generators ran,
then those types were removed on refresh. This was not a problem as long as
a user only has one "session" at a time open. However, in 2.0 out of
process development mode encourages patterns where users might concurrently
access the same module in multiple tabs, or multiple browsers. In this
case, unpredictable behavior was likely to occur. This problem also
manifested trying to do a JUnit test using multiple concurrent dev mode
browsers.
+
+For more pre-2.0 background, please see CompilationUnit_1_5.
+
+== Basic Solution ==
+We create multiple instances of {{{CompilationState}}}, one per active dev
mode sesions, or per concurrent compile. Each {{{CompilationState}}} state
has its own {{{TypeOracle}}}, and neither of these can be reset
or "refreshed". They operate in a forward-only fashion.. new types can be
added via generators, but existing types cannot be removed or changed.
+
+== Problem - Building {{{CompilationState}}} is too expensive ==
+However, {{{CompilationState}}} is extremely expensive to build from
scratch. One possible solution to the problem could have been to simply
recycle old {{{CompilationState}}}s when a page is refreshed, using the old
{{{CompilationState}}} refresh logic; if a user opened multiple tabs we'd
simply have to spend tons of CPU creating multiple instances of
{{{CompilationState}}}. However, this solution is not workable because
generally speaking, we don't know when a page is being refreshed. In the
current dev mode architecture, old sessions may hang around for several
seconds after a new session is up and running, thwarting a recycling
strategy. Something more sophisticated is needed.
+
+== Refinement #1 ==
+In GWT 2.0, multiple independent {{{CompilationState}}} intances are all
built from a shared singleton subsystem, {{{CompilationState}}}Builder. It
makes sense for it to be a singleton, because ultimately all Java source
files (henceforth, compilation units) that can possibly be managed, exist
on the singleton system class path. {{{CompilationState}}}Builder is
responsible for building new {{{CompilationState}}} instances using
compiled compilation units that reflect the current contents of some subset
of Java source files on the system class path. It also efficiently caches
already-compiled units, and correctly and smartly recompiles stale units
(and any units which depend on stale units). Ensuring cache correctness is
a two-fold problem. First, {{{CompilationState}}}Builder must ensure that
any time an underlying source file changes, the associated
{{{CompilationUnit}}} is seen as stale. Secondly,
{{{CompilationState}}}Builder must accurately track dependencies between
units, such that one stale unit causes a cascading series of recompiles of
units that transitively depend on the stale unit.
+
+{{{CompilationState}}}Builder solves the first problem by mapping compiled
units by {{{ContentId}}}. A {{{ContentId}}} consists of the main type name
of a {{{CompilationUnit}}}, and a strong hash of the source file contents.
This combination ensures that changes to source files can be identified in
a lightweight manner. In addition, the {{{ContentId}}} instances themselves
are cached by absolute resource location and last modified timestamp. As
long as a resource's timstamp has not changed, the associated
{{{ContentId}}} is assumed to be fresh.
+
+{{{CompilationState}}}Builder solves the second problem using the existing
dependency tracking mechanisms that have been in place since GWT 1.5 and
earlier, and in addition it now also tracks JSNI references between units.
These dependencies are recorded as {{{ContentId}}} instances, so that one
{{{CompilationUnit}}}'s dependencies can be uniquely resolved to one
particular revision of another {{{CompilationUnit}}}.
+
+== Problem - {{{TypeOracle}}} is built from JDT data structures ==
+Priior to GWT 2.0, JDT data structures were directly used to build
{{{TypeOracle}}}. The JDT structures were transient and only existed until
{{{TypeOracle}}} finished builder. If {{{TypeOracle}}} was later refreshed,
old {{{TypeOracle}}} components would simply be reused. However, in the new
design, multiple instances of a {{{TypeOracle}}} need to coexist, and new
{{{TypeOracle}}} instances may be built from cache at a later time. Keeping
the JDT structures cached in order to later build new {{{TypeOracle}}} data
structures would consume an unacceptable amount of memory, and those JDT
structures cannot natively be serialized to disk. One possible solution
could have been to modify {{{TypeOracle}}} so that individual components
can be serialized, and a new {{{TypeOracle}}} could be built from
deserialized pieces. However, the design of {{{TypeOracle}}} was not
amenable to this approach.
+
+== Refinement #2 ==
+Fortunately, John Tamplin already did the work necessary to build
{{{TypeOracle}}} instances directly from bytecode as part of the Instant
Hosted Mode effort. We took this one piece of IHM and thereby eliminated
the need to cache JDT structures, instead we only cache bytecode.
+
+== Problem - Generated Compilation Units ==
+Prior to GWT 2.0, we cached the {{{CompilationUnit}}} for generated source
files. If the same source file was regenerated with the exact same
contents, the old {{{CompilationUnit}}} could be reused without
recompiling. (This process was referred to as the GRAVEYARD in the existing
wiki doc). We need to translate this idea to allow
{{{CompilationState}}}Builder to also reuse generated units with the same
source.
+
+== Refinement #3 ==
+Generated units are simply cached by {{{ContentId}}} without regard to
modification time. When a new source file is generated, a {{{ContentId}}}
is created and an existing unit can simply be looked up in the cache.
--~--~---------~--~----~------------~-------~--~----~
http://groups.google.com/group/Google-Web-Toolkit-Contributors
-~----------~----~----~----~------~----~------~--~---
