IMPLEMENTATION NOTES This patch updates the Linker API to be shardable. It updates the compiler to deal with this new API, and it updates the built-in linkers to support it.
The main API change is that Linker.link has an extra boolean argument. That method is to be called once per permutation with the argument set to "true". It is then called at the end of compilation with the argument set to "false". Linker classes that don't implement this new method are still supported, but they go through "simulated sharding" code paths. Additionally, Compiler and GWTCompiler always use simulated sharding. Additionally, to make multi-computer builds more efficient, artifacts are not provided by default to the final call to Linker.link. Instead, artifacts go through a thinning-down process. All EmittedArtifacts are turned into BinaryEmittedArtifacts, thus dropping any extra fields the original artifacts might have had. Other artifacts are simply dropped, unless they are marked as @Transferable. The API change itself is in classes Linker and in the two new annotations Transferable and Shardable. The bulk of the real changes are in class Link. Instead of doLink, there is now doSimulatedShardingLink and doLinkFinal. The former is used if any linkers haven't been updated and so need to have calls to the non-sharding API mixed in. The latter is used when all linkers have been updated and the Precompile/CompilePerms/Link entry points are being used. Link also has new methods related to thinning out the artifact set for transfer from CompilePerms computers to the Link computer: linkOnePermutationToJar and scanJars are the pair used for true sharding, and simulateTransferThinning is used for simulated sharding. CompilePerms and Precompile are updated to support sharded linking in addition to their old code paths. If all linkers have been updated, then Precompile only counts the permutations, CompilePerms does a precompile, and CompilePerms calls the per-permutation part of the new Linker API. If any linker hasn't been updated, then a non-sharded compile is still used. I believe the linker changes are straightforward. Each one is updated to be shardable, and to transfer the minimum possible from CompilePerms to Link. The rest of the patch is a variety of little odds and ends. There is a new little hierarchy of BinaryEmittedArtifacts. Implementations of that abstract class exist for jar entries and for wrapping a preexisting EmittedArtifact. There is one skimpy test case. It checks that if one final linker produces an artifact that other final linkers will see it. This is also a test that PRE/PRIMARY/POST is respected for final linkers. To factor out Link a little bit, the patch adds an OutputFileSet interface. That interface can target a directory or a jar. LinkerContext now has getLinkerShortName(Class) for reasoning about the emitted file names of private linker artifacts. This is needed in final calls to Linker.link because they deal with binary emitted artifacts. Linkers can't know their own short name, because it depends on what the module's gwt.xml file has. LinkerContext also has getModuleScriptReferences() and getModuleStylesheetReferences(). This means we don't have to create artifacts that get passed all the way through to the final link step. I can add it back if necessary, but this way is simple and should run fast. http://gwt-code-reviews.appspot.com/78805 --~--~---------~--~----~------------~-------~--~----~ http://groups.google.com/group/Google-Web-Toolkit-Contributors -~----------~----~----~----~------~----~------~--~---
