On Fri, Feb 12, 2010 at 7:00 PM, Lex Spoon <sp...@google.com> wrote: > On Fri, Feb 12, 2010 at 9:50 AM, Alex Moffat <alex.mof...@gmail.com>wrote: > >> Where can I read a description of what -XshardPrecompile, or see the >> code for it, it sounds very useful to me personally? > > > -XshardPrecompile is an experiment that everyone wants to change, so it > seems unlikely to be released in its current form. We can talk about it if > it helps, but I would propose that we focus more on what we want to do for > real. >
It seemed relevant because it sounded like you propose to essentially make -XshardPrecompile the default (only?) behavior for Precompile? Or did I misread? The reason that makes me cautious has to do with a desire for a future change to the Generator API to support things like minimal rebuild. I imagine a world where the work each Generator does could be sharded out in a way that's independent of the number of permutations. > >> - I'm not sure why development mode wouldn't run a sharded link first. >> Wouldn't it make sense if development mode works just like production >> compile, it just runs a single "development mode" permutation shard link >> before running the final link? >> > > Sure, we can do that. Note, though, that they will be running against an > empty ArtifactSet, because there aren't any compiles for them to look at. > Thus, they won't typically do anything. > Do public resources and generated resources show up during the sharded phase? > > >> 2) Instead of trying to do automatic thinning, we just let the linkers >> themselves do the thinning. For example, one of the most >> serialization-expensive things we do is serialize/deserialze symbolMaps. To >> avoid this, we update SymbolMapsLinker to do most of its work during >> sharding, and update IFrameLinker (et al) to remove the CompilationResult >> during the sharded link so it never gets sent across to the final link. >> > > In addition to the other issues pointed out, note that this adds ordering > constraints among the linkers. Any linker that deletes something must run > after every linker that wants to look at it. Your example wouldn't work as > is, because it would mean no POST linker can look at CompilationResults. It > also wouldn't work to put the deletion in a POST linker, for the same > reason. We'd have to work out a way for the deletions to happen last, after > all the normal linkage activity. > > Suppose, continuing that idea, we add a POSTPOST order that is used only > for deletion. If it's really only for deletion, then the usual link() API > is overly general, because it lets linkers both add and remove artifacts > during POSTPOST, which is not desired. So, we want a POSTPOST API that is > only for deletion. Linkers somehow or another mark artifacts for deletion, > but not anything else. At this point, though, isn't it pretty much the same > as the automated thinning in the initial proposal? > It does start to sound like a big mess when you put it that way. And Ray makes a good point about forcing people to write linkers that won't blow up when you shard them. It sounds like, on a high level, there's a broad agreement that changing the linker API isn't really a problem. That we should basically just redesign it to do the exact best thing and it's okay, since not many linkers exist. To be honest, I hadn't really approached the problem from that point of view. Now that I am thinking along those lines, it almost begs the question. If we are willing to break the world, is this the best possible way to model new link process? In other words, it seems worth re-examining the design without regard to the existing API and asking ourselves if it's the thing we'd have designed from scratch. Maybe you guys all already did that and I'm the only one late to the party. For example, if we're going from scratch, then we could avoid the transition entirely and just mandate what the new rules are. We wouldn't need a @Shardable annotation since all linkers would need to be sharding aware. We might rather have two separate methods for sharded vs. non-sharded link than a boolean parameter. We might revisit the whole PRE, PRIMARY, POST thing with regards to sharding and decide the right answer is SHARD, PRE, PRIMARY, POST. Or something. I don't know what the right answers are. All I'm saying is, breaking things is awesome when you're doing something revolutionary and the end result is awesome. I just want to be sure, if we're going to break things, that we believe we'll end up somewhere revolutionary and awesome as opposed to evolutionary and incremental, but less than awesome. --Scott -- http://groups.google.com/group/Google-Web-Toolkit-Contributors
