I spent quite a bit of time trying out this new permutation worker
framework (mostly spent stepping through it in my debugger, as I was
having issues). Here are my thoughts:
1). The mechanism for designating the number of threads or processes
needs to be cleaned up or rethought. The parameter "-localWorkers N"
ends up spawning 1 ThreadedPermutationWorker and N-1
ExternalPermutationWorkers. This is counter-intuitive, because when I
set something to N, I'm expecting to see N homogeneous objects.
That's because if you don't specify a value for
gwt.jjs.permutationWorkerFactory, the default is to start creating
threaded workers up to maxThreads and use external workers for the
rest.
I have a dual-core CPU, so I wanted to use 2 threads. I eventually
figured out that the way to accomplish this is to either
a). set "-Dgwt.jjs.maxThreads=2" *and* "-localWorkers 2"
simultaneously, or
b). set "-
Dgwt.jjs.permutationWorkerFactory=com.google.gwt.dev.ThreadedPermutationWorkerFactory"and
"-localWorkers 2"
A better way of managing this would be to have two command-line
parameters, nProcesses and n nThreadsPerProcess.
2). I couldn't get ExternalPermutationWorkerFactory to work:
java.net.ConnectException: Connection refused: connect
at com.google.gwt.dev.CompilePermsServer.run
(CompilePermsServer.java:243)
3). Finally, I'm sorry to say this, but this framework is mostly
useless if you have fewer than 8 cores and more than one module to
compile.
There is very little performance gain when I compile with 2 threads
instead of 1 (only 10% per module on a dual-core CPU with 5
permutations per module).
Although the permutations can now be compiled using possibly more than
one thread, the modules are still processed sequentially.
If you have more than one output module (and I suspect most developers
for whom compiler speed is an issue fall into this category), then
it's much faster to run the compiler in parallel (i.e. invoke multiple
instances of the complier with 1 module each, instead of one compiler
process with a list of modules). This gives you a 100% boost on a
dual-core CPU. I've been doing this using the ForTask from ant-
contrib for the past half year and it halved my build time.
In other words, the task of compiling modules is embarrassingly
parallel, but the the task of generating permutations within a module
doesn't seem to be so.
So basically, as much as this new framework looks cool, it's totally
useless to me when I have to build more than one module.
If I had a computer with 2 quad-core CPUs (like Bob ;) ), on the other
hand, I might spawn 2 compiler instances with 4 threads each.
Some food for thought here - perhaps you should think about using the
external workers for modules and threads for permutations.
Alex
On Jan 21, 6:45 pm, Alex Epshteyn <alexander.epsht...@gmail.com>
wrote:
> This stuff looks great, guys. Is there any chance of actually setting
> up a distributed build cluster using this code?
>
> Back when I had an older laptop, I've thought about the idea of
> spawning one or more EC2 instances to do builds. Now, with a top of
> the line laptop, I was happy for a while, but I still only have 2
> cores, and build times are starting to bother me again.
>
> On Nov 24 2008, 12:00 pm, Lex Spoon <sp...@google.com> wrote:
>
> > On Thu, Nov 20, 2008 at 8:58 PM, Scott Blum <sco...@google.com> wrote:
> > > - We ended up going with Lex's ThreadPoolExecutor stuff. Mainly because
> > > he
> > > was physically here, and I think both of the TPE-based implementations
> > > suck.
>
> > Well, we eliminated TPE itself. However, we went with message-passing
> > coordination rather than checking multiple counters and queues and/or
> > checking the interrupted states.
>
> > -Lex
--~--~---------~--~----~------------~-------~--~----~
http://groups.google.com/group/Google-Web-Toolkit-Contributors
-~----------~----~----~----~------~----~------~--~---
