Lance Amundsen wrote:
Example: let's say I have 10K one second jobs and I want the whole thing to run 2 seconds. I currently see no way for Hadoop to achieve this,
That's right. That has not been a design goal to date. Tasks are typically expected to last at least several seconds. To fix this we'd need to permit a single JVM to handle a sequence of tasks, which would be possible. Currently tasks are always run in a separate JVM since they load user code and may fail in arbitrary ways. So if someone wishes to implement this, it should probably be optional.
Doug
