I have a scheduling problem, and I'm not sure how to address it with the currently available tools. We're using Hadoop-0.20.1.
For most of our needs, I like the Fair Scheduler. It allocates resources reasonably between multiple users. However, I have a process which occasionally needs to run a small job without delay. According to the documentation for the Fair Scheduler, I can create a separate queue for these high priority jobs, and allocate resources. However, if there is nothing in that queue, the resources will be released for use by other queues, and will not be available when I need them. Without preemptive scheduling, I'm not sure how to address this. It looks like the best solution may be to apply the patch from MAPREDUCE-551, which adds preemption. Anyone have a better idea? I'm a little concerned about the stability of code when applying patches that may be part of a release, without updating to a full release. Thanks! David
