Doug Cutting wrote:
Paul Sutter wrote:
(1) Allow submission times in the future, enabling the creation of
"background" jobs. My understanding is that job submission times are
used to
prioritize scheduling. All tasks from a job submitted early run to
completion before those of a job submitted later. If we could submit any
days-long jobs with a submission time in the future, say the year
2010, and
any short hours-long jobs with the current time, that short job would be
able to interrupt the long job. Hack? Yes. Useful? I think so.
I think this is equivalent to adding a job priority, where tasks with
the highest priority job are run first. If jobs are at the same
priority, then the first submitted would run. Adding priority would
add a bit more complexity, but would also be less of a hack.
Hmm.. If you compare it to a Unix scheduler, processes at the same
priority have even chances of being run, regardless of which was started
first - not only that, processes undergo a "priority decay", in that if
they are running longer then their priority is lowered - this enables
new processes to start quickly (and maybe quickly finish), and then
fairly compete with other processes.
In our case, this would mean that jobs with the same priority would
execute concurrently, sharing available map/reduce slots, and long
running jobs would be gradually de-prioritized. This also means that the
first job will slow down when the second one is started, but the second
job will have a chance to make a good start (and perhaps quickly finish)
and then, subject to the priority decay, run in parallel with other jobs
(albeit slower) instead of being stuck in the wait queue.
And if the second job is started with a higher priority, it should
preempt the first job (i.e. it should get proportionally more slots than
the first job). If you need all cluster resources for a specific job,
and don't want any other jobs to run, just set the priority to the
highest value, thus preempting all other jobs (actually, it would
suspend other already executing jobs, which would resume when your job
is done - not a bad feature either!).
I think this is a relatively simple and well understood mechanism.
(2) Have a per-job total task count limit. Currently, we establish the
number of tasks each node runs, and how many map or reduce tasks we have
total in a given job. But it would be great if we could set a ceiling
on the
number of tasks that run concurrently for a given job. This may help
with
Andrzej's fetcher (since it is bandwidth constrained, maybe fewer
concurrent
jobs would be fine?).
I like this idea. So if the highest-priority job is already running
at its task limit, then tasks can be run from the next
highest-priority job. Should there be separate limits for maps and
reduces?
I like this idea too. I think a similar setting for the minimum number
of tasks would be needed too? That would solve my problem. In fact, it
would be probably better than the schema I described above, because it
would guarantee certain minimum tasks running at any time.
This reminds me of the "idle time" and "real time" policies in BSD
scheduler ... man 1 rtprio. The "real time" policy prevents the priority
decay that normally occurs, and "idle time" policy allows processes to
run only if CPU is idle.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
