Doug Cutting wrote:
> Paul Sutter wrote:
>> (1) Allow submission times in the future, enabling the creation of
>> "background" jobs. My understanding is that job submission
times are
>> used to
>> prioritize scheduling. All tasks from a job submitted early run to
>> completion before those of a job submitted later. If we could
submit
any
>> days-long jobs with a submission time in the future, say the year
>> 2010, and
>> any short hours-long jobs with the current time, that short job
would
be
>> able to interrupt the long job. Hack? Yes. Useful? I think so.
>
> I think this is equivalent to adding a job priority, where tasks
with
> the highest priority job are run first. If jobs are at the same
> priority, then the first submitted would run. Adding priority
would
> add a bit more complexity, but would also be less of a hack.
Hmm.. If you compare it to a Unix scheduler, processes at the same
priority have even chances of being run, regardless of which was
started
first - not only that, processes undergo a "priority decay", in
that if
they are running longer then their priority is lowered - this enables
new processes to start quickly (and maybe quickly finish), and then
fairly compete with other processes.
In our case, this would mean that jobs with the same priority would
execute concurrently, sharing available map/reduce slots, and long
running jobs would be gradually de-prioritized. This also means
that the
first job will slow down when the second one is started, but the
second
job will have a chance to make a good start (and perhaps quickly
finish)
and then, subject to the priority decay, run in parallel with
other jobs
(albeit slower) instead of being stuck in the wait queue.
And if the second job is started with a higher priority, it should
preempt the first job (i.e. it should get proportionally more
slots than
the first job). If you need all cluster resources for a specific job,
and don't want any other jobs to run, just set the priority to the
highest value, thus preempting all other jobs (actually, it would
suspend other already executing jobs, which would resume when your
job
is done - not a bad feature either!).
I think this is a relatively simple and well understood mechanism.
>> (2) Have a per-job total task count limit. Currently, we
establish the
>> number of tasks each node runs, and how many map or reduce
tasks we
have
>> total in a given job. But it would be great if we could set a
ceiling
>> on the
>> number of tasks that run concurrently for a given job. This may
help
>> with
>> Andrzej's fetcher (since it is bandwidth constrained, maybe fewer
>> concurrent
>> jobs would be fine?).
>
> I like this idea. So if the highest-priority job is already
running
> at its task limit, then tasks can be run from the next
> highest-priority job. Should there be separate limits for maps and
> reduces?
I like this idea too. I think a similar setting for the minimum
number
of tasks would be needed too? That would solve my problem. In
fact, it
would be probably better than the schema I described above,
because it
would guarantee certain minimum tasks running at any time.
This reminds me of the "idle time" and "real time" policies in BSD
scheduler ... man 1 rtprio. The "real time" policy prevents the
priority
decay that normally occurs, and "idle time" policy allows
processes to
run only if CPU is idle.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
