[
https://issues.apache.org/jira/browse/HADOOP-4513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644943#action_12644943
]
Vivek Ratan commented on HADOOP-4513:
-------------------------------------
We should have a thread (call it the init thread) within the Capacity Scheduler
that initializes jobs (i.e., calls initTasks() on jobs even when the job is not
ready to run, as per the scheduler). The following features are desirable:
* We want pre-initialized jobs in all the queues, since the Scheduler can
possibly look at all queues between consecutive runs of this thread.
* Since initTasks() may take a fair bit of time for some jobs, we don't want to
block on this call as much as possible.
* Remember that the init thread is really predicting which jobs the scheduler
will run in the next many heartbeats. It's hard to make this prediction as the
decision of which job gets a free slot depends on the state of the running jobs
at that very instant (to compute user quotas, for example). But there are some
constraints that the thread can consider:
** the waiting queue is sorted, and the scheduler considers jobs in that sort
order. So the init thread should pre-initialize jobs in the same order.
** User quotas are tricky. Suppose a queue supports N concurrent users. The
init thread should really pre-initialize jobs from these N users only, as
described earlier. But this list of N users can change often, depending on
which jobs from which users finish. So, to be safe, the init thread should
pre-initialize jobs from, say N+2 users, to be safe.
* Since we're keeping the run-queue sorted (HADOOP-4471), and that jobs are
moved from the wait queue to the run queue once initTasks is called and the
setup task is complete, the Scheduler only needs to look at jobs in its run
queue. If it doesn't find any job that can accept a task, it moves on to the
next queue.
* If the init thread runs periodically, there may be a brief startup delay.
Suppose the thread runs every 5 seconds. Suppose a job is submitted to an empty
queue just after the thread runs. That job will not be initialized till the
init thread runs next (a max delay of 5 secs). This is not too bad, given that
there will be a delay in running the setup task of that job too. Worst case,
the Scheduler can always cause the init thread to run as soon as a job is
submitted to an empty queue, but this may be an overkill.
> Capacity scheduler should initialize tasks asynchronously
> ---------------------------------------------------------
>
> Key: HADOOP-4513
> URL: https://issues.apache.org/jira/browse/HADOOP-4513
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Sreekanth Ramakrishnan
>
> Currently, the capacity scheduler initializes tasks on demand, as opposed to
> the eager initialization technique used by the default scheduler. This is
> done in order to save JT memory footprint. However, the initialization is
> done in the {{assignTasks}} API which is not a good idea as task
> initialization could be a time consuming operation. This JIRA is to move out
> the initialization outside the {{assignTasks}} API and do it asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
