[
https://issues.apache.org/jira/browse/HADOOP-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718665#action_12718665
]
Matei Zaharia commented on HADOOP-5659:
---------------------------------------
Hi Dhruba,
A question about this feature: How do you want to decide when to kill the
wellcare job? Do you want the tasks specified as a %, as a fixed number, or
what?
Matei
> Fair share schduler may support preemption only with a specific pool
> --------------------------------------------------------------------
>
> Key: HADOOP-5659
> URL: https://issues.apache.org/jira/browse/HADOOP-5659
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
>
> There are a set of jobs that helps to keep the cluster resources being used
> optimally. For example, there are data sets that are made of a multiple files
> in a directory. These part-xxx files could be concatenated to a relatively
> few files (to reduce memory pressure on the namenode). Also, there are files
> that could be compressed more efficiently (e.g. bzip2) to reduce save on disk
> usage. These are kind of system-wellcare jobs that should run only if it does
> not impact any other "real" user of the cluster. On an idle cluster, these
> wellcare jobs should use all availale system resources. When a real user
> submits a job, the wellcare job(s) should be pre-empted. If a scheduler can
> support pre-emption only for jobs in a specified pool, then I can submit
> these well-care jobs to that special pool. Real user's jobs will never get
> pre-empted;but the wellcare jobs can get pre-empted as soon as there is
> resource contention. If a task of well-care jobs is pre-empted more than a
> configured max, the entire wellcare job will fail.. that this is the
> behaviour I want. The wellcare jobs would run in idle slots as long as all
> user-submitted jobs have been satisfied, but would be preempted as soon as
> user jobs require any of those slots.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
