[
https://issues.apache.org/jira/browse/HADOOP-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613304#action_12613304
]
Amar Kamat commented on HADOOP-3740:
------------------------------------
bq. allow users to override part of it dynamically
I dont think allowing user code to run in JT is a good idea. I think it makes
more sense to keep the scheduling code as is and spawn tasks irrespective of
the resource. The M/R code should (as a first step) try and get the license
(e.g in {{configure()}}) and block on the {{resource.get()}} call. It might
seem like wastage of resources but probably better than running user code in
Hadoop daemons. Also license is one type of resource. There can be different
types of resources and hence having separate hooks might not make sense. Having
a protocol that defines a communication between the scheduler (as of today the
JIP) and implemented by the _resource-server_ might be another way to tackle
this problem.
> Make JobInProgress pluggable
> ----------------------------
>
> Key: HADOOP-3740
> URL: https://issues.apache.org/jira/browse/HADOOP-3740
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Environment: all
> Reporter: Alejandro Abdelnur
>
> By allowing a pluggable JobInProgess it will be possible for provide
> implementations that can do a sophisticated task provisioning to the
> JobTracker.
> For example, by providing alternate implementations of the
> {{obtainNewMapTask}}, {{obtainNewReduceTask}} and {{updateTaskInProgress}} it
> would be possible to implement a license server that allows to throttle use
> of external resources (ie webservices, databases) so at any given time there
> are not more than N tasks using a given resource. For this a task could be
> tagged with the names of external resources and the license server would keep
> track of the tasks running per tag, if the counter reaches zero then the
> {{obtainNew*Task}} method could return NULL instead of a task.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
