[
https://issues.apache.org/jira/browse/MAPREDUCE-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Radwan updated MAPREDUCE-2836:
------------------------------------
Attachment: MAPREDUCE-2836_rev5.patch
Thanks Todd for you detailed comments.
- In jobAdded(), it actually calls poolMgr.addJob(..) before constructing the
JobInfo and putting it in the infos map (FairScheduler.java: lines 171 - 173).
So in case of throwing an exception, no orphaned jobs will exist.
- In case of reloading the allocation file, only new pool names are added,
while existing ones will remain (even if removed from the allocation file). I
thought this is the required behavior (i.e., only previously existing pools
will get removed if the scheduler is restarted). I agree with you, I updated
the patch to clear the pools map with every reload, so only pools in the
reloaded file will exist.
- The JobTracker.addJob(..) actually throws the exception (JobTracker.java:
line 4030). If you mean the call to addJob (line 3985), the exception is
caught, but it is logged and thrown again (line 3991).
- I have modified the new test case to use the MiniMRCluster. The behavior
remains the same; the exception propagates all the way to the JobClient.
I have also modified the property name per your comment to not use camelCase.
> Provide option to fail jobs when submitted to non-existent pools.
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-2836
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2836
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: Jeff Bean
> Assignee: Ahmed Radwan
> Priority: Minor
> Attachments: MAPREDUCE-2836.patch, MAPREDUCE-2836_rev2.patch,
> MAPREDUCE-2836_rev3.patch, MAPREDUCE-2836_rev4.patch,
> MAPREDUCE-2836_rev5.patch
>
>
> In some environments, it might be desirable to explicitly specify the fair
> scheduler pools and to explicitly fail jobs that are not submitted to any of
> the pools.
> Current behavior of the fair scheduler is to submit jobs to a default pool if
> a pool name isn't specified or to create a pool with the new name if the pool
> name doesn't already exist. There should be a configuration option for the
> fair scheduler that causes it to noisily fail the job if it's submitted to a
> pool that isn't pre-specified or if the specified pool doesn't exist.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
