[
https://issues.apache.org/jira/browse/MAPREDUCE-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098489#comment-13098489
]
Todd Lipcon commented on MAPREDUCE-2836:
----------------------------------------
Hi Ahmed,
I took another detailed look at this and some potential issues.
- In jobAdded(), it constructs a new JobInfo and puts it in the {{infos}} map
before it calls {{poolMgr.addJob}}. So, in the case that an exception is
thrown, the job will end up orphaned in the {{infos}} map.
- this implementation has the bug that, if the allocations file is reloaded
with one of the pools removed, users will still be allowed to submit jobs to
it. That doesn't seem quite right.
- in JobTracker.addJob, it looks like it just catches and warns if the jobAdded
notification fails. It doesn't actually reject the job submission. I think your
test doesn't catch this since it's using a fake JT and not the real
implementation.
- Maybe you can add a functional test that uses MiniMRCluster to see the above
bug?
One minor thing I noticed: we don't usually use camelCase in the configs.
Perhaps {{mapred.fairscheduler.allow.undeclared.pools}} would be more
consistent.
I think I agree with Matei's earlier comment that it would be better to add a
hook in the scheduler API like "checkJobSubmission(JobInProgress job)" that
could be called after aclsManager.checkAccess and checkQueueValidity in
submitJob. Then, to fix the other bug with pool reloading, add a new
{{{Set<String> validPools}} which is loaded in {{reloadAllocs}}. At submission
time you can simply check if the pool exists in that list. Then getPool() etc
wouldn't need to be changed at all. Does that sound reasonable?
> Provide option to fail jobs when submitted to non-existent pools.
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-2836
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2836
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: Jeff Bean
> Assignee: Ahmed Radwan
> Priority: Minor
> Attachments: MAPREDUCE-2836.patch, MAPREDUCE-2836_rev2.patch,
> MAPREDUCE-2836_rev3.patch, MAPREDUCE-2836_rev4.patch
>
>
> In some environments, it might be desirable to explicitly specify the fair
> scheduler pools and to explicitly fail jobs that are not submitted to any of
> the pools.
> Current behavior of the fair scheduler is to submit jobs to a default pool if
> a pool name isn't specified or to create a pool with the new name if the pool
> name doesn't already exist. There should be a configuration option for the
> fair scheduler that causes it to noisily fail the job if it's submitted to a
> pool that isn't pre-specified or if the specified pool doesn't exist.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
