[
https://issues.apache.org/jira/browse/TEZ-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14104668#comment-14104668
]
Siddharth Seth commented on TEZ-1363:
-------------------------------------
[~jeagles] - the general approach looks good to me.
Question: Is there an advantage of creating a new interface
TezFrameworkAMRMClientAsync, over just extending TezAMRMClientAsync that is in
YARN ? I'm assuming all the methods in this are the same as the ones in
AMRMClientAysnc (YARN)
This is very useful, since it simulates cluster behaviour better - in terms of
object cache usage, container re-use etc. Will have to be careful with handling
of request removals - since the TaskScheduler asks for a container anyway, and
consequently removes the request once it's able to re-use. In terms of location
hints - I don't think that'll be handled at all, since LocalFS doesn't provide
any location information. It should just work if LocalMode happens to run
against an HDFS cluster.
Unrelated, but location hints for one-to-one edges in localmode will be
interesting. Will have to look at how this wold work if the source doesn't have
any location information.
Is the LocalContainerAllocator similar to LocalTaskScheduler ? We should remove
the LocalTaskScheduler as part of this patch itself.
bq. Blacklisting needs to be disabled in local mode
I think this needs to go in anyway.
Is this ready for a detailed review, beyond the structural changes ?
> Make use of the regular scheduler when running in LocalMode
> -----------------------------------------------------------
>
> Key: TEZ-1363
> URL: https://issues.apache.org/jira/browse/TEZ-1363
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Jonathan Eagles
> Attachments: TEZ-1363-v1.patch, TEZ-1363-v2.patch, TEZ-1363-v3.patch
>
>
> In TEZ-708, we decided to introduce a new scheduler for local mode - to keep
> things simple initially, and get local mode working.
> Eventually, however, scheduling should go through the regular task scheduler
> - which should be able to get containers from YARN / LocalAllocator / other
> sources - and treat them as a regular container for scheduling purposes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
