[jira] [Commented] (TEZ-707) Create LocalContainerLauncher
[ https://issues.apache.org/jira/browse/TEZ-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14077747#comment-14077747 ] Chen He commented on TEZ-707: - I have spent much time on this and want it to be done by myself. Thank you for the suggestion, [~sseth]. Create LocalContainerLauncher - Key: TEZ-707 URL: https://issues.apache.org/jira/browse/TEZ-707 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.3.0 Reporter: Chen He Assignee: Chen He Priority: Blocker Attachments: TEZ-707-2014-7015.patch, TEZ-707-2014-7015.patch.review, TEZ-707-v3.patch, TEZ-707-v4.patch, TEZ-707.patch, TEZ-707.patch, TEZ-707.patch, Tez-707.patch.v2, tez-707.patch Create LocalContainerLauncher and make it work for a single stage DAG. The TaskSchedulerEventHandler still asks RM for new container but LocalContainerLauncher will run TezTask in form of thread instead of using this container from yarn. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-707) Create LocalContainerLauncher
[ https://issues.apache.org/jira/browse/TEZ-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14077977#comment-14077977 ] Siddharth Seth commented on TEZ-707: [~airbots] - sure, can you work on getting the last bits you mentioned in today. Basic code cleanup - get rid of the statics, fix the overrides (and conf handling), local-dir setup (this will likely be another config parameter which is setup by the local-client). I believe the patch will work for cases where the DAG executes correctly and without errors - so we can commit this to get started. Create LocalContainerLauncher - Key: TEZ-707 URL: https://issues.apache.org/jira/browse/TEZ-707 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.3.0 Reporter: Chen He Assignee: Chen He Priority: Blocker Attachments: TEZ-707-2014-7015.patch, TEZ-707-2014-7015.patch.review, TEZ-707-v3.patch, TEZ-707-v4.patch, TEZ-707.patch, TEZ-707.patch, TEZ-707.patch, Tez-707.patch.v2, tez-707.patch Create LocalContainerLauncher and make it work for a single stage DAG. The TaskSchedulerEventHandler still asks RM for new container but LocalContainerLauncher will run TezTask in form of thread instead of using this container from yarn. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-707) Create LocalContainerLauncher
[ https://issues.apache.org/jira/browse/TEZ-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14078090#comment-14078090 ] Chen He commented on TEZ-707: - Sure. About the local-dir setup, Yarn uses yarn.nodemanager.local-dirs to get the local directories for localized files and job related temp files. It may be flexible if we have tez.local-dirs in TezConfiguration without dependency on nm related configuration. What you feel, [~sseth]? Create LocalContainerLauncher - Key: TEZ-707 URL: https://issues.apache.org/jira/browse/TEZ-707 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.3.0 Reporter: Chen He Assignee: Chen He Priority: Blocker Attachments: TEZ-707-2014-7015.patch, TEZ-707-2014-7015.patch.review, TEZ-707-v3.patch, TEZ-707-v4.patch, TEZ-707.patch, TEZ-707.patch, TEZ-707.patch, Tez-707.patch.v2, tez-707.patch Create LocalContainerLauncher and make it work for a single stage DAG. The TaskSchedulerEventHandler still asks RM for new container but LocalContainerLauncher will run TezTask in form of thread instead of using this container from yarn. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-707) Create LocalContainerLauncher
[ https://issues.apache.org/jira/browse/TEZ-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14078245#comment-14078245 ] Siddharth Seth commented on TEZ-707: Instead of tez-local-dirs, let's call it something related to local mode. tez.local.execution.work.dir Create LocalContainerLauncher - Key: TEZ-707 URL: https://issues.apache.org/jira/browse/TEZ-707 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.3.0 Reporter: Chen He Assignee: Chen He Priority: Blocker Attachments: TEZ-707-2014-7015.patch, TEZ-707-2014-7015.patch.review, TEZ-707-v3.patch, TEZ-707-v4.patch, TEZ-707.patch, TEZ-707.patch, TEZ-707.patch, Tez-707.patch.v2, tez-707.patch Create LocalContainerLauncher and make it work for a single stage DAG. The TaskSchedulerEventHandler still asks RM for new container but LocalContainerLauncher will run TezTask in form of thread instead of using this container from yarn. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-707) Create LocalContainerLauncher
[ https://issues.apache.org/jira/browse/TEZ-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14078253#comment-14078253 ] Chen He commented on TEZ-707: - Got it. Would you mind I create a JIRA to fix this? Create LocalContainerLauncher - Key: TEZ-707 URL: https://issues.apache.org/jira/browse/TEZ-707 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.3.0 Reporter: Chen He Assignee: Chen He Priority: Blocker Attachments: TEZ-707-2014-7015.patch, TEZ-707-2014-7015.patch.review, TEZ-707-v3.patch, TEZ-707-v4.patch, TEZ-707.patch, TEZ-707.patch, TEZ-707.patch, Tez-707.patch.v2, tez-707.patch Create LocalContainerLauncher and make it work for a single stage DAG. The TaskSchedulerEventHandler still asks RM for new container but LocalContainerLauncher will run TezTask in form of thread instead of using this container from yarn. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-707) Create LocalContainerLauncher
[ https://issues.apache.org/jira/browse/TEZ-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14078744#comment-14078744 ] Chen He commented on TEZ-707: - BTW, patch tested with latest TEZ-717 together. It successfully run one vertex. For the more than one vertex, we need TEZ-709. Create LocalContainerLauncher - Key: TEZ-707 URL: https://issues.apache.org/jira/browse/TEZ-707 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.3.0 Reporter: Chen He Assignee: Chen He Priority: Blocker Attachments: TEZ-707-2014-7015.patch, TEZ-707-2014-7015.patch.review, TEZ-707-v10.patch, TEZ-707-v3.patch, TEZ-707-v4.patch, TEZ-707.patch, TEZ-707.patch, TEZ-707.patch, Tez-707.patch.v2, tez-707.patch Create LocalContainerLauncher and make it work for a single stage DAG. The TaskSchedulerEventHandler still asks RM for new container but LocalContainerLauncher will run TezTask in form of thread instead of using this container from yarn. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-707) Create LocalContainerLauncher
[ https://issues.apache.org/jira/browse/TEZ-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14078907#comment-14078907 ] Siddharth Seth commented on TEZ-707: Committing this. Thanks [~airbots] Create LocalContainerLauncher - Key: TEZ-707 URL: https://issues.apache.org/jira/browse/TEZ-707 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.3.0 Reporter: Chen He Assignee: Chen He Priority: Blocker Attachments: TEZ-707-2014-7015.patch, TEZ-707-2014-7015.patch.review, TEZ-707-v10.patch, TEZ-707-v3.patch, TEZ-707-v4.patch, TEZ-707.patch, TEZ-707.patch, TEZ-707.patch, Tez-707.patch.v2, tez-707.patch Create LocalContainerLauncher and make it work for a single stage DAG. The TaskSchedulerEventHandler still asks RM for new container but LocalContainerLauncher will run TezTask in form of thread instead of using this container from yarn. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-707) Create LocalContainerLauncher
[ https://issues.apache.org/jira/browse/TEZ-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14076348#comment-14076348 ] Siddharth Seth commented on TEZ-707: [~airbots] - the number of threads in the pool does not need to be set high to maintain pending items. The queue that is backing the executor takes care of this. Create LocalContainerLauncher - Key: TEZ-707 URL: https://issues.apache.org/jira/browse/TEZ-707 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.3.0 Reporter: Chen He Assignee: Chen He Priority: Blocker Attachments: TEZ-707-2014-7015.patch, TEZ-707-2014-7015.patch.review, TEZ-707-v3.patch, TEZ-707-v4.patch, TEZ-707.patch, TEZ-707.patch, Tez-707.patch.v2, tez-707.patch Create LocalContainerLauncher and make it work for a single stage DAG. The TaskSchedulerEventHandler still asks RM for new container but LocalContainerLauncher will run TezTask in form of thread instead of using this container from yarn. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-707) Create LocalContainerLauncher
[ https://issues.apache.org/jira/browse/TEZ-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14075775#comment-14075775 ] Chen He commented on TEZ-707: - I am still confusing about why we need future type of tasks running in local mode after I investigate the Callback and ListeningServiceFuture mechanism. Callback mechanism allows us to get return of task, ListeningServiceFuture provides manageable async way to run tasks. We can get return of each running task by using above mechanisms. However, In current version of LocalContainerLauncher, threadspool is used to control the parallelism of running tasks. We do blocking if running tasks reach the poolsize and sleep TezSubTaskRunner. Actually, tasks come and go, we care the parallelism more than the return of each task. Since a task failed, it will record failure in log (since we are running in a single process). Please correct me if there is misunderstanding. What you feel, [~sseth]? Create LocalContainerLauncher - Key: TEZ-707 URL: https://issues.apache.org/jira/browse/TEZ-707 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.3.0 Reporter: Chen He Assignee: Chen He Priority: Blocker Attachments: TEZ-707-2014-7015.patch, TEZ-707-2014-7015.patch.review, TEZ-707-v3.patch, TEZ-707-v4.patch, TEZ-707.patch, TEZ-707.patch, Tez-707.patch.v2, tez-707.patch Create LocalContainerLauncher and make it work for a single stage DAG. The TaskSchedulerEventHandler still asks RM for new container but LocalContainerLauncher will run TezTask in form of thread instead of using this container from yarn. -- This message was sent by Atlassian JIRA (v6.2#6252)
