[
https://issues.apache.org/jira/browse/TEZ-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated TEZ-2212:
--------------------------------
Attachment: TEZ-2212.1.txt
The patch informs the ContainerLauncher and TaskAttemptListener about dag
completion. It also performs some basic cleanup like removing the cached
entries in *ID classes, AMContainerHelpers and TaskHeartbeatHandler.
For cleanup, there's more to be done - which will be separate jiras - cleanup
the data structures in TaskAttemptListener, potentially update Node failure
counts, eventually inform containers so that they can delete data.
Haven't included any tests; they would primarily just ensure the methods are
invoked. The existing tests should cover functionality.
[~hitesh] - mind taking a look. One thing to look at - is it safe to make these
invocations like it's being done in the patch, or do they need to go onto the
Central Dispatcher. Would the queue have been emptied by the time the
DAG_FINISHED event is processed.
> Notify components on DAG completion
> -----------------------------------
>
> Key: TEZ-2212
> URL: https://issues.apache.org/jira/browse/TEZ-2212
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Attachments: TEZ-2212.1.txt
>
>
> Various components, especially the ones that are made pluggable by TEZ-2003,
> should be informed about DAG completion, so that they can perform required
> cleanup - ContainerLauncher, TaskAttemptListener, Scheduler (already
> informed). In addition ID caches etc should be cleaned up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
