[
https://issues.apache.org/jira/browse/TEZ-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503828#comment-14503828
]
Siddharth Seth commented on TEZ-2212:
-------------------------------------
It could be done directly in the submitDagHandler. It should only be invoked
after the appContext has been updated, which makes the dagId etc available via
the context.
The main intent for this was to serve information off of the DAG instance setup
by the dagComplete invocation. After this, information will be served out again
from the appContext once dagSubmitted is invoked. It ends up providing a sync
point after which any queries on dag / vertex information will be the new DAG.
>From the offline discussion we just had, serving data from the old dag during
>the dagComplete invocation into user code may be sufficient.
After this, user code should ideally not be querying any data for the dag.
One option is to have the dagSubmitted notification as a sync point, after
which user code starts getting valid data again.
The other option is to just keep serving off the appContext - in which case the
user code doesn't know that a new dag is available, and it queries for
information only on receiving specific events.
At the moment, I don't think there's an explicit requirement in the
TaskCommunicator to query asynchronously (i.e. without the framework invoking
the user code). So both options would work. I'd still lean towards informing
the user code in case they need to take some action.
> Notify components on DAG completion
> -----------------------------------
>
> Key: TEZ-2212
> URL: https://issues.apache.org/jira/browse/TEZ-2212
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: TEZ-2212.1.txt, TEZ-2212.2.txt, TEZ-2212.3.txt
>
>
> Various components, especially the ones that are made pluggable by TEZ-2003,
> should be informed about DAG completion, so that they can perform required
> cleanup - ContainerLauncher, TaskAttemptListener, Scheduler (already
> informed). In addition ID caches etc should be cleaned up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
