swapna267 commented on PR #28002: URL: https://github.com/apache/flink/pull/28002#issuecomment-4337737116
Specific to lineage listener , this seems to break for code using the flink cli to submit the job to session cluster. As per my understanding, StreamGraph is generated and serialized over network in those use cases, for example, https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/deployment/executors/AbstractSessionClusterExecutor.java#L104 LineageGraph is transient and is not serialized over. https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/streaming/api/graph/StreamGraph.java#L166 . So LineageGraph details are not available on Dispatcher running as part of JobManage JVM. Please correct me if my understanding is not right. I see in https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener, `In this FLIP the JobStatusChangedListener will be in Client and JobMaster, which will report lineage information and job status independently. Another proposal is to **put the lineage information into JobGraph and then report it in JobMaster** . Considering that the lineage information may be relatively large and affect JobGraph , we will report separately in the first phase and then consider merging in the future.` That seems to be the right way to go about if we want to ensure, 1. Single Lineage listener context 2. JobCreationEvent is sent before any JobTransitionEvent. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
