fapaul commented on pull request #18428: URL: https://github.com/apache/flink/pull/18428#issuecomment-1024099669
@gaoyunhaii thanks for your review. > I have one major concern regarding currently we might add the expanded transformations to the original environment. This seems to might cause problems if users call execute multiple times with the same env like > > ``` > DataStream a = ... > DataStream b = ... > b.addSink(new xx()); > > // The following would first clear all the current > // transformations, but when translating the sink, > // it would add the expanded transformations here. > env.execute() > > b.map(...); > // Then here it would execute a graph also with > // a -> some sink transformations > env.execute() > ``` > > Is it possible we use a separate `env` for the sink translation ? We could wrap the input transformation with a DataStream in the new environment, and at last we get all the transformations and transform them. We could also disable some methods inside adding topology, like switch runtime mode or configure slot resources. Maybe I overlooked something but I thought after every `StreamExecutionEnvironment#execute` the transformations are cleared. [1][2][3] Regarding your failover region comment https://github.com/apache/flink/pull/18428#discussion_r794246463 (for some reason I cannot answer your comment in the UI...) I assumed that the Batch exchange will never have an effect in streaming mode and therefore we could handle them equally. If that is not the case I also tend to do your suggestion. [1] https://github.com/apache/flink/blob/1704b1336bcbcdd42405f5e6e3e09c0176925210/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#L2137 [2] https://github.com/apache/flink/blob/1704b1336bcbcdd42405f5e6e3e09c0176925210/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#L2120 [3] https://github.com/apache/flink/blob/1704b1336bcbcdd42405f5e6e3e09c0176925210/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#L1950 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
