afedulov commented on PR #23777: URL: https://github.com/apache/flink/pull/23777#issuecomment-1910235989
@leonardBang , thanks for sharing your concerns, I understand where they are coming from. Here are my thoughts: - This is probably the trickiest source to migrate, since it artificially imposes synchronization requirement between the sources and the sinks. This is not the normal case and I do not expect users or connector implementors to have to deal with these intricate details. The aim of this PR is solely to provide a drop-in replacement for the FiniteTestSource that then can be utilized as is. This is how the "normal" case looks like: https://github.com/apache/flink/pull/21774/files - The complexity of the implementation stems from two main aspects: - The inherent complexity of the new Source API - The fact that the execution model is completely different. The "pull" loop is outside of our reach. In the new Source API we are essentially implementing an async functions that are, by design, supposed to be oblivious of the context it which it is called. There is no checkpoint lock anymore that we can hold to prevent the data from being pulled while the sink finishes flushing data. - To be honest I feel we are a bit lucky that this is was at all possible to achieve feature parity with the SourceFunction without the the checkpoint lock and without modifying the core of the new Source API itself :) Maybe as we go further we can detect more common patterns to further abstract away, but at the moment I do not have ideas for something meaningful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
