becketqin commented on pull request #13574: URL: https://github.com/apache/flink/pull/13574#issuecomment-717837286
@StephanEwen I am trying to address the blocking close() method issue. However, it is a little more complicated than I thought. I want to align some design principle with you and see what might be the solution there. From what I understand the current design principle for the `SplitEnumerator` is that all the exception handling are synchronous, i.e. the implementation should throw exception when a method is invoked. If no exception was thrown, the method invocation is considered successful. In this case, if we want to allow the asynchronous handling and exception propagation in `SplitEnumerator`, such as asynchronous close(), we will need to have a `failJob()` method in the `SplitEnumeratorContext`. So users can call `failJob()` asynchronously instead of throw exceptions when closing in a non-blocking manner. The only caveat to this solution is that now users can either throw exception or call `failJob()` when the methods are invoked and people may wonder what is the difference. To avoid the above caveat, I was thinking that we can add the async close to the `SourceCoordinator` so we don't have to add `failJob()` method to the `SplitEnumerator`. However, in practice, sometimes the previous instance of `SplitEnumerator` must be successfully closed before the next instance can be created. Otherwise there might be conflicts. Therefore, naively having non-blocking closing in the `SourceCoordinator` won't work in all cases. Given all the above thoughts, I am falling back to the solution of adding a `failJob()` method to the `SplitEnumeratorContext` so the `SplitEnumerator` implementations can decide by themselves what to do in each method. And any exception thrown from the method invocation will just result in the job failure. Do you have any suggestion? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
