becketqin commented on pull request #15461: URL: https://github.com/apache/flink/pull/15461#issuecomment-811668592
I share the same concerns as @StephanEwen. `TopicListSubscriber` is probably by far the most used subscriber, so improving the performance there is rewarding. Maybe we can do the following: 1. Pass the `AdminClient` to the subscriber to get the interested partitions. 2. The worker thread hands over the interested partitions to the coordinator thread. 3. The coordinator thread filters out the assigned / pending assigned partitions, then hand over the new partitions to the worker thread for offsets lookup. 4. The worker thread looks up the offsets and hands over the Partition Split to the coordinator thread for assignment. It is a bit more complicated than the current solution, but seems not too bad. Basically `callAsync()` will be invoked twice in order to complete an assignment. The first invocation retrieves the interested partitions and finds out the new partitions, and the second invocation retrieves the offsets for new partitions and complete the assignment. This way we can make sure the coordinator thread won't perform blocking operations. And because the state is only maintained by the coordinator thread, we don't need to worry about the race condition either. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
