becketqin commented on issue #10487: [FLINK-15100][connector/source] Add a base implementation for SourceReader. URL: https://github.com/apache/flink/pull/10487#issuecomment-605050095 Thanks for the review @StephanEwen. I'll merge this PR and submit the SourceCoordinator PR then. The reason I am worrying about the batch size was because in old high-level Kafka consumer, the client imply returns a bunch of iterators and thus there was no concept of record batch. But I agree that it doesn't seem a common pattern. > (Side note: The Handover in the FlinkKafkaConsumer is effectively a size one queue, supporting wakeup() to avoid interrupts on the fetcher thread, because we found that many connectors behave erratically when the Thread is interrupted while inside the connector call stack. Both Kafka and HDFS had quite some issues with that in the past.) Completely agree that the wakeup is tricky. It took me quite a while to handle that in the `SplitFetcher`.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
