StephanEwen edited a comment on issue #10487: [FLINK-15100][connector/source] Add a base implementation for SourceReader. URL: https://github.com/apache/flink/pull/10487#issuecomment-604955694 Let's keep the mocks where they are then. About the queue and size: I think this is not a performance issue, so lock-free tricks are probably not the most important concern. - We know we don't need it for Kafka, a size-one handover of `ConsumerRecords` works well and has a manageable memory footprint. - I believe for Kinesis it is somewhat similar, with the exception that it is one FetchBatch per subscribed shard. - For files, let's see. We can only validate this in an actual implementation, but I think we will typically hand over one of the following: - the equivalent of either one I/O buffer, in deserialized form (record-based file formats) - one set of column vectors, when working with a format like ORC and Parquet. My suggestion would be to leave it as is now, maybe set the default value of the queue sie to `1`. Then we try to eliminate this in follow-up implementations of the sources as above, and if we find that we indeed don't need this, then let's replace it by a `Handover` like in the `FlinkKafkaConsumer`. So, from my side, +1 to go ahead and merge this. *(Side note: The Handover in the FlinkKafkaConsumer is effectively a size one queue, supporting `wakeup()` to avoid interrupts on the fetcher thread, because we found that many connectors behave erratically when the Thread is interrupted while inside the connector call stack. Both Kafka and HDFS had quite some issues with that in the past.)*
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
