Hello everyone, I have a few questions about the current status and future of the Kafka consumers.
We have been working to adding Kafka support in Spring XD [1], currently using the high level consumer via Spring Integration Kafka [2]. We are working on adding features such as: - the ability to control offsets/replay topics; - the ability to control partition allocation across multiple consumers; We are currently at version 0.8.1.1, so using the simple consumer is a pretty straightforward choice right now. However, in the light of the upcoming consumer changes for 0.8.2 and 0.9, I have a few questions: 1) With respect to the consumer redesign for 0.9, what is the future of the Simple Consumer and High Level Consumer? To my best understanding, the existing high level consumer API will be deprecated in favour of the new consumer API. What is the future of the Simple Consumer, in this case? it will continue to exist as a low-level API implementing the Kafka protocol [3] and providing the building blocks for the new consumer, or will it be deprecated as well? 2) Regarding the new consumer: the v0.8.2 codebase contains an early implementation of it, but since this a feature scheduled only for 0.9, what is its status as well? Is it included only as a future reference and for stabilizing the API? 3) Obviously, offset management is a concern if using the simple consumer, so - wondering about the Offset Management API as well. The Kafka protocol document specifically indicates that it will be fully functional in 0.8.2 [4] - however, a functional implementation is already available in 0.8.1.1 (accessible via the SimpleConsumer API but not documented in [5]). Again, trying to understand the extent of what 0.8.1.1 already supports (ostensibly, the offset manager support seems to have been added only in 0.8.2 - please correct me if I am wrong), and whether if it is recommended for use in production in any form (with the caveats that accompany the use of ZooKeeper). 4) Trying to interpret the existing examples in [6] and the comments on [7] - the version of the Offset Management API that exists in 0.8.1.1 is using ZooKeeper - whereas ZooKeeper will be optional in 0.8.2 - to be replaced by Kafka, and phased out if possible. To my understanding, the switch between the two will be controlled by the broker configuration (along with other parameters that control the performance of offset queues. Is that correct? 5) Also, wondering about the timeline of 0.8.2 - according to the roadmaps it should be released relatively shortly. Is that correct? Thanks, Marius [1] http://projects.spring.io/spring-xd/ [2] https://github.com/spring-projects/spring-integration-kafka [3] https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol [4] https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommit/FetchAPI [5] http://kafka.apache.org/082/documentation.html#simpleconsumerapi [6] https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka [7] https://issues.apache.org/jira/browse/KAFKA-1729
