I will be working on rewriting the kafka input operator. Here is the ticket https://malhar.atlassian.net/browse/MLHR-1904
Here is some comments on the ticket The RC2 is out here https://people.apache.org/~junrao/kafka-0.9.0.0-candidate2/ We will keep most features of the old input operator but the internal mechanism will be changed, for example, using new API to refresh the metadata The bugs that will be fixed: - Synchronized offset checkpoint - Transient offsetmanager New features: - Support customized partition schema - Default OffsetManager using new Improvement - Add window id and application name to OffsetManager interface - Support multi-topic - Easy configuration Please leave thoughts here or on the ticket. Thanks Best, Siyuan
