Hey guys, I have just created a PR for the new Kafka input operator using new Kafka consumer API
Some highlights of the operator 1. Out-of-box One-to-one and one-to-many partition scheme support plus customizable partition schema 2. Fault-tolerant 3. At-least-once semantics for operator recovery (no matter which operator fails) 4. At-least-once semantics for cold restart (no data loss even if you restart the application) 5. Multi-cluster support, one operator can consume data from more than one kafka clusters 6. Multi-topic support, one operator can subscribe multiple topics 7. Throughput control support, you can throttle number of tuple for each streaming window Things needs to be done, More unit test for different partition scheme, application restart More integration test Metrics for operator Bring back idempotency
