Sorry, forgot the link https://github.com/apache/incubator-apex-malhar/pull/134
On Tue, Dec 15, 2015 at 10:42 AM, Siyuan Hua <[email protected]> wrote: > Hey guys, > > I have just created a PR for the new Kafka input operator using new Kafka > consumer API > > Some highlights of the operator > > 1. Out-of-box One-to-one and one-to-many partition scheme support plus > customizable partition schema > 2. Fault-tolerant > 3. At-least-once semantics for operator recovery (no matter which > operator fails) > 4. At-least-once semantics for cold restart (no data loss even if you > restart the application) > 5. Multi-cluster support, one operator can consume data from more than > one kafka clusters > 6. Multi-topic support, one operator can subscribe multiple topics > 7. Throughput control support, you can throttle number of tuple for > each streaming window > > Things needs to be done, > More unit test for different partition scheme, application restart > More integration test > Metrics for operator > Bring back idempotency > >
