Hi Gwen, Please see inline answers. I¹ll update them in the KIP as well.
Thanks. Jiangjie (Becket) Qin On 1/20/15, 6:39 PM, "Gwen Shapira" <gshap...@cloudera.com> wrote: >Thanks for the detailed document, Jiangjie. Super helpful. > >Few questions: > >1. You mention that "A ConsumerRebalanceListener class is created and >could be wired into ZookeeperConsumerConnector to avoid duplicate >messages when consumer rebalance occurs in mirror maker." > >Is this something the user needs to do or configure? or is the wiring >of rebalance listener into the zookeeper consumer will be part of the >enhancement? >In other words, will we need to do anything extra to avoid duplicates >during rebalance in MirrorMaker? For ZookeeperConsumerConnector in general, users need to wire in listener by themselves in code. For Mirror Maker, an internal rebalance listener has been wired in by default to avoid duplicates on consumer rebalance. User could still specify a custom listener class in command line argument, the internal rebalance listener will call that listener after it finishes the default logic. > >2. "The only source of truth for offsets in consume-then-send pattern >is end user." - I assume you don't mean an actual person, right? So >what does "end user" refer to? Can you clarify when will the offset >commit thread commit offsets? And which JIRA implements this? By end user I mean the target cluster here. The offset commit thread commit thread periodically. It only commit the offsets that have been acked. > >3. Maintaining message order - In which JIRA do we implement this part? KAFKA-1650 > >Again, thanks a lot for documenting this and even more for the >implementation - it is super important for many use cases. > >Gwen > > >Gwen > >On Tue, Jan 20, 2015 at 4:31 PM, Jiangjie Qin <j...@linkedin.com.invalid> >wrote: >> Hi Kafka Devs, >> >> We are working on Kafka Mirror Maker enhancement. A KIP is posted to >>document and discuss on the followings: >> 1. KAFKA-1650: No Data loss mirror maker change >> 2. KAFKA-1839: To allow partition aware mirror. >> 3. KAFKA-1840: To allow message filtering/format conversion >> Feedbacks are welcome. Please let us know if you have any questions or >>concerns. >> >> Thanks. >> >> Jiangjie (Becket) Qin
