Re: [DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

Oh, seems I missed your comment saying the default would be "auto." Hmm... If that's safe, then it sounds good to me. -Jason On Mon, Mar 13, 2017 at 2:32 PM, Jason Gustafson wrote: > Hey Onur, > > >> Regarding 1: I've been considering something like this for a while now. >>

Re: [DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

Hey Onur, > Regarding 1: I've been considering something like this for a while now. > KIP-122 has a similar issue and I brought up some hacks in that discussion > to work around it (http://markmail.org/message/kk4ng74riejidify). While > solving this problem would help loosen the requirements for

Re: [DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

I forgot to mention that in that above idea, the "coordination.migration.mode" config would default to "auto". On Fri, Mar 10, 2017 at 1:08 AM, Onur Karaman wrote: > Hey Jason. > > Thanks for the comments! > > Regarding 1: I've been considering something like this

Re: [DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

Hey Jason. Thanks for the comments! Regarding 1: I've been considering something like this for a while now. KIP-122 has a similar issue and I brought up some hacks in that discussion to work around it (http://markmail.org/message/kk4ng74riejidify). While solving this problem would help loosen

Re: [DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

Hey Onur, Sorry for the late reply. Thanks for the well-written KIP! I think the proposal makes sense. The only thing I was wondering is whether the process is a bit complex for most users. You'd probably have no trouble at LI (especially given you're implementing it!), but I'm not so sure about

Re: [DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

Yeah, I agree it is a bit complex to do that approach for a one-time migration. Probably not worth it. Here is another idea to reduce, but not eliminate, the amount of message duplication during migration. I am fine with not doing it. Just want to see the opinion from open source community. The

Re: [DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

Regarding (2) - yes that's a good point. @Onur - I think the KIP should explicitly call this out. It is something that we did consider and decided against optimizing for. i.e., we just wrote that off as a minor caveat of the upgrade path in that there will be a few duplicates, but not too many

Re: [DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

Regarding 1: We won't lose the offset from zookeeper upon partition transfer from OZKCC/MDZKCC to MEZKCC because MEZKCC has "dual.commit.enabled" set to true as well as "offsets.storage" set to kafka. The combination of these configs results in the consumer fetching offsets from both kafka and

Re: [DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

Hey Onur, Thanks for the well-written KIP! I have two questions below. 1) In the process of migrating from OZKCCs and MDZKCCs to MEZKCCs, we will may a mix of OZKCCs, MDZKCCs and MEZKCCs. OZKCC and MDZKCC will only commit to zookeeper and MDZKCC will use kafka-based offset storage. Would we lose

[DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

Hey everyone. I made a KIP that provides a mechanism for migrating from ZookeeperConsumerConnector to KafkaConsumer as well as a mechanism for rolling back from KafkaConsumer to ZookeeperConsumerConnector: