Re: KIP-122: Add a tool to Reset Consumer Group Offsets

; wrote: > >> \cc from dev >> >> >> ---- Forwarded Message ---- >> Subject: Re: KIP-122: Add a tool to Reset Consumer Group Offsets >> Date: Thu, 23 Feb 2017 10:13:39 -0800 >> From: Matthias J. Sax <matth...@confluent.io> >> Organizatio

Re: Re: KIP-122: Add a tool to Reset Consumer Group Offsets

+1 for such a tool. It would be of great help in a lot of use cases. On Thu, Feb 23, 2017 at 11:44 PM, Matthias J. Sax <matth...@confluent.io> wrote: > \cc from dev > > > Forwarded Message ---- > Subject: Re: KIP-122: Add a tool to Reset Consumer Group Offset

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

@Matthias about the point 9: What about keeping only the --topic option, and support this format: `--topic t1:0,1,2 --topic t2 --topic t3:2` In this case topics t1, t2, and t3 will be selected: topic t1 with partitions 0,1 and 2; topic t2 with all its partitions; and topic t3, with only

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Thanks for the feedback Matthias. * 1. You're right. I'll reorder the scenarios. * 2. Agree. I'll update the KIP. * 3. I like it, updating to `reset-offsets` * 4. Agree, removing the `reset-` part * 5. Yes, 1.e option without --execute or --export will print out current offset, and the new

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Hi, thanks for updating the KIP. Couple of follow up comments: * Nit: Why is "Reset to Earliest" and "Reset to Latest" a "reset by time" option -- IMHO it belongs to "reset by position"? * Nit: Description of "Reset to Earliest" > using Kafka Consumer's `auto.offset.reset` to `earliest` I

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Hi, according to the feedback, I've updated the KIP: - We have added and ordered the scenarios, scopes and executions of the Reset Offset tool. - Consider it as an extension to the current `ConsumerGroupCommand` tool - Execution will be possible without generating JSON files.

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Yeah, that's a good point. Some of the operations might make sense on multiple partitions at once. Moving to a timestamp might apply to all partitions, moving backwards and forwards by N offsets might apply to all partitions. However, moving to a specific offset ("set to offset 43") would most

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

@gmail.com> To: d...@kafka.apache.org, Users <users@kafka.apache.org> Date: 02/08/2017 02:23 PM Subject: Re: KIP-122: Add a tool to Reset Consumer Group Offsets Great. I think I got the idea. What about this options: Scenarios: 1. Current status ´kafka-consumer-groups.sh -

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Just to clarify, we'll need to allow specifying topic and partition. I don't think we want this on ALL partitions at once. On Wed, Feb 8, 2017 at 3:35 PM, Gwen Shapira wrote: > That's what I'd like to see. For example, suppose a Connect task fails > because it can't

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

That's what I'd like to see. For example, suppose a Connect task fails because it can't deserialize an event from a partition. Stop connector, move offset forward, start connector. Boom! On Wed, Feb 8, 2017 at 3:22 PM, Matthias J. Sax wrote: > I am not sure about

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

I am not sure about --reset-plus and --reset-minus Would this skip n messages forward/backward for each partitions? -Matthias On 2/8/17 2:23 PM, Jorge Esteban Quilcate Otoya wrote: > Great. I think I got the idea. What about this options: > > Scenarios: > > 1. Current status > >

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Great. I think I got the idea. What about this options: Scenarios: 1. Current status ´kafka-consumer-groups.sh --reset-offset --group cg1´ 2. To Datetime ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-datetime 2017-01-01T00:00:00.000´ 3. To Period ´kafka-consumer-groups.sh

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Hi, Just my few thoughts: does it need to be json? the old zkOffset tool had a nice format, very easy to manipulate on cli very powerfull: changes as many consumergroups/topics/partitions in one go as you want maybe allow -1 and -2 to indicate earliest and latest reset regardless of what

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Yes - using a tool like this to skip a set of consumer groups over a corrupt/bad message is definitely appealing. B On Wed, Feb 8, 2017 at 9:37 PM Gwen Shapira wrote: > I like the --reset-to-earliest and --reset-to-latest. In general, > since the JSON route is the most

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

I like the --reset-to-earliest and --reset-to-latest. In general, since the JSON route is the most challenging for users, we want to provide a lot of ways to do useful things without going there. Two things that can help: 1. A lot of times, users want to skip few messages that cause issues and

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Thanks for the feedback! @Onur, @Gwen: Agree. Actually at the first draft I considered to have it inside ´kafka-consumer-groups.sh´, but I decide to propose it as a standalone tool to describe it clearly and focus it on reset functionality. But now that you mentioned, it does make sense to have

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

As long as the CLI is a bit consistent? Like, not just adding 3 arguments and a JSON parser to the existing tool, right? On Tue, Feb 7, 2017 at 10:29 PM, Onur Karaman wrote: > I think it makes sense to just add the feature to kafka-consumer-groups.sh > > On Tue, Feb

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

I think it makes sense to just add the feature to kafka-consumer-groups.sh On Tue, Feb 7, 2017 at 10:24 PM, Gwen Shapira wrote: > Thanks for the KIP. I'm super happy about adding the capability. > > I hate the interface, though. It looks exactly like the replica > assignment

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Thanks for the KIP. I'm super happy about adding the capability. I hate the interface, though. It looks exactly like the replica assignment tool. A tool everyone loves so much that there are multiple projects, open and closed, that try to fix it. Can we swap it with something that looks a bit

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Hey Jorge, Thanks for the KIP. I have some quick comments: - Should we allow user to use wildcard to reset offset of all groups for a given topic as well? - Should we allow user to specify timestamp per topic partition in the json file as well? - Should the script take some credential file to

KIP-122: Add a tool to Reset Consumer Group Offsets

Hi all, I would like to propose a KIP to Add a tool to Reset Consumer Group Offsets. https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+a+tool+to+Reset+Consumer+Group+Offsets Please, take a look at the proposal and share your feedback. Thanks, Jorge.