That's what I'd like to see. For example, suppose a Connect task fails because it can't deserialize an event from a partition. Stop connector, move offset forward, start connector. Boom!
On Wed, Feb 8, 2017 at 3:22 PM, Matthias J. Sax <matth...@confluent.io> wrote: > I am not sure about --reset-plus and --reset-minus > > Would this skip n messages forward/backward for each partitions? > > > -Matthias > > On 2/8/17 2:23 PM, Jorge Esteban Quilcate Otoya wrote: >> Great. I think I got the idea. What about this options: >> >> Scenarios: >> >> 1. Current status >> >> ´kafka-consumer-groups.sh --reset-offset --group cg1´ >> >> 2. To Datetime >> >> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-datetime >> 2017-01-01T00:00:00.000´ >> >> 3. To Period >> >> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-period P2D´ >> >> 4. To Earliest >> >> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-earliest´ >> >> 5. To Latest >> >> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-latest´ >> >> 6. Minus 'n' offsets >> >> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-minus n´ >> >> 7. Plus 'n' offsets >> >> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-plus n´ >> >> 8. To specific offset >> >> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to x´ >> >> Scopes: >> >> a. All topics used by Consumer Group >> >> Don't specify --topics >> >> b. Specific List of Topics >> >> Add list of values in --topics t1,t2,tn >> >> c. One Topic, all Partitions >> >> Add one topic and no partitions values: --topic t1 >> >> d. One Topic, List of Partitions >> >> Add one topic and partitions values: --topic t1 --partitions 0,1,2 >> >> About Reset Plan (JSON file): >> >> I think is still valid to have the option to persist reset configuration as >> a file, but I agree to give the option to run the tool without going down >> to the JSON file. >> >> Execution options: >> >> 1. Without execution argument (No args): >> >> Print out results (reset plan) >> >> 2. With --execute argument: >> >> Run reset process >> >> 3. With --output argument: >> >> Save result in a JSON format. >> >> 4. Only with --execute option and --reset-file (path to JSON) >> >> Reset based on file >> >> 4. Only with --verify option and --reset-file (path to JSON) >> >> Verify file values with current offsets >> >> I think we can remove --generate-and-execute because is a bit clumsy. >> >> With this options we will be able to execute with manual JSON configuration. >> >> >> El mié., 8 feb. 2017 a las 22:43, Ben Stopford (<b...@confluent.io>) >> escribió: >> >>> Yes - using a tool like this to skip a set of consumer groups over a >>> corrupt/bad message is definitely appealing. >>> >>> B >>> >>> On Wed, Feb 8, 2017 at 9:37 PM Gwen Shapira <g...@confluent.io> wrote: >>> >>>> I like the --reset-to-earliest and --reset-to-latest. In general, >>>> since the JSON route is the most challenging for users, we want to >>>> provide a lot of ways to do useful things without going there. >>>> >>>> Two things that can help: >>>> >>>> 1. A lot of times, users want to skip few messages that cause issues >>>> and continue. maybe just specifying the topic, partition and delta >>>> will be better than having to find the offset and write a JSON and >>>> validate the JSON etc. >>>> >>>> 2. Thinking if there are other common use-cases that we can make easy >>>> rather than just one generic but not very usable method. >>>> >>>> Gwen >>>> >>>> On Wed, Feb 8, 2017 at 3:25 AM, Jorge Esteban Quilcate Otoya >>>> <quilcate.jo...@gmail.com> wrote: >>>>> Thanks for the feedback! >>>>> >>>>> @Onur, @Gwen: >>>>> >>>>> Agree. Actually at the first draft I considered to have it inside >>>>> ´kafka-consumer-groups.sh´, but I decide to propose it as a standalone >>>> tool >>>>> to describe it clearly and focus it on reset functionality. >>>>> >>>>> But now that you mentioned, it does make sense to have it in >>>>> ´kafka-consumer-groups.sh´. How would be a consistent way to introduce >>>> it? >>>>> >>>>> Maybe something like this: >>>>> >>>>> ´kafka-consumer-groups.sh --reset-offset --generate --group cg1 >>> --topics >>>> t1 >>>>> --reset-from 2017-01-01T00:00:00.000 --output plan.json´ >>>>> >>>>> ´kafka-consumer-groups.sh --reset-offset --verify --reset-json-file >>>>> plan.json´ >>>>> >>>>> ´kafka-consumer-groups.sh --reset-offset --execute --reset-json-file >>>>> plan.json´ >>>>> >>>>> ´kafka-consumer-groups.sh --reset-offset --generate-and-execute --group >>>> cg1 >>>>> --topics t1 --reset-from 2017-01-01T00:00:00.000´ >>>>> >>>>> @Gwen: >>>>> >>>>>> It looks exactly like the replica assignment tool >>>>> >>>>> It was influenced by ;-) I use the generate-verify-execute process here >>>> to >>>>> make sure user will be aware of the result of this operation. At the >>>>> beginning we considered only add a couple of options to Consumer Group >>>>> Command: >>>>> >>>>> --rewind-to-timestamp and --rewind-to-period >>>>> >>>>> @Onur: >>>>> >>>>>> You can actually get away with overriding while members of the group >>>> are live >>>>> with method 2 by using group information from DescribeGroupsRequest. >>>>> >>>>> This means that we need to have Consumer Group stopped before executing >>>> and >>>>> start a new consumer internally to do this? Therefore, we won't be able >>>> to >>>>> consider executing reset when ConsumerGroup is active? (trying to >>> relate >>>> it >>>>> with @Dong 5th question) >>>>> >>>>> @Dong: >>>>> >>>>>> Should we allow user to use wildcard to reset offset of all groups >>> for a >>>>> given topic as well? >>>>> >>>>> I haven't thought about this scenario. Could be interesting. Following >>>> the >>>>> recommendation to add it into Consumer Group Command, in this case >>> Group >>>>> argument will be optional if there are only 1 topic. I think for >>> multiple >>>>> topic won't be that useful. >>>>> >>>>>> Should we allow user to specify timestamp per topic partition in the >>>> json >>>>> file as well? >>>>> >>>>> Don't think this could be a valid from the tool, but if Reset Plan is >>>>> generated, and user want to set the offset for a specific partition to >>>>> other offset (eventually based on another timestamp), and execute it, >>> it >>>>> will be up to her/him. >>>>> >>>>>> Should the script take some credential file to make sure that this >>>>> operation is authenticated given the potential impact of this >>> operation? >>>>> >>>>> Haven't tried to secure brokers yet, but the tool should support >>>>> authorization if it's enabled in the broker. >>>>> >>>>>> Should we provide constant to reset committed offset to >>> earliest/latest >>>>> offset of a partition, e.g. -1 indicates earliest offset and -2 >>> indicates >>>>> latest offset. >>>>> >>>>> I will go for something like ´--reset-to-earliest´ and >>>> ´--reset-to-latest´ >>>>> >>>>>> Should we allow dynamic change of the comitted offset when consumer >>> are >>>>> running, such that consumer will seek to the newly committed offset and >>>>> start consuming from there? >>>>> >>>>> Not sure about this. I will recommend to keep it simple and ask user to >>>>> stop consumers first. But I would considered it if the trade-offs are >>>>> clear. >>>>> >>>>> @Matthias >>>>> >>>>> Added :). And thanks a lot for your help to define this KIP! >>>>> >>>>> >>>>> >>>>> El mié., 8 feb. 2017 a las 7:47, Gwen Shapira (<g...@confluent.io>) >>>>> escribió: >>>>> >>>>>> As long as the CLI is a bit consistent? Like, not just adding 3 >>>>>> arguments and a JSON parser to the existing tool, right? >>>>>> >>>>>> On Tue, Feb 7, 2017 at 10:29 PM, Onur Karaman >>>>>> <onurkaraman.apa...@gmail.com> wrote: >>>>>>> I think it makes sense to just add the feature to >>>>>> kafka-consumer-groups.sh >>>>>>> >>>>>>> On Tue, Feb 7, 2017 at 10:24 PM, Gwen Shapira <g...@confluent.io> >>>> wrote: >>>>>>> >>>>>>>> Thanks for the KIP. I'm super happy about adding the capability. >>>>>>>> >>>>>>>> I hate the interface, though. It looks exactly like the replica >>>>>>>> assignment tool. A tool everyone loves so much that there are >>>> multiple >>>>>>>> projects, open and closed, that try to fix it. >>>>>>>> >>>>>>>> Can we swap it with something that looks a bit more like the >>> consumer >>>>>>>> group tool? or the kafka streams reset tool? Consistency is helpful >>>> in >>>>>>>> such cases. I spent some time learning existing tools and learning >>>> yet >>>>>>>> another one is a deterrent. >>>>>>>> >>>>>>>> Gwen >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Feb 7, 2017 at 6:43 PM, Jorge Esteban Quilcate Otoya >>>>>>>> <quilcate.jo...@gmail.com> wrote: >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I would like to propose a KIP to Add a tool to Reset Consumer >>> Group >>>>>>>> Offsets. >>>>>>>>> >>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>>>>>>> 122%3A+Add+a+tool+to+Reset+Consumer+Group+Offsets >>>>>>>>> >>>>>>>>> Please, take a look at the proposal and share your feedback. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Jorge. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Gwen Shapira >>>>>>>> Product Manager | Confluent >>>>>>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760> >>> <(650)%20450-2760> | @gwenshap >>>>>>>> Follow us: Twitter | blog >>>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Gwen Shapira >>>>>> Product Manager | Confluent >>>>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760> >>> <(650)%20450-2760> | @gwenshap >>>>>> Follow us: Twitter | blog >>>>>> >>>> >>>> >>>> >>>> -- >>>> Gwen Shapira >>>> Product Manager | Confluent >>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760> | @gwenshap >>>> Follow us: Twitter | blog >>>> >>> >> > -- Gwen Shapira Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter | blog
