Damian, > resets everything and clears up >> the state stores.
That's not correct. The reset tool does not touch local store. For this, we have `KafkaStreams#clenup` -- otherwise, you would need to run the tool in every machine you run an application instance. With regard to state, the tool only deletes the underlying changelog topics. Just to clarify. The idea of allowing to rest with different offset is to clear all state but just use a different start offset (instead of zero). This is for use case where your topic has a larger retention time than the amount of data you want to reprocess. But we always need to start with an empty state. (Resetting with consistent state is something we might do at some point, but it's much hard and not part of this KIP) > @matthias, could we remove the ZK dependency from the streams reset tool > now? I think so. The new AdminClient provide the feature we need AFAIK. I guess we can picky back this into the KIP (we would need a KIP anyway because we deprecate `--zookeeper` what is an public API change). I just want to point out, that even without ZK dependency, I prefer to wrap the consumer offset tool and keep two tools. -Matthias On 6/29/17 9:14 AM, Damian Guy wrote: > Hi, > > Thanks for the KIP. What is not clear is how is this going to handle state > stores? Right now the streams reset tool, resets everything and clears up > the state stores. What are we going to do if we reset to a particular > offset? If we clear the state then we've lost any previously aggregated > values (which may or may not be what is expected). If we don't clear them, > then we will end up with incorrect aggregates. > > @matthias, could we remove the ZK dependency from the streams reset tool > now? > > Thanks, > Damian > > On Thu, 29 Jun 2017 at 15:22 Jorge Esteban Quilcate Otoya < > quilcate.jo...@gmail.com> wrote: > >> You're right, I was not considering Zookeeper dependency. >> >> I'm starting to like the idea to wrap `reset-offset` from >> `streams-application-reset`. >> >> I will wait a bit for more feedback and work on a draft with this changes. >> >> >> El mié., 28 jun. 2017 a las 0:20, Matthias J. Sax (<matth...@confluent.io >>> ) >> escribió: >> >>> I agree, that we should not duplicate functionality. >>> >>> However, I am worried, that a non-streams users using the offset reset >>> tool might delete topics unintentionally (even if the changes are pretty >>> small). Also, currently the stream reset tool required Zookeeper while >>> the offset reset tool does not -- I don't think we should add this >>> dependency to the offset reset tool. >>> >>> Thus, it think it might be better to keep both tools, but internally >>> rewrite the streams reset entry class, to reuse as much as possible from >>> the offset reset tool. Ie. the streams tool would be a wrapper around >>> the offset tool and add some functionality it needs that is Streams >>> specific. >>> >>> I also think, that keeping separate tools for consumers and streams is a >>> good thing. We might want to add new features that don't apply to plain >>> consumers -- note, a Streams applications is more than just a client. >>> >>> WDYT? >>> >>> Would be good to get some feedback from others, too. >>> >>> >>> -Matthias >>> >>> >>> On 6/27/17 9:05 AM, Jorge Esteban Quilcate Otoya wrote: >>>> Thanks for the feedback Matthias! >>>> >>>> The main idea is to use only 1 tool to reset offsets and don't >> replicate >>>> functionality between tools. >>>> Reset Offset (KIP-122) tool not only reset but support execute the >> reset >>>> but also export, import from files, etc. >>>> If we extend the current tool (kafka-streams-application-reset.sh) we >>> will >>>> have to duplicate all this functionality also. >>>> Maybe another option is to move the current implementation into >>>> `kafka-consumer-group` and add a new command `--reset-offset-streams` >>> with >>>> the current implementation functionality and add `--reset-offset` >> options >>>> for input topics. Does this make sense? >>>> >>>> >>>> El lun., 26 jun. 2017 a las 23:32, Matthias J. Sax (< >>> matth...@confluent.io>) >>>> escribió: >>>> >>>>> Jorge, >>>>> >>>>> thanks a lot for this KIP. Allowing the reset streams applications >> with >>>>> arbitrary start offset is something we got multiple requests already. >>>>> >>>>> Couple of clarification question: >>>>> >>>>> - why do you want to deprecate the current tool instead of extending >>>>> the current tool with the stuff the offset reset tool can do (ie, use >>>>> the offset reset tool internally) >>>>> >>>>> - you suggest to extend the offset reset tool to replace the stream >>>>> reset tool: how would the reset tool know if it is resetting a streams >>>>> applications or a regular consumer group? >>>>> >>>>> >>>>> >>>>> -Matthias >>>>> >>>>> >>>>> On 6/26/17 1:28 PM, Jorge Esteban Quilcate Otoya wrote: >>>>>> Hi all, >>>>>> >>>>>> I'd like to start the discussion to add reset offset tooling for >> Stream >>>>>> applications. >>>>>> The KIP can be found here: >>>>>> >>>>> >>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-171+-+Extend+Consumer+Group+Reset+Offset+for+Stream+Application >>>>>> >>>>>> Thanks, >>>>>> Jorge. >>>>>> >>>>> >>>>> >>>> >>> >>> >> >
signature.asc
Description: OpenPGP digital signature
