Hello there, I’m just wondering if there are real world use cases for maintaining this default behavior. It’s a bit counter intuitive and sometimes results in serious production issues. ( We had a similar issue when changing the topic name, and resulting reading every message twice - both from the old one and from the new).
Cheers, Feng Le mer. 13 févr. 2019 à 17:56, Tzu-Li (Gordon) Tai <tzuli...@apache.org> a écrit : > Hi, > > Partition offsets stored in state will always be respected when the > consumer is restored from checkpoints / savepoints. > AFAIK, this seems to have been the behaviour for quite some time now (since > FlinkKafkaConsumer08). > > I think in the past there were some discussion to at least allow some way > to ignore restored partition offsets. > One way to enable this is to filter the restored partition offsets based on > the configured list of specified topics / topic regex pattern in the > current execution. This should work, since this can only be modified when > restoring from savepoints (i.e. manual restores). > To avoid breaking the current behaviour, we can maybe add a > `filterRestoredPartitionOffsetState()` configuration on the consumer, which > by default is disabled to match the current behaviour. > > What do you think? > > Cheers, > Gordon > > On Wed, Feb 13, 2019 at 11:59 PM Gyula Fóra <gyula.f...@gmail.com> wrote: > > > Hi! > > > > I have run into a weird issue which I could have sworn that it wouldnt > > happen :D > > I feel there was a discussion about this in the past but maybe im wrong, > > but I hope someone can point me to a ticket. > > > > Lets say you create a kafka consumer that consumes (t1,t2,t3), you take a > > savepoint and deploy a new version that only consumes (t1). > > > > The restore logic now still starts to consume (t1,t2,t3) which feels very > > unintuitive as those were explicitly removed from the list. It is also > hard > > to debug as the topics causing the problem are not defined anywhere in > your > > job, configs etc. > > > > Has anyone run into this issue? Should we change this default behaviour > or > > at least have an option to not do this? > > > > Cheers, > > Gyula > > >
