[
https://issues.apache.org/jira/browse/KAFKA-7506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650532#comment-16650532
]
Matthias J. Sax commented on KAFKA-7506:
----------------------------------------
[~niklas.lonn] If you want a back-port, pleas comment on the other ticket.
However, it's not a bug fix (the other ticket is marked as "improvement"), but
a default config change and thus should not be part of a bug fix release IMHO.
Note that you don't really need the bug fix though. You can reconfigure the
repartition topics manually after Streams created them via `bin/kafka-topic.sh`
command. As an alternative, you can also pass in topic-level configs that
overwrite default config via StreamsConfig (note, that those configs apply to
all internal topic; thus, you need to double check if you can use this or not):
https://cwiki.apache.org/confluence/display/KAFKA/KIP-173%3A+Add+prefix+to+StreamsConfig+to+enable+setting+default+internal+topic+configs
> KafkaStreams repartition topic settings not suitable for processing old
> records
> -------------------------------------------------------------------------------
>
> Key: KAFKA-7506
> URL: https://issues.apache.org/jira/browse/KAFKA-7506
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 1.1.0, 1.1.1, 2.0.0
> Reporter: Niklas Lönn
> Priority: Major
> Attachments: kafka-7506.patch
>
>
> Hi, We are using Kafka Streams to process a compacted store, when resetting
> the application/processing from scratch the default topic configuration for
> repartition topics is 50MB and 10min segment sizes.
>
> As the retention.ms is undefined, this leads to default retention.ms and log
> cleaner starts competing with the application, effectively causing the
> streams app to skip records.
> {{Application logs the following:}}
> {{Fetch offset 213792 is out of range for partition
> app-id-KTABLE-AGGREGATE-STATE-STORE-0000000015-repartition-7, resetting
> offset}}
> {{Fetch offset 110227 is out of range for partition
> app-id-KTABLE-AGGREGATE-STATE-STORE-0000000015-repartition-2, resetting
> offset}}
> {{Resetting offset for partition
> app-id-KTABLE-AGGREGATE-STATE-STORE-0000000015-repartition-7 to offset
> 233302.}}
> {{Resetting offset for partition
> app-id-KTABLE-AGGREGATE-STATE-STORE-0000000015-repartition-2 to offset
> 119914.}}
> By adding the following configuration to RepartitionTopicConfig.java the
> issue is solved
> {{tempTopicDefaultOverrides.put(TopicConfig.RETENTION_MS_CONFIG, "-1"); //
> Infinite}}
>
> My understanding is that this should be safe as KafkaStreams uses the admin
> API to delete segments.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
