[
https://issues.apache.org/jira/browse/KAFKA-8650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885707#comment-16885707
]
Matthias J. Sax commented on KAFKA-8650:
----------------------------------------
Splitting out initialization is not easy. The problem is that one might start
multiple instances at once, but only one instance should do the initialization.
Furthermore, processing can only start after the initialization is completed.
Atm, we rely on consumer rebalance protocol to ensure this property: the group
leader will do the initialization. If we separate both, there are two problems
if you start multiple instances at the same time: (1) which instance should do
the initialization. (2) How do other instances know that the initialization is
finished and that they can start processing?
Maybe, it would be possible to enhance the reset-tool though: it might be
possible to hand in the `StreamsBuilder` and the reset-tool could setup the
topics accordingly (may open question raises though; it's just a rough idea).
> Streams does not work as expected with auto.offset.reset=none
> -------------------------------------------------------------
>
> Key: KAFKA-8650
> URL: https://issues.apache.org/jira/browse/KAFKA-8650
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 2.3.0
> Reporter: Raman Gupta
> Priority: Major
> Labels: needs-kip
>
> The auto.offset.reset policy of none is useful as a safety measure,
> especially when
> * exactly-once processing is desired, or
> * at-least-once is desired, but it is expensive to reprocess from the
> beginning.
> In this case, using "none" forces the ops team to explicitly set the offset
> before the stream can re-start processing, in the (hopefully rare) situations
> in which the stream consumer offset has been lost for some reason, or in the
> case of a new stream that should not start processing from the beginning or
> the end, but somewhere in the middle (this scenario might occur during topic
> migrations).
> Kafka streams really only supports auto.offset.reset of earliest or latest
> (see the `Topology.AutoOffsetReset` enum). It is also possible to use the
> auto.offset.reset configuration value, but this works suboptimally because if
> the streams application reset tool is used (even with a specific offset
> specified), the offset is set for the input topic, but it is not, and cannot
> be, set for the internal topics, which won't exist yet.
> The internal topics are created by Kafka streams at startup time, but because
> the auto.offset.reset policy of "none" is passed to the consumer of those
> internal topics, the Kafka stream fails to start with a
> "NoOffsetForPartitionException".
> Proposals / options:
> 1) Allow auto.offset.reset=none to be specified in Consumed.with() so that it
> affects the input topics, but not the internal topics.
> 2) Allow streams to be configured with auto.offset.reset=none, but explicitly
> set the offset to 0 for newly created internal topics.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
