[
https://issues.apache.org/jira/browse/KAFKA-8650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885640#comment-16885640
]
Raman Gupta commented on KAFKA-8650:
------------------------------------
There is also a related API change / improvement that I believe should be in
the KIP if one is created.
Currently, a new (or reset) stream initializes itself when "start" is called
i.e. "start" is overloaded to mean "initialize and start".
If "initialize" was possible without start, then it would be easy to call
"initialize" to create the stream's topology, then use the application reset
tool to set the desired offsets, then use "start" to actually start the stream
at the desired place. This would allow the user to carry out this process
without actually relying on the system to throw an exception at stream startup
time, post-init without an offset.
This can also be done in advance of allowing auto.offset.reset = none for
streams, which, when done, would just become a safety on the stream.start
trigger.
> Streams does not work as expected with auto.offset.reset=none
> -------------------------------------------------------------
>
> Key: KAFKA-8650
> URL: https://issues.apache.org/jira/browse/KAFKA-8650
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 2.3.0
> Reporter: Raman Gupta
> Priority: Major
> Labels: needs-kip
>
> The auto.offset.reset policy of none is useful as a safety measure,
> especially when
> * exactly-once processing is desired, or
> * at-least-once is desired, but it is expensive to reprocess from the
> beginning.
> In this case, using "none" forces the ops team to explicitly set the offset
> before the stream can re-start processing, in the (hopefully rare) situations
> in which the stream consumer offset has been lost for some reason, or in the
> case of a new stream that should not start processing from the beginning or
> the end, but somewhere in the middle (this scenario might occur during topic
> migrations).
> Kafka streams really only supports auto.offset.reset of earliest or latest
> (see the `Topology.AutoOffsetReset` enum). It is also possible to use the
> auto.offset.reset configuration value, but this works suboptimally because if
> the streams application reset tool is used (even with a specific offset
> specified), the offset is set for the input topic, but it is not, and cannot
> be, set for the internal topics, which won't exist yet.
> The internal topics are created by Kafka streams at startup time, but because
> the auto.offset.reset policy of "none" is passed to the consumer of those
> internal topics, the Kafka stream fails to start with a
> "NoOffsetForPartitionException".
> Proposals / options:
> 1) Allow auto.offset.reset=none to be specified in Consumed.with() so that it
> affects the input topics, but not the internal topics.
> 2) Allow streams to be configured with auto.offset.reset=none, but explicitly
> set the offset to 0 for newly created internal topics.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
