Raman Gupta created KAFKA-8650:
----------------------------------
Summary: Streams does not work as expected with
auto.offset.reset=none
Key: KAFKA-8650
URL: https://issues.apache.org/jira/browse/KAFKA-8650
Project: Kafka
Issue Type: Improvement
Components: streams
Affects Versions: 2.3.0
Reporter: Raman Gupta
The auto.offset.reset policy of none is useful as a safety measure, especially
when
* exactly-once processing is desired, or
* at-least-once is desired, but it is expensive to reprocess from the beginning.
In this case, using "none" forces the ops team to explicitly set the offset
before the stream can re-start processing, in the (hopefully rare) situations
in which the stream consumer offset has been lost for some reason, or in the
case of a new stream that should not start processing from the beginning or the
end, but somewhere in the middle (this scenario might occur during topic
migrations).
Kafka streams really only supports auto.offset.reset of earliest or latest (see
the `Topology.AutoOffsetReset` enum). It is also possible to use the
auto.offset.reset configuration value, but this works suboptimally because if
the streams application reset tool is used (even with a specific offset
specified), the offset is set for the input topic, but it is not, and cannot
be, set for the internal topics, which won't exist yet.
The internal topics are created by Kafka streams at startup time, but because
the auto.offset.reset policy of "none" is passed to the consumer of those
internal topics, the Kafka stream fails to start with a
"NoOffsetForPartitionException".
Proposals / options:
1) Allow auto.offset.reset=none to be specified in Consumed.with() so that it
affects the input topics, but not the internal topics.
2) Allow streams to be configured with auto.offset.reset=none, but explicitly
set the offset to 0 for newly created internal topics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
