[
https://issues.apache.org/jira/browse/FLINK-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15657244#comment-15657244
]
Juho Autio commented on FLINK-3123:
-----------------------------------
Nice one, looking forward to this!
{quote}
If a subscribed partition is not defined a specific offset (does not have a
corresponding entry in the specificStartOffsets map), then the startup
behaviour for that particular partition fallbacks to the default group offset
behaviour (look for offset in ZK / Kafka for that partition, or use
"auto.offset.reset" if none can be found).
{quote}
My use case is skipping to the latest offsets while keeping the same consumer
group id. What happens if I define all topics & partitions with offset -1?
Because that's the special value for fetching the latest offset (and -2 is
earliest). I wonder if there's some code that translates -1 to 0 or -2, which
would be a problem.
Even better in my case would be that I can just say:
consumer.setStartFromOffset(-1) which would automatically discover all
partitions for all topics and initialize the offsets by fetching from Kafka
with "current offset" -1. It would be nice if this reset the offsets even when
a Flink snapshot state exists with some offset positions (I suppose that's how
your setStartFromSpecificOffsets works any way).
> Allow setting custom start-offsets for the Kafka consumer
> ---------------------------------------------------------
>
> Key: FLINK-3123
> URL: https://issues.apache.org/jira/browse/FLINK-3123
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
> Reporter: Robert Metzger
> Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.0.0, 1.2.0
>
>
> Currently, the Kafka consumer only allows to start reading from the earliest
> available offset or the current offset.
> Sometimes, users want to set a specific start offset themselves.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
