[jira] [Commented] (KAFKA-4114) Allow for different "auto.offset.reset" strategies for different input streams
[
https://issues.apache.org/jira/browse/KAFKA-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15820146#comment-15820146
]
ASF GitHub Bot commented on KAFKA-4114:
---
Github user asfgit closed the pull request at:
https://github.com/apache/kafka/pull/2007
> Allow for different "auto.offset.reset" strategies for different input streams
> --
>
> Key: KAFKA-4114
> URL: https://issues.apache.org/jira/browse/KAFKA-4114
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Matthias J. Sax
>Assignee: Bill Bejeck
> Fix For: 0.10.2.0
>
>
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> However, it would be useful to improve this settings to allow users have
> finer control over different input stream. For example, with two input
> streams, one of them always reading from offset 0 upon (re)-starting, and the
> other reading for log end offset.
> This JIRA requires to extend {{KStreamBuilder}} API for methods
> {{.stream(...)}} and {{.table(...)}} to add a new parameter that indicate the
> initial offset to be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4114) Allow for different "auto.offset.reset" strategies for different input streams
[
https://issues.apache.org/jira/browse/KAFKA-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15564050#comment-15564050
]
Bill Bejeck commented on KAFKA-4114:
Fair enough doing so now.
> Allow for different "auto.offset.reset" strategies for different input streams
> --
>
> Key: KAFKA-4114
> URL: https://issues.apache.org/jira/browse/KAFKA-4114
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Matthias J. Sax
>Assignee: Bill Bejeck
>
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> However, it would be useful to improve this settings to allow users have
> finer control over different input stream. For example, with two input
> streams, one of them always reading from offset 0 upon (re)-starting, and the
> other reading for log end offset.
> This JIRA requires to extend {{KStreamBuilder}} API for methods
> {{.stream(...)}} and {{.table(...)}} to add a new parameter that indicate the
> initial offset to be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4114) Allow for different "auto.offset.reset" strategies for different input streams
[
https://issues.apache.org/jira/browse/KAFKA-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563941#comment-15563941
]
Matthias J. Sax commented on KAFKA-4114:
Just open an PR whenever you are ready. :)
> Allow for different "auto.offset.reset" strategies for different input streams
> --
>
> Key: KAFKA-4114
> URL: https://issues.apache.org/jira/browse/KAFKA-4114
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Matthias J. Sax
>Assignee: Bill Bejeck
>
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> However, it would be useful to improve this settings to allow users have
> finer control over different input stream. For example, with two input
> streams, one of them always reading from offset 0 upon (re)-starting, and the
> other reading for log end offset.
> This JIRA requires to extend {{KStreamBuilder}} API for methods
> {{.stream(...)}} and {{.table(...)}} to add a new parameter that indicate the
> initial offset to be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4114) Allow for different "auto.offset.reset" strategies for different input streams
[
https://issues.apache.org/jira/browse/KAFKA-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563934#comment-15563934
]
Bill Bejeck commented on KAFKA-4114:
I have a working solution, just waiting for resolution to KAFKA-4269 so I can
merge those changes into the PR for this Jira ticket.
> Allow for different "auto.offset.reset" strategies for different input streams
> --
>
> Key: KAFKA-4114
> URL: https://issues.apache.org/jira/browse/KAFKA-4114
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Matthias J. Sax
>Assignee: Bill Bejeck
>
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> However, it would be useful to improve this settings to allow users have
> finer control over different input stream. For example, with two input
> streams, one of them always reading from offset 0 upon (re)-starting, and the
> other reading for log end offset.
> This JIRA requires to extend {{KStreamBuilder}} API for methods
> {{.stream(...)}} and {{.table(...)}} to add a new parameter that indicate the
> initial offset to be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4114) Allow for different "auto.offset.reset" strategies for different input streams
[
https://issues.apache.org/jira/browse/KAFKA-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531701#comment-15531701
]
Matthias J. Sax commented on KAFKA-4114:
No problem. Just shoot your questions. :)
> Allow for different "auto.offset.reset" strategies for different input streams
> --
>
> Key: KAFKA-4114
> URL: https://issues.apache.org/jira/browse/KAFKA-4114
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Matthias J. Sax
>Assignee: Bill Bejeck
>
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> However, it would be useful to improve this settings to allow users have
> finer control over different input stream. For example, with two input
> streams, one of them always reading from offset 0 upon (re)-starting, and the
> other reading for log end offset.
> This JIRA requires to extend {{KStreamBuilder}} API for methods
> {{.stream(...)}} and {{.table(...)}} to add a new parameter that indicate the
> initial offset to be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4114) Allow for different "auto.offset.reset" strategies for different input streams
[
https://issues.apache.org/jira/browse/KAFKA-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531675#comment-15531675
]
Bill Bejeck commented on KAFKA-4114:
No worries on the response.
Thanks for clearing things up. I knew the first case was covered by the
StreamsConfigs settings but my confusion stemmed from when I took a look at the
{{Consumer}} interface, I completely missed the {{seekToEnd()}} and
{{seekToBegining()}} methods taking a {{Collection}}
parameter, thus enabling the 'fine grained' control per topic. I think this is
precisely the approach to take vs multiple consumers.
>From now on I think I will institute a self-imposed 24 hour hold on all
>questions.
Thanks for the comments.
> Allow for different "auto.offset.reset" strategies for different input streams
> --
>
> Key: KAFKA-4114
> URL: https://issues.apache.org/jira/browse/KAFKA-4114
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Matthias J. Sax
>Assignee: Bill Bejeck
>
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> However, it would be useful to improve this settings to allow users have
> finer control over different input stream. For example, with two input
> streams, one of them always reading from offset 0 upon (re)-starting, and the
> other reading for log end offset.
> This JIRA requires to extend {{KStreamBuilder}} API for methods
> {{.stream(...)}} and {{.table(...)}} to add a new parameter that indicate the
> initial offset to be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4114) Allow for different "auto.offset.reset" strategies for different input streams
[
https://issues.apache.org/jira/browse/KAFKA-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531570#comment-15531570
]
Matthias J. Sax commented on KAFKA-4114:
Sorry for the late reply. Case #1 is already supported right now. Because there
is a single Consumer, it's reset policy is configured via eg
{{StreamConfig#put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");}} which
applies for all stream/table that are consumed. This JIRA should enable case
#2. However, I think we do not need multiple consumers. I would rather set
default value for this consumer config as {{"none"}} and use
{{seekToBeginning()}} and {{seekToEnd()}} according to the provided reset
strategy per stream/table to the manually reset the offset instead of relying
on consumer's internal reset. WDYT? Maybe [~guozhang] also want to comment?
> Allow for different "auto.offset.reset" strategies for different input streams
> --
>
> Key: KAFKA-4114
> URL: https://issues.apache.org/jira/browse/KAFKA-4114
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Matthias J. Sax
>Assignee: Bill Bejeck
>
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> However, it would be useful to improve this settings to allow users have
> finer control over different input stream. For example, with two input
> streams, one of them always reading from offset 0 upon (re)-starting, and the
> other reading for log end offset.
> This JIRA requires to extend {{KStreamBuilder}} API for methods
> {{.stream(...)}} and {{.table(...)}} to add a new parameter that indicate the
> initial offset to be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4114) Allow for different "auto.offset.reset" strategies for different input streams
[
https://issues.apache.org/jira/browse/KAFKA-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513498#comment-15513498
]
Bill Bejeck commented on KAFKA-4114:
Hi Matthias,
I've started working on this and wanted to confirm my understanding.
The addtional parameter for auto-reset will need to be the same for all calls
to {{KStreamBuilder.stream}} and {{KStreamBuilder.table}} for a given single
instance of {{KStreamBuilder}}, but can differ for separate instances.
For example given two {{KStreamBuilder}} instances {{kb1}} and {{kb2}} you
could do the following:
{{kb1.stream("foo", "earliest")}}
{{kb1.stream("bar", "earliest")}}
{{kb2.stream("baz", "latest")}}
{{kb2.table("bar", "latest")}}
but not this:
{{kb1.stream("foo", "latest")}}
{{kb1.stream("bar", "earliest")}}
as this second case above would reguire multiple consumers in the underlying
{{StreamThread}} of the {{KafkaStreams}} instance obtained from using {{kb1}}
as a construction parameter
Is this correct?
> Allow for different "auto.offset.reset" strategies for different input streams
> --
>
> Key: KAFKA-4114
> URL: https://issues.apache.org/jira/browse/KAFKA-4114
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Matthias J. Sax
>Assignee: Bill Bejeck
>
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> However, it would be useful to improve this settings to allow users have
> finer control over different input stream. For example, with two input
> streams, one of them always reading from offset 0 upon (re)-starting, and the
> other reading for log end offset.
> This JIRA requires to extend {{KStreamBuilder}} API for methods
> {{.stream(...)}} and {{.table(...)}} to add a new parameter that indicate the
> initial offset to be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
