[
https://issues.apache.org/jira/browse/STORM-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Mingmin updated STORM-2340:
------------------------------
Description:
What's the issue?
When Storm topology is run on 'At-Most-Once' mode, with ack executor=0,
KafkaSpout cannot emit records after some time.
What's the cause?
Function poll() controls whether it need to pull more data from Kafka cluster.
With the condition !waitingToEmit() && numUncommittedOffsets <
maxUncommittedOffsets, it's always FALSE after reaching threshold, as
numUncommittedOffsets is creasing and never reset on 'At-Most-Once' mode.
What's the solution?
I change the condition to !waitingToEmit() && (numUncommittedOffsets <
maxUncommittedOffsets || consumerAutoCommitMode), that return TRUE with
AutoCommitMode regarding of UnCommittedOffset, see the comments inline.
Also, it's not required to track emitted(msgId) and numUncommittedOffsets with
AutoCommitMode .
was:
What's the issue?
When Storm topology is run on 'At-Most-Once' mode, with ack executor=0,
KafkaSpout cannot emit records after some time.
What's the cause?
Function poll() controls whether it need to pull more data from Kafka cluster.
With the condition !waitingToEmit() && numUncommittedOffsets <
maxUncommittedOffsets, it's always FALSE after reaching threshold, as
numUncommittedOffsets is creasing and never reset on 'At-Most-Once' mode.
What's the solution?
I change the condition to !waitingToEmit() && (numUncommittedOffsets <
maxUncommittedOffsets || consumerAutoCommitMode), that return TRUE with
AutoCommitMode regarding of UnCommittedOffset, see the comments inline.
Also, it's not required to track emitted(msgId) and numUncommittedOffsets with
AutoCommitMode .
Summary: fix At-Most-Once issue in KafkaSpout (was: KafkaSpout is
blocked in AutoCommitMode)
> fix At-Most-Once issue in KafkaSpout
> ------------------------------------
>
> Key: STORM-2340
> URL: https://issues.apache.org/jira/browse/STORM-2340
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-kafka-client
> Reporter: Xu Mingmin
> Time Spent: 10m
> Remaining Estimate: 0h
>
> What's the issue?
> When Storm topology is run on 'At-Most-Once' mode, with ack executor=0,
> KafkaSpout cannot emit records after some time.
> What's the cause?
> Function poll() controls whether it need to pull more data from Kafka
> cluster. With the condition !waitingToEmit() && numUncommittedOffsets <
> maxUncommittedOffsets, it's always FALSE after reaching threshold, as
> numUncommittedOffsets is creasing and never reset on 'At-Most-Once' mode.
> What's the solution?
> I change the condition to !waitingToEmit() && (numUncommittedOffsets <
> maxUncommittedOffsets || consumerAutoCommitMode), that return TRUE with
> AutoCommitMode regarding of UnCommittedOffset, see the comments inline.
> Also, it's not required to track emitted(msgId) and numUncommittedOffsets
> with AutoCommitMode .
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
