[
https://issues.apache.org/jira/browse/SPARK-31451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084790#comment-17084790
]
Gabor Somogyi commented on SPARK-31451:
---------------------------------------
In DStreams offset commit is best effort and Spark is not stating the opposite.
Additionally in the next micro-batch the offset commit will be retried (of
course the increased one). Waiting on the result and doing retry would have
negative effect of the overall processing time.
> Kafka connector does not retry in case of RetriableException
> ------------------------------------------------------------
>
> Key: SPARK-31451
> URL: https://issues.apache.org/jira/browse/SPARK-31451
> Project: Spark
> Issue Type: Bug
> Components: DStreams
> Affects Versions: 2.4.5
> Reporter: Chaoran Yu
> Priority: Major
>
> The Spark DStream API uses spark-streaming-kafka-0-10 to talk to Kafka. The
> method in the connector code that's responsible for committing offset,
> commitAll, calls commitAsync in Kafka client to commit the offsets.
> commitAsync tries to find the group coordinator and sends the commits in case
> of success, or throws a RetriableCommitFailedException in case of failure and
> doesn't retry. This behavior was introduced in KAFKA-4034. The reason for not
> attempting retry was written there as: "we don't want recursive retries which
> can cause offset commits to arrive out of order". From the Spark side though,
> we should be able to retry when running into a RetriableException. The issue
> of potentially committing offsets out of order can be addressed by keeping a
> monotonically increasing sequence number every time a commit happens and
> including this number in the callback function of commitAsync.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
