[
https://issues.apache.org/jira/browse/SPARK-31451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085110#comment-17085110
]
Chaoran Yu commented on SPARK-31451:
------------------------------------
[~gsomogyi] Thanks for the reply. The issue is that Spark will simply fail when
running into a RetriableException and won't recover:
shaded.org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset
commit failed with a retriable exception. You should retry committing offsets.
Caused by:
shaded.org.apache.kafka.common.errors.GroupCoordinatorNotAvailableException:
The group coordinator is not available.
> Kafka connector does not retry in case of RetriableException
> ------------------------------------------------------------
>
> Key: SPARK-31451
> URL: https://issues.apache.org/jira/browse/SPARK-31451
> Project: Spark
> Issue Type: Bug
> Components: DStreams
> Affects Versions: 2.4.5
> Reporter: Chaoran Yu
> Priority: Major
>
> The Spark DStream API uses spark-streaming-kafka-0-10 to talk to Kafka. The
> method in the connector code that's responsible for committing offset,
> commitAll, calls commitAsync in Kafka client to commit the offsets.
> commitAsync tries to find the group coordinator and sends the commits in case
> of success, or throws a RetriableCommitFailedException in case of failure and
> doesn't retry. This behavior was introduced in KAFKA-4034. The reason for not
> attempting retry was written there as: "we don't want recursive retries which
> can cause offset commits to arrive out of order". From the Spark side though,
> we should be able to retry when running into a RetriableException. The issue
> of potentially committing offsets out of order can be addressed by keeping a
> monotonically increasing sequence number every time a commit happens and
> including this number in the callback function of commitAsync.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
