[
https://issues.apache.org/jira/browse/SPARK-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224571#comment-14224571
]
Tathagata Das commented on SPARK-2383:
--------------------------------------
This issue has been resolved in Spark 1.2.0. I am marking this JIRA as
duplicate of that SPARK-2492 and closing this.
> With auto.offset.reset, KafkaReceiver potentially deletes Consumer nodes from
> Zookeeper
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-2383
> URL: https://issues.apache.org/jira/browse/SPARK-2383
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Reporter: Tobias Pfeiffer
>
> When auto.offset.reset is set in the Kafka configuration, then
> {{KafkaReceiver}}'s {{tryZookeeperConsumerGroupCleanup()}} will delete the
> whole /consume/<groupId> tree in Zookeeper before creating consumer nodes. If
> there are already consumer nodes present (this may happen when multiple
> KafkaReceivers in the same consumer group are launched), they are deleted as
> well, leading to subsequent NoNode exceptions, for example, on rebalance.
> There should be a check before the delete like {{if (zk.countChildren(dir +
> "/ids") == 0) ...}} (ideally in an atomic way) in order to prevent deleting
> existing consumer nodes.
> (Also note that the behavior of auto.offset.reset as realized by Spark's
> Kafka receiver differs from the behavior defined in Kafka's documentation.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
