anandchangediya commented on issue #21038: [SPARK-22968][DStream] Throw an exception on partition revoking issue URL: https://github.com/apache/spark/pull/21038#issuecomment-541690014 Hey @koeninger thanks for the reply The issue with my application is I have one topic with 3 partitions once I start my application (Spark consumer) it listens to all the 3 partitions LOG found as below `Setting newly assigned partitions [topic.partition-2, topic.partition-1, topic.partition-0]` When I start another instance of the same application with the same group id I can see there is rebalance in spark and one partition is assigned to the second application instance LOG as follows in the first application instance `Setting newly assigned partitions [topic.partition-2, topic.partition-0]` So we can see the topic.partition 1 is assigned to the second instance of an application in the rebalancing process But just after the above-mentioned log there is an exception as follows `java.lang.IllegalStateException: Previously tracked partitions [topic.partition-1] been revoked by Kafka because of consumer rebalance. This is mostly due to another stream with same group id joined, please check if there're different streaming application misconfigure to use the same group id. Fundamentally different stream should use different group id` And the application exits. How can I have multiple consumers with same groupId for different partitions? I also provided Assingmnt strategy as RoundRobin `kafkaParam.put("partition.assignment.strategy", "org.apache.kafka.clients.consumer.RoundRobinAssignor");`
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
