koeninger commented on issue #21038: [SPARK-22968][DStream] Throw an exception on partition revoking issue URL: https://github.com/apache/spark/pull/21038#issuecomment-541793444 Don't start another copy of the application with the same group ID. Spark is already giving as much parallelism as possible, by having consumers on the workers. Have you read or watched the information linked from https://github.com/koeninger/kafka-exactly-once On Mon, Oct 14, 2019, 8:54 AM Anand Changediya <[email protected]> wrote: > Hey @koeninger <https://github.com/koeninger> thanks for the reply > The issue with my application is I have one topic with 3 partitions once I > start my application (Spark consumer) it listens to all the 3 partitions > LOG found as below > Setting newly assigned partitions [topic.partition-2, topic.partition-1, > topic.partition-0] > > When I start another instance of the same application with the same group > id I can see there is rebalance in spark and one partition is assigned to > the second application instance > > LOG as follows in the first application instance > > Setting newly assigned partitions [topic.partition-2, topic.partition-0] > > So we can see the topic.partition 1 is assigned to the second instance of > an application in the rebalancing process > But just after the above-mentioned log there is an exception as follows > > java.lang.IllegalStateException: Previously tracked partitions > [topic.partition-1] been revoked by Kafka because of consumer rebalance. > This is mostly due to another stream with same group id joined, please > check if there're different streaming application misconfigure to use the > same group id. Fundamentally different stream should use different group id > > And the application exits. > How can I have multiple consumers with same groupId for different > partitions? > I also provided Assingmnt strategy as RoundRobin > kafkaParam.put("partition.assignment.strategy", > "org.apache.kafka.clients.consumer.RoundRobinAssignor"); > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/spark/pull/21038?email_source=notifications&email_token=AAAYAB6VDNLNWPUDCJS2QBLQOR22ZA5CNFSM4EZ57NIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBEYRHQ#issuecomment-541690014>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAAYABZOMPDVNT6RKTH4UMLQOR22ZANCNFSM4EZ57NIA> > . >
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
