Re: KafkaConsumer keeps getting InstanceAlreadyExistsException

Hi Rong, The issue here is that the PartitionDiscoverer has an internal KafkaConsumer which reuses the client.id set by the users for the actual fetching KafkaConsumer. Different KafkaConsumers distinguish their metrics by client.id, therefore if there are two KafkaConsumers in the same JVM with

Re: KafkaConsumer keeps getting InstanceAlreadyExistsException

Hi Becket/Till, Thanks for the detail explanation. Just to confirm: the issue in FLINK-8093 refers to multiple Kafka consumer within the same TM - thus the fix should be to make consumer client.id unique for different tasks ? and the issue here is an issue internal to the Kafka consumer, where

Re: KafkaConsumer keeps getting InstanceAlreadyExistsException

Actually it might be better to create another ticket, FLINK-8093 was mainly complaining about the JMX bean collision when there are multiple tasks running in the same TM. Jiangjie (Becket) Qin On Tue, Mar 17, 2020 at 6:33 PM Becket Qin wrote: > Hi Till, > > It looks FLINK-8093

Re: KafkaConsumer keeps getting InstanceAlreadyExistsException

Hi Till, It looks FLINK-8093 reports the same issue, although the reported information is not exactly correct, as this should not cause the producer to fail. I'll take care of the ticket. Thanks, Jiangjie (Becket) Qin On Tue, Mar 17, 2020 at

Re: KafkaConsumer keeps getting InstanceAlreadyExistsException

@Becket do we already have a JIRA ticket to track this effort? Cheers, Till On Mon, Mar 16, 2020 at 4:07 AM Becket Qin wrote: > Hi Sidney, > > The WARN logging you saw was from the AbstractPartitionDiscoverer which is > created by FlinkKafkaConsumer itself. It has an internal consumer which >

Re: KafkaConsumer keeps getting InstanceAlreadyExistsException

Hi Sidney, The WARN logging you saw was from the AbstractPartitionDiscoverer which is created by FlinkKafkaConsumer itself. It has an internal consumer which shares the client.id of the actual consumer fetching data. This is a bug that we should fix. As Rong said, this won't affect the normal

Re: KafkaConsumer keeps getting InstanceAlreadyExistsException

We also had seen this issue before running Flink apps in a shared cluster environment. Basically, Kafka is trying to register a JMX MBean[1] for application monitoring. This is only a WARN suggesting that you are registering more than one MBean with the same client id "consumer-1", it should not

KafkaConsumer keeps getting InstanceAlreadyExistsException

Hey, I've been using Flink for a while now without any problems when running apps with a FlinkKafkaConsumer. All my apps have the same overall logic (consume from kafka -> transform event -> write to file) and the only way they differ from each other is the topic they read (remaining kafka