Re: Kafka Consumers Partition Discovery doesn't work

Thanks Gordon, here's the ticket: https://issues.apache.org/jira/browse/FLINK-9334 If you'd like me to have a stab at it, feel free to assign the ticket to me. On Thu, Apr 12, 2018 at 10:28 PM, Tzu-Li (Gordon) Tai wrote: > Hi Juno, > > Thanks for reporting back, glad to

Re: Kafka Consumers Partition Discovery doesn't work

Hi Juno, Thanks for reporting back, glad to know that it's not an issue :) In general, connector specific configurations should always happen at the connector level, per-connector. The flink-conf.yaml file is usually for cluster wide configurations. And yes, it might be helpful to have a code

Re: Kafka Consumers Partition Discovery doesn't work

Ahhh looks like I had simply misunderstood where that property should go. The docs correctly say: > To enable it, set a non-negative value for flink.partition-discovery.interval-millis in the __provided properties config__ So it should be set in the Properties that are passed in the constructor

Re: Kafka Consumers Partition Discovery doesn't work

Still not working after I had a fresh build from https://github.com/apache/flink/tree/release-1.5. When the job starts this is logged: 2018-04-05 09:29:38,157 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property:

Re: Kafka Consumers Partition Discovery doesn't work

Hi, I think you’ve made a good point: there is currently no logs that tell anything about discovering a new partition. We should probably add this. And yes, it would be great if you can report back on this using either the latest master, release-1.5 or release-1.4 branches. On 22 March 2018

Re: Kafka Consumers Partition Discovery doesn't work

Thanks, that sounds promising. I don't know how to check if it's consuming all partitions? For example I couldn't find any logs about discovering a new partition. However, did I understand correctly that this is also fixed in Flink dev? If yes, I could rebuild my 1.5-SNAPSHOT and try again. On

Re: Kafka Consumers Partition Discovery doesn't work

Hi Juho, Can you confirm that the new partition is consumed, but only that Flink’s reported metrics do not include them? If yes, then I think your observations can be explained by this issue:  https://issues.apache.org/jira/browse/FLINK-8419 This issue should have been fixed in the recently

Kafka Consumers Partition Discovery doesn't work

According to the docs*, flink.partition-discovery.interval-millis can be set to enable automatic partition discovery. I'm testing this, apparently it doesn't work. I'm using Flink Version: 1.5-SNAPSHOT Commit: 8395508 and FlinkKafkaConsumer010. I had my flink stream running, consuming an