Re[2]: Kafka behavior when consuming a topic which doesn't exist?
Please ensure that you set the desired minimum partition and replication settings in the server.properties file. These are the ones which will be used for auto creation of topics. KR, -- Original Message -- From: "Ali Akhtar" To: users@kafka.apache.org Sent: 7/8/2017 4:27:55 PM Subject: Re: Kafka behavior when consuming a topic which doesn't exist? Oh gotcha, thanks. So a topic will be created if topic creation is enabled. On Sat, Jul 8, 2017 at 8:14 PM, M. Manna wrote: Please check my previous email. On Sat, 8 Jul 2017 at 2:32 am, Ali Akhtar wrote: > What happens if auto creation is enabled but the topic doesn't exist. > Consumers subscribe to that topic which doesn't exist. Then messages are > posted to that topic. > > Will the consumer receive those messages in this scenario? > > > > On 8 Jul 2017 4:38 a.m., "M. Manna" wrote: > > That depends. > > If auto creation of non-existent topic enabled (check docs), then it will > simple use the minimum partiotion and replication settings defined in > broker config to create a topic. If auto creation is disabled, your > consumer group won't do anything. > > With auto creation enable - It's the same if you publish to a topic that > doesn't yet exist. But that means topic will get created first, and then > the messages are sent on the bus. If you subscribe to that topic later you > will get messages. Check below: > > http://grokbase.com/t/kafka/users/1648kbr04c/subscribe-on- > a-topic-that-does-not-exist > > > > On 7 Jul 2017 9:46 pm, "Ali Akhtar" wrote: > > > Sometimes I see warnings in my logs if i create a consumer for a topic > > which doesn't exist. Such as: > > > > org.apache.kafka.clients.NetworkClient - Error while fetching metadata > > with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE} > > > > If later messages are posted to that topic (which will create it), will > my > > consumer receive those messages? > > >
Re: Kafka behavior when consuming a topic which doesn't exist?
Oh gotcha, thanks. So a topic will be created if topic creation is enabled. On Sat, Jul 8, 2017 at 8:14 PM, M. Manna wrote: > Please check my previous email. > > On Sat, 8 Jul 2017 at 2:32 am, Ali Akhtar wrote: > > > What happens if auto creation is enabled but the topic doesn't exist. > > Consumers subscribe to that topic which doesn't exist. Then messages are > > posted to that topic. > > > > Will the consumer receive those messages in this scenario? > > > > > > > > On 8 Jul 2017 4:38 a.m., "M. Manna" wrote: > > > > That depends. > > > > If auto creation of non-existent topic enabled (check docs), then it will > > simple use the minimum partiotion and replication settings defined in > > broker config to create a topic. If auto creation is disabled, your > > consumer group won't do anything. > > > > With auto creation enable - It's the same if you publish to a topic that > > doesn't yet exist. But that means topic will get created first, and then > > the messages are sent on the bus. If you subscribe to that topic later > you > > will get messages. Check below: > > > > http://grokbase.com/t/kafka/users/1648kbr04c/subscribe-on- > > a-topic-that-does-not-exist > > > > > > > > On 7 Jul 2017 9:46 pm, "Ali Akhtar" wrote: > > > > > Sometimes I see warnings in my logs if i create a consumer for a topic > > > which doesn't exist. Such as: > > > > > > org.apache.kafka.clients.NetworkClient - Error while fetching > metadata > > > with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE} > > > > > > If later messages are posted to that topic (which will create it), will > > my > > > consumer receive those messages? > > > > > >
Re: Kafka behavior when consuming a topic which doesn't exist?
Please check my previous email. On Sat, 8 Jul 2017 at 2:32 am, Ali Akhtar wrote: > What happens if auto creation is enabled but the topic doesn't exist. > Consumers subscribe to that topic which doesn't exist. Then messages are > posted to that topic. > > Will the consumer receive those messages in this scenario? > > > > On 8 Jul 2017 4:38 a.m., "M. Manna" wrote: > > That depends. > > If auto creation of non-existent topic enabled (check docs), then it will > simple use the minimum partiotion and replication settings defined in > broker config to create a topic. If auto creation is disabled, your > consumer group won't do anything. > > With auto creation enable - It's the same if you publish to a topic that > doesn't yet exist. But that means topic will get created first, and then > the messages are sent on the bus. If you subscribe to that topic later you > will get messages. Check below: > > http://grokbase.com/t/kafka/users/1648kbr04c/subscribe-on- > a-topic-that-does-not-exist > > > > On 7 Jul 2017 9:46 pm, "Ali Akhtar" wrote: > > > Sometimes I see warnings in my logs if i create a consumer for a topic > > which doesn't exist. Such as: > > > > org.apache.kafka.clients.NetworkClient - Error while fetching metadata > > with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE} > > > > If later messages are posted to that topic (which will create it), will > my > > consumer receive those messages? > > >
Re: Kafka behavior when consuming a topic which doesn't exist?
What happens if auto creation is enabled but the topic doesn't exist. Consumers subscribe to that topic which doesn't exist. Then messages are posted to that topic. Will the consumer receive those messages in this scenario? On 8 Jul 2017 4:38 a.m., "M. Manna" wrote: That depends. If auto creation of non-existent topic enabled (check docs), then it will simple use the minimum partiotion and replication settings defined in broker config to create a topic. If auto creation is disabled, your consumer group won't do anything. With auto creation enable - It's the same if you publish to a topic that doesn't yet exist. But that means topic will get created first, and then the messages are sent on the bus. If you subscribe to that topic later you will get messages. Check below: http://grokbase.com/t/kafka/users/1648kbr04c/subscribe-on- a-topic-that-does-not-exist On 7 Jul 2017 9:46 pm, "Ali Akhtar" wrote: > Sometimes I see warnings in my logs if i create a consumer for a topic > which doesn't exist. Such as: > > org.apache.kafka.clients.NetworkClient - Error while fetching metadata > with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE} > > If later messages are posted to that topic (which will create it), will my > consumer receive those messages? >
Re: Kafka behavior when consuming a topic which doesn't exist?
That depends. If auto creation of non-existent topic enabled (check docs), then it will simple use the minimum partiotion and replication settings defined in broker config to create a topic. If auto creation is disabled, your consumer group won't do anything. With auto creation enable - It's the same if you publish to a topic that doesn't yet exist. But that means topic will get created first, and then the messages are sent on the bus. If you subscribe to that topic later you will get messages. Check below: http://grokbase.com/t/kafka/users/1648kbr04c/subscribe-on-a-topic-that-does-not-exist On 7 Jul 2017 9:46 pm, "Ali Akhtar" wrote: > Sometimes I see warnings in my logs if i create a consumer for a topic > which doesn't exist. Such as: > > org.apache.kafka.clients.NetworkClient - Error while fetching metadata > with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE} > > If later messages are posted to that topic (which will create it), will my > consumer receive those messages? >
Kafka behavior when consuming a topic which doesn't exist?
Sometimes I see warnings in my logs if i create a consumer for a topic which doesn't exist. Such as: org.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE} If later messages are posted to that topic (which will create it), will my consumer receive those messages?