Re: HELP PLEASE->Kafka 0.9.0.0 create topic throwing ERROR kafka.admin.AdminOperationException: replication factor: 1 larger than available brokers: 0 with zookeeper 3.4.6
Hi Gwen, Thanks for responding back. Issue was with my Init.d script. Now its resolved. Thanks once again for your valuable time and help. Best regards, Dilpreet On 1/27/16, 2:24 PM, "Gwen Shapira" <g...@confluent.io> wrote: >Did you check your brokers are running? > >On Wed, Jan 27, 2016 at 1:30 PM, Sandhu, Dilpreet <dsan...@illumina.com> >wrote: > >> Hi all, >> I am using Kafka 0.9.0.0 with Zookeeper 3.4.6. I am not sure if >>I >> am missing anything :( >> When I try to create any topic I get the following error:- >> >> >> Error while executing topic command : replication factor: 1 larger than >> available brokers: 0 >> >> [2016-01-27 20:35:53,738] ERROR kafka.admin.AdminOperationException: >> replication factor: 1 larger than available brokers: 0 >> >> at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:77) >> >> at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:236) >> >> at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:105) >> >> at kafka.admin.TopicCommand$.main(TopicCommand.scala:60) >> >> at kafka.admin.TopicCommand.main(TopicCommand.scala) >> >> (kafka.admin.TopicCommand$) >> >> It works with kafka 0.8.2.1 >> >> Zookeeper is running. >> >> Any help will be greatly appreciated. >> >> Thanks. >> Best regards, >> Dilpreet >> >>
Kafka 0.9.0.0 create topic throwing KeeperErrorCode = NoNode for /brokers/ids with zookeeper 3.4.6
Hi all, I am using Kafka 0.9.0.0 with Zookeeper 3.4.6. I am not sure if I am missing anything :( When I try to create any topic I get the following error:- Error while executing topic command : replication factor: 1 larger than available brokers: 0 [2016-01-27 20:35:53,738] ERROR kafka.admin.AdminOperationException: replication factor: 1 larger than available brokers: 0 at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:77) at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:236) at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:105) at kafka.admin.TopicCommand$.main(TopicCommand.scala:60) at kafka.admin.TopicCommand.main(TopicCommand.scala) (kafka.admin.TopicCommand$) It works with kafka 0.8.2.1 Zookeeper is running. Any help will be greatly appreciated. Thanks. Best regards, Dilpreet
Re: Kafka 0.9.0.0 create topic throwing ERROR kafka.admin.AdminOperationException: replication factor: 1 larger than available brokers: 0 with zookeeper 3.4.6
Ignore my last email sorry >Hi all, >I am using Kafka 0.9.0.0 with Zookeeper 3.4.6. I am not sure if I >am missing anything :( >When I try to create any topic I get the following error:- > > >Error while executing topic command : replication factor: 1 larger than >available brokers: 0 > >[2016-01-27 20:35:53,738] ERROR kafka.admin.AdminOperationException: >replication factor: 1 larger than available brokers: 0 > >at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:77) > >at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:236) > >at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:105) > >at kafka.admin.TopicCommand$.main(TopicCommand.scala:60) > >at kafka.admin.TopicCommand.main(TopicCommand.scala) > > (kafka.admin.TopicCommand$) > >It works with kafka 0.8.2.1 > >Zookeeper is running. > >Any help will be greatly appreciated. > >Thanks. >Best regards, >Dilpreet
Sorry. Figured it was Init.d script.
Sorry for spamming you all with questions. Figured out it was my init.d script that was broken. :( On 1/27/16, 1:30 PM, "Sandhu, Dilpreet" <dsan...@illumina.com> wrote: >Hi all, >I am using Kafka 0.9.0.0 with Zookeeper 3.4.6. I am not sure if I >am missing anything :( >When I try to create any topic I get the following error:- > > >Error while executing topic command : replication factor: 1 larger than >available brokers: 0 > >[2016-01-27 20:35:53,738] ERROR kafka.admin.AdminOperationException: >replication factor: 1 larger than available brokers: 0 > >at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:77) > >at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:236) > >at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:105) > >at kafka.admin.TopicCommand$.main(TopicCommand.scala:60) > >at kafka.admin.TopicCommand.main(TopicCommand.scala) > >(kafka.admin.TopicCommand$) > >It works with kafka 0.8.2.1 > >Zookeeper is running. > >Any help will be greatly appreciated. > >Thanks. >Best regards, >Dilpreet >
Re: Kafka log retention questions
Thanks a lot Gwen and Raju. This information was really helpful. I used the following example:- kafka-topics.sh --zookeeper localhost:13003 --alter --topic MyTopic --config retention.ms=1000 And it works as expected. By setting its limit to a particular value, it prunes messages after that limit. I just want to make sure I am looking at right property. Kindly provide me with feedback, if I am on right direction. Thanks once again. Your help is much appreciated. Best regards, Dilpreet On 11/13/15, 1:24 AM, "Raju Bairishetti" <raju@gmail.com> wrote: >Adding some more info inline. > >On Fri, Nov 13, 2015 at 10:43 AM, Gwen Shapira <g...@confluent.io> wrote: > >> See answers inline >> >> On Thu, Nov 12, 2015 at 2:53 PM, Sandhu, Dilpreet <dsan...@illumina.com> >> wrote: >> >> > Hi all, >> >I am new to Kafka usage. Here are some questions that I have in >> > mind. Kindly help me understand it better. If some questions make no >> sense >> > feel free to call it out. >> > 1. Is it possible to prune log offsets (messages)older than certain >>date >> > in a partition? >> > >> >> You can configure the retention time for a topic, all older messages >>will >> be pruned. This configuration is at topic level, not for a specific >> partition. >> > >*IMO, we can set the retention in bytes(**retention.bytes**) at partition >level also. Whenever the partition log grows more than configured >retention >bytes then it deletes the older messages from the partition log. **Please >correct me if I am wrong.* > >> >> >> > 2. Will Kafka delete a topic regardless of unconsumed messages, if log >> > retention hours limit is met? >> > >> >> Yes >> > >Kafka does not care & does not know whether the message is consumed or >not. Kafka provides a mechanism to store all messages from multiple >producer machines to a central location. > >> >> >> > 3. Is there a way to just delete messages in Kafka? >> > >> >> Normally, we are only deleting messages when they are old or when size >> limit is exceeded. >> There is a special "compaction" mode where the last message for any key >>is >> stored forever. In this mode you can delete messages by sending a new >> message with that key and a "null" value. However, this also just takes >> place after a certain amount of time. >> >> >> > >> > Any help will be much appreciated. >> > >> > Thanks. >> > Best regards, >> > Dilpreet >> > >> > > > >-- >Thanks >Raju Bairishetti, > >www.inmobi.com > > > > >*2012 ME Best Mobile Ad Network Award Winner > >[image: http://www.inmobi.com/press-releases/files/2013/02/MIT.png]
Re: Kafka log retention questions
Thank you so much Gwen.This is really useful information :) On 11/13/15, 1:24 PM, "Gwen Shapira" <g...@confluent.io> wrote: >Mostly correct, yes. > >Just one additional point: >You configured the topic to retain data for 1s. However, there is also a >property for how often the broker will look for something to delete: >log.retention.check.interval.ms. >This defaults to 5 minute. So unless you tweak it, the old messages will >only be deleted every 5 minutes. > >Gwen > > >On Fri, Nov 13, 2015 at 11:07 AM, Sandhu, Dilpreet <dsan...@illumina.com> >wrote: > >> Thanks a lot Gwen and Raju. This information was really helpful. >> >> I used the following example:- >> kafka-topics.sh --zookeeper localhost:13003 --alter --topic MyTopic >> --config retention.ms=1000 >> >> >> And it works as expected. >> By setting its limit to a particular value, it prunes messages after >>that >> limit. I just want to make sure I am looking at right property. >> >> >> Kindly provide me with feedback, if I am on right direction. >> Thanks once again. >> >> Your help is much appreciated. >> Best regards, >> Dilpreet >> >> >> >> On 11/13/15, 1:24 AM, "Raju Bairishetti" <raju....@gmail.com> wrote: >> >> >Adding some more info inline. >> > >> >On Fri, Nov 13, 2015 at 10:43 AM, Gwen Shapira <g...@confluent.io> >>wrote: >> > >> >> See answers inline >> >> >> >> On Thu, Nov 12, 2015 at 2:53 PM, Sandhu, Dilpreet >><dsan...@illumina.com >> > >> >> wrote: >> >> >> >> > Hi all, >> >> >I am new to Kafka usage. Here are some questions that I >>have in >> >> > mind. Kindly help me understand it better. If some questions make >>no >> >> sense >> >> > feel free to call it out. >> >> > 1. Is it possible to prune log offsets (messages)older than >>certain >> >>date >> >> > in a partition? >> >> > >> >> >> >> You can configure the retention time for a topic, all older messages >> >>will >> >> be pruned. This configuration is at topic level, not for a specific >> >> partition. >> >> >> > >> >*IMO, we can set the retention in bytes(**retention.bytes**) at >>partition >> >level also. Whenever the partition log grows more than configured >> >retention >> >bytes then it deletes the older messages from the partition log. >>**Please >> >correct me if I am wrong.* >> > >> >> >> >> >> >> > 2. Will Kafka delete a topic regardless of unconsumed messages, if >>log >> >> > retention hours limit is met? >> >> > >> >> >> >> Yes >> >> >> > >> >Kafka does not care & does not know whether the message is >>consumed or >> >not. Kafka provides a mechanism to store all messages from multiple >> >producer machines to a central location. >> > >> >> >> >> >> >> > 3. Is there a way to just delete messages in Kafka? >> >> > >> >> >> >> Normally, we are only deleting messages when they are old or when >>size >> >> limit is exceeded. >> >> There is a special "compaction" mode where the last message for any >>key >> >>is >> >> stored forever. In this mode you can delete messages by sending a new >> >> message with that key and a "null" value. However, this also just >>takes >> >> place after a certain amount of time. >> >> >> >> >> >> > >> >> > Any help will be much appreciated. >> >> > >> >> > Thanks. >> >> > Best regards, >> >> > Dilpreet >> >> > >> >> >> > >> > >> > >> >-- >> >Thanks >> >Raju Bairishetti, >> > >> >www.inmobi.com >> > >> > >> > >> > >> >*2012 ME Best Mobile Ad Network Award Winner >> > >> >[image: http://www.inmobi.com/press-releases/files/2013/02/MIT.png] >> >>
Kafka log retention questions
Hi all, I am new to Kafka usage. Here are some questions that I have in mind. Kindly help me understand it better. If some questions make no sense feel free to call it out. 1. Is it possible to prune log offsets (messages)older than certain date in a partition? 2. Will Kafka delete a topic regardless of unconsumed messages, if log retention hours limit is met? 3. Is there a way to just delete messages in Kafka? Any help will be much appreciated. Thanks. Best regards, Dilpreet
