Todd, I actually only intend to keep each topic valid for 3 days most. Each of our topic has 3 partitions, so its around 3*240*3 =2160 partitions. Since there is no api for deleting topic, i guess i could set up a cron job deleting the out dated topics(folders) from zookeeper.. do you know when the delete topic api will be available in kafka? Chen
On Mon, Aug 11, 2014 at 3:47 PM, Todd Palino <tpal...@linkedin.com.invalid> wrote: > You need to consider your total partition count as you do this. After 30 > days, assuming 1 partition per topic, you have 7200 partitions. Depending > on how many brokers you have, this can start to be a problem. We just > found an issue on one of our clusters that has over 70k partitions that > there¹s now a problem with doing actions like a preferred replica election > for all topics because the JSON object that gets written to the zookeeper > node to trigger it is too large for Zookeeper¹s default 1 MB data size. > > You also need to think about the number of open file handles. Even with no > data, there will be open files for each topic. > > -Todd > > > On 8/11/14, 2:19 PM, "Chen Wang" <chen.apache.s...@gmail.com> wrote: > > >Folks, > >Is there any potential issue with creating 240 topics every day? Although > >the retention of each topic is set to be 2 days, I am a little concerned > >that since right now there is no delete topic api, the zookeepers might be > >overloaded. > >Thanks, > >Chen > >
