In my organisation I have 3 machine cluster of Kafka and each topic
assigned two machine for storing there data.
There is one topic for which I get lot of data from clients thats data
exceeds my disk space in one machine because that machine is a leader of
that topic, when I look into kafka-logs seeing all topic data there is only
one topic whose data has consumed lot of space.
In one month I have 36 GB data from that topic. What I have thought so far
is kafka logs will be delete after 24 hours and new data will be retained
for 24 hours and my consumer will consume all data with in 24 hours, so
there will be no problem of disk space.
I have enable everything in server.properties to clean kafka-logs but still
data is there. Below data is my settings in server.properties file.
*delete.topic.enable = true*
*auto.create.topics.enable = true*
Please help me in this so that my machine can able to handle large number
of request and data well using 3 machine cluster.
*First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798