We have 3 DC's and created 5 node Kafka cluster in each DC, connected these 3
DC's using Mirror Maker for replication. We were conducting performance testing
using Kafka Producer Performance tool to load 100 million rows into 7 topics.
We expected that data will be loaded evenly across 7 topics but 4 topics got
loaded with ~2 million messages and remaining 3 topics loaded with 90 million
messages. The nodes that were leaders of those 3 topics ran out of disk space
and nodes went down.
We tried to bring back these 2 nodes by doing following
1. Stopped Kafka Service 2. Deleted couple of topics that were taking up too
much space i.e. /var/kafka/logs/{topic$}/ and file system showed 47% available
3. Brought back the Kafka nodes
As soon as nodes are back, we started observing the file system growing and in
15 minutes the mount point became full again. Deleted topics got recreated and
taking up space again. Looking at kafka.log, it shows many of the following
messages. Ultimately the node goes down. We don't need to recover data now, we
would like to bring nodes back. What are the steps to bring back these nodes?
[2015-03-11 20:52:36,323] INFO Rolled new log segment for 'dc2-perf-topic5-0'
in 3 ms. (kafka.log.Log)
[2015-03-11 15:58:07,321] INFO [Kafka Server 1021124614], started
(kafka.server.KafkaServer)
[2015-03-11 15:58:07,882] INFO Completed load of log dc2-perf-topic5-0 with log
end offset 0 (kafka.log.Log)
[2015-03-11 15:58:07,900] INFO Created log for partition [dc2-perf-topic5,0] in
/var/kafka/log with properties {segment.index.bytes -> 10485760,
file.delete.delay.ms -> 60000, segment.bytes -> 1073741824, flush.ms ->
9223372036854775807, delete.retention.ms -> 3600000, index.interval.bytes ->
4096, retention.bytes -> -1, cleanup.policy -> delete, segment.ms -> 604800000,
max.message.bytes -> 1000012, flush.messages -> 9223372036854775807,
min.cleanable.dirty.ratio -> 0.5, retention.ms -> 604800000}.
(kafka.log.LogManager)
[2015-03-11 15:58:07,914] INFO Completed load of log dc2-perf-topic2-0 with log
end offset 0 (kafka.log.Log)
[2015-03-11 15:58:07,916] INFO Created log for partition [dc2-perf-topic2,0] in
/var/kafka/log with properties {segment.index.bytes -> 10485760,
file.delete.delay.ms -> 60000, segment.bytes -> 1073741824, flush.ms ->
9223372036854775807, delete.retention.ms -> 3600000, index.interval.bytes ->
4096, retention.bytes -> -1, cleanup.policy -> delete, segment.ms -> 604800000,
max.message.bytes -> 1000012, flush.messages -> 9223372036854775807,
min.cleanable.dirty.ratio -> 0.5, retention.ms -> 604800000}.
(kafka.log.LogManager)
[2015-03-11 15:58:07,935] INFO Completed load of log dc2-perf-topic9-0 with log
end offset 0 (kafka.log.Log)
SP Naidu
