Log loading is taking time. By default, Kafka uses one thread per data.dir to load the logs. You can try by increasing "num.recovery.threads.per.data.dir" broker config property.
On Mon, Jul 31, 2017 at 8:21 AM, Vinod KC <v...@hortonworks.com> wrote: > Hi , > > Our business use case requires to keep all kafka logs in topics for ever > and based on suggestion provided in http://grokbase.com/t/kafka/ > users/152ws9gbvx/unlimited-log-retention ,we have set > log.retention.hours=2147483647" (245,146 years) > > We have around 45 TB data in 3 nodes kafka 0.8.2.2 cluster, but now each > kafka broker is taking more than 1 hour to restart, do you think it is > because of high data retention value? > > Please refer the below time difference from shutdown to log loading > completion > > >cat /kafka-2.out | grep "Controlled shutdown succeeded" > [2017-06-12 05:18:28,190] INFO [Kafka Server 1002], Controlled shutdown > succeeded (kafka.server.KafkaServer) > > cat /kafka-2.out | grep "INFO starting (kafka.server.KafkaServer)" > [2017-06-12 05:23:17,691] INFO starting (kafka.server.KafkaServer) > >cat /kafka-2.out | grep "INFO Logs loading complete. > (kafka.log.LogManager)" > [2017-06-12 06:29:28,171] INFO Logs loading complete. > (kafka.log.LogManager) > > Thanks > Vinod > >
