The configs look reasonable. Currently, we don't repartition existing data. Only new messages will consider the newly added partitions.
Thanks Jun On Mon, Nov 19, 2012 at 9:34 AM, Muthukumar <zenmu...@gmail.com> wrote: > Hi, > > We're currently having 1 broker to 10 consumers setup pointing to 5 ZK > node. Load would be in 500GB data / day, and following is the generic > configuration we're using as, > > brokerid=0 > port=9092 > num.threads=8 > socket.send.buffer=1048576 > socket.receive.buffer=1048576 > max.socket.request.bytes=104857600 > log.dir=/data/kafka/server/logs > num.partitions=16 > > topic.partition.count.map=topic1:16,topic2:16,topic3:16,topic4:16,topic5:16,topic6:16,topic7:46,topic8:16 > log.flush.interval=10000 > log.default.flush.interval.ms=1000 > log.default.flush.scheduler.interval.ms=1000 > log.retention.hours=48 > log.file.size=536870912 > log.cleanup.interval.mins=1 > enable.zookeeper=true > zk.connectiontimeout.ms=1000000 > > 1) Please let us know if this configuration is holds good or needs a > change? > 2) We're seeing a huge backlog of 320GB for topic6 partitioned for 46. > Is it possible to re-partition since 21 partition were newly added and > data lag is almost 0 there. > > Let me know if more details are required. And thanks for your great help. > > -Muthu >
