Here's an example from our server.properties

log.segment.bytes=1073741824
message.max.bytes=5242880
num.partitions=3
log.retention.bytes=5000000000000
num.network.threads=6
num.io.threads=16
replica.fetch.max.bytes=6242880
default.replication.factor=3
zookeeper.connection.timeout.ms=60000
auto.leader.rebalance.enable=true
delete.topic.enable=true


log.retention.bytes=5000000000000 is the setting, we have 5.5TB available,
so we start deleting at 5TB space used.




On Mon, Oct 17, 2016 at 10:37 AM, Kunal Gupta <kunal.gu...@cube26.com>
wrote:

> I didn't get it ...
>
> Can you explain me in form of example or something for which you are
> feasible
>
> On Oct 17, 2016 2:08 PM, "Ben Davison" <ben.davi...@7digital.com> wrote:
>
> > We have it setup so that both log ms is set to 7 days and log delete
> > bytes(can't remember exactly what the setting is called. So we never run
> > out of space (don't set the value to something like 99% of your disk, as
> > the log cleaner thread might not kick in time, we leave it at 90% of
> disks
> > space)
> >
> > On Monday, 17 October 2016, Kunal Gupta <kunal.gu...@cube26.com> wrote:
> >
> > > Please help me :(
> > >
> > > *Thanks, Kunal*
> > > *+91-9958189589*
> > > *Data Analyst*
> > > *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798
> > > <http://dl.acm.org/citation.cfm?id=2790798> *
> > > *Blog:- **http://learnhardwithkunalgupta.blogspot.in
> > > <http://learnhardwithkunalgupta.blogspot.in> *
> > >
> > >
> > >
> > >
> > >
> > > On Sun, Oct 16, 2016 at 11:23 AM, Kunal Gupta <kunal.gu...@cube26.com
> > > <javascript:;>>
> > > wrote:
> > >
> > > >
> > > > In my organisation I have 3 machine cluster of Kafka and each topic
> > > > assigned two machine for storing there data.
> > > >
> > > > There is one topic for which I get lot of data from clients thats
> data
> > > > exceeds my disk space in one machine because that machine is a leader
> > of
> > > > that topic, when I look into kafka-logs seeing all topic data there
> is
> > > only
> > > > one topic whose data has consumed lot of space.
> > > >
> > > > In one month I have 36 GB data from that topic. What I have thought
> so
> > > far
> > > > is kafka logs will be delete after 24 hours and new data will be
> > retained
> > > > for 24 hours and my consumer will consume all data with in 24 hours,
> so
> > > > there will be no problem of disk space.
> > > >
> > > > I have enable everything in server.properties to clean kafka-logs but
> > > > still data is there. Below data is my settings in server.properties
> > file.
> > > >
> > > > *log.retention.minutes=1440*
> > > > *log.retention.hours=24*
> > > > *log.retention.ms <http://log.retention.ms>=86400000*
> > > > *log.cleaner.delete.retention.ms
> > > > <http://log.cleaner.delete.retention.ms>=24*
> > > > *log.segment.bytes=1048576*
> > > > *log.retention.check.interval.ms
> > > > <http://log.retention.check.interval.ms>=3000*
> > > > *log.cleaner.enable=true*
> > > > *zookeeper.connection.timeout.ms
> > > > <http://zookeeper.connection.timeout.ms>=30000*
> > > > *delete.topic.enable = true*
> > > > *auto.create.topics.enable = true*
> > > > *default.replication.factor=2*
> > > > *auto.leader.rebalance.enable=true*
> > > > *controlled.shutdown.enable=true*
> > > > *controller.socket.timeout.ms <http://controller.socket.timeout.ms
> > > >=120000*
> > > >
> > > > Please help me in this so that my machine can able to handle large
> > number
> > > > of request and data well using 3 machine cluster.
> > > >
> > > > *Thanks, Kunal*
> > > > *+91-9958189589*
> > > > *Data Analyst*
> > > > *First Paper Publication : **http://dl.acm.org/citation.
> cfm?id=2790798
> > > > <http://dl.acm.org/citation.cfm?id=2790798> *
> > > > *Blog:- **http://learnhardwithkunalgupta.blogspot.in
> > > > <http://learnhardwithkunalgupta.blogspot.in> *
> > > >
> > >
> >
> > --
> >
> >
> > This email, including attachments, is private and confidential. If you
> have
> > received this email in error please notify the sender and delete it from
> > your system. Emails are not secure and may contain viruses. No liability
> > can be accepted for viruses that might be transferred by this email or
> any
> > attachment. Any unauthorised copying of this message or unauthorised
> > distribution and publication of the information contained herein are
> > prohibited.
> >
> > 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
> > Registered in England and Wales. Registered No. 04843573.
> >
>

-- 


This email, including attachments, is private and confidential. If you have 
received this email in error please notify the sender and delete it from 
your system. Emails are not secure and may contain viruses. No liability 
can be accepted for viruses that might be transferred by this email or any 
attachment. Any unauthorised copying of this message or unauthorised 
distribution and publication of the information contained herein are 
prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

Reply via email to