try bin/kafka-server-start.sh ./config/server.properties Also check how many segments have got created in logs dir and size of each. May be set the retention to 1 hour or 15 minutes. 10 sec seems very less.
On Wed, Nov 23, 2016 at 11:21 AM, Eranga Heshan <eranga....@gmail.com> wrote: > As you suggested I did these changes to the server.properties file on all > nodes. > > log.retention.check.interval.ms=10000 > log.retention.ms=10000 > log.segment.bytes=10490000 > > How I start each server is by, > > /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/bin/kafka-server-start.sh > -daemon > /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/config/server.properties > > However, I still get about 4GB log files. Did I miss anything? > > Thanks, > Regards, > > > Eranga Heshan > *Undergraduate* > Computer Science & Engineering > University of Moratuwa > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > Email: era...@wso2.com <osanda.des...@gmail.com> > <https://www.facebook.com/erangaheshan> <https://twitter.com/ > erangaheshan> > <https://www.linkedin.com/in/erangaheshan> > > On Wed, Nov 23, 2016 at 10:14 AM, Sachin Mittal <sjmit...@gmail.com> > wrote: > > > Check these in server.properties > > ############################# Log Retention Policy > > ############################# > > > > # The following configurations control the disposal of log segments. The > > policy can > > # be set to delete segments after a period of time, or after a given size > > has accumulated. > > # A segment will be deleted whenever *either* of these criteria are met. > > Deletion always happens > > # from the end of the log. > > > > # The minimum age of a log file to be eligible for deletion > > log.retention.hours=168 > > > > # A size-based retention policy for logs. Segments are pruned from the > log > > as long as the remaining > > # segments don't drop below log.retention.bytes. > > #log.retention.bytes=1073741824 > > > > # The maximum size of a log segment file. When this size is reached a new > > log segment will be created. > > log.segment.bytes=1073741824 > > > > Try by setting hours as 1 and set segment bytes as say 10 MB. You should > be > > able to see multiple segments created of size 20 MB under you kafka logs > > dir > > say: tmp\kafka-logs\<topic>\*.log > > > > As the files age > 1 hour it will be marked for deletion and be deleted > > next time delete thread runs. By default the delete thread check is 5 > > minutes. > > > > Also make sure when you start server you specify this property as the > > argument. > > > > bin\windows\kafka-server-start.bat config\server.properties > > > > > > > > On Wed, Nov 23, 2016 at 9:50 AM, Eranga Heshan <eranga....@gmail.com> > > wrote: > > > > > This is the exact parameter I added to server.config file on all the > > nodes. > > > > > > log.retention.ms=1000 > > > > > > Thanks, > > > Regards, > > > > > > > > > Eranga Heshan > > > *Undergraduate* > > > Computer Science & Engineering > > > University of Moratuwa > > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > > Email: era...@wso2.com <osanda.des...@gmail.com> > > > <https://www.facebook.com/erangaheshan> <https://twitter.com/ > > > erangaheshan> > > > <https://www.linkedin.com/in/erangaheshan> > > > > > > On Wed, Nov 23, 2016 at 9:43 AM, Eranga Heshan <eranga....@gmail.com> > > > wrote: > > > > > > > I have 4 node cluster and node1 has zookeeper running. I added those > > > > parameters on each of my server.properties files in kafka nodes. > > > > > > > > > > > > Eranga Heshan > > > > *Undergraduate* > > > > Computer Science & Engineering > > > > University of Moratuwa > > > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > > > Email: era...@wso2.com <osanda.des...@gmail.com> > > > > <https://www.facebook.com/erangaheshan> > > > > <https://twitter.com/erangaheshan> > > > > <https://www.linkedin.com/in/erangaheshan> > > > > > > > > On Wed, Nov 23, 2016 at 9:36 AM, Sachin Mittal <sjmit...@gmail.com> > > > wrote: > > > > > > > >> Kafka logs are not kept based on data consumption. They are deleted > > > after > > > >> it reaches a particular size and retention time. > > > >> Even if they are consumed multiple times in that duration, they are > > > still > > > >> kept. > > > >> > > > >> Where have you added those params and how are you starting your > > > zookeeper > > > >> and server. > > > >> > > > >> > > > >> On Wed, Nov 23, 2016 at 8:00 AM, Eranga Heshan < > eranga....@gmail.com> > > > >> wrote: > > > >> > > > >> > Sorry for repeatedly asking the same question, but although I > tried > > > all > > > >> > those parameters; my log file still at large about 2.5GB. Is it > due > > to > > > >> the > > > >> > data inside my topic is flushed to disk? I meant that there might > be > > > >> data > > > >> > inside topic which is not consumed yet or not been deleted even > > after > > > >> > consumed. > > > >> > > > > >> > Thanks, > > > >> > Regards, > > > >> > > > > >> > > > > >> > Eranga Heshan > > > >> > *Undergraduate* > > > >> > Computer Science & Engineering > > > >> > University of Moratuwa > > > >> > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > > >> > Email: era...@wso2.com <osanda.des...@gmail.com> > > > >> > <https://www.facebook.com/erangaheshan> <https://twitter.com/ > > > >> > erangaheshan> > > > >> > <https://www.linkedin.com/in/erangaheshan> > > > >> > > > > >> > On Tue, Nov 22, 2016 at 2:15 PM, Sachin Mittal < > sjmit...@gmail.com> > > > >> wrote: > > > >> > > > > >> > > Set some value to log.retention.ms which is ms time to retain > log > > > and > > > >> > also > > > >> > > log.segment.bytes which is size of a single log file. > > > >> > > delete.topic.enable is needed if you want to delete the topic > > > itself. > > > >> > > > > > >> > > Thanks > > > >> > > Sachin > > > >> > > > > > >> > > > > > >> > > On Tue, Nov 22, 2016 at 1:52 PM, Eranga Heshan < > > > eranga....@gmail.com> > > > >> > > wrote: > > > >> > > > > > >> > > > I tried log.retention.bytes=1049000 which is about 10MB. But > it > > > did > > > >> not > > > >> > > > work. Is it because I had not given any value to > > > >> delete.topic.enable? > > > >> > > > > > > >> > > > > > > >> > > > Eranga Heshan > > > >> > > > *Undergraduate* > > > >> > > > Computer Science & Engineering > > > >> > > > University of Moratuwa > > > >> > > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > > >> > > > Email: era...@wso2.com <osanda.des...@gmail.com> > > > >> > > > <https://www.facebook.com/erangaheshan> < > https://twitter.com/ > > > >> > > > erangaheshan> > > > >> > > > <https://www.linkedin.com/in/erangaheshan> > > > >> > > > > > > >> > > > On Tue, Nov 22, 2016 at 1:47 PM, Sachin Mittal < > > > sjmit...@gmail.com> > > > >> > > wrote: > > > >> > > > > > > >> > > > > Check > > > >> > > > > http://kafka.apache.org/documentation.html#brokerconfigs > > > >> > > > > log.retention.bytes > > > >> > > > > log.retention.ms > > > >> > > > > log.segment.bytes > > > >> > > > > delete.topic.enable > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > On Tue, Nov 22, 2016 at 1:36 PM, Eranga Heshan < > > > >> eranga....@gmail.com > > > >> > > > > > >> > > > > wrote: > > > >> > > > > > > > >> > > > > > Hi all, > > > >> > > > > > > > > >> > > > > > I want to keep the size of kafka-logs reduced as much as > > > >> possible. > > > >> > To > > > >> > > > do > > > >> > > > > it > > > >> > > > > > I need to set it to delete its unwanted logs frequently. > Is > > > >> there a > > > >> > > way > > > >> > > > > to > > > >> > > > > > do it? > > > >> > > > > > > > > >> > > > > > Thanks, > > > >> > > > > > Regards, > > > >> > > > > > > > > >> > > > > > Eranga Heshan > > > >> > > > > > *Undergraduate* > > > >> > > > > > Computer Science & Engineering > > > >> > > > > > University of Moratuwa > > > >> > > > > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > > >> > > > > > Email: era...@wso2.com <osanda.des...@gmail.com> > > > >> > > > > > <https://www.facebook.com/erangaheshan> < > > > https://twitter.com/ > > > >> > > > > > erangaheshan> > > > >> > > > > > <https://www.linkedin.com/in/erangaheshan> > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > > > > > >
