Re: Log Retention Period of Kafka Messages

[James Cheng]

> On Mar 12, 2018, at 10:13 AM, Kyle Tinker  
> wrote:
> 
> You have a couple options:
> 1) You can adjust log.segment.bytes to make the segments smaller so that 
> individual segments can be trimmed
> 2) You can set log.roll.hours, which will roll to a new log segment even if 
> the size hasn't been reached
> Note: #1 and #2 also have per-topic setting controls
> 

About #2, it's possible that segments will only roll when a message has been 
received, after log.roll.hours (or log.roll.ms, etc). That is, if the time has 
passed but no new message has been received, the log roll might not happen.

You should do some testing to double check.

-James


> https://kafka.apache.org/08/documentation/#brokerconfigs
> 
> Thanks,
> --
> 
> 
> KYLE TINKER
> Lead Architect/Team Lead | WorkForce Software 
> 
> T: +1 734 742 2616 | E: ktin...@workforcesoftware.com
> 
> 
> -Original Message-
> From: Muruganandham, Ashokkumar [mailto:ashokkumar.muruganand...@fluke.com] 
> Sent: Monday, March 12, 2018 1:19 AM
> To: dev@kafka.apache.org
> Subject: Log Retention Period of Kafka Messages
> 
> Hi Team,
> 
> Form the documentation I could see that log retention period is relative to 
> the partition segments. So unless the segment is closed (1Gb/1 Week) the 
> message will never be deleted.
> 
> Consider I have a very critical use case where I will need to delete the data 
> every day and my segment reaches 1GB after three days. So according the above 
> statement, then there is no way I could configure Kafka to delete the data in 
> one day.
> 
> Is my understanding correct ?
> 
> Regards,
> AK
> 
> 
> Please be advised that this email may contain confidential information. If 
> you are not the intended recipient, please notify us by email by replying to 
> the sender and delete this message. The sender disclaims that the content of 
> this email constitutes an offer to enter into, or the acceptance of, any 
> agreement; provided that the foregoing does not invalidate the binding effect 
> of any digital or other electronic reproduction of a manual signature that is 
> included in any attachment.
> 
> 
> 
> This message is intended exclusively for the individual or entity to which it 
> is addressed. This communication may contain information that is proprietary, 
> privileged, confidential or otherwise legally exempt from disclosure. If you 
> are not the named addressee, or have been inadvertently and erroneously 
> referenced in the address line, you are not authorized to read, print, 
> retain, copy or disseminate this message or any part of it. If you have 
> received this message in error, please notify the sender immediately by 
> e-mail and delete all copies of the message. (ID m031214)

RE: Log Retention Period of Kafka Messages

[Kyle Tinker]

You have a couple options:
1) You can adjust log.segment.bytes to make the segments smaller so that 
individual segments can be trimmed
2) You can set log.roll.hours, which will roll to a new log segment even if the 
size hasn't been reached
Note: #1 and #2 also have per-topic setting controls

https://kafka.apache.org/08/documentation/#brokerconfigs

Thanks,
--


KYLE TINKER
Lead Architect/Team Lead | WorkForce Software 

T: +1 734 742 2616 | E: ktin...@workforcesoftware.com


-Original Message-
From: Muruganandham, Ashokkumar [mailto:ashokkumar.muruganand...@fluke.com] 
Sent: Monday, March 12, 2018 1:19 AM
To: dev@kafka.apache.org
Subject: Log Retention Period of Kafka Messages

Hi Team,

Form the documentation I could see that log retention period is relative to the 
partition segments. So unless the segment is closed (1Gb/1 Week) the message 
will never be deleted.

Consider I have a very critical use case where I will need to delete the data 
every day and my segment reaches 1GB after three days. So according the above 
statement, then there is no way I could configure Kafka to delete the data in 
one day.

Is my understanding correct ?

Regards,
AK


Please be advised that this email may contain confidential information. If you 
are not the intended recipient, please notify us by email by replying to the 
sender and delete this message. The sender disclaims that the content of this 
email constitutes an offer to enter into, or the acceptance of, any agreement; 
provided that the foregoing does not invalidate the binding effect of any 
digital or other electronic reproduction of a manual signature that is included 
in any attachment.



This message is intended exclusively for the individual or entity to which it 
is addressed. This communication may contain information that is proprietary, 
privileged, confidential or otherwise legally exempt from disclosure. If you 
are not the named addressee, or have been inadvertently and erroneously 
referenced in the address line, you are not authorized to read, print, retain, 
copy or disseminate this message or any part of it. If you have received this 
message in error, please notify the sender immediately by e-mail and delete all 
copies of the message. (ID m031214)