Re: log retention bytes and log segment bytes

[Peter Bukowinski]

Hi Amit,

This is from the broker config section of the very good documentation on the 
kafka web site: https://kafka.apache.org/0100/documentation.html#brokerconfigs

log.segment.bytes: The maximum size of a single log file (default 1GB)

log.retention.bytes: The maximum size of the log before deleting it (default 
unlimited)

My explanation:

The log.segment.bytes parameter refers to individual segments of the whole 
topic partition log on the broker. Kafka will create a new log segment for the 
partition once the segment limit is reached.

The log.retention.bytes parameter refers to the cumulative size of all the 
partition log segments on the broker. If this value is set, then kafka will 
purge the oldest segments until the total size falls beneath the maximum.

In your example, you will get a new topic partition log after only 100 bytes, 
and you can have up to 50 logs until total size reaches 5000 bytes. At that 
point the oldest logs will start to be purged.

-- Peter (from phone)

> On Apr 12, 2018, at 10:10 AM, amit mishra  wrote:
> 
> Hi all ,
> 
> I am using kafka 0.10.
> 
>log.retention.bytes = 5000
>log.retention.check.interval.ms = 6000
>log.retention.hours = 24
>log.retention.minutes = null
>log.retention.ms = null
>log.roll.hours = 168
>log.roll.jitter.hours = 0
>log.roll.jitter.ms = null
>log.roll.ms = null
>log.segment.bytes = 100
> 
> Please let me know what does log.retention.bytes and log.segment.bytes
> denotes ?
> 
> Regards,
> amit

Re: log retention bytes and log segment bytes

[Ted Yu]

For log.retention.bytes :
A size-based retention policy for logs. Segments are pruned from the log
unless the remaining segments drop below log.retention.bytes
This config is per partition.

For log.segment.bytes :
The maximum size of a log segment file. When this size is reached a new log
segment will be created.



On Thu, Apr 12, 2018 at 10:10 AM, amit mishra  wrote:

> Hi all ,
>
> I am using kafka 0.10.
>
> log.retention.bytes = 5000
> log.retention.check.interval.ms = 6000
> log.retention.hours = 24
> log.retention.minutes = null
> log.retention.ms = null
> log.roll.hours = 168
> log.roll.jitter.hours = 0
> log.roll.jitter.ms = null
> log.roll.ms = null
> log.segment.bytes = 100
>
> Please let me know what does log.retention.bytes and log.segment.bytes
> denotes ?
>
> Regards,
> amit
>

log retention bytes and log segment bytes

[amit mishra]

Hi all ,

I am using kafka 0.10.

log.retention.bytes = 5000
log.retention.check.interval.ms = 6000
log.retention.hours = 24
log.retention.minutes = null
log.retention.ms = null
log.roll.hours = 168
log.roll.jitter.hours = 0
log.roll.jitter.ms = null
log.roll.ms = null
log.segment.bytes = 100

Please let me know what does log.retention.bytes and log.segment.bytes
denotes ?

Regards,
amit