Hi,
you might want to have a look here:
http://kafka.apache.org/documentation.html#topic-config
_segment.ms_ and _segment.bytes _ should allow you to control the
time/size when segments are rolled.
Best
Jan
On 16.06.2015 14:05, Shayne S wrote:
Some further information, and is this a bug?
Ah misread that sorry!
On 17.06.2015 14:26, Shayne S wrote:
Right, you can see I've got segment.ms set. The trick is that they don't
actually roll over until something new arrives. If your topic is idle (not
receiving messages), it won't ever roll over to a new segment, and thus the
last
Some further information, and is this a bug? I'm using 0.8.2.1.
Log compaction will only occur on the non active segments. Intentional or
not, it seems that the last segment is always the active segment. In other
words, an expired segment will not be cleaned until a new segment has been
Hi,
Your observation is correct. we never compact the active segment.
Some improvements are proposed here,
https://issues.apache.org/jira/browse/KAFKA-1981
Manikumar
On Tue, Jun 16, 2015 at 5:35 PM, Shayne S shaynest...@gmail.com wrote:
Some further information, and is this a bug?
Ok..I got your point. Currently we check the log segment constraints
(segment.bytes, segment.ms)
only before appending new messages. So we will not create a new log segment
until new data comes.
In your case, your approach(sending periodic dummy/ping message) should be
fine.
On Tue, Jun 16,
Thank you for the response!
Unfortunately, those improvements would not help. It is the lack of
activity resulting in a new segment that prevents compaction.
I was confused by what qualifies as the active segment. The active segment
is the last segment as opposed to the segment that would be
Hi, I'm new to Kafka and having trouble with log compaction.
I'm attempting to set up topics that will aggressively compact, but so far
I'm having trouble getting complete compaction at all. The topic is
configured like so:
Topic:beer_archive PartitionCount:20 ReplicationFactor:1
