This is correct, compression isn't used for the offsets at all. If, for
some reason, you do have either a compressed or a corrupt message somewhere
in the topic, the method I mentioned previously will flush it out. We
haven't seen that as a recurring problem, so fixing it once is sufficient.
-Todd
Just to clarify too, if the only use case for log-compaction we use is for
the __consumer_offsets, we should be ok, correct? I assume compression is
not used by default for consumer offsets?
Jason
On Fri, Sep 25, 2015 at 12:15 AM, Todd Palino wrote:
> For now, that's the way it is. Historicall
For now, that's the way it is. Historically, we've only monitored the lag
for our infrastructure applications. Other users are responsible for their
own checking, typically using the maxlag mbean or some application specific
metric. Besides the core, we've probably got a dozen or so consumers moved
> On Sep 24, 2015, at 8:11 PM, Todd Palino wrote:
>
> Well, in general you can't currently use compressed messages in any topic
> that has compaction turned on regardless of whether or not you are using
> Kafka-committed offsets. The log compaction thread will die either way.
> There's only one c
Well, in general you can't currently use compressed messages in any topic
that has compaction turned on regardless of whether or not you are using
Kafka-committed offsets. The log compaction thread will die either way.
There's only one compression thread for the broker that runs on all topics
that
On Sep 18, 2015, at 10:25 AM, Todd Palino wrote:
> I think the last major issue with log compaction (that it couldn't handle
> compressed messages) was committed as part of
> https://issues.apache.org/jira/browse/KAFKA-1374 in August, but I'm not
> certain what version this will end up in. It ma
It looks like that fix will not be included in a release until 0.9.0.0.
I'm thinking maybe it makes sense not to switch to kafka storage for
offsets until then?
Jason
On Fri, Sep 18, 2015 at 1:25 PM, Todd Palino wrote:
> I think the last major issue with log compaction (that it couldn't handle
I think the last major issue with log compaction (that it couldn't handle
compressed messages) was committed as part of
https://issues.apache.org/jira/browse/KAFKA-1374 in August, but I'm not
certain what version this will end up in. It may be part of 0.8.2.2.
Regardless, you'll probably be OK now
Thanks!
I did what you suggested and it worked except it was necessary for me to
remove the cleaner-offset-checkpoint file from the data directory and
restart the servers. The log indicates all is well.
Do you know what version the fix to this will be in? I'm not looking
forward to dealing with
Yes, this is a known concern, and it should be fixed with recent commits.
In the meantime, you'll have to do a little manual cleanup.
The problem you're running into is a corrupt message in the offsets topic.
We've seen this a lot. What you need to do is set the topic configuration
to remove the c
I've been experiencing this issue across several of our environments ever
since we enabled the log cleaner for the __consumer_offsets topic.
We are on version 0.8.2.1 of kafka, using the new producer. All of our
consumers are set to commit to kafka only.
Below is the stack trace in the log I've
11 matches
Mail list logo
