[
https://issues.apache.org/jira/browse/KAFKA-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772175#comment-16772175
]
Ninth Nails commented on KAFKA-7951:
------------------------------------
As a temporary fix, I moved the partition on question {{__consumer_offsets-37}}
onto another broker. This allow the cleaner do to it's work on the faulty
broker, but I'm expecting the other broker's log cleaner to fails as well,
eventually.
> Log Cleaner thread stop with "Varint is too long" error
> -------------------------------------------------------
>
> Key: KAFKA-7951
> URL: https://issues.apache.org/jira/browse/KAFKA-7951
> Project: Kafka
> Issue Type: Bug
> Components: log
> Affects Versions: 0.11.0.3
> Environment: Amazon Linux AMI 2017.09
> confluent-platform-oss-2.11-3.3.2
> OpenJDK Runtime Environment (build 1.8.0_191-b12)
> EC2 instance type i3.xlarge
> Reporter: Ninth Nails
> Priority: Blocker
>
> I have one broker log cleaner thread dying because of a failing type
> conversion. My cluster is running on 0.11.03-cp1 (Confluent Platform). Here's
> the relevant logs in log-cleaner.log
> {code:java}
> [2019-02-19 15:08:23,387] priority=INFO message="[kafka-log-cleaner-thread-0]:
> Log cleaner thread 0 cleaned log __consumer_offsets-36 (dirty section
> = [931148954, 931148954])
> 13,051.5 MB of log processed in 159.0 seconds (82.1 MB/sec).
> Indexed 13,051.5 MB in 81.2 seconds (160.8 Mb/sec, 51.1% of total
> time)
> Buffer utilization: 0.0%
> Cleaned 13,051.5 MB in 77.8 seconds (167.7 Mb/sec, 48.9% of total
> time)
> Start size: 13,051.5 MB (166,676,749 messages)
> End size: 0.0 MB (159 messages)
> 100.0% size reduction (100.0% fewer messages)
> " category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Beginning
> cleaning of log __consumer_offsets-37." category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Building offset
> map for __consumer_offsets-37..." category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,410] priority=INFO message="Cleaner 0: Building offset
> map for log __consumer_offsets-37 for 87 segments in offset range [211566448,
> 308245257)." category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,945] priority=ERROR
> message="[kafka-log-cleaner-thread-0]: Error due to"
> category=kafka.log.LogCleaner
> java.lang.IllegalArgumentException: Varint is too long, the most significant
> bit in the 5th byte is set, converted value: 445d418c
> at
> org.apache.kafka.common.utils.ByteUtils.illegalVarintException(ByteUtils.java:326)
> at
> org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:148)
> at
> org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:305)
> at
> org.apache.kafka.common.record.DefaultRecordBatch$2.readNext(DefaultRecordBatch.java:299)
> at
> org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:563)
> at
> org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:532)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:752)
> at
> kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:741)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> kafka.log.Cleaner.kafka$log$Cleaner$$buildOffsetMapForSegment(LogCleaner.scala:741)
> at
> kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:707)
> at
> kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:704)
> at
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:704)
> at kafka.log.Cleaner.doClean(LogCleaner.scala:373)
> at kafka.log.Cleaner.clean(LogCleaner.scala:361)
> at
> kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:256)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:236)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
> [2019-02-19 15:08:23,964] priority=INFO
> message="[kafka-log-cleaner-thread-0]: Stopped"
> category=kafka.log.LogCleaner{code}
> This is the first time we are seeing this error and coincidentally we have
> changed our JVM from Oracle to OpenJDK, both 1.8 Java.
>
> Please note that I can't upgrade to newer version of Kafka yet.
> Thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
