[
https://issues.apache.org/jira/browse/KAFKA-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578161#comment-16578161
]
Alastair Munro edited comment on KAFKA-7282 at 8/13/18 12:00 PM:
-----------------------------------------------------------------
These seem related; it seems to be related to rolling new logs; we use a small
log size of 100Mb.
https://issues.apache.org/jira/browse/KAFKA-6292
https://issues.apache.org/jira/browse/KAFKA-7130
Here are the settings we override:
{code}
delete.topic.enable=true
auto.create.topics.enable=false
log.cleanup.policy=compact
log.cleaner.min.compaction.lag.ms=3600000
log.segment.bytes=104857600
log.roll.hours=8
log.dir=/tmp/kafka-logs/kafka
log.dirs=/tmp/kafka-logs/kafka
inter.broker.protocol.version=2.0.0
{code}
See the issue on 0.11.0.2, 1.1.1 and 2.0.0.
We use 4 kafka instances and use isr of 3. The issue does not seem to affect
leader balancing or isr's; this all seems fine.
On 2.0.0, we also get these errors, without stack traces:
{code:java}
[2018-08-13 11:54:32,947] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
fetcherId=0] Error for partition http-request-v1-2 at offset 116705
(kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownServerException: The server experienced
an unexpected error when processing the request
[2018-08-13 11:54:40,946] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
fetcherId=0] Error for partition http-request-v1-14 at offset 96796
(kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownServerException: The server experienced
an unexpected error when processing the request
[2018-08-13 11:54:40,980] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
fetcherId=0] Error for partition http-response-v1-14 at offset 96795
(kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownServerException: The server experienced
an unexpected error when processing the request
[2018-08-13 11:55:04,954] ERROR [ReplicaFetcher replicaId=0, leaderId=1,
fetcherId=0] Error for partition http-request-v1-29 at offset 116407
(kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownServerException: The server experienced
an unexpected error when processing the request
[2018-08-13 11:55:07,947] ERROR [ReplicaFetcher replicaId=0, leaderId=3,
fetcherId=0] Error for partition http-request-v1-39 at offset 116853
(kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownServerException: The server experienced
an unexpected error when processing the request
[2018-08-13 11:55:07,950] ERROR [ReplicaFetcher replicaId=0, leaderId=1,
fetcherId=0] Error for partition http-request-v1-21 at offset 116443
(kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownServerException: The server experienced
an unexpected error when processing the request
{code}
was (Author: amunro):
These seem related; it seems to be related to rolling new logs; we use a small
log size of 100Mb.
https://issues.apache.org/jira/browse/KAFKA-6292
https://issues.apache.org/jira/browse/KAFKA-7130
Here are the settings we override:
delete.topic.enable=true
auto.create.topics.enable=false
log.cleanup.policy=compact
log.cleaner.min.compaction.lag.ms=3600000
log.segment.bytes=104857600
log.roll.hours=8
log.dir=/tmp/kafka-logs/kafka
log.dirs=/tmp/kafka-logs/kafka
inter.broker.protocol.version=2.0.0
We use 4 kafka instances and use isr of 3. The issue does not seem to affect
leader balancing or isr's; this all seems fine.
On 2.0.0, we also get these errors, without stack traces:
{code:java}
[2018-08-13 11:54:32,947] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
fetcherId=0] Error for partition http-request-v1-2 at offset 116705
(kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownServerException: The server experienced
an unexpected error when processing the request
[2018-08-13 11:54:40,946] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
fetcherId=0] Error for partition http-request-v1-14 at offset 96796
(kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownServerException: The server experienced
an unexpected error when processing the request
[2018-08-13 11:54:40,980] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
fetcherId=0] Error for partition http-response-v1-14 at offset 96795
(kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownServerException: The server experienced
an unexpected error when processing the request
[2018-08-13 11:55:04,954] ERROR [ReplicaFetcher replicaId=0, leaderId=1,
fetcherId=0] Error for partition http-request-v1-29 at offset 116407
(kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownServerException: The server experienced
an unexpected error when processing the request
[2018-08-13 11:55:07,947] ERROR [ReplicaFetcher replicaId=0, leaderId=3,
fetcherId=0] Error for partition http-request-v1-39 at offset 116853
(kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownServerException: The server experienced
an unexpected error when processing the request
[2018-08-13 11:55:07,950] ERROR [ReplicaFetcher replicaId=0, leaderId=1,
fetcherId=0] Error for partition http-request-v1-21 at offset 116443
(kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownServerException: The server experienced
an unexpected error when processing the request
{code}
> Failed to read `log header` from file channel
> ---------------------------------------------
>
> Key: KAFKA-7282
> URL: https://issues.apache.org/jira/browse/KAFKA-7282
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.11.0.2, 1.1.1, 2.0.0
> Environment: Linux
> Reporter: Alastair Munro
> Priority: Major
>
> Full stack trace:
> {code:java}
> [2018-08-13 11:22:01,635] ERROR [ReplicaManager broker=2] Error processing
> fetch operation on partition segmenter-evt-v1-14, offset 96745
> (kafka.server.ReplicaManager)
> org.apache.kafka.common.KafkaException: java.io.EOFException: Failed to read
> `log header` from file channel `sun.nio.ch.FileChannelImpl@6e6d8ddd`.
> Expected to read 17 bytes, but reached end of file after reading 0 bytes.
> Started read from position 25935.
> at
> org.apache.kafka.common.record.RecordBatchIterator.makeNext(RecordBatchIterator.java:40)
> at
> org.apache.kafka.common.record.RecordBatchIterator.makeNext(RecordBatchIterator.java:24)
> at
> org.apache.kafka.common.utils.AbstractIterator.maybeComputeNext(AbstractIterator.java:79)
> at
> org.apache.kafka.common.utils.AbstractIterator.hasNext(AbstractIterator.java:45)
> at
> org.apache.kafka.common.record.FileRecords.searchForOffsetWithSize(FileRecords.java:286)
> at kafka.log.LogSegment.translateOffset(LogSegment.scala:254)
> at kafka.log.LogSegment.read(LogSegment.scala:277)
> at kafka.log.Log$$anonfun$read$2.apply(Log.scala:1159)
> at kafka.log.Log$$anonfun$read$2.apply(Log.scala:1114)
> at kafka.log.Log.maybeHandleIOException(Log.scala:1837)
> at kafka.log.Log.read(Log.scala:1114)
> at
> kafka.server.ReplicaManager.kafka$server$ReplicaManager$$read$1(ReplicaManager.scala:912)
> at
> kafka.server.ReplicaManager$$anonfun$readFromLocalLog$1.apply(ReplicaManager.scala:974)
> at
> kafka.server.ReplicaManager$$anonfun$readFromLocalLog$1.apply(ReplicaManager.scala:973)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at kafka.server.ReplicaManager.readFromLocalLog(ReplicaManager.scala:973)
> at kafka.server.ReplicaManager.readFromLog$1(ReplicaManager.scala:802)
> at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:815)
> at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:678)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:107)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:69)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
