[
https://issues.apache.org/jira/browse/KAFKA-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chaos updated KAFKA-13251:
--------------------------
Description:
Disk error occurred in broker(=42),and then Shrinking ISR,
why Shrinking ISR to error broker ?
kafka logs:
broker42:
[2021-08-26 20:20:55,640] ERROR [ReplicaManager broker=42] Error processing
fetch with max size 1048576 from consumer on partition topic_xx-123:
(fetchOffset=11061228956, logStartOffset=-1, maxBytes=1048576,
currentLeaderEpoch=Optional.empty) (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.CorruptRecordException: Found record size 0
smaller than minimum record overhead (14) in file
/data4/kafka-logs/topic_xx-123/00000000011060934646.log.
[2021-08-26 20:20:55,640] ERROR Error while appending records to topic_xx-123
in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
[2021-08-26 20:20:55,645] ERROR Error while deleting segments for topic_xx-123
in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException:
/data4/kafka-logs/topic_xx-123/00000000011040402299.log ->
/data4/kafka-logs/topic_xx-123/00000000011040402299.log.deleted: Read-only file
system
Suppressed: java.nio.file.FileSystemException:
/data4/kafka-logs/topic_xx-123/00000000011040402299.log ->
/data4/kafka-logs/topic_xx-123/00000000011040402299.log.deleted: Read-only file
system
[2021-08-26 20:20:55,644] ERROR Error while appending records to topic_xx-123
in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
[2021-08-26 20:20:55,652] INFO [Partition topic_xx-123 broker=42] Shrinking ISR
from 55,42 to 42. Leader: (highWatermark: 11061228956, endOffset: 11061228965).
Out of sync replicas: (brokerId: 55, endOffset: 11061228956).
(kafka.cluster.Partition)
disk error is:
Aug 26 20:20:55 kafka-zk-19 kernel: sd 0:2:5:0: [sdf] tag#33 FAILED Result:
hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
was:
[2021-08-26 20:20:55,640] ERROR [ReplicaManager broker=42] Error processing
fetch with max size 1048576 from consumer on partition topic_xx-123:
(fetchOffset=11061228956, logStartOffset=-1, maxBytes=1048576,
currentLeaderEpoch=Optional.empty) (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.CorruptRecordException: Found record size 0
smaller than minimum record overhead (14) in file
/data4/kafka-logs/topic_xx-123/00000000011060934646.log.
[2021-08-26 20:20:55,640] ERROR Error while appending records to topic_xx-123
in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
[2021-08-26 20:20:55,645] ERROR Error while deleting segments for topic_xx-123
in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException:
/data4/kafka-logs/topic_xx-123/00000000011040402299.log ->
/data4/kafka-logs/topic_xx-123/00000000011040402299.log.deleted: Read-only file
system
Suppressed: java.nio.file.FileSystemException:
/data4/kafka-logs/topic_xx-123/00000000011040402299.log ->
/data4/kafka-logs/topic_xx-123/00000000011040402299.log.deleted: Read-only file
system
[2021-08-26 20:20:55,644] ERROR Error while appending records to topic_xx-123
in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
[2021-08-26 20:20:55,644] ERROR Error while appending records to topic_xx-123
in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
[2021-08-26 20:20:55,644] ERROR Error while appending records to topic_xx-123
in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
[2021-08-26 20:20:55,652] INFO [Partition topic_xx-123 broker=42] Shrinking ISR
from 55,42 to 42. Leader: (highWatermark: 11061228956, endOffset: 11061228965).
Out of sync replicas: (brokerId: 55, endOffset: 11061228956).
(kafka.cluster.Partition)
disk errors are:
> 2.4.0
> -----
>
> Key: KAFKA-13251
> URL: https://issues.apache.org/jira/browse/KAFKA-13251
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 2.4.0
> Environment: linux 4.1.0
> Reporter: chaos
> Priority: Major
>
> Disk error occurred in broker(=42),and then Shrinking ISR,
> why Shrinking ISR to error broker ?
>
> kafka logs:
> broker42:
> [2021-08-26 20:20:55,640] ERROR [ReplicaManager broker=42] Error processing
> fetch with max size 1048576 from consumer on partition topic_xx-123:
> (fetchOffset=11061228956, logStartOffset=-1, maxBytes=1048576,
> currentLeaderEpoch=Optional.empty) (kafka.server.ReplicaManager)
> org.apache.kafka.common.errors.CorruptRecordException: Found record size 0
> smaller than minimum record overhead (14) in file
> /data4/kafka-logs/topic_xx-123/00000000011060934646.log.
> [2021-08-26 20:20:55,640] ERROR Error while appending records to topic_xx-123
> in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
> [2021-08-26 20:20:55,645] ERROR Error while deleting segments for
> topic_xx-123 in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
> java.nio.file.FileSystemException:
> /data4/kafka-logs/topic_xx-123/00000000011040402299.log ->
> /data4/kafka-logs/topic_xx-123/00000000011040402299.log.deleted: Read-only
> file system
> Suppressed: java.nio.file.FileSystemException:
> /data4/kafka-logs/topic_xx-123/00000000011040402299.log ->
> /data4/kafka-logs/topic_xx-123/00000000011040402299.log.deleted: Read-only
> file system
> [2021-08-26 20:20:55,644] ERROR Error while appending records to topic_xx-123
> in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
> [2021-08-26 20:20:55,652] INFO [Partition topic_xx-123 broker=42] Shrinking
> ISR from 55,42 to 42. Leader: (highWatermark: 11061228956, endOffset:
> 11061228965). Out of sync replicas: (brokerId: 55, endOffset: 11061228956).
> (kafka.cluster.Partition)
>
> disk error is:
> Aug 26 20:20:55 kafka-zk-19 kernel: sd 0:2:5:0: [sdf] tag#33 FAILED Result:
> hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
