[
https://issues.apache.org/jira/browse/KAFKA-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104144#comment-18104144
]
Santosh Kumar Sahu commented on KAFKA-20911:
--------------------------------------------
Hi [~hudeqi] [~viktorsomogyi] [~borisvu] [~little brother ma], [~alchimie],
Can you brief like what disk problems?
Testcase we are running is that, we are doing sequential restarts of worker
nodes. And when one of the worker node is restarted which has kafka pod
scheduled, then this issue is coming.
This is more frequent now, 3 times we could reproduce.
Also any WA can you suggest? We had tried below WA but we need your inputs on
the WA steps-
also need input whether below WA will introduce any data loss or not, see 3rd
step for that.
Below are the steps that can be tried in case of CorruptRecordException error -
# Check whether other kafka pods than faulty pod still in healthy quorum and
serving the requests or not. If its other 2 pods are healthy and serving
requests, then go with next steps. Otherwise its complete cluster shutdown due
to data corruption.
# Check the logs of kafka pod where this issue is seen (pod-0 in your case)
and find out topic name for which its complaining
# Check and ensure if faulty pod i.e. pod-0 is not leader pod for topic_name.
You can check this with kafka-topics.sh --describe command. In your case, pod-0
was not leader for any of the topic because pod-0 was cut off from quorum due
to this fatal exception tranferring the leadership to other pods in healthy
quorum. But its worth ensuring if faulty pod is not leader for that topic_name.
# Delete topic_name.log file from /var/lib/kafka/data/logs folder, because
topic_name.log is corrupted. After deleting , topic_name.log file will sync and
will be recreated from the cluster with other 2 healthy kafka pods
# After deleting topic_name.log file, restart faulty kafka pod and check
whether issue is solved or if faulty pod is again crashing due to some other
faulty data pointing to some other topic name.
# If its still failing for other topic_name, then repeat steps from step 2
for that new topic name. If few topic related data deletion solves the issue
and there is no CorruptRecordException then pod-0 will sync from other 2 pods
and whole quorum will be healthy with 3 kafka pods.
# But if issue is not getting solved by deleting few topic related data, and
pod-0 keeps complaining with new topic_name with CorruptRecordException error
each time. It means whole PVC data is corrupted and then delete all .log files
related to topic that you created as part of your application from
/var/lib/kafka/data/logs. And then let all topic data get synced from other 2
pods quorum. It will solve the issue. (but also verify step 3 before deleting
all topic.log files)
Thanks
Rahul
> CorruptRecordException: Found record size 0 smaller than minimum record
> overhead
> --------------------------------------------------------------------------------
>
> Key: KAFKA-20911
> URL: https://issues.apache.org/jira/browse/KAFKA-20911
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 3.9.0
> Reporter: Santosh Kumar Sahu
> Priority: Major
>
> We were performing the sequential node reboot TC.
> kafka-0 pod was scheduled on worker-1. when worker-1 is rebooted, kafka-0 was
> also terminated. after node came up, the same pod was scheduled on worker-1
> but it is not coming to running state. it is on continuous restart and POD is
> not recovered.
> Pod-0 was restarting with below fatal exception
> {"version": "1.2.0", "timestamp": "2026-07-03T10:23:06.427+00:00",
> "severity": "error", "service_id": "pcc-data-message-bus-kf", "message":
> "Exiting Kafka due to fatal exception during startup.
> org.apache.kafka.common.errors.CorruptRecordException: Found record size 0
> smaller than minimum record overhead (14) in file
> /opt/kafka/data/logs/pcc-south-log-ccm-mfs-363-0/00000000000000202197.log.",
> "metadata": \{"pod_name":"dc8vpod2pcc-cn-message-bus-kf-0", "container_name":
> "messagebuskf", "namespace":"pcc"}, "extra_data": \{"file": "Kafka.scala",
> "line": "28"}}
> {"version": "1.2.0", "timestamp": "2026-07-03T10:23:06.427+00:00",
> "severity": "error", "service_id": "pcc-data-message-bus-kf", "message":
> "Exiting Kafka due to fatal exception during startup.
> org.apache.kafka.common.errors.CorruptRecordException: Found record size 0
> smaller than minimum record overhead (14) in file
> /opt/kafka/data/logs/pcc-south-log-ccm-mfs-363-0/00000000000000202197.log.",
> "metadata": \{"pod_name":"dc8vpod2pcc-cn-message-bus-kf-0", "container_name":
> "messagebuskf", "namespace":"pcc"}, "extra_data": \{"file": "Kafka.scala",
> "line": "28"}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
