[
https://issues.apache.org/jira/browse/KAFKA-9613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17905673#comment-17905673
]
Teddy Yan edited comment on KAFKA-9613 at 12/14/24 5:25 AM:
------------------------------------------------------------
let me show the details, I have 3 servers Kafka cluster: master (partition 3
leader), worker01 (consumer), worker02 (partition 3 follower)
{code:java}
// code placeholder
kafka-topics --bootstrap-server localhost:9092 --topic df-flow --describe
Topic: df-flow TopicId: ZEV1NX-XRvCHlp7s3wRUKw PartitionCount: 6
ReplicationFactor: 2 Configs:
min.insync.replicas=1,segment.bytes=1073741824,retention.ms=14400000
Topic: df-flowPartition: 0Leader: 90883781Replicas: 90883781,608469468Isr:
90883781,608469468
Topic: df-flowPartition: 1Leader: 90883781Replicas: 90883781,181419666Isr:
90883781,181419666
Topic: df-flowPartition: 2Leader: 181419666Replicas: 181419666,608469468Isr:
181419666,608469468
Topic: df-flowPartition: 3Leader: 608469468Replicas: 608469468,181419666Isr:
608469468,181419666
Topic: df-flowPartition: 4Leader: 90883781Replicas: 90883781,608469468Isr:
90883781,608469468
Topic: df-flowPartition: 5Leader: 181419666Replicas: 181419666,90883781Isr:
181419666,90883781 {code}
kafka topic is df-flow
{code:java}
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group
df-classifydGROUP TOPIC PARTITION CURRENT-OFFSET
LOG-END-OFFSET LAG CONSUMER-ID
HOST CLIENT-ID
df-classifyd df-flow 0 64289212 64291422 2210
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 1 60414657 60415643 986
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 4 59941504 59946812 5308
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 5 59769487 59774060 4573
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 3 60380160 60623749
243589 worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 2 60711101 60714732 3631
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0 {code}
I was trying to break the kafka log on master (partition 3)
{code:java}
watch ls /pipedream/kafka/df-flow-3 -las
total 43320
4 drwxr-xr-x 2 kafka kafka 4096 Dec 14 05:09 .
4 drwxrwxr-x 85 kafka kafka 4096 Dec 14 05:14 ..
104 -rw-r--r-- 1 kafka kafka 105160 Dec 14 05:09
00000000000030533604.index
26156 -rw-r--r-- 1 kafka kafka 1073708343 Dec 14 05:09 00000000000030533604.log
4 -rw-r--r-- 1 kafka kafka 10 Dec 13 23:56
00000000000030533604.snapshot
140 -rw-r--r-- 1 kafka kafka 139428 Dec 14 05:09
00000000000030533604.timeindex
4 -rw-r--r-- 1 kafka kafka 10485760 Dec 14 05:14
00000000000061153545.index
16888 -rw-r--r-- 1 kafka kafka 17288401 Dec 14 05:14 00000000000061153545.log
4 -rw-r--r-- 1 kafka kafka 10 Dec 14 05:09
00000000000061153545.snapshot
4 -rw-r--r-- 1 kafka kafka 10485756 Dec 14 05:14
00000000000061153545.timeindex
4 -rwxr-xr-x 1 root root 96 Dec 14 05:02 break-log.sh
4 -rw-r--r-- 1 kafka kafka 15 Dec 14 05:06 leader-epoch-checkpoint
{code}
run this
{code:java}
#!/bin/bash
{ printf '\x00'; tail -c +0 00000000000030533604.log; } >
00000000000030533604.log {code}
after that,
I got errors in kafka logs:
{code:java}
worker02:
[2024-12-13 18:42:09,256] WARN No meta.properties file under dir
/pipedream/kafka/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2024-12-14 05:02:58,007] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:02:59,032] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:00,517] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:01,774] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:03,077] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:04,155] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:05,271] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:06,317] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:07,320] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:08,426] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:09,680] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:10,888] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:11,892] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:13,111] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:14,121] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:15,224] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:16,467] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:17,903] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:18,988] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:20,386] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:21,442] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:22,571] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:23,784] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:24,792] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:25,796] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:26,935] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:28,103] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:29,201] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:30,304] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:31,317] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:32,320] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:33,323] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:34,716] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:35,963] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:37,207] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:38,510] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:39,838] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:40,963] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:42,203] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:43,624] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:44,709] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
worker01:
[2024-12-13 18:42:09,188] WARN No meta.properties file under dir
/pipedream/kafka/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2024-12-13 18:45:10,068] WARN [ReplicaFetcher replicaId=90883781,
leaderId=608469468, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition __consumer_offsets-16. This error may be returned
transiently when the partition is being created or deleted, but it is not
expected to persist. (kafka.server.ReplicaFetcherThread)
[2024-12-13 18:45:10,068] WARN [ReplicaFetcher replicaId=90883781,
leaderId=608469468, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition __consumer_offsets-46. This error may be returned
transiently when the partition is being created or deleted, but it is not
expected to persist. (kafka.server.ReplicaFetcherThread)
[2024-12-13 18:45:10,068] WARN [ReplicaFetcher replicaId=90883781,
leaderId=608469468, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition __consumer_offsets-10. This error may be returned
transiently when the partition is being created or deleted, but it is not
expected to persist. (kafka.server.ReplicaFetcherThread)
[2024-12-13 18:45:10,068] WARN [ReplicaFetcher replicaId=90883781,
leaderId=608469468, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition __consumer_offsets-40. This error may be returned
transiently when the partition is being created or deleted, but it is not
expected to persist. (kafka.server.ReplicaFetcherThread)
[2024-12-13 18:45:10,068] WARN [ReplicaFetcher replicaId=90883781,
leaderId=608469468, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition __consumer_offsets-34. This error may be returned
transiently when the partition is being created or deleted, but it is not
expected to persist. (kafka.server.ReplicaFetcherThread)
[2024-12-13 18:45:10,068] WARN [ReplicaFetcher replicaId=90883781,
leaderId=608469468, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition __consumer_offsets-4. This error may be returned
transiently when the partition is being created or deleted, but it is not
expected to persist. (kafka.server.ReplicaFetcherThread)
[2024-12-13 18:45:10,068] WARN [ReplicaFetcher replicaId=90883781,
leaderId=608469468, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition __consumer_offsets-28. This error may be returned
transiently when the partition is being created or deleted, but it is not
expected to persist. (kafka.server.ReplicaFetcherThread)
[2024-12-13 18:45:10,068] WARN [ReplicaFetcher replicaId=90883781,
leaderId=608469468, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition __consumer_offsets-22. This error may be returned
transiently when the partition is being created or deleted, but it is not
expected to persist. (kafka.server.ReplicaFetcherThread)
master:
[2024-12-13 18:42:09,166] WARN No meta.properties file under dir
/pipedream/kafka/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2024-12-13 18:45:09,812] WARN [ReplicaFetcher replicaId=608469468,
leaderId=90883781, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition connect-database-cluster-configs-0. This error may be
returned transiently when the partition is being created or deleted, but it is
not expected to persist. (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:02:57,997] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:02:59,030] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:00,516] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:01,773] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:03,076] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:04,154] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:05,270] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:06,316] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:07,320] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:08,425] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:09,679] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:10,888] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:11,892] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:13,110] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:14,120] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:15,223] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:16,467] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:17,903] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:18,987] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:20,385] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:21,442] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:22,570] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:23,783] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:24,792] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:25,795] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:26,935] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:28,102] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:29,201] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:30,304] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:31,316] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:32,319] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:33,322] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:34,715] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:35,963] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:37,206] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:38,509] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:39,838] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:40,962] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:42,202] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:43,623] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:44,708] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager) {code}
and you can see the partition 3 offset stuck at 60380160, lag was increasing
{code:java}
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group
df-classifydGROUP TOPIC PARTITION CURRENT-OFFSET
LOG-END-OFFSET LAG CONSUMER-ID
HOST CLIENT-ID
df-classifyd df-flow 0 64289212 64291422 2210
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 1 60414657 60415643 986
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 4 59941504 59946812 5308
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 5 59769487 59774060 4573
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 3 60380160 60623749
243589 worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 2 60711101 60714732 3631
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0 {code}
I try to use this command to delete records: 60380161
{code:java}
cat delete-records.json
{
"partitions": [
{
"topic": "df-flow",
"partition": 3,
"offset": 60380161
}
],
"version": 1
} {code}
I got timeout error
{code:java}
kafka-delete-records --bootstrap-server localhost:9092 --offset-json-file
delete-records.json
Executing records delete operation
Records delete operation completed:
partition: df-flow-3 error: org.apache.kafka.common.errors.TimeoutException:
The request timed out. {code}
master (partition leader master), it just jump the offset from xxx159 to
xxxx161.
{code:java}
[2024-12-14 05:06:29,324] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:06:29,388] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 2097152 from consumer on partition df-flow-3:
PartitionData(fetchOffset=60380159, logStartOffset=-1, maxBytes=2097152,
currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)
(kafka.server.ReplicaManager)
[2024-12-14 05:06:29,889] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 2097152 from consumer on partition df-flow-3:
PartitionData(fetchOffset=60380159, logStartOffset=-1, maxBytes=2097152,
currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)
(kafka.server.ReplicaManager)
[2024-12-14 05:06:30,390] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 2097152 from consumer on partition df-flow-3:
PartitionData(fetchOffset=60380159, logStartOffset=-1, maxBytes=2097152,
currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)
(kafka.server.ReplicaManager)
[2024-12-14 05:06:30,406] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:06:30,892] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 2097152 from consumer on partition df-flow-3:
PartitionData(fetchOffset=60380159, logStartOffset=-1, maxBytes=2097152,
currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)
(kafka.server.ReplicaManager)
[2024-12-14 05:06:31,393] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 2097152 from consumer on partition df-flow-3:
PartitionData(fetchOffset=60380159, logStartOffset=-1, maxBytes=2097152,
currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)
(kafka.server.ReplicaManager)
[2024-12-14 05:06:31,409] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:06:31,894] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 2097152 from consumer on partition df-flow-3:
PartitionData(fetchOffset=60380159, logStartOffset=-1, maxBytes=2097152,
currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)
(kafka.server.ReplicaManager)
[2024-12-14 05:06:32,615] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380161, logStartOffset=60380161,
maxBytes=1048576, currentLeaderEpoch=Optional[0],
lastFetchedEpoch=Optional.empty) (kafka.server.ReplicaManager)
[2024-12-14 05:06:32,898] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 2097152 from consumer on partition df-flow-3:
PartitionData(fetchOffset=60380161, logStartOffset=-1, maxBytes=2097152,
currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)
(kafka.server.ReplicaManager)
[2024-12-14 05:06:33,399] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 2097152 from consumer on partition df-flow-3:
PartitionData(fetchOffset=60380161, logStartOffset=-1, maxBytes=2097152,
currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)
(kafka.server.ReplicaManager) {code}
worker02 (worker02 follower's logs)
{code:java}
[2024-12-14 05:06:31,410] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:06:32,605] WARN [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Reset fetch offset for partition df-flow-3
from 60380159 to current leader's start offset 60380161
(kafka.server.ReplicaFetcherThread)
[2024-12-14 05:06:32,615] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380161 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:06:33,659] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380161 (kafka.server.ReplicaFetcherThread) {code}
The partition lag was increasing, delete record didn't work.
was (Author: JIRAUSER307572):
let me show the details, I have 3 servers Kafka cluster: master, worker01,
worker02
{code:java}
// code placeholder
kafka-topics --bootstrap-server localhost:9092 --topic df-flow --describe
Topic: df-flow TopicId: ZEV1NX-XRvCHlp7s3wRUKw PartitionCount: 6
ReplicationFactor: 2 Configs:
min.insync.replicas=1,segment.bytes=1073741824,retention.ms=14400000
Topic: df-flowPartition: 0Leader: 90883781Replicas: 90883781,608469468Isr:
90883781,608469468
Topic: df-flowPartition: 1Leader: 90883781Replicas: 90883781,181419666Isr:
90883781,181419666
Topic: df-flowPartition: 2Leader: 181419666Replicas: 181419666,608469468Isr:
181419666,608469468
Topic: df-flowPartition: 3Leader: 608469468Replicas: 608469468,181419666Isr:
608469468,181419666
Topic: df-flowPartition: 4Leader: 90883781Replicas: 90883781,608469468Isr:
90883781,608469468
Topic: df-flowPartition: 5Leader: 181419666Replicas: 181419666,90883781Isr:
181419666,90883781 {code}
kafka topic is df-flow
{code:java}
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group
df-classifydGROUP TOPIC PARTITION CURRENT-OFFSET
LOG-END-OFFSET LAG CONSUMER-ID
HOST CLIENT-ID
df-classifyd df-flow 0 64289212 64291422 2210
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 1 60414657 60415643 986
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 4 59941504 59946812 5308
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 5 59769487 59774060 4573
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 3 60380160 60623749
243589 worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 2 60711101 60714732 3631
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0 {code}
I was trying to break the kafka log on master (partition 3)
{code:java}
watch ls /pipedream/kafka/df-flow-3 -las
total 43320
4 drwxr-xr-x 2 kafka kafka 4096 Dec 14 05:09 .
4 drwxrwxr-x 85 kafka kafka 4096 Dec 14 05:14 ..
104 -rw-r--r-- 1 kafka kafka 105160 Dec 14 05:09
00000000000030533604.index
26156 -rw-r--r-- 1 kafka kafka 1073708343 Dec 14 05:09 00000000000030533604.log
4 -rw-r--r-- 1 kafka kafka 10 Dec 13 23:56
00000000000030533604.snapshot
140 -rw-r--r-- 1 kafka kafka 139428 Dec 14 05:09
00000000000030533604.timeindex
4 -rw-r--r-- 1 kafka kafka 10485760 Dec 14 05:14
00000000000061153545.index
16888 -rw-r--r-- 1 kafka kafka 17288401 Dec 14 05:14 00000000000061153545.log
4 -rw-r--r-- 1 kafka kafka 10 Dec 14 05:09
00000000000061153545.snapshot
4 -rw-r--r-- 1 kafka kafka 10485756 Dec 14 05:14
00000000000061153545.timeindex
4 -rwxr-xr-x 1 root root 96 Dec 14 05:02 break-log.sh
4 -rw-r--r-- 1 kafka kafka 15 Dec 14 05:06 leader-epoch-checkpoint
{code}
run this
{code:java}
#!/bin/bash
{ printf '\x00'; tail -c +0 00000000000030533604.log; } >
00000000000030533604.log {code}
after that,
I got errors in kafka logs:
{code:java}
worker02:
[2024-12-13 18:42:09,256] WARN No meta.properties file under dir
/pipedream/kafka/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2024-12-14 05:02:58,007] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:02:59,032] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:00,517] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:01,774] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:03,077] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:04,155] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:05,271] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:06,317] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:07,320] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:08,426] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:09,680] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:10,888] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:11,892] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:13,111] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:14,121] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:15,224] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:16,467] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:17,903] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:18,988] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:20,386] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:21,442] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:22,571] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:23,784] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:24,792] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:25,796] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:26,935] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:28,103] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:29,201] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:30,304] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:31,317] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:32,320] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:33,323] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:34,716] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:35,963] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:37,207] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:38,510] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:39,838] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:40,963] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:42,203] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:43,624] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:03:44,709] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
worker01:
[2024-12-13 18:42:09,188] WARN No meta.properties file under dir
/pipedream/kafka/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2024-12-13 18:45:10,068] WARN [ReplicaFetcher replicaId=90883781,
leaderId=608469468, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition __consumer_offsets-16. This error may be returned
transiently when the partition is being created or deleted, but it is not
expected to persist. (kafka.server.ReplicaFetcherThread)
[2024-12-13 18:45:10,068] WARN [ReplicaFetcher replicaId=90883781,
leaderId=608469468, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition __consumer_offsets-46. This error may be returned
transiently when the partition is being created or deleted, but it is not
expected to persist. (kafka.server.ReplicaFetcherThread)
[2024-12-13 18:45:10,068] WARN [ReplicaFetcher replicaId=90883781,
leaderId=608469468, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition __consumer_offsets-10. This error may be returned
transiently when the partition is being created or deleted, but it is not
expected to persist. (kafka.server.ReplicaFetcherThread)
[2024-12-13 18:45:10,068] WARN [ReplicaFetcher replicaId=90883781,
leaderId=608469468, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition __consumer_offsets-40. This error may be returned
transiently when the partition is being created or deleted, but it is not
expected to persist. (kafka.server.ReplicaFetcherThread)
[2024-12-13 18:45:10,068] WARN [ReplicaFetcher replicaId=90883781,
leaderId=608469468, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition __consumer_offsets-34. This error may be returned
transiently when the partition is being created or deleted, but it is not
expected to persist. (kafka.server.ReplicaFetcherThread)
[2024-12-13 18:45:10,068] WARN [ReplicaFetcher replicaId=90883781,
leaderId=608469468, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition __consumer_offsets-4. This error may be returned
transiently when the partition is being created or deleted, but it is not
expected to persist. (kafka.server.ReplicaFetcherThread)
[2024-12-13 18:45:10,068] WARN [ReplicaFetcher replicaId=90883781,
leaderId=608469468, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition __consumer_offsets-28. This error may be returned
transiently when the partition is being created or deleted, but it is not
expected to persist. (kafka.server.ReplicaFetcherThread)
[2024-12-13 18:45:10,068] WARN [ReplicaFetcher replicaId=90883781,
leaderId=608469468, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition __consumer_offsets-22. This error may be returned
transiently when the partition is being created or deleted, but it is not
expected to persist. (kafka.server.ReplicaFetcherThread)
master:
[2024-12-13 18:42:09,166] WARN No meta.properties file under dir
/pipedream/kafka/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2024-12-13 18:45:09,812] WARN [ReplicaFetcher replicaId=608469468,
leaderId=90883781, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the
leader for partition connect-database-cluster-configs-0. This error may be
returned transiently when the partition is being created or deleted, but it is
not expected to persist. (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:02:57,997] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:02:59,030] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:00,516] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:01,773] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:03,076] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:04,154] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:05,270] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:06,316] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:07,320] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:08,425] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:09,679] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:10,888] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:11,892] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:13,110] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:14,120] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:15,223] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:16,467] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:17,903] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:18,987] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:20,385] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:21,442] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:22,570] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:23,783] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:24,792] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:25,795] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:26,935] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:28,102] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:29,201] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:30,304] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:31,316] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:32,319] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:33,322] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:34,715] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:35,963] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:37,206] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:38,509] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:39,838] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:40,962] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:42,202] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:43,623] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:03:44,708] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager) {code}
and you can see the partition 3 offset stuck at 60380160:
{code:java}
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group
df-classifydGROUP TOPIC PARTITION CURRENT-OFFSET
LOG-END-OFFSET LAG CONSUMER-ID
HOST CLIENT-ID
df-classifyd df-flow 0 64289212 64291422 2210
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 1 60414657 60415643 986
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 4 59941504 59946812 5308
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 5 59769487 59774060 4573
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 3 60380160 60623749
243589 worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0
df-classifyd df-flow 2 60711101 60714732 3631
worker01-classifyd-0-0-27095c61-2471-4fec-8cd0-f707854f6636
/10.253.2.75 worker01-classifyd-0-0 {code}
I try to use this command to delete records: 60380161
{code:java}
cat delete-records.json
{
"partitions": [
{
"topic": "df-flow",
"partition": 3,
"offset": 60380161
}
],
"version": 1
} {code}
I got timeout error
{code:java}
kafka-delete-records --bootstrap-server localhost:9092 --offset-json-file
delete-records.json
Executing records delete operation
Records delete operation completed:
partition: df-flow-3 error: org.apache.kafka.common.errors.TimeoutException:
The request timed out. {code}
master (partition leader master)
{code:java}
[2024-12-14 05:06:29,324] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:06:29,388] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 2097152 from consumer on partition df-flow-3:
PartitionData(fetchOffset=60380159, logStartOffset=-1, maxBytes=2097152,
currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)
(kafka.server.ReplicaManager)
[2024-12-14 05:06:29,889] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 2097152 from consumer on partition df-flow-3:
PartitionData(fetchOffset=60380159, logStartOffset=-1, maxBytes=2097152,
currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)
(kafka.server.ReplicaManager)
[2024-12-14 05:06:30,390] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 2097152 from consumer on partition df-flow-3:
PartitionData(fetchOffset=60380159, logStartOffset=-1, maxBytes=2097152,
currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)
(kafka.server.ReplicaManager)
[2024-12-14 05:06:30,406] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:06:30,892] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 2097152 from consumer on partition df-flow-3:
PartitionData(fetchOffset=60380159, logStartOffset=-1, maxBytes=2097152,
currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)
(kafka.server.ReplicaManager)
[2024-12-14 05:06:31,393] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 2097152 from consumer on partition df-flow-3:
PartitionData(fetchOffset=60380159, logStartOffset=-1, maxBytes=2097152,
currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)
(kafka.server.ReplicaManager)
[2024-12-14 05:06:31,409] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380159, logStartOffset=30533699,
maxBytes=1048576, currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional[0])
(kafka.server.ReplicaManager)
[2024-12-14 05:06:31,894] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 2097152 from consumer on partition df-flow-3:
PartitionData(fetchOffset=60380159, logStartOffset=-1, maxBytes=2097152,
currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)
(kafka.server.ReplicaManager)
[2024-12-14 05:06:32,615] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 1048576 from replica [181419666] on partition
df-flow-3: PartitionData(fetchOffset=60380161, logStartOffset=60380161,
maxBytes=1048576, currentLeaderEpoch=Optional[0],
lastFetchedEpoch=Optional.empty) (kafka.server.ReplicaManager)
[2024-12-14 05:06:32,898] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 2097152 from consumer on partition df-flow-3:
PartitionData(fetchOffset=60380161, logStartOffset=-1, maxBytes=2097152,
currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)
(kafka.server.ReplicaManager)
[2024-12-14 05:06:33,399] ERROR [ReplicaManager broker=608469468] Error
processing fetch with max size 2097152 from consumer on partition df-flow-3:
PartitionData(fetchOffset=60380161, logStartOffset=-1, maxBytes=2097152,
currentLeaderEpoch=Optional[0], lastFetchedEpoch=Optional.empty)
(kafka.server.ReplicaManager) {code}
worker02 (worker02 follower's logs)
{code:java}
[2024-12-14 05:06:31,410] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380159 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:06:32,605] WARN [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Reset fetch offset for partition df-flow-3
from 60380159 to current leader's start offset 60380161
(kafka.server.ReplicaFetcherThread)
[2024-12-14 05:06:32,615] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380161 (kafka.server.ReplicaFetcherThread)
[2024-12-14 05:06:33,659] ERROR [ReplicaFetcher replicaId=181419666,
leaderId=608469468, fetcherId=0] Error for partition df-flow-3 at offset
60380161 (kafka.server.ReplicaFetcherThread) {code}
> CorruptRecordException: Found record size 0 smaller than minimum record
> overhead
> --------------------------------------------------------------------------------
>
> Key: KAFKA-9613
> URL: https://issues.apache.org/jira/browse/KAFKA-9613
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 2.6.2
> Reporter: Amit Khandelwal
> Assignee: hudeqi
> Priority: Major
> Attachments: image-2024-11-13-14-02-45-768.png
>
>
> 20200224;21:01:38: [2020-02-24 21:01:38,615] ERROR [ReplicaManager broker=0]
> Error processing fetch with max size 1048576 from consumer on partition
> SANDBOX.BROKER.NEWORDER-0: (fetchOffset=211886, logStartOffset=-1,
> maxBytes=1048576, currentLeaderEpoch=Optional.empty)
> (kafka.server.ReplicaManager)
> 20200224;21:01:38: org.apache.kafka.common.errors.CorruptRecordException:
> Found record size 0 smaller than minimum record overhead (14) in file
> /data/tmp/kafka-topic-logs/SANDBOX.BROKER.NEWORDER-0/00000000000000000000.log.
> 20200224;21:05:48: [2020-02-24 21:05:48,711] INFO [GroupMetadataManager
> brokerId=0] Removed 0 expired offsets in 1 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)
> 20200224;21:10:22: [2020-02-24 21:10:22,204] INFO [GroupCoordinator 0]:
> Member
> xxxxxxxx_011-9e61d2c9-ce5a-4231-bda1-f04e6c260dc0-StreamThread-1-consumer-27768816-ee87-498f-8896-191912282d4f
> in group yyyyyyyyy_011 has failed, removing it from the group
> (kafka.coordinator.group.GroupCoordinator)
>
> [https://stackoverflow.com/questions/60404510/kafka-broker-issue-replica-manager-with-max-size#]
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
