Xi Yang created KAFKA-15591:
-------------------------------
Summary: Trogdor produce workload reports errors in KRaft mode
Key: KAFKA-15591
URL: https://issues.apache.org/jira/browse/KAFKA-15591
Project: Kafka
Issue Type: Bug
Environment: Linux
Reporter: Xi Yang
The Kafka benchmark in the Dacapo Benchmark Suite uses the Trogdor's exec mode
([https://github.com/dacapobench/dacapobench/pull/224)] to test the Kafka
broker.
I am trying to update the benchmark to use the KRaft protocol. We use single
Kafka instant that plays both controller and broker following the guide in
Kafka README.md
(https://github.com/apache/kafka#running-a-kafka-broker-in-kraft-mode).
However, the Trogdor producing workload (tests/spec/simple_produce_bench.json)
reports the NOT_LEADER_OR_FOLLOWER error. The errors are gone after many time
of retry. Is this caused by that in KRaft protocal, Kafka doesn't not elect
leaders immediately after a new topic created but rather do that on-demand
after receiving the first message on the topic? If this is the root cause, Is
there a way to ask Kafka to elect the leader after creating the topic?
{code:java}
// code placeholder
./bin/trogdor.sh agent -n node0 -c ./config/trogdor.conf --exec
./tests/spec/simple_produce_bench.json
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/xyang/code/kafka/tools/build/dependant-libs-2.13.12/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/xyang/code/kafka/trogdor/build/dependant-libs-2.13.12/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
Oct 12, 2023 12:30:50 AM org.glassfish.jersey.server.wadl.WadlFeature configure
WARNING: JAXBContext implementation could not be found. WADL feature is
disabled.
Oct 12, 2023 12:30:50 AM org.glassfish.jersey.internal.inject.Providers
checkProviderRuntime
WARNING: A provider org.apache.kafka.trogdor.agent.AgentRestResource registered
in SERVER runtime does not implement any provider interfaces applicable in the
SERVER runtime. Due to constraint configuration problems the provider
org.apache.kafka.trogdor.agent.AgentRestResource will be ignored.
Waiting for completion of task:{
"class" : "org.apache.kafka.trogdor.workload.ProduceBenchSpec",
"startMs" : 1697070650540,
"durationMs" : 10000000,
"producerNode" : "node0",
"bootstrapServers" : "localhost:9092",
"targetMessagesPerSec" : 10000,
"maxMessages" : 50000,
"keyGenerator" : {
"type" : "sequential",
"size" : 4,
"startOffset" : 0
},
"valueGenerator" : {
"type" : "constant",
"size" : 512,
"value" :
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="
},
"activeTopics" : {
"foo[1-3]" : {
"numPartitions" : 10,
"replicationFactor" : 1
}
},
"inactiveTopics" : {
"foo[4-5]" : {
"numPartitions" : 10,
"replicationFactor" : 1
}
},
"useConfiguredPartitioner" : false,
"skipFlush" : false
}
[2023-10-12 00:30:50,862] WARN [Producer clientId=producer-1] Got error produce
response with correlation id 6 on topic-partition foo1-8, retrying (2147483646
attempts left). Error: NOT_LEADER_OR_FOLLOWER
(org.apache.kafka.clients.producer.internals.Sender)
[2023-10-12 00:30:50,862] WARN [Producer clientId=producer-1] Received invalid
metadata error in produce request on partition foo1-8 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-10-12 00:30:50,870] WARN [Producer clientId=producer-1] Got error produce
response with correlation id 8 on topic-partition foo2-8, retrying (2147483646
attempts left). Error: NOT_LEADER_OR_FOLLOWER
(org.apache.kafka.clients.producer.internals.Sender)
[2023-10-12 00:30:50,870] WARN [Producer clientId=producer-1] Received invalid
metadata error in produce request on partition foo2-8 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-10-12 00:30:50,875] WARN [Producer clientId=producer-1] Got error produce
response with correlation id 9 on topic-partition foo3-8, retrying (2147483646
attempts left). Error: NOT_LEADER_OR_FOLLOWER
(org.apache.kafka.clients.producer.internals.Sender)
[2023-10-12 00:30:50,875] WARN [Producer clientId=producer-1] Received invalid
metadata error in produce request on partition foo3-8 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2023-10-12 00:30:50,876] WARN [Producer clientId=producer-1] Got error produce
response with correlation id 9 on topic-partition foo2-4, retrying (2147483646
attempts left). Error: NOT_LEADER_OR_FOLLOWER
(org.apache.kafka.clients.producer.internals.Sender)
[2023-10-12 00:30:50,876] WARN [Producer clientId=producer-1] Received invalid
metadata error in produce request on partition foo2-4 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
