Ruimin MA created KAFKA-21107:
---------------------------------
Summary: String-valued class configuration fails when the context
class loader cannot see kafka-clients
Key: KAFKA-21107
URL: https://issues.apache.org/jira/browse/KAFKA-21107
Project: Kafka
Issue Type: Bug
Components: clients
Affects Versions: 3.4.0
Reporter: Ruimin MA
Assignee: Ruimin MA
String-valued class configuration fails when the context class loader cannot
see kafka-clients
Type: Bug
Component: clients
Environment
Original report: kafka-clients 3.0.0 in a Spring Boot executable JAR.
Reproduced on Kafka trunk at commit 56baca2b65251318a17154034a638b6f37654c26.
The minimal reproduction does not require Spring Boot or a Kafka broker.
Problem
Utils.loadClass() uses the thread context class loader (TCCL) whenever it is
non-null. If the TCCL cannot find a configured class, Kafka does not retry with
its own defining class loader.
Consequently, a string-valued class configuration can fail even though the
requested class is available to Kafka's class loader.
Example producer configuration:
key.serializer=org.apache.kafka.common.serialization.StringSerializer
The failure occurs during local configuration parsing, before the producer
connects to a broker:
ConfigException: Invalid value
org.apache.kafka.common.serialization.StringSerializer for configuration
key.serializer: Class org.apache.kafka.common.serialization.StringSerializer
could not be found.
Real-world trigger
The issue was observed in a Spring Boot executable JAR when the first
KafkaTemplate.send() call occurred in this pattern:
CompletableFuture
.supplyAsync(() -> handleMessage(message), applicationExecutor)
.whenCompleteAsync((result, exception) ->
kafkaTemplate.send(topic, key, serialize(result)))
.exceptionally(exception -> null);
Although supplyAsync() uses an explicit application executor,
whenCompleteAsync() does not receive one. Its callback therefore uses the
default asynchronous execution facility, normally ForkJoinPool.commonPool().
KafkaTemplate creates its producer lazily. When this callback performs the
first send, Kafka parses the serializer class name using the common-pool
worker's TCCL. In an executable Spring Boot JAR, that loader may not see
dependencies in BOOT-INF/lib, while the class loader that loaded Kafka can
resolve StringSerializer.
Relevant call path:
CompletableFuture.whenCompleteAsync
-> ForkJoinPool.commonPool worker
-> KafkaTemplate.send
-> DefaultKafkaProducerFactory.createProducer
-> KafkaProducer constructor
-> ConfigDef.parseType(Type.CLASS)
-> Utils.loadClass
-> ClassNotFoundException
The final exceptionally() can hide the failure from the calling code, but it is
not the cause.
Minimal reproduction
The issue can be reproduced without Spring Boot or a broker:
Confirm that Kafka's defining class loader can load StringSerializer.
Install a non-null TCCL that deliberately rejects only the StringSerializer
class name.
Parse that class name through ConfigDef.parseType(..., Type.CLASS).
Restore the original TCCL in a finally block.
ClassLoader original = Thread.currentThread().getContextClassLoader();
String className = StringSerializer.class.getName();
ClassLoader rejectingLoader = new ClassLoader(original) {
@Override
public Class<?> loadClass(String name) throws ClassNotFoundException {
if (className.equals(name)) {
throw new ClassNotFoundException(name);
}
return super.loadClass(name);
}
};
try {
Thread.currentThread().setContextClassLoader(rejectingLoader);
ConfigDef.parseType("key.serializer", className, ConfigDef.Type.CLASS);
} finally {
Thread.currentThread().setContextClassLoader(original);
}
Before the proposed fix, ConfigDef.parseType() throws ConfigException, wrapping
ClassNotFoundException. At the same time, this lookup succeeds:
Utils.class.getClassLoader().loadClass(className);
Expected behavior
Kafka should preserve TCCL precedence. If the initial lookup throws
ClassNotFoundException, Kafka should try its defining class loader before
reporting that the class is unavailable.
Initialization, linkage, and type-check failures should not trigger a fallback.
Workarounds
Configure StringSerializer.class instead of its class name.
Supply serializer instances to the producer factory.
Pass an explicit executor to whenCompleteAsync() whose TCCL can see the
application dependencies.
The first two workarounds avoid class-name resolution. The third only avoids
the triggering class-loader mismatch.
Proposed fix and validation
The proposed implementation retries the initial lookup with Kafka's defining
class loader only after ClassNotFoundException. It preserves TCCL precedence,
alias handling, and non-lookup failures.
A ConfigDefTest regression fails on the original implementation and passes with
the fix. Additional UtilsTest cases cover loader priority, fallback
diagnostics, a null TCCL, linkage errors, and initialization lookup failures.
Local validation:
126 tests passed across UtilsTest and ConfigDefTest.
Client Checkstyle, SpotBugs, and Spotless checks passed.
The fork CI Compile and Check (Merge Ref) job passed.
Prototype pull request:
https://github.com/3pacccccc/kafka/pull/1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
