[
https://issues.apache.org/jira/browse/KAFKA-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruimin MA updated KAFKA-21107:
------------------------------
Description:
String-valued class configuration fails when the TCCL cannot see kafka-clients
Type: Bug
Component: clients
Problem
Utils.loadClass() only uses the thread context class loader (TCCL) when it is
non-null. If that loader cannot find a configured class, Kafka fails without
trying its own defining class loader.
This can make a built-in serializer fail to load even though Kafka's class
loader can resolve it:
key.serializer=org.apache.kafka.common.serialization.StringSerializer
Observed trigger
{code:java}
CompletableFuture
.supplyAsync(() -> handleMessage(message), applicationExecutor)
.whenCompleteAsync((result, error) ->
kafkaTemplate.send(topic, key, serialize(result)))
.exceptionally(error -> null);
{code}
whenCompleteAsync() has no executor argument, so it normally runs on
ForkJoinPool.commonPool(). If this callback performs the first send,
KafkaTemplate lazily creates the producer on that worker. In a Spring Boot
executable JAR, the worker's TCCL may not see BOOT-INF/lib, causing
ConfigDef.parseType() to throw ConfigException for StringSerializer.
was:
String-valued class configuration fails when the context class loader cannot
see kafka-clients
Type: Bug
Component: clients
Environment
Original report: kafka-clients 3.0.0 in a Spring Boot executable JAR.
Reproduced on Kafka trunk at commit 56baca2b65251318a17154034a638b6f37654c26.
The minimal reproduction does not require Spring Boot or a Kafka broker.
Problem
Utils.loadClass() uses the thread context class loader (TCCL) whenever it is
non-null. If the TCCL cannot find a configured class, Kafka does not retry with
its own defining class loader.
Consequently, a string-valued class configuration can fail even though the
requested class is available to Kafka's class loader.
Example producer configuration:
key.serializer=org.apache.kafka.common.serialization.StringSerializer
The failure occurs during local configuration parsing, before the producer
connects to a broker:
ConfigException: Invalid value
org.apache.kafka.common.serialization.StringSerializer for configuration
key.serializer: Class org.apache.kafka.common.serialization.StringSerializer
could not be found.
Real-world trigger
The issue was observed in a Spring Boot executable JAR when the first
KafkaTemplate.send() call occurred in this pattern:
CompletableFuture
.supplyAsync(() -> handleMessage(message), applicationExecutor)
.whenCompleteAsync((result, exception) ->
kafkaTemplate.send(topic, key, serialize(result)))
.exceptionally(exception -> null);
Although supplyAsync() uses an explicit application executor,
whenCompleteAsync() does not receive one. Its callback therefore uses the
default asynchronous execution facility, normally ForkJoinPool.commonPool().
KafkaTemplate creates its producer lazily. When this callback performs the
first send, Kafka parses the serializer class name using the common-pool
worker's TCCL. In an executable Spring Boot JAR, that loader may not see
dependencies in BOOT-INF/lib, while the class loader that loaded Kafka can
resolve StringSerializer.
Relevant call path:
CompletableFuture.whenCompleteAsync
-> ForkJoinPool.commonPool worker
-> KafkaTemplate.send
-> DefaultKafkaProducerFactory.createProducer
-> KafkaProducer constructor
-> ConfigDef.parseType(Type.CLASS)
-> Utils.loadClass
-> ClassNotFoundException
The final exceptionally() can hide the failure from the calling code, but it is
not the cause.
Minimal reproduction
The issue can be reproduced without Spring Boot or a broker:
Confirm that Kafka's defining class loader can load StringSerializer.
Install a non-null TCCL that deliberately rejects only the StringSerializer
class name.
Parse that class name through ConfigDef.parseType(..., Type.CLASS).
Restore the original TCCL in a finally block.
ClassLoader original = Thread.currentThread().getContextClassLoader();
String className = StringSerializer.class.getName();
ClassLoader rejectingLoader = new ClassLoader(original) {
@Override
public Class<?> loadClass(String name) throws ClassNotFoundException {
if (className.equals(name)) {
throw new ClassNotFoundException(name);
}
return super.loadClass(name);
}
};
try {
Thread.currentThread().setContextClassLoader(rejectingLoader);
ConfigDef.parseType("key.serializer", className, ConfigDef.Type.CLASS);
} finally {
Thread.currentThread().setContextClassLoader(original);
}
Before the proposed fix, ConfigDef.parseType() throws ConfigException, wrapping
ClassNotFoundException. At the same time, this lookup succeeds:
Utils.class.getClassLoader().loadClass(className);
Expected behavior
Kafka should preserve TCCL precedence. If the initial lookup throws
ClassNotFoundException, Kafka should try its defining class loader before
reporting that the class is unavailable.
Initialization, linkage, and type-check failures should not trigger a fallback.
Workarounds
Configure StringSerializer.class instead of its class name.
Supply serializer instances to the producer factory.
Pass an explicit executor to whenCompleteAsync() whose TCCL can see the
application dependencies.
The first two workarounds avoid class-name resolution. The third only avoids
the triggering class-loader mismatch.
Proposed fix and validation
The proposed implementation retries the initial lookup with Kafka's defining
class loader only after ClassNotFoundException. It preserves TCCL precedence,
alias handling, and non-lookup failures.
A ConfigDefTest regression fails on the original implementation and passes with
the fix. Additional UtilsTest cases cover loader priority, fallback
diagnostics, a null TCCL, linkage errors, and initialization lookup failures.
Local validation:
126 tests passed across UtilsTest and ConfigDefTest.
Client Checkstyle, SpotBugs, and Spotless checks passed.
The fork CI Compile and Check (Merge Ref) job passed.
Prototype pull request:
https://github.com/3pacccccc/kafka/pull/1
> String-valued class configuration fails when the context class loader cannot
> see kafka-clients
> ----------------------------------------------------------------------------------------------
>
> Key: KAFKA-21107
> URL: https://issues.apache.org/jira/browse/KAFKA-21107
> Project: Kafka
> Issue Type: Bug
> Components: clients
> Affects Versions: 3.4.0
> Reporter: Ruimin MA
> Assignee: Ruimin MA
> Priority: Major
>
> String-valued class configuration fails when the TCCL cannot see kafka-clients
> Type: Bug
> Component: clients
> Problem
> Utils.loadClass() only uses the thread context class loader (TCCL) when it is
> non-null. If that loader cannot find a configured class, Kafka fails without
> trying its own defining class loader.
> This can make a built-in serializer fail to load even though Kafka's class
> loader can resolve it:
> key.serializer=org.apache.kafka.common.serialization.StringSerializer
> Observed trigger
> {code:java}
> CompletableFuture
> .supplyAsync(() -> handleMessage(message), applicationExecutor)
> .whenCompleteAsync((result, error) ->
> kafkaTemplate.send(topic, key, serialize(result)))
> .exceptionally(error -> null);
> {code}
> whenCompleteAsync() has no executor argument, so it normally runs on
> ForkJoinPool.commonPool(). If this callback performs the first send,
> KafkaTemplate lazily creates the producer on that worker. In a Spring Boot
> executable JAR, the worker's TCCL may not see BOOT-INF/lib, causing
> ConfigDef.parseType() to throw ConfigException for StringSerializer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
