This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 967614e1ebf9 CAMEL-27130: Camel JBang downloads the serde jars of the
Kafka schema registry vendors, and knows --repos=confluent (#30296)
967614e1ebf9 is described below
commit 967614e1ebf97842739f7e8d01f2178cc306639e
Author: Claus Ibsen <[email protected]>
AuthorDate: Tue Nov 17 21:47:43 2026 +0100
CAMEL-27130: Camel JBang downloads the serde jars of the Kafka schema
registry vendors, and knows --repos=confluent (#30296)
Camel JBang downloads the serde jars of the chosen schemaRegistryVendor, so
no --dep is needed, and camel run --repos=confluent adds the Confluent
repository (the Confluent serdes are not on Maven Central).
- camel-kamelet-main maps every serializer and deserializer class that
schemaRegistryVendor/schemaFormat set (Confluent, Apicurio, AWS Glue,
Azure) to its Maven artifact. camel-kafka already resolves these classes
through the CamelContext ClassResolver, which under Camel JBang downloads
the jar.
- A mapping can list several artifacts that are downloaded together as one
dependency graph: the Azure serde comes with the azure-core of the Azure
SDK BOM, as its own azure-core 1.35 fails with NoSuchMethodError in
azure-identity.
- The class resolver does not retry a known dependency that failed to
download, so a missing Confluent repository is reported once.
- When the serde of the schemaRegistryVendor cannot be loaded, the producer,
consumer and share consumer fail at start with the Maven artifact to add
(only a WARN with a custom client factory).
- parent/pom.xml: confluent-serializer-version 8.3.3,
aws-glue-schema-registry-version 2.0.2,
azure-schemaregistry-kafka-avro-version 1.1.4. The Apicurio, AWS Glue and
Azure serdes are test dependencies of camel-kafka, and a test loads every
class schemaRegistryVendor sets, so a renamed class fails the build.
- The schema format is normalized once, and the consumer checks the
deserializer once instead of for each consumer.
- Docs: the known repository names in camel-jbang-running.adoc and the
camel-kafka Schema registry section.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Claus Ibsen <[email protected]>
---
.../apache/camel/catalog/docs/kafka-component.adoc | 13 +-
.../component/kafka/KafkaClientConfiguration.java | 29 +++++
.../camel/component/kafka/KafkaSchemaRegistry.java | 67 ++++++++--
.../component/kafka/share/KafkaShareConsumer.java | 4 +-
.../component/kafka/share/KafkaShareEndpoint.java | 19 +++
.../share/KafkaShareSchemaRegistrySerdeTest.java | 76 +++++++++++
components/camel-kafka/pom.xml | 34 +++++
.../camel-kafka/src/main/docs/kafka-component.adoc | 13 +-
.../camel/component/kafka/KafkaConsumer.java | 7 +-
.../camel/component/kafka/KafkaEndpoint.java | 19 +++
.../camel/component/kafka/KafkaProducer.java | 1 +
.../KafkaSchemaRegistryVendorClassesTest.java | 56 +++++++++
.../kafka/KafkaSchemaRegistryVendorTest.java | 139 +++++++++++++++++++++
.../modules/ROOT/pages/camel-jbang-running.adoc | 13 ++
dsl/camel-kamelet-main/pom.xml | 13 ++
.../DependencyDownloaderClassResolver.java | 72 ++++++-----
.../DependencyDownloaderPropertiesComponent.java | 22 +---
...ependencyDownloaderPropertyBindingListener.java | 16 +--
.../main/download/JavaKnownImportsDownloader.java | 6 +-
.../main/download/KnownDependenciesResolver.java | 65 +++++++++-
.../main/known-third-party-libraries.properties | 36 +++++-
.../download/KnownDependenciesResolverTest.java | 80 ++++++++++++
.../MavenDependencyDownloaderBatchTest.java | 16 +++
parent/pom.xml | 3 +
.../main/resources/known-maven-repos.properties | 3 +-
.../maven/DefaultRepositoryResolverTest.java | 48 +++++++
.../maven/packaging/PrepareKameletMainMojo.java | 135 +++++++++++++-------
27 files changed, 869 insertions(+), 136 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/kafka-component.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/kafka-component.adoc
index 8df8afd264f4..406fa76ad9dd 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/kafka-component.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/kafka-component.adoc
@@ -1000,7 +1000,7 @@ Leave `schemaRegistryVendor` unset to configure the
serializer, deserializer and
The body of the message to send must be a value that the serializer of the
vendor accepts, such as an Avro `GenericRecord` or a class generated from the
Avro schema.
-IMPORTANT: Camel does not include the serde libraries of the vendors: add the
serde jar of the vendor and format to your application.
+IMPORTANT: Camel does not include the serde libraries of the vendors: add the
serde jar of the vendor and format to your application (Camel JBang downloads
it, see below).
The Confluent serde jars are not in Maven Central: add the Confluent Maven
repository `https://packages.confluent.io/maven/`, or have the jars in a
repository of your own.
[width="100%",cols="1m,4",options="header"]
@@ -1026,17 +1026,21 @@ For example with Maven, for Confluent and Avro:
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
- <version>8.0.0</version>
+ <version>8.3.3</version>
</dependency>
----
-And with Camel JBang:
+Camel JBang downloads the serde jar of the vendor and format when the producer
or consumer starts, so you do not add it with `--dep`.
+The Confluent serde jars are not in Maven Central, so add the Confluent Maven
repository with `--repos=confluent` (Camel JBang knows the `confluent`
repository by name):
[source,bash]
----
-camel run MyRoute.java --repos=https://packages.confluent.io/maven/
--dep=io.confluent:kafka-avro-serializer:8.0.0
+camel run MyRoute.java --repos=confluent
----
+When the serializer or deserializer of the vendor cannot be loaded (the jar is
not on the classpath, or Camel JBang cannot download it, such as a Confluent
jar without `--repos=confluent`),
+the producer or consumer fails to start with a message that names the Maven
artifact to add.
+
*Confluent Schema Registry*
NOTE: Supports Avro, JSON Schema and Protobuf. The serde jars are only in the
Confluent Maven repository, not in Maven Central. The Avro serde and the
registry client are licensed under the Apache License 2.0; the JSON Schema and
Protobuf serdes are also listed under the Confluent Community License, so check
their license for your use. Registries that implement the Confluent registry
API, such as Karapace and Redpanda, work with `confluent` too.
@@ -1151,6 +1155,7 @@ The Azure serde reads `auto.register.schemas`,
`use.azure.credential` and `speci
The dependencies of `azure-schemaregistry-kafka-avro` ask for different
versions of `com.azure:azure-core`, and Maven picks the older one, which lacks
`TokenRequestContext.isCaeEnabled()` that Azure Identity calls: the first send
then fails with a `NoSuchMethodError`.
Add a recent `com.azure:azure-core` next to the serde, or import the
`com.azure:azure-sdk-bom`.
+Camel JBang downloads the `com.azure:azure-core` of the Azure SDK BOM together
with the serde.
The options to connect to Event Hubs with the Kafka protocol are left out of
the example.
[tabs]
diff --git
a/components/camel-kafka-common/src/main/java/org/apache/camel/component/kafka/KafkaClientConfiguration.java
b/components/camel-kafka-common/src/main/java/org/apache/camel/component/kafka/KafkaClientConfiguration.java
index e8c02cea5149..e2bd89a5d56e 100644
---
a/components/camel-kafka-common/src/main/java/org/apache/camel/component/kafka/KafkaClientConfiguration.java
+++
b/components/camel-kafka-common/src/main/java/org/apache/camel/component/kafka/KafkaClientConfiguration.java
@@ -43,6 +43,7 @@ import org.apache.camel.support.jsse.TrustManagersParameters;
import org.apache.camel.util.ObjectHelper;
import org.apache.camel.util.StringHelper;
import org.apache.kafka.clients.CommonClientConfigs;
+import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.config.SaslConfigs;
import org.apache.kafka.common.config.SslConfigs;
import org.apache.kafka.common.config.internals.BrokerSecurityConfigs;
@@ -472,6 +473,34 @@ public abstract class KafkaClientConfiguration implements
Cloneable, HeaderFilte
}
}
+ /**
+ * Checks the value serializer or deserializer that the
schemaRegistryVendor sets. The endpoint replaces the class
+ * names of the serializers and deserializers by the classes that the
class resolver of the CamelContext loads
+ * (which downloads the jar with Camel JBang), so the class name of the
vendor that is still a String is a class
+ * that none of the class loaders of Camel and Kafka has, and Kafka would
fail to load it too.
+ *
+ * @param props the properties of the client, after the endpoint replaced
the class names by the classes
+ * @param key value.serializer or value.deserializer
+ * @return a message that names the Maven artifact of the class to
add, or null if the class was loaded, no
+ * schemaRegistryVendor is set, or the class is not the one
of the vendor (such as an explicit
+ * valueSerializer)
+ */
+ public String missingSchemaRegistrySerde(Properties props, String key) {
+ if (ObjectHelper.isEmpty(schemaRegistryVendor) || !(props.get(key)
instanceof String name)) {
+ return null;
+ }
+ String vendor = resolveSchemaRegistryVendor();
+ boolean serializer =
ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG.equals(key);
+ String expected = serializer
+ ? KafkaSchemaRegistry.serializer(vendor, getSchemaFormat())
+ : KafkaSchemaRegistry.deserializer(vendor, getSchemaFormat());
+ if (!expected.equals(name.trim())) {
+ return null;
+ }
+ return KafkaSchemaRegistry.missingSerdeMessage(vendor,
getSchemaFormat(),
+ serializer ? "serializer" : "deserializer", expected);
+ }
+
private String resolveSchemaRegistryVendor() {
return KafkaSchemaRegistry.vendor(getSchemaRegistryVendor(),
getSchemaFormat());
}
diff --git
a/components/camel-kafka-common/src/main/java/org/apache/camel/component/kafka/KafkaSchemaRegistry.java
b/components/camel-kafka-common/src/main/java/org/apache/camel/component/kafka/KafkaSchemaRegistry.java
index b45bf1aab155..ba2cd9216034 100644
---
a/components/camel-kafka-common/src/main/java/org/apache/camel/component/kafka/KafkaSchemaRegistry.java
+++
b/components/camel-kafka-common/src/main/java/org/apache/camel/component/kafka/KafkaSchemaRegistry.java
@@ -29,8 +29,8 @@ import org.apache.kafka.clients.producer.ProducerConfig;
* <p/>
* The serializer and deserializer are class names and the properties are keys
of the vendor serde libraries, so that
* Camel does not depend on any vendor library: the user adds the serde jar of
the vendor. The keys and class names are
- * the ones of the vendor libraries (Confluent kafka-avro-serializer 8.0,
Apicurio Registry 3.x serdes, AWS Glue
- * schema-registry-serde 1.1 and Azure azure-schemaregistry-kafka-avro 1.1).
+ * the ones of the vendor libraries (Confluent kafka-avro-serializer 8.x,
Apicurio Registry 3.x serdes, AWS Glue
+ * schema-registry-serde 1.1 and 2.0, and Azure
azure-schemaregistry-kafka-avro 1.1).
*/
final class KafkaSchemaRegistry {
@@ -112,7 +112,10 @@ final class KafkaSchemaRegistry {
* The value serializer class name of the vendor and format.
*/
static String serializer(String vendor, String format) {
- String f = format(format);
+ return serializerOf(vendor, format(format));
+ }
+
+ private static String serializerOf(String vendor, String f) {
return switch (vendor) {
case CONFLUENT -> switch (f) {
case JSON ->
"io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer";
@@ -133,7 +136,10 @@ final class KafkaSchemaRegistry {
* The value deserializer class name of the vendor and format.
*/
static String deserializer(String vendor, String format) {
- String f = format(format);
+ return deserializerOf(vendor, format(format));
+ }
+
+ private static String deserializerOf(String vendor, String f) {
return switch (vendor) {
case CONFLUENT -> switch (f) {
case JSON ->
"io.confluent.kafka.serializers.json.KafkaJsonSchemaDeserializer";
@@ -150,6 +156,50 @@ final class KafkaSchemaRegistry {
};
}
+ /**
+ * The Maven artifact (groupId:artifactId) of the serializer and
deserializer of the vendor and format.
+ */
+ static String artifact(String vendor, String format) {
+ String f = format(format);
+ return switch (vendor) {
+ case CONFLUENT -> switch (f) {
+ case JSON -> "io.confluent:kafka-json-schema-serializer";
+ case PROTOBUF -> "io.confluent:kafka-protobuf-serializer";
+ default -> "io.confluent:kafka-avro-serializer";
+ };
+ case APICURIO -> switch (f) {
+ case JSON ->
"io.apicurio:apicurio-registry-jsonschema-serde-kafka";
+ case PROTOBUF ->
"io.apicurio:apicurio-registry-protobuf-serde-kafka";
+ default -> "io.apicurio:apicurio-registry-avro-serde-kafka";
+ };
+ case AWS_GLUE -> "software.amazon.glue:schema-registry-serde";
+ default -> "com.microsoft.azure:azure-schemaregistry-kafka-avro";
+ };
+ }
+
+ /**
+ * The message for a serializer or deserializer of the vendor that cannot
be loaded, which names the Maven artifact
+ * to add (and for Confluent, the repository).
+ *
+ * @param kind serializer or deserializer
+ * @param className the class that cannot be loaded
+ */
+ static String missingSerdeMessage(String vendor, String format, String
kind, String className) {
+ StringBuilder sb = new StringBuilder(256);
+ sb.append("Cannot load the value ").append(kind).append('
').append(className)
+ .append(" of schemaRegistryVendor ").append(vendor).append("
and schemaFormat ").append(format(format))
+ .append(". Add the Maven artifact ").append(artifact(vendor,
format)).append(" to the classpath");
+ switch (vendor) {
+ case CONFLUENT -> sb.append(". The Confluent serde jars are not in
Maven Central but in the Confluent Maven"
+ + " repository
https://packages.confluent.io/maven/ (with Camel JBang:"
+ + " camel run --repos=confluent).");
+ case APICURIO -> sb.append(" (version 3.x).");
+ case AZURE -> sb.append(", with a recent com.azure:azure-core (its
dependencies ask for an older one).");
+ default -> sb.append('.');
+ }
+ return sb.toString();
+ }
+
/**
* Adds the registry URL under the key of the vendor. Without a vendor,
the URL is set as schema.registry.url, the
* key of Confluent and of the registries that follow its serde
configuration.
@@ -174,14 +224,15 @@ final class KafkaSchemaRegistry {
* Sets the value serializer (unless the user set one) and the producer
properties of the vendor.
*/
static void applyProducer(Properties props, String vendor, String format,
boolean explicitSerializer) {
+ String f = format(format);
if (!explicitSerializer) {
- props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
serializer(vendor, format));
+ props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
serializerOf(vendor, f));
}
switch (vendor) {
// Confluent registers the schemas by default
(auto.register.schemas=true)
case APICURIO -> props.put(APICURIO_AUTO_REGISTER, Boolean.TRUE);
case AWS_GLUE -> {
- props.put(GLUE_DATA_FORMAT,
format(format).toUpperCase(Locale.ROOT));
+ props.put(GLUE_DATA_FORMAT, f.toUpperCase(Locale.ROOT));
props.put(GLUE_AUTO_REGISTRATION, Boolean.TRUE);
}
case AZURE -> {
@@ -200,10 +251,10 @@ final class KafkaSchemaRegistry {
*/
static void applyConsumer(
Properties props, String vendor, String format, boolean
specificAvroReader, boolean explicitDeserializer) {
+ String f = format(format);
if (!explicitDeserializer) {
- props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
deserializer(vendor, format));
+ props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
deserializerOf(vendor, f));
}
- String f = format(format);
boolean avro = AVRO.equals(f);
switch (vendor) {
case APICURIO -> {
diff --git
a/components/camel-kafka-share/src/main/java/org/apache/camel/component/kafka/share/KafkaShareConsumer.java
b/components/camel-kafka-share/src/main/java/org/apache/camel/component/kafka/share/KafkaShareConsumer.java
index ae6749392178..b81a9a64463c 100644
---
a/components/camel-kafka-share/src/main/java/org/apache/camel/component/kafka/share/KafkaShareConsumer.java
+++
b/components/camel-kafka-share/src/main/java/org/apache/camel/component/kafka/share/KafkaShareConsumer.java
@@ -99,7 +99,9 @@ public class KafkaShareConsumer extends DefaultConsumer {
}
// the share consumer rejects the consumer group options, fail now
rather than on every attempt to create it
- KafkaShareConfiguration.validateShareConsumerProperties(getProps());
+ Properties props = getProps();
+ KafkaShareConfiguration.validateShareConsumerProperties(props);
+ endpoint.checkSchemaRegistrySerde(props);
executor = endpoint.createExecutor(this);
diff --git
a/components/camel-kafka-share/src/main/java/org/apache/camel/component/kafka/share/KafkaShareEndpoint.java
b/components/camel-kafka-share/src/main/java/org/apache/camel/component/kafka/share/KafkaShareEndpoint.java
index 30093f15cf05..9bdef3f0e581 100644
---
a/components/camel-kafka-share/src/main/java/org/apache/camel/component/kafka/share/KafkaShareEndpoint.java
+++
b/components/camel-kafka-share/src/main/java/org/apache/camel/component/kafka/share/KafkaShareEndpoint.java
@@ -179,6 +179,25 @@ public class KafkaShareEndpoint extends DefaultEndpoint
implements MultipleConsu
}
}
+ /**
+ * Fails when the value deserializer that the schemaRegistryVendor sets
cannot be loaded, with a message that names
+ * the Maven artifact to add, rather than when Kafka fails to load it.
With a custom
+ * {@link KafkaShareClientFactory}, which may create the client without
loading the class by its name, only logs the
+ * message as a WARN.
+ *
+ * @param props the properties of the share consumer, after {@link
#updateClassProperties(Properties)}
+ */
+ void checkSchemaRegistrySerde(Properties props) {
+ String message = configuration.missingSchemaRegistrySerde(props,
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG);
+ if (message != null) {
+ if (kafkaShareClientFactory == null
+ || kafkaShareClientFactory.getClass() ==
DefaultKafkaShareClientFactory.class) {
+ throw new IllegalStateException(message);
+ }
+ LOG.warn("{}", message);
+ }
+ }
+
ExecutorService createExecutor(Object source) {
return
getCamelContext().getExecutorServiceManager().newFixedThreadPool(source,
"KafkaShareConsumer[" + configuration.getTopic() + "]",
configuration.getConsumersCount());
diff --git
a/components/camel-kafka-share/src/test/java/org/apache/camel/component/kafka/share/KafkaShareSchemaRegistrySerdeTest.java
b/components/camel-kafka-share/src/test/java/org/apache/camel/component/kafka/share/KafkaShareSchemaRegistrySerdeTest.java
new file mode 100644
index 000000000000..207fd4f2aa5a
--- /dev/null
+++
b/components/camel-kafka-share/src/test/java/org/apache/camel/component/kafka/share/KafkaShareSchemaRegistrySerdeTest.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.kafka.share;
+
+import java.util.Properties;
+
+import org.apache.camel.CamelContext;
+import org.apache.camel.Consumer;
+import org.apache.camel.impl.DefaultCamelContext;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.junit.jupiter.api.Test;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatCode;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+/**
+ * The value deserializer that the schemaRegistryVendor sets and that cannot
be loaded fails the share consumer, with
+ * the Maven artifact to add (the serde jars of the vendors are not on the
test classpath).
+ */
+class KafkaShareSchemaRegistrySerdeTest {
+
+ private static final String URI
+ =
"kafka-share:orders?brokers=localhost:39092&groupId=g&schemaRegistryVendor=apicurio&schemaFormat=protobuf";
+
+ @Test
+ void shareConsumerFailsToStartWithoutTheDeserializerAndNamesTheArtifact()
throws Exception {
+ try (CamelContext camel = new DefaultCamelContext()) {
+ camel.start();
+ KafkaShareEndpoint endpoint = camel.getEndpoint(URI,
KafkaShareEndpoint.class);
+ Consumer consumer = endpoint.createConsumer(exchange -> {
+ });
+ try {
+ assertThatThrownBy(consumer::start)
+ .isInstanceOf(IllegalStateException.class)
+ .hasMessageStartingWith("Cannot load the value
deserializer "
+ +
"io.apicurio.registry.serde.protobuf.ProtobufKafkaDeserializer "
+ + "of schemaRegistryVendor
apicurio and schemaFormat protobuf")
+ .hasMessageContaining(
+ "Add the Maven artifact
io.apicurio:apicurio-registry-protobuf-serde-kafka to the classpath");
+ } finally {
+ consumer.stop();
+ }
+ }
+ }
+
+ @Test
+ void customClientFactoryOnlyWarnsAboutAMissingDeserializer() throws
Exception {
+ // a custom client factory may create the share consumer without
loading the class by its name
+ try (CamelContext camel = new DefaultCamelContext()) {
+ camel.start();
+ KafkaShareEndpoint endpoint = camel.getEndpoint(URI,
KafkaShareEndpoint.class);
+ endpoint.setKafkaShareClientFactory(new
DefaultKafkaShareClientFactory() {
+ });
+ Properties props =
endpoint.getConfiguration().createShareConsumerProperties();
+
+
assertThat(endpoint.getConfiguration().missingSchemaRegistrySerde(props,
+
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG)).isNotNull();
+ assertThatCode(() ->
endpoint.checkSchemaRegistrySerde(props)).doesNotThrowAnyException();
+ }
+ }
+}
diff --git a/components/camel-kafka/pom.xml b/components/camel-kafka/pom.xml
index 9d5e76b9719c..2590b6fe5e88 100644
--- a/components/camel-kafka/pom.xml
+++ b/components/camel-kafka/pom.xml
@@ -105,6 +105,40 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
+
+ <!-- The serdes of the schema registry vendors, to check the class
names that schemaRegistryVendor sets
+ (KafkaSchemaRegistryVendorClassesTest), and so that their
versions in the parent pom are kept up to date.
+ The Confluent serdes are not here: they are only in the Confluent
Maven repository. -->
+ <dependency>
+ <groupId>io.apicurio</groupId>
+ <artifactId>apicurio-registry-avro-serde-kafka</artifactId>
+ <version>${apicurio-registry-sdk-version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>io.apicurio</groupId>
+ <artifactId>apicurio-registry-jsonschema-serde-kafka</artifactId>
+ <version>${apicurio-registry-sdk-version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>io.apicurio</groupId>
+ <artifactId>apicurio-registry-protobuf-serde-kafka</artifactId>
+ <version>${apicurio-registry-sdk-version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>software.amazon.glue</groupId>
+ <artifactId>schema-registry-serde</artifactId>
+ <version>${aws-glue-schema-registry-version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.microsoft.azure</groupId>
+ <artifactId>azure-schemaregistry-kafka-avro</artifactId>
+ <version>${azure-schemaregistry-kafka-avro-version}</version>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
diff --git a/components/camel-kafka/src/main/docs/kafka-component.adoc
b/components/camel-kafka/src/main/docs/kafka-component.adoc
index 8df8afd264f4..406fa76ad9dd 100644
--- a/components/camel-kafka/src/main/docs/kafka-component.adoc
+++ b/components/camel-kafka/src/main/docs/kafka-component.adoc
@@ -1000,7 +1000,7 @@ Leave `schemaRegistryVendor` unset to configure the
serializer, deserializer and
The body of the message to send must be a value that the serializer of the
vendor accepts, such as an Avro `GenericRecord` or a class generated from the
Avro schema.
-IMPORTANT: Camel does not include the serde libraries of the vendors: add the
serde jar of the vendor and format to your application.
+IMPORTANT: Camel does not include the serde libraries of the vendors: add the
serde jar of the vendor and format to your application (Camel JBang downloads
it, see below).
The Confluent serde jars are not in Maven Central: add the Confluent Maven
repository `https://packages.confluent.io/maven/`, or have the jars in a
repository of your own.
[width="100%",cols="1m,4",options="header"]
@@ -1026,17 +1026,21 @@ For example with Maven, for Confluent and Avro:
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
- <version>8.0.0</version>
+ <version>8.3.3</version>
</dependency>
----
-And with Camel JBang:
+Camel JBang downloads the serde jar of the vendor and format when the producer
or consumer starts, so you do not add it with `--dep`.
+The Confluent serde jars are not in Maven Central, so add the Confluent Maven
repository with `--repos=confluent` (Camel JBang knows the `confluent`
repository by name):
[source,bash]
----
-camel run MyRoute.java --repos=https://packages.confluent.io/maven/
--dep=io.confluent:kafka-avro-serializer:8.0.0
+camel run MyRoute.java --repos=confluent
----
+When the serializer or deserializer of the vendor cannot be loaded (the jar is
not on the classpath, or Camel JBang cannot download it, such as a Confluent
jar without `--repos=confluent`),
+the producer or consumer fails to start with a message that names the Maven
artifact to add.
+
*Confluent Schema Registry*
NOTE: Supports Avro, JSON Schema and Protobuf. The serde jars are only in the
Confluent Maven repository, not in Maven Central. The Avro serde and the
registry client are licensed under the Apache License 2.0; the JSON Schema and
Protobuf serdes are also listed under the Confluent Community License, so check
their license for your use. Registries that implement the Confluent registry
API, such as Karapace and Redpanda, work with `confluent` too.
@@ -1151,6 +1155,7 @@ The Azure serde reads `auto.register.schemas`,
`use.azure.credential` and `speci
The dependencies of `azure-schemaregistry-kafka-avro` ask for different
versions of `com.azure:azure-core`, and Maven picks the older one, which lacks
`TokenRequestContext.isCaeEnabled()` that Azure Identity calls: the first send
then fails with a `NoSuchMethodError`.
Add a recent `com.azure:azure-core` next to the serde, or import the
`com.azure:azure-sdk-bom`.
+Camel JBang downloads the `com.azure:azure-core` of the Azure SDK BOM together
with the serde.
The options to connect to Event Hubs with the Kafka protocol are left out of
the example.
[tabs]
diff --git
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaConsumer.java
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaConsumer.java
index ab8c264c3c82..4657064b74e1 100644
---
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaConsumer.java
+++
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaConsumer.java
@@ -211,8 +211,13 @@ public class KafkaConsumer extends DefaultConsumer
}
try {
for (int i = 0; i <
endpoint.getConfiguration().getConsumersCount(); i++) {
+ Properties props = getProps(i);
+ if (i == 0) {
+ // the deserializer is the same for every consumer, so
check it once
+ endpoint.checkSchemaRegistrySerde(props,
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG);
+ }
KafkaFetchRecords task = new KafkaFetchRecords(
- this, bridge, topic, pattern, Integer.toString(i),
getProps(i), consumerListener);
+ this, bridge, topic, pattern, Integer.toString(i),
props, consumerListener);
// if camel has not been fully started yet then delay starting
this consumer to avoid
// process incoming message before camel is fully started
diff --git
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaEndpoint.java
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaEndpoint.java
index 5dd197f8495a..9bcffeb83adc 100644
---
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaEndpoint.java
+++
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaEndpoint.java
@@ -223,6 +223,25 @@ public class KafkaEndpoint extends DefaultEndpoint
implements MultipleConsumersS
}
}
+ /**
+ * Fails when the value serializer or deserializer that the
schemaRegistryVendor sets cannot be loaded, with a
+ * message that names the Maven artifact to add, rather than when Kafka
fails to load it. With a custom
+ * {@link KafkaClientFactory}, which may create the client without loading
the class by its name, only logs the
+ * message as a WARN.
+ *
+ * @param props the properties of the client, after {@link
#updateClassProperties(Properties)}
+ * @param key value.serializer or value.deserializer
+ */
+ void checkSchemaRegistrySerde(Properties props, String key) {
+ String message = configuration.missingSchemaRegistrySerde(props, key);
+ if (message != null) {
+ if (kafkaClientFactory == null || kafkaClientFactory.getClass() ==
DefaultKafkaClientFactory.class) {
+ throw new IllegalStateException(message);
+ }
+ LOG.warn("{}", message);
+ }
+ }
+
public ExecutorService createExecutor(Object source) {
return
getCamelContext().getExecutorServiceManager().newFixedThreadPool(source,
"KafkaConsumer[" + configuration.getTopic() + "]",
configuration.getConsumersCount());
diff --git
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaProducer.java
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaProducer.java
index e43c8dbd748e..d3ab90b44368 100755
---
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaProducer.java
+++
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaProducer.java
@@ -194,6 +194,7 @@ public class KafkaProducer extends DefaultAsyncProducer
implements RouteIdAware
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, transactionId);
}
if (kafkaProducer == null) {
+ endpoint.checkSchemaRegistrySerde(props,
ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG);
createProducer(props);
}
diff --git
a/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/KafkaSchemaRegistryVendorClassesTest.java
b/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/KafkaSchemaRegistryVendorClassesTest.java
new file mode 100644
index 000000000000..9fdfee1c483a
--- /dev/null
+++
b/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/KafkaSchemaRegistryVendorClassesTest.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.kafka;
+
+import java.util.stream.Stream;
+
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.Serializer;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Checks that the serializer and deserializer classes that
schemaRegistryVendor sets exist in the serde jars of the
+ * vendors, which are test dependencies, so that a renamed class fails the
build when a vendor version is updated.
+ * Confluent is not checked here, as its serdes are only in the Confluent
Maven repository.
+ */
+class KafkaSchemaRegistryVendorClassesTest {
+
+ static Stream<Arguments> vendorFormats() {
+ return Stream.of(
+ Arguments.of("apicurio", "avro"),
+ Arguments.of("apicurio", "json"),
+ Arguments.of("apicurio", "protobuf"),
+ Arguments.of("aws-glue", "avro"),
+ Arguments.of("aws-glue", "json"),
+ Arguments.of("aws-glue", "protobuf"),
+ Arguments.of("azure", "avro"));
+ }
+
+ @ParameterizedTest
+ @MethodSource("vendorFormats")
+ void testSerdeClassesExist(String vendor, String format) throws Exception {
+ Class<?> serializer =
Class.forName(KafkaSchemaRegistry.serializer(vendor, format));
+ assertTrue(Serializer.class.isAssignableFrom(serializer), serializer +
" is not a Kafka Serializer");
+
+ Class<?> deserializer =
Class.forName(KafkaSchemaRegistry.deserializer(vendor, format));
+ assertTrue(Deserializer.class.isAssignableFrom(deserializer),
deserializer + " is not a Kafka Deserializer");
+ }
+}
diff --git
a/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/KafkaSchemaRegistryVendorTest.java
b/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/KafkaSchemaRegistryVendorTest.java
index 44114633ac38..8525822b57f9 100644
---
a/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/KafkaSchemaRegistryVendorTest.java
+++
b/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/KafkaSchemaRegistryVendorTest.java
@@ -24,13 +24,17 @@ import java.util.TreeMap;
import java.util.stream.Stream;
import org.apache.camel.CamelContext;
+import org.apache.camel.Consumer;
import org.apache.camel.Exchange;
import org.apache.camel.Producer;
+import org.apache.camel.impl.DefaultCamelContext;
+import org.apache.camel.impl.engine.DefaultClassResolver;
import org.apache.camel.support.DefaultExchange;
import org.apache.camel.test.infra.core.CamelContextExtension;
import org.apache.camel.test.infra.core.DefaultCamelContextExtension;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.StringSerializer;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
@@ -40,8 +44,11 @@ import org.junit.jupiter.params.provider.MethodSource;
import org.mockito.Mockito;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -416,6 +423,138 @@ class KafkaSchemaRegistryVendorTest {
}
}
+ static Stream<Arguments> artifacts() {
+ return Stream.of(
+ Arguments.of("confluent", "avro",
"io.confluent:kafka-avro-serializer"),
+ Arguments.of("confluent", "json",
"io.confluent:kafka-json-schema-serializer"),
+ Arguments.of("confluent", "protobuf",
"io.confluent:kafka-protobuf-serializer"),
+ Arguments.of("apicurio", "avro",
"io.apicurio:apicurio-registry-avro-serde-kafka"),
+ Arguments.of("apicurio", "json",
"io.apicurio:apicurio-registry-jsonschema-serde-kafka"),
+ Arguments.of("apicurio", "protobuf",
"io.apicurio:apicurio-registry-protobuf-serde-kafka"),
+ Arguments.of("aws-glue", "json",
"software.amazon.glue:schema-registry-serde"),
+ Arguments.of("azure", "avro",
"com.microsoft.azure:azure-schemaregistry-kafka-avro"));
+ }
+
+ // the serde jars of Apicurio, AWS Glue and Azure are test dependencies
(KafkaSchemaRegistryVendorClassesTest),
+ // so only the Confluent ones, which are not in Maven Central, are missing
from the test classpath
+ static Stream<Arguments> missingArtifacts() {
+ return artifacts().filter(a -> "confluent".equals(a.get()[0]));
+ }
+
+ @ParameterizedTest(name = "{0} {1}")
+ @MethodSource("artifacts")
+ void missingSerdeMessageNamesTheArtifact(String vendor, String format,
String artifact) {
+ assertEquals(artifact, KafkaSchemaRegistry.artifact(vendor, format));
+ String className = KafkaSchemaRegistry.serializer(vendor, format);
+ String msg = KafkaSchemaRegistry.missingSerdeMessage(vendor, format,
"serializer", className);
+ assertTrue(msg.startsWith("Cannot load the value serializer " +
className + " of schemaRegistryVendor " + vendor
+ + " and schemaFormat " + format + ". Add the
Maven artifact " + artifact
+ + " to the classpath"),
+ msg);
+ assertEquals("confluent".equals(vendor), msg.contains("camel run
--repos=confluent"), msg);
+ }
+
+ @ParameterizedTest(name = "{0} {1}")
+ @MethodSource("missingArtifacts")
+ void producerFailsToStartWithoutTheSerializerAndNamesTheArtifact(String
vendor, String format, String artifact)
+ throws Exception {
+ // the Confluent serde jars are not on the test classpath
+ KafkaEndpoint endpoint = context.getEndpoint(
+ "kafka:mytopic?brokers=localhost:39092&schemaRegistryVendor="
+ vendor + "&schemaFormat=" + format,
+ KafkaEndpoint.class);
+ Producer producer = endpoint.createProducer();
+ try {
+ IllegalStateException e =
assertThrows(IllegalStateException.class, producer::start);
+ assertTrue(e.getMessage().startsWith("Cannot load the value
serializer "
+ +
KafkaSchemaRegistry.serializer(vendor, format)
+ + " of schemaRegistryVendor "
+ vendor + " and schemaFormat " + format
+ + ". Add the Maven artifact "
+ artifact + " to the classpath"),
+ e.getMessage());
+ assertEquals("confluent".equals(vendor),
e.getMessage().contains("camel run --repos=confluent"),
+ e.getMessage());
+ } finally {
+ producer.stop();
+ }
+ }
+
+ @ParameterizedTest(name = "{0} {1}")
+ @MethodSource("missingArtifacts")
+ void consumerFailsToStartWithoutTheDeserializerAndNamesTheArtifact(String
vendor, String format, String artifact)
+ throws Exception {
+ KafkaEndpoint endpoint = context.getEndpoint(
+
"kafka:mytopic?brokers=localhost:39092&groupId=g&schemaRegistryVendor=" +
vendor + "&schemaFormat="
+ + format,
+ KafkaEndpoint.class);
+ Consumer consumer = endpoint.createConsumer(exchange -> {
+ });
+ try {
+ IllegalStateException e =
assertThrows(IllegalStateException.class, consumer::start);
+ assertTrue(e.getMessage().startsWith("Cannot load the value
deserializer "
+ +
KafkaSchemaRegistry.deserializer(vendor, format)),
+ e.getMessage());
+ assertTrue(e.getMessage().contains("Add the Maven artifact " +
artifact + " to the classpath"),
+ e.getMessage());
+ } finally {
+ consumer.stop();
+ }
+ }
+
+ @Test
+ void serializerOfTheVendorIsLoadedThroughTheClassResolver() throws
Exception {
+ // Camel JBang downloads the jar of a known class that the class
resolver is asked for and is not on the
+ // classpath: the class the resolver returns is the one Kafka gets
+ try (CamelContext camel = new DefaultCamelContext()) {
+ camel.setClassResolver(new DefaultClassResolver(camel) {
+ @Override
+ public Class<?> resolveClass(String name) {
+ return
"io.apicurio.registry.serde.avro.AvroKafkaSerializer".equals(name)
+ ? StringSerializer.class :
super.resolveClass(name);
+ }
+ });
+ camel.start();
+ KafkaEndpoint endpoint = camel.getEndpoint(
+
"kafka:mytopic?brokers=localhost:39092&schemaRegistryVendor=apicurio",
KafkaEndpoint.class);
+ KafkaProducer producer = new KafkaProducer(endpoint);
+
+ Properties props = producer.getProps();
+ assertSame(StringSerializer.class,
props.get(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG));
+
assertNull(endpoint.getConfiguration().missingSchemaRegistrySerde(props,
+ ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG));
+ endpoint.checkSchemaRegistrySerde(props,
ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG);
+ }
+ }
+
+ @Test
+ void customClientFactoryOnlyWarnsAboutAMissingSerializer() {
+ // a custom client factory may create the client without loading the
class by its name
+ KafkaEndpoint endpoint = context.getEndpoint(
+
"kafka:mytopic?brokers=localhost:39092&schemaRegistryVendor=confluent",
KafkaEndpoint.class);
+ endpoint.setKafkaClientFactory(new DefaultKafkaClientFactory() {
+ });
+ Properties props =
endpoint.getConfiguration().createProducerProperties();
+
+
assertNotNull(endpoint.getConfiguration().missingSchemaRegistrySerde(props,
+ ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG));
+ assertDoesNotThrow(() -> endpoint.checkSchemaRegistrySerde(props,
ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG));
+ }
+
+ @Test
+ void onlyTheMissingSerdeOfTheVendorIsReported() {
+ // no vendor
+ KafkaConfiguration config = new KafkaConfiguration();
+ Properties props = new Properties();
+ props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"com.example.MySerializer");
+ assertNull(config.missingSchemaRegistrySerde(props,
ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG));
+
+ // an explicit serializer that is not the one of the vendor is left to
Kafka
+ config.setSchemaRegistryVendor("confluent");
+ assertNull(config.missingSchemaRegistrySerde(props,
ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG));
+
+ // the serializer of the vendor that is still a class name could not
be loaded
+ props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"io.confluent.kafka.serializers.KafkaAvroSerializer");
+ assertNotNull(config.missingSchemaRegistrySerde(props,
ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG));
+ }
+
/**
* The properties that are not Kafka client options with a fixed value in
these tests: the registry properties.
*/
diff --git a/docs/user-manual/modules/ROOT/pages/camel-jbang-running.adoc
b/docs/user-manual/modules/ROOT/pages/camel-jbang-running.adoc
index 71e34598309c..6ad17c4119fb 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-jbang-running.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-jbang-running.adoc
@@ -306,6 +306,19 @@ camel run foo.java
--repos=https://packages.atlassian.com/maven-external
TIP: Separate multiple repositories with commas.
+Some repositories are known by name, so you can use the name instead of the
URL, such as `--repos=confluent`:
+
+[width="100%",cols="1m,3",options="header"]
+|===
+| Name | Repository
+| central | `https://repo1.maven.org/maven2`
+| apache-snapshot | `https://repository.apache.org/snapshots`
+| atlassian | `https://packages.atlassian.com/maven-external`
+| confluent | `https://packages.confluent.io/maven/` (the Confluent
serializers and deserializers for Kafka, such as
`io.confluent:kafka-avro-serializer`)
+| jboss |
`https://repository.jboss.org/nexus/service/local/repositories/thirdparty-releases/content`
+| redhat | `https://maven.repository.redhat.com/ga`
+|===
+
You can also configure repositories in `application.properties`:
[source,properties]
diff --git a/dsl/camel-kamelet-main/pom.xml b/dsl/camel-kamelet-main/pom.xml
index c981972f75d2..1f9e1af37d16 100644
--- a/dsl/camel-kamelet-main/pom.xml
+++ b/dsl/camel-kamelet-main/pom.xml
@@ -230,6 +230,19 @@
</execution>
</executions>
</plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <configuration>
+ <!-- the versions of parent/pom.xml that
camel-thirdparty-known-dependencies.properties must use -->
+ <systemPropertyVariables>
+
<confluent-serializer-version>${confluent-serializer-version}</confluent-serializer-version>
+
<apicurio-registry-sdk-version>${apicurio-registry-sdk-version}</apicurio-registry-sdk-version>
+
<aws-glue-schema-registry-version>${aws-glue-schema-registry-version}</aws-glue-schema-registry-version>
+
<azure-schemaregistry-kafka-avro-version>${azure-schemaregistry-kafka-avro-version}</azure-schemaregistry-kafka-avro-version>
+ </systemPropertyVariables>
+ </configuration>
+ </plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>templating-maven-plugin</artifactId>
diff --git
a/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/DependencyDownloaderClassResolver.java
b/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/DependencyDownloaderClassResolver.java
index 3a560dd558b4..20da93ef8e7e 100644
---
a/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/DependencyDownloaderClassResolver.java
+++
b/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/DependencyDownloaderClassResolver.java
@@ -19,6 +19,8 @@ package org.apache.camel.main.download;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
import org.apache.camel.CamelContext;
import org.apache.camel.catalog.CamelCatalog;
@@ -36,6 +38,8 @@ public final class DependencyDownloaderClassResolver extends
DefaultClassResolve
private final KnownDependenciesResolver knownDependenciesResolver;
private final DependencyDownloader downloader;
private final boolean silent;
+ // the known dependencies that could not be downloaded, which are not
attempted again
+ private final Set<String> failed = ConcurrentHashMap.newKeySet();
public DependencyDownloaderClassResolver(CamelContext camelContext,
KnownDependenciesResolver
knownDependenciesResolver,
@@ -60,22 +64,7 @@ public final class DependencyDownloaderClassResolver extends
DefaultClassResolve
if (answer == null) {
// okay maybe the class is from a known GAV that we can download
first and then load the class
- MavenGav gav = knownDependenciesResolver.mavenGavForClass(uri);
- if (gav == null) {
- // okay maybe its a known pojo-bean from the catalog
- // lookup via class name without package
- String last = StringHelper.afterLast(uri, ".", uri);
- PojoBeanModel model = catalog.pojoBeanModel(last);
- if (model != null && uri.equals(model.getJavaType())) {
- gav = MavenGav.fromCoordinates(model.getGroupId(),
model.getArtifactId(), model.getVersion(), null, null);
- }
- }
- if (gav != null) {
- if (!downloader.alreadyOnClasspath(gav.getGroupId(),
gav.getArtifactId(),
- gav.getVersion())) {
- downloader.downloadDependency(gav.getGroupId(),
gav.getArtifactId(),
- gav.getVersion());
- }
+ if (downloadKnownDependencies(uri)) {
try {
answer = super.loadResourceAsStream(uri);
} catch (Exception e) {
@@ -98,22 +87,7 @@ public final class DependencyDownloaderClassResolver extends
DefaultClassResolve
if (answer == null) {
// okay maybe the class is from a known GAV that we can download
first and then load the class
- MavenGav gav = knownDependenciesResolver.mavenGavForClass(name);
- if (gav == null) {
- // okay maybe its a known pojo-bean from the catalog
- // lookup via class name without package
- String last = StringHelper.afterLast(name, ".", name);
- PojoBeanModel model = catalog.pojoBeanModel(last);
- if (model != null && name.equals(model.getJavaType())) {
- gav = MavenGav.fromCoordinates(model.getGroupId(),
model.getArtifactId(), model.getVersion(), null, null);
- }
- }
- if (gav != null) {
- if (!downloader.alreadyOnClasspath(gav.getGroupId(),
gav.getArtifactId(),
- gav.getVersion())) {
- downloader.downloadDependency(gav.getGroupId(),
gav.getArtifactId(),
- gav.getVersion());
- }
+ if (downloadKnownDependencies(name)) {
try {
answer = ObjectHelper.loadClass(name, loader);
} catch (Exception e) {
@@ -125,6 +99,40 @@ public final class DependencyDownloaderClassResolver
extends DefaultClassResolve
return answer;
}
+ /**
+ * Downloads the known dependencies of the class (or resource), such as
the serializer of a Kafka schema registry
+ * that camel-kafka resolves by name, or a known pojo-bean of the catalog.
+ *
+ * @return true if the class has known dependencies (which are now on the
classpath, unless the download failed)
+ */
+ private boolean downloadKnownDependencies(String name) {
+ List<MavenGav> gavs =
knownDependenciesResolver.mavenGavsForClass(name);
+ if (gavs.isEmpty()) {
+ // okay maybe its a known pojo-bean from the catalog
+ // lookup via class name without package
+ String last = StringHelper.afterLast(name, ".", name);
+ PojoBeanModel model = catalog.pojoBeanModel(last);
+ if (model != null && name.equals(model.getJavaType())) {
+ gavs = List.of(
+ MavenGav.fromCoordinates(model.getGroupId(),
model.getArtifactId(), model.getVersion(), null, null));
+ }
+ }
+ if (gavs.isEmpty()) {
+ return false;
+ }
+ MavenGav first = gavs.get(0);
+ String key = first.toString();
+ if (failed.contains(key)) {
+ // the class is looked up through several class loaders: report a
failed download once, not for each
+ return false;
+ }
+ KnownDependenciesResolver.download(downloader, gavs);
+ if (!downloader.alreadyOnClasspath(first.getGroupId(),
first.getArtifactId(), first.getVersion())) {
+ failed.add(key);
+ }
+ return true;
+ }
+
private class KNativeHttpServerFactory implements ResourceResolverListener
{
@Override
diff --git
a/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/DependencyDownloaderPropertiesComponent.java
b/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/DependencyDownloaderPropertiesComponent.java
index 0bd2a87733a1..9721071da39d 100644
---
a/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/DependencyDownloaderPropertiesComponent.java
+++
b/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/DependencyDownloaderPropertiesComponent.java
@@ -28,7 +28,6 @@ import org.apache.camel.spi.Language;
import org.apache.camel.support.DefaultExchange;
import org.apache.camel.support.SimpleEventNotifierSupport;
import org.apache.camel.support.service.ServiceSupport;
-import org.apache.camel.tooling.maven.MavenGav;
import org.apache.camel.util.IOHelper;
import org.apache.camel.util.ObjectHelper;
import org.apache.camel.util.OrderedLocationProperties;
@@ -82,25 +81,10 @@ public class DependencyDownloaderPropertiesComponent
extends ServiceSupport impl
for (String key : properties.stringPropertyNames()) {
// check both key and values (and combined)
String value = properties.getProperty(key);
- MavenGav gav = knownDependenciesResolver.mavenGavForClass(key);
- if (gav != null) {
- downloadLoader(gav.getGroupId(), gav.getArtifactId(),
gav.getVersion());
- }
- gav = knownDependenciesResolver.mavenGavForClass(value);
- if (gav != null) {
- downloadLoader(gav.getGroupId(), gav.getArtifactId(),
gav.getVersion());
- }
+ KnownDependenciesResolver.download(downloader,
knownDependenciesResolver.mavenGavsForClass(key));
+ KnownDependenciesResolver.download(downloader,
knownDependenciesResolver.mavenGavsForClass(value));
String line = key + "=" + value;
- gav = knownDependenciesResolver.mavenGavForClass(line);
- if (gav != null) {
- downloadLoader(gav.getGroupId(), gav.getArtifactId(),
gav.getVersion());
- }
- }
- }
-
- private void downloadLoader(String groupId, String artifactId, String
version) {
- if (!downloader.alreadyOnClasspath(groupId, artifactId, version)) {
- downloader.downloadDependency(groupId, artifactId, version);
+ KnownDependenciesResolver.download(downloader,
knownDependenciesResolver.mavenGavsForClass(line));
}
}
diff --git
a/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/DependencyDownloaderPropertyBindingListener.java
b/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/DependencyDownloaderPropertyBindingListener.java
index 215455408196..053490666c7d 100644
---
a/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/DependencyDownloaderPropertyBindingListener.java
+++
b/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/DependencyDownloaderPropertyBindingListener.java
@@ -16,6 +16,8 @@
*/
package org.apache.camel.main.download;
+import java.util.List;
+
import org.apache.camel.CamelContext;
import org.apache.camel.support.PropertyBindingListener;
import org.apache.camel.tooling.maven.MavenGav;
@@ -35,18 +37,12 @@ public class DependencyDownloaderPropertyBindingListener
implements PropertyBind
public void bindProperty(Object target, String key, Object value) {
if (value instanceof String) {
String s = (String) value;
- MavenGav gav = knownDependenciesResolver.mavenGavForClass(s);
- if (gav == null) {
+ List<MavenGav> gavs =
knownDependenciesResolver.mavenGavsForClass(s);
+ if (gavs.isEmpty()) {
String line = target.getClass().getName() + ":" + key + "=" +
s;
- gav = knownDependenciesResolver.mavenGavForClass(line);
- }
- if (gav != null) {
- if (!downloader.alreadyOnClasspath(gav.getGroupId(),
gav.getArtifactId(),
- gav.getVersion())) {
- downloader.downloadDependency(gav.getGroupId(),
gav.getArtifactId(),
- gav.getVersion());
- }
+ gavs = knownDependenciesResolver.mavenGavsForClass(line);
}
+ KnownDependenciesResolver.download(downloader, gavs);
}
}
diff --git
a/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/JavaKnownImportsDownloader.java
b/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/JavaKnownImportsDownloader.java
index 9f7dffc9fd2b..d3b1c8674942 100644
---
a/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/JavaKnownImportsDownloader.java
+++
b/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/JavaKnownImportsDownloader.java
@@ -52,9 +52,9 @@ public class JavaKnownImportsDownloader implements
CompilePreProcessor {
List<String> imports = determineImports(code);
for (String imp : imports) {
// attempt known dependency resolver first
- MavenGav gav = knownDependenciesResolver.mavenGavForImport(imp);
- if (gav != null) {
- downloadLoader(gav.getGroupId(), gav.getArtifactId(),
gav.getVersion());
+ List<MavenGav> gavs =
knownDependenciesResolver.mavenGavsForImport(imp);
+ if (!gavs.isEmpty()) {
+ KnownDependenciesResolver.download(downloader, gavs);
} else {
// is this a known bean then we can determine the dependency
for (String n : catalog.findBeansNames()) {
diff --git
a/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/KnownDependenciesResolver.java
b/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/KnownDependenciesResolver.java
index ced1bb4c50b2..0a0e22ccd047 100644
---
a/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/KnownDependenciesResolver.java
+++
b/dsl/camel-kamelet-main/src/main/java/org/apache/camel/main/download/KnownDependenciesResolver.java
@@ -18,9 +18,12 @@ package org.apache.camel.main.download;
import java.io.InputStream;
import java.net.URL;
+import java.util.ArrayList;
+import java.util.Collections;
import java.util.Enumeration;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Set;
@@ -106,8 +109,24 @@ public final class KnownDependenciesResolver {
this.mappings.putAll(mappings);
}
+ /**
+ * The dependency of a class, or the first one when the mapping of the
class lists more (see
+ * {@link #mavenGavsForClass(String)}).
+ */
public MavenGav mavenGavForClass(String className) {
- return toMavenGav(findGav(mappings, className));
+ List<MavenGav> answer = mavenGavsForClass(className);
+ return answer.isEmpty() ? null : answer.get(0);
+ }
+
+ /**
+ * The dependencies of a class: one, or more when the mapping lists more,
separated by commas (such as a library and
+ * a recent version of a library it needs), which are to be downloaded
together with
+ * {@link #download(DependencyDownloader, List)}.
+ *
+ * @return the dependencies, or an empty list when the class is not known
+ */
+ public List<MavenGav> mavenGavsForClass(String className) {
+ return toMavenGavs(findGav(mappings, className));
}
/**
@@ -118,11 +137,53 @@ public final class KnownDependenciesResolver {
* component.
*/
public MavenGav mavenGavForImport(String className) {
+ List<MavenGav> answer = mavenGavsForImport(className);
+ return answer.isEmpty() ? null : answer.get(0);
+ }
+
+ /**
+ * The dependencies of a class a source imports: as {@link
#mavenGavForImport(String)}, with all the dependencies of
+ * the mapping (see {@link #mavenGavsForClass(String)}).
+ */
+ public List<MavenGav> mavenGavsForImport(String className) {
String gav = findGav(mappings, className);
if (gav == null) {
gav = findGav(componentPackages, className);
}
- return toMavenGav(gav);
+ return toMavenGavs(gav);
+ }
+
+ /**
+ * Downloads the dependencies of a mapping that are not on the classpath.
Several dependencies are downloaded
+ * together, as one dependency graph, so that their versions win over the
versions that the first one asks for.
+ */
+ static void download(DependencyDownloader downloader, List<MavenGav> gavs)
{
+ List<MavenGav> todo = new ArrayList<>();
+ for (MavenGav gav : gavs) {
+ if (!downloader.alreadyOnClasspath(gav.getGroupId(),
gav.getArtifactId(), gav.getVersion())) {
+ todo.add(gav);
+ }
+ }
+ if (todo.size() == 1) {
+ MavenGav gav = todo.get(0);
+ downloader.downloadDependency(gav.getGroupId(),
gav.getArtifactId(), gav.getVersion());
+ } else if (!todo.isEmpty()) {
+ downloader.downloadDependencies(todo, false);
+ }
+ }
+
+ private List<MavenGav> toMavenGavs(String gavs) {
+ if (gavs == null) {
+ return Collections.emptyList();
+ }
+ List<MavenGav> answer = new ArrayList<>();
+ for (String gav : gavs.split(",")) {
+ MavenGav g = toMavenGav(gav.trim());
+ if (g != null) {
+ answer.add(g);
+ }
+ }
+ return answer;
}
private MavenGav toMavenGav(String gav) {
diff --git
a/dsl/camel-kamelet-main/src/main/known-third-party-libraries.properties
b/dsl/camel-kamelet-main/src/main/known-third-party-libraries.properties
index 9a3f70137679..e54c45fc38fb 100644
--- a/dsl/camel-kamelet-main/src/main/known-third-party-libraries.properties
+++ b/dsl/camel-kamelet-main/src/main/known-third-party-libraries.properties
@@ -20,12 +20,15 @@
# camel-thirdparty-known-dependencies.properties into target/classes with the
versions resolved; that file is
# shipped in the jar and never committed, this one is the source.
#
-# One line per library: <package> = <groupId>:<artifactId>:<version>
+# One line per library: <package or class> = <groupId>:<artifactId>:<version>
# <version> is ${a-version-property} of parent/pom.xml (checked at build
time; a literal version is refused),
# or @bom(<groupId>:<artifactId>:${bom-version-property}) for a library
whose version a BOM manages.
-# The runtime matches the class name and then each enclosing package, so map
the library's own package and
-# never a shared parent such as org.apache.commons or com.google.cloud, or the
walk up the package picks the
-# wrong jar. With -Dcamel.known-dependencies.verify=true the build resolves
every jar and checks the package.
+# A line can list more dependencies, separated by commas: they are
downloaded together as one dependency graph,
+# so their versions win over the versions that the first one asks for (such
as a recent library it needs).
+# The runtime matches the class name and then each enclosing package, so map
the library's own package (or its
+# classes) and never a shared parent such as org.apache.commons or
com.google.cloud, or the walk up the package
+# picks the wrong jar. With -Dcamel.known-dependencies.verify=true the build
resolves every jar and checks that the
+# package or class is in the first one.
# JDBC drivers, datasources and connection pools
org.postgresql = org.postgresql:postgresql:${pgjdbc-driver-version}
@@ -69,6 +72,31 @@ org.eclipse.paho.client.mqttv3 =
org.eclipse.paho:org.eclipse.paho.client.mqttv3
org.eclipse.paho.mqttv5.client =
org.eclipse.paho:org.eclipse.paho.mqttv5.client:${paho-version}
com.hivemq.client = com.hivemq:hivemq-mqtt-client:${hivemq-mqtt-client-version}
+# Kafka schema registry serializers and deserializers, which the
schemaRegistryVendor and schemaFormat options of
+# camel-kafka set by class name (camel-kafka loads them through the class
resolver before it creates the client).
+# Mapped by class, as the serdes of the formats share packages with other jars
of the vendor.
+# The Confluent jars are only in the Confluent Maven repository (camel run
--repos=confluent).
+io.confluent.kafka.serializers.KafkaAvroSerializer =
io.confluent:kafka-avro-serializer:${confluent-serializer-version}
+io.confluent.kafka.serializers.KafkaAvroDeserializer =
io.confluent:kafka-avro-serializer:${confluent-serializer-version}
+io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer =
io.confluent:kafka-json-schema-serializer:${confluent-serializer-version}
+io.confluent.kafka.serializers.json.KafkaJsonSchemaDeserializer =
io.confluent:kafka-json-schema-serializer:${confluent-serializer-version}
+io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializer =
io.confluent:kafka-protobuf-serializer:${confluent-serializer-version}
+io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer =
io.confluent:kafka-protobuf-serializer:${confluent-serializer-version}
+# the serdes are released with the Apicurio Registry Java SDK that
camel-apicurio-registry uses
+io.apicurio.registry.serde.avro.AvroKafkaSerializer =
io.apicurio:apicurio-registry-avro-serde-kafka:${apicurio-registry-sdk-version}
+io.apicurio.registry.serde.avro.AvroKafkaDeserializer =
io.apicurio:apicurio-registry-avro-serde-kafka:${apicurio-registry-sdk-version}
+io.apicurio.registry.serde.jsonschema.JsonSchemaKafkaSerializer =
io.apicurio:apicurio-registry-jsonschema-serde-kafka:${apicurio-registry-sdk-version}
+io.apicurio.registry.serde.jsonschema.JsonSchemaKafkaDeserializer =
io.apicurio:apicurio-registry-jsonschema-serde-kafka:${apicurio-registry-sdk-version}
+io.apicurio.registry.serde.protobuf.ProtobufKafkaSerializer =
io.apicurio:apicurio-registry-protobuf-serde-kafka:${apicurio-registry-sdk-version}
+io.apicurio.registry.serde.protobuf.ProtobufKafkaDeserializer =
io.apicurio:apicurio-registry-protobuf-serde-kafka:${apicurio-registry-sdk-version}
+com.amazonaws.services.schemaregistry.serializers.GlueSchemaRegistryKafkaSerializer
=
software.amazon.glue:schema-registry-serde:${aws-glue-schema-registry-version}
+com.amazonaws.services.schemaregistry.deserializers.GlueSchemaRegistryKafkaDeserializer
=
software.amazon.glue:schema-registry-serde:${aws-glue-schema-registry-version}
+# the dependencies of the Azure serde ask for an old azure-core (1.35) that
lacks TokenRequestContext.isCaeEnabled(),
+# which the azure-identity it also brings calls (NoSuchMethodError on the
first send): azure-core of the Azure SDK BOM
+# is downloaded with it
+com.microsoft.azure.schemaregistry.kafka.avro.KafkaAvroSerializer =
com.microsoft.azure:azure-schemaregistry-kafka-avro:${azure-schemaregistry-kafka-avro-version},
com.azure:azure-core:@bom(com.azure:azure-sdk-bom:${azure-sdk-bom-version})
+com.microsoft.azure.schemaregistry.kafka.avro.KafkaAvroDeserializer =
com.microsoft.azure:azure-schemaregistry-kafka-avro:${azure-schemaregistry-kafka-avro-version},
com.azure:azure-core:@bom(com.azure:azure-sdk-bom:${azure-sdk-bom-version})
+
# Databases, search and caches
com.mongodb = org.mongodb:mongodb-driver-sync:${mongo-java-driver-version}
redis.clients.jedis = redis.clients:jedis:${jedis-client-version}
diff --git
a/dsl/camel-kamelet-main/src/test/java/org/apache/camel/main/download/KnownDependenciesResolverTest.java
b/dsl/camel-kamelet-main/src/test/java/org/apache/camel/main/download/KnownDependenciesResolverTest.java
index d45b980ddf86..0ddc6c7f24ea 100644
---
a/dsl/camel-kamelet-main/src/test/java/org/apache/camel/main/download/KnownDependenciesResolverTest.java
+++
b/dsl/camel-kamelet-main/src/test/java/org/apache/camel/main/download/KnownDependenciesResolverTest.java
@@ -16,10 +16,12 @@
*/
package org.apache.camel.main.download;
+import java.util.List;
import java.util.Map;
import org.apache.camel.catalog.DefaultCamelCatalog;
import org.apache.camel.impl.engine.SimpleCamelContext;
+import org.apache.camel.main.util.VersionHelper;
import org.apache.camel.tooling.maven.MavenGav;
import org.junit.jupiter.api.Test;
@@ -27,6 +29,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
public class KnownDependenciesResolverTest {
@@ -176,6 +179,83 @@ public class KnownDependenciesResolverTest {
assertImport(resolver,
"org.apache.camel.component.typesafeai.TypeSafeAiEndpoint",
"camel-typesafe-ai");
}
+ @Test
+ void theShippedMappingResolvesTheKafkaSchemaRegistrySerdes() {
+ // the serializers and deserializers that the schemaRegistryVendor and
schemaFormat options of camel-kafka set,
+ // with the versions of parent/pom.xml (passed by surefire)
+ KnownDependenciesResolver resolver = new KnownDependenciesResolver(new
SimpleCamelContext(), null, null);
+ resolver.loadKnownDependencies();
+
+ String confluent = System.getProperty("confluent-serializer-version");
+ String apicurio = System.getProperty("apicurio-registry-sdk-version");
+ String glue = System.getProperty("aws-glue-schema-registry-version");
+ String azure =
System.getProperty("azure-schemaregistry-kafka-avro-version");
+ assertNotNull(confluent, "confluent-serializer-version");
+
+ for (String c : new String[] { "KafkaAvroSerializer",
"KafkaAvroDeserializer" }) {
+ assertGav(resolver, "io.confluent.kafka.serializers." + c,
"io.confluent:kafka-avro-serializer:" + confluent);
+ }
+ for (String c : new String[] { "KafkaJsonSchemaSerializer",
"KafkaJsonSchemaDeserializer" }) {
+ assertGav(resolver, "io.confluent.kafka.serializers.json." + c,
+ "io.confluent:kafka-json-schema-serializer:" + confluent);
+ }
+ for (String c : new String[] { "KafkaProtobufSerializer",
"KafkaProtobufDeserializer" }) {
+ assertGav(resolver, "io.confluent.kafka.serializers.protobuf." + c,
+ "io.confluent:kafka-protobuf-serializer:" + confluent);
+ }
+ for (String c : new String[] { "AvroKafkaSerializer",
"AvroKafkaDeserializer" }) {
+ assertGav(resolver, "io.apicurio.registry.serde.avro." + c,
+ "io.apicurio:apicurio-registry-avro-serde-kafka:" +
apicurio);
+ }
+ for (String c : new String[] { "JsonSchemaKafkaSerializer",
"JsonSchemaKafkaDeserializer" }) {
+ assertGav(resolver, "io.apicurio.registry.serde.jsonschema." + c,
+ "io.apicurio:apicurio-registry-jsonschema-serde-kafka:" +
apicurio);
+ }
+ for (String c : new String[] { "ProtobufKafkaSerializer",
"ProtobufKafkaDeserializer" }) {
+ assertGav(resolver, "io.apicurio.registry.serde.protobuf." + c,
+ "io.apicurio:apicurio-registry-protobuf-serde-kafka:" +
apicurio);
+ }
+ assertGav(resolver,
"com.amazonaws.services.schemaregistry.serializers.GlueSchemaRegistryKafkaSerializer",
+ "software.amazon.glue:schema-registry-serde:" + glue);
+ assertGav(resolver,
"com.amazonaws.services.schemaregistry.deserializers.GlueSchemaRegistryKafkaDeserializer",
+ "software.amazon.glue:schema-registry-serde:" + glue);
+
+ // the Azure serde comes with a recent azure-core, as its dependencies
ask for an old one
+ for (String c : new String[] { "KafkaAvroSerializer",
"KafkaAvroDeserializer" }) {
+ List<MavenGav> gavs =
resolver.mavenGavsForClass("com.microsoft.azure.schemaregistry.kafka.avro." +
c);
+ assertEquals(2, gavs.size(), c);
+
assertEquals("com.microsoft.azure:azure-schemaregistry-kafka-avro:" + azure,
gavs.get(0).toString());
+ assertEquals("com.azure", gavs.get(1).getGroupId());
+ assertEquals("azure-core", gavs.get(1).getArtifactId());
+ // TokenRequestContext.isCaeEnabled() that azure-identity calls is
in azure-core 1.54.1 and later
+ assertTrue(VersionHelper.isGE(gavs.get(1).getVersion(), "1.54.1"),
gavs.get(1).getVersion());
+ // a single dependency for the callers that take one
+ assertEquals(gavs.get(0).toString(),
+
resolver.mavenGavForClass("com.microsoft.azure.schemaregistry.kafka.avro." +
c).toString());
+ }
+ // the other classes of the vendors are not mapped
+
assertNull(resolver.mavenGavForClass("io.confluent.kafka.serializers.Other"));
+
assertNull(resolver.mavenGavForClass("io.apicurio.registry.serde.Other"));
+ }
+
+ @Test
+ void aMappingCanListMoreDependencies() {
+ KnownDependenciesResolver resolver = new KnownDependenciesResolver(new
SimpleCamelContext(), null, null);
+ resolver.addMappings(Map.of("org.example.Both",
"org.example:first:1.0, org.example:second:2.0"));
+
+ List<MavenGav> gavs = resolver.mavenGavsForClass("org.example.Both");
+ assertEquals(List.of("org.example:first:1.0",
"org.example:second:2.0"),
+ gavs.stream().map(MavenGav::toString).toList());
+ assertEquals("org.example:first:1.0",
resolver.mavenGavForClass("org.example.Both").toString());
+ assertEquals(gavs.toString(),
resolver.mavenGavsForImport("org.example.Both").toString());
+
assertTrue(resolver.mavenGavsForClass("org.example.Unknown").isEmpty());
+ }
+
+ private static void assertGav(KnownDependenciesResolver resolver, String
className, String gav) {
+ List<MavenGav> gavs = resolver.mavenGavsForClass(className);
+ assertEquals(List.of(gav),
gavs.stream().map(MavenGav::toString).toList(), className);
+ }
+
private static void assertImport(KnownDependenciesResolver resolver,
String className, String artifactId) {
MavenGav gav = resolver.mavenGavForImport(className);
assertNotNull(gav, className);
diff --git
a/dsl/camel-kamelet-main/src/test/java/org/apache/camel/main/download/MavenDependencyDownloaderBatchTest.java
b/dsl/camel-kamelet-main/src/test/java/org/apache/camel/main/download/MavenDependencyDownloaderBatchTest.java
index e48a85f39b17..3aee9632d85c 100644
---
a/dsl/camel-kamelet-main/src/test/java/org/apache/camel/main/download/MavenDependencyDownloaderBatchTest.java
+++
b/dsl/camel-kamelet-main/src/test/java/org/apache/camel/main/download/MavenDependencyDownloaderBatchTest.java
@@ -113,6 +113,22 @@ class MavenDependencyDownloaderBatchTest {
assertEquals(List.of("org.example.batch:a:1.0"), downloaded);
}
+ @Test
+ void knownDependenciesOfAMappingAreDownloadedTogether() {
+ // as the Kafka serde of Azure with a recent azure-core: the version
of the mapping wins over the version that
+ // the first dependency asks for
+ MavenGav common = MavenGav.fromCoordinates("org.example.batch",
"common", "2.0", null, null);
+ KnownDependenciesResolver.download(downloader, List.of(gav("a"),
common));
+
+ assertEquals(List.of("org.example.batch:a:1.0",
"org.example.batch:common:2.0"), downloaded);
+ assertEquals(List.of("a-1.0.jar", "common-2.0.jar"), jars());
+
+ // what is on the classpath is not downloaded again
+ downloaded.clear();
+ KnownDependenciesResolver.download(downloader, List.of(gav("a"),
common));
+ assertEquals(List.of(), downloaded);
+ }
+
private static MavenGav gav(String artifactId) {
return MavenGav.fromCoordinates("org.example.batch", artifactId,
"1.0", null, null);
}
diff --git a/parent/pom.xml b/parent/pom.xml
index 680c2d52f837..b9b6ac43382b 100644
--- a/parent/pom.xml
+++ b/parent/pom.xml
@@ -95,8 +95,10 @@
<avro-ipc-jetty-version>1.12.2</avro-ipc-jetty-version>
<avro-ipc-netty-version>1.12.2</avro-ipc-netty-version>
<awaitility-version>4.3.0</awaitility-version>
+
<aws-glue-schema-registry-version>2.0.2</aws-glue-schema-registry-version>
<aws-java-sdk2-version>2.57.11</aws-java-sdk2-version>
<azure-iot-device-client-version>2.5.1</azure-iot-device-client-version>
+
<azure-schemaregistry-kafka-avro-version>1.1.4</azure-schemaregistry-kafka-avro-version>
<azure-sdk-bom-version>1.3.3</azure-sdk-bom-version>
<azure-storage-blob-changefeed-version>12.0.0-beta.40</azure-storage-blob-changefeed-version>
<beanio-version>3.2.1</beanio-version>
@@ -141,6 +143,7 @@
<commons-text-version>1.15.0</commons-text-version>
<commons-validator-version>1.11.0</commons-validator-version>
<compress-lzf-version>1.2.1</compress-lzf-version>
+ <confluent-serializer-version>8.3.3</confluent-serializer-version>
<consul-client-version>1.12.2</consul-client-version>
<couchbase-client-version>3.13.1</couchbase-client-version>
<curator-version>5.9.0</curator-version>
diff --git
a/tooling/camel-tooling-maven/src/main/resources/known-maven-repos.properties
b/tooling/camel-tooling-maven/src/main/resources/known-maven-repos.properties
index a7713de3ea1c..b3be0b98ab2c 100644
---
a/tooling/camel-tooling-maven/src/main/resources/known-maven-repos.properties
+++
b/tooling/camel-tooling-maven/src/main/resources/known-maven-repos.properties
@@ -19,4 +19,5 @@ central = https://repo1.maven.org/maven2
apache-snapshot = https://repository.apache.org/snapshots
atlassian = https://packages.atlassian.com/maven-external
jboss =
https://repository.jboss.org/nexus/service/local/repositories/thirdparty-releases/content
-redhat = https://maven.repository.redhat.com/ga
\ No newline at end of file
+redhat = https://maven.repository.redhat.com/ga
+confluent = https://packages.confluent.io/maven/
diff --git
a/tooling/camel-tooling-maven/src/test/java/org/apache/camel/tooling/maven/DefaultRepositoryResolverTest.java
b/tooling/camel-tooling-maven/src/test/java/org/apache/camel/tooling/maven/DefaultRepositoryResolverTest.java
new file mode 100644
index 000000000000..3a0287699fda
--- /dev/null
+++
b/tooling/camel-tooling-maven/src/test/java/org/apache/camel/tooling/maven/DefaultRepositoryResolverTest.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.tooling.maven;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+/**
+ * The known repository names that --repos takes, such as --repos=confluent.
+ */
+class DefaultRepositoryResolverTest {
+
+ @Test
+ void knownNamesResolveToTheirUrl() {
+ DefaultRepositoryResolver resolver = new DefaultRepositoryResolver();
+ resolver.build();
+
+ assertEquals("https://packages.confluent.io/maven/",
resolver.resolveRepository("confluent"));
+ assertEquals("https://maven.repository.redhat.com/ga",
resolver.resolveRepository("redhat"));
+ assertEquals("https://repo1.maven.org/maven2",
resolver.resolveRepository("central"));
+ }
+
+ @Test
+ void otherValuesAreKept() {
+ DefaultRepositoryResolver resolver = new DefaultRepositoryResolver();
+ resolver.build();
+
+ assertEquals("https://repo.example.com/maven2",
resolver.resolveRepository("https://repo.example.com/maven2"));
+ assertEquals("my=https://repo.example.com/maven2",
resolver.resolveRepository("my=https://repo.example.com/maven2"));
+ assertNull(resolver.resolveRepository(" "));
+ }
+}
diff --git
a/tooling/maven/camel-package-maven-plugin/src/main/java/org/apache/camel/maven/packaging/PrepareKameletMainMojo.java
b/tooling/maven/camel-package-maven-plugin/src/main/java/org/apache/camel/maven/packaging/PrepareKameletMainMojo.java
index 1180b0c23340..fcb1ab54787d 100644
---
a/tooling/maven/camel-package-maven-plugin/src/main/java/org/apache/camel/maven/packaging/PrepareKameletMainMojo.java
+++
b/tooling/maven/camel-package-maven-plugin/src/main/java/org/apache/camel/maven/packaging/PrepareKameletMainMojo.java
@@ -111,6 +111,14 @@ public class PrepareKameletMainMojo extends AbstractMojo {
@Parameter(defaultValue = "false", property =
"camel.known-dependencies.verify")
protected boolean verifyThirdPartyJars;
+ /**
+ * The repositories (id=url, separated by commas) that
verifyThirdPartyJars uses in addition to the ones of the
+ * project, for the libraries that are not in Maven Central.
+ */
+ @Parameter(defaultValue = "confluent=https://packages.confluent.io/maven/",
+ property = "camel.known-dependencies.verify.repos")
+ protected String verifyRepositories;
+
private final RepositorySystem repoSystem;
@Parameter(defaultValue = "${repositorySystemSession}", readonly = true,
required = true)
@@ -161,11 +169,12 @@ public class PrepareKameletMainMojo extends AbstractMojo {
/**
* Generates camel-thirdparty-known-dependencies.properties from the
curated list of third-party libraries
- * (CAMEL-24809). Each input line maps a package to
groupId:artifactId:version, where the version is a ${property}
- * of the project (inherited from camel-parent) or
@bom(groupId:artifactId:${property}) for a library whose version
- * a BOM manages. The version is resolved here, so the runtime needs no
lookup, and a property that does not exist
- * or a literal version fails the build. With verifyThirdPartyJars every
jar is resolved and the mapped package must
- * be found in it.
+ * (CAMEL-24809). Each input line maps a package or a class to
groupId:artifactId:version (or several, separated by
+ * commas, that are downloaded together), where the version is a
${property} of the project (inherited from
+ * camel-parent) or @bom(groupId:artifactId:${property}) for a library
whose version a BOM manages. The version is
+ * resolved here, so the runtime needs no lookup, and a property that does
not exist or a literal version fails the
+ * build. With verifyThirdPartyJars every jar is resolved and the mapped
package or class must be found in the first
+ * one.
*/
protected void updateKnownThirdPartyDependencies() throws Exception {
File input = thirdPartyLibraries.isAbsolute()
@@ -182,43 +191,15 @@ public class PrepareKameletMainMojo extends AbstractMojo {
List<String> problems = new ArrayList<>();
Map<String, String> resolved = new TreeMap<>();
for (String pkg : in.stringPropertyNames()) {
- String gav = in.getProperty(pkg).trim();
- int i = gav.indexOf(':');
- int j = gav.indexOf(':', i + 1);
- if (i < 0 || j < 0) {
- problems.add(pkg + " = " + gav + ": expected
groupId:artifactId:version");
- continue;
- }
- String groupId = gav.substring(0, i);
- String artifactId = gav.substring(i + 1, j);
- String version = gav.substring(j + 1);
- Matcher pm = PROPERTY_VERSION.matcher(version);
- Matcher bm = BOM_VERSION.matcher(version);
- if (pm.matches()) {
- String value =
project.getProperties().getProperty(pm.group(1));
- if (value == null) {
- problems.add(pkg + ": no property " + pm.group(1) + " in
parent/pom.xml");
- continue;
- }
- version = value;
- } else if (bm.matches()) {
- String bomVersion =
project.getProperties().getProperty(bm.group(3));
- if (bomVersion == null) {
- problems.add(pkg + ": no property " + bm.group(3) + " in
parent/pom.xml");
- continue;
- }
- String bomKey = bm.group(1) + ":" + bm.group(2) + ":" +
bomVersion;
- version = managedVersion(boms, bomKey, groupId, artifactId);
- if (version == null) {
- problems.add(pkg + ": " + groupId + ":" + artifactId + "
is not managed by " + bomKey);
- continue;
+ // one or more dependencies, separated by commas
+ List<String> gavs = new ArrayList<>();
+ for (String gav : in.getProperty(pkg).split(",")) {
+ String r = resolveThirdPartyGav(pkg, gav.trim(), boms,
problems);
+ if (r != null) {
+ gavs.add(r);
}
- } else {
- problems.add(pkg + " = " + gav + ": a literal version; add a
<" + artifactId.toLowerCase(Locale.ROOT)
- + "-version> property to parent/pom.xml and use
${...}");
- continue;
}
- resolved.put(pkg, groupId + ":" + artifactId + ":" + version);
+ resolved.put(pkg, String.join(",", gavs));
}
if (!problems.isEmpty()) {
throw new MojoFailureException("Problems in " + input + ":\n " +
String.join("\n ", problems));
@@ -229,8 +210,8 @@ public class PrepareKameletMainMojo extends AbstractMojo {
List<String> lines = new ArrayList<>();
lines.add("# Generated by
camel-package-maven-plugin:prepare-kamelet-main from src/main/"
+ input.getName() + " (CAMEL-24809). Do not edit.");
- lines.add("# Third-party libraries camel run downloads on demand,
mapped by package; the resolver matches the class"
- + " and then each enclosing package.");
+ lines.add("# Third-party libraries camel run downloads on demand,
mapped by package or class; the resolver matches"
+ + " the class and then each enclosing package.");
for (Map.Entry<String, String> e : resolved.entrySet()) {
lines.add(e.getKey() + " = " + e.getValue());
}
@@ -247,6 +228,51 @@ public class PrepareKameletMainMojo extends AbstractMojo {
getLog().info("Generated " + out.getFileName() + " with " +
resolved.size() + " libraries");
}
+ /**
+ * Resolves the version of one groupId:artifactId:version of the
third-party libraries.
+ *
+ * @return the dependency with its version, or null when there is a
problem (added to the problems)
+ */
+ private String resolveThirdPartyGav(String pkg, String gav, Map<String,
String> boms, List<String> problems)
+ throws Exception {
+ int i = gav.indexOf(':');
+ int j = gav.indexOf(':', i + 1);
+ if (i < 0 || j < 0) {
+ problems.add(pkg + " = " + gav + ": expected
groupId:artifactId:version");
+ return null;
+ }
+ String groupId = gav.substring(0, i);
+ String artifactId = gav.substring(i + 1, j);
+ String version = gav.substring(j + 1);
+ Matcher pm = PROPERTY_VERSION.matcher(version);
+ Matcher bm = BOM_VERSION.matcher(version);
+ if (pm.matches()) {
+ String value = project.getProperties().getProperty(pm.group(1));
+ if (value == null) {
+ problems.add(pkg + ": no property " + pm.group(1) + " in
parent/pom.xml");
+ return null;
+ }
+ version = value;
+ } else if (bm.matches()) {
+ String bomVersion =
project.getProperties().getProperty(bm.group(3));
+ if (bomVersion == null) {
+ problems.add(pkg + ": no property " + bm.group(3) + " in
parent/pom.xml");
+ return null;
+ }
+ String bomKey = bm.group(1) + ":" + bm.group(2) + ":" + bomVersion;
+ version = managedVersion(boms, bomKey, groupId, artifactId);
+ if (version == null) {
+ problems.add(pkg + ": " + groupId + ":" + artifactId + " is
not managed by " + bomKey);
+ return null;
+ }
+ } else {
+ problems.add(pkg + " = " + gav + ": a literal version; add a <" +
artifactId.toLowerCase(Locale.ROOT)
+ + "-version> property to parent/pom.xml and use
${...}");
+ return null;
+ }
+ return groupId + ":" + artifactId + ":" + version;
+ }
+
private String managedVersion(Map<String, String> cache, String bomKey,
String groupId, String artifactId)
throws Exception {
String key = bomKey + "->" + groupId + ":" + artifactId;
@@ -292,14 +318,26 @@ public class PrepareKameletMainMojo extends AbstractMojo {
}
private void verifyPackagesInJars(Map<String, String> resolved) throws
Exception {
+ List<RemoteRepository> repos = new ArrayList<>(repositories);
+ if (verifyRepositories != null) {
+ for (String repo : verifyRepositories.split(",")) {
+ int eq = repo.indexOf('=');
+ if (eq > 0) {
+ repos.add(new RemoteRepository.Builder(
+ repo.substring(0, eq).trim(), "default",
+ repo.substring(eq + 1).trim()).build());
+ }
+ }
+ }
List<String> problems = new ArrayList<>();
Map<String, File> jars = new LinkedHashMap<>();
for (Map.Entry<String, String> e : resolved.entrySet()) {
- String gav = e.getValue();
+ // the package or class is in the first dependency, the others
come with it
+ String gav = e.getValue().split(",")[0];
File jar = jars.get(gav);
if (jar == null) {
String[] parts = gav.split(":");
- ArtifactRequest req = new
ArtifactRequest().setRepositories(repositories)
+ ArtifactRequest req = new
ArtifactRequest().setRepositories(repos)
.setArtifact(new DefaultArtifact(parts[0], parts[1],
"jar", parts[2]));
try {
jar = repoSystem.resolveArtifact(repoSession,
req).getArtifact().getFile();
@@ -309,13 +347,16 @@ public class PrepareKameletMainMojo extends AbstractMojo {
}
jars.put(gav, jar);
}
+ // a package, or a class
String dir = e.getKey().replace('.', '/') + "/";
+ String clazz = e.getKey().replace('.', '/') + ".class";
boolean found;
try (ZipFile zip = new ZipFile(jar)) {
- found = zip.stream().anyMatch(z -> z.getName().startsWith(dir)
&& z.getName().endsWith(".class"));
+ found = zip.stream().anyMatch(z -> z.getName().equals(clazz)
+ || z.getName().startsWith(dir) &&
z.getName().endsWith(".class"));
}
if (!found) {
- problems.add(e.getKey() + ": no classes under " + dir + " in "
+ gav);
+ problems.add(e.getKey() + ": no class " + clazz + " nor
classes under " + dir + " in " + gav);
}
}
if (!problems.isEmpty()) {
@@ -323,7 +364,7 @@ public class PrepareKameletMainMojo extends AbstractMojo {
"Third-party known dependencies do not match their jars:\n
"
+ String.join("\n ", problems));
}
- getLog().info("Verified " + resolved.size() + " third-party packages
against " + jars.size() + " jars");
+ getLog().info("Verified " + resolved.size() + " third-party packages
and classes against " + jars.size() + " jars");
}
protected void updateKnownDependencies() throws Exception {