This is an automated email from the ASF dual-hosted git repository.
capistrant pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new b81ebcee78e Upgrade Kubernetes clients and related dependencies in
`kubernetes-overlord-extensions` and `druid-kubernetes-extensions` (#19071)
b81ebcee78e is described below
commit b81ebcee78eab89df68e1f8b08a6bcdb4ad88ddb
Author: Lucas Capistrant <[email protected]>
AuthorDate: Tue Mar 10 11:34:49 2026 -0500
Upgrade Kubernetes clients and related dependencies in
`kubernetes-overlord-extensions` and `druid-kubernetes-extensions` (#19071)
* fabric8 bump checkpoint
* expose webclientoptions in configuration and override additionalconfig
for vertx
* document new power
* Remove unnecessary complication of the druid vertx factory wrapper
* use junit5
* bump okhttp to try and fix a dependency enforcer
* update licenses.yaml
* nit spacing
* min dep enforcer requires bump of kotlin-stdlib
* tinkering with licenses yaml
* Bump kubernetes-extensions k8s dep so we can align on okhttp
also get a bunch of other licenses in line
* cleanup object mapper for k8s-ol-ext
* minor dep updates + exclude protobuf from druid-kubernetes-extensions
* Keep working on getting these dependencies to play nicely together
* Update aws sdk in licenses
* fix kubernetes-extensions pom
* Working on static checks still
* k8s overlord ext pom
---
docs/development/extensions-core/k8s-jobs.md | 29 ++++++
embedded-tests/pom.xml | 1 -
extensions-core/kubernetes-extensions/pom.xml | 61 +++++++----
.../druid/k8s/discovery/DefaultK8sApiClient.java | 49 ++++++++-
.../kubernetes-overlord-extensions/pom.xml | 13 ++-
.../k8s/overlord/KubernetesOverlordModule.java | 9 +-
.../DruidKubernetesVertxHttpClientConfig.java | 11 ++
.../DruidKubernetesVertxHttpClientFactory.java | 38 +++++--
.../overlord/KubernetesTaskRunnerFactoryTest.java | 2 +-
.../DruidKubernetesVertxHttpClientConfigTest.java | 80 +++++++++++++++
.../DruidKubernetesVertxHttpClientFactoryTest.java | 113 +++++++++++++++++++++
.../DruidPeonClientIntegrationTest.java | 2 +-
licenses.yaml | 78 +++++++++-----
pom.xml | 44 ++++++--
14 files changed, 456 insertions(+), 74 deletions(-)
diff --git a/docs/development/extensions-core/k8s-jobs.md
b/docs/development/extensions-core/k8s-jobs.md
index a9c32370d60..222a9bac2df 100644
--- a/docs/development/extensions-core/k8s-jobs.md
+++ b/docs/development/extensions-core/k8s-jobs.md
@@ -911,6 +911,35 @@ was
[okhttp](https://github.com/fabric8io/kubernetes-client/tree/main/httpclient
|`druid.indexer.runner.k8sAndWorker.http.vertx.eventLoopPoolSize`|`Integer`|...|`2
* number cores`|No|
|`druid.indexer.runner.k8sAndWorker.http.vertx.internalBlockingPoolSize`|`Integer`|...|20|No|
+##### `WebClientOptions` pass-through
+
+The vert.x HTTP client also supports a generic pass-through for any property
on the underlying Vert.x
[`WebClientOptions`](https://vertx.io/docs/apidocs/io/vertx/ext/web/client/WebClientOptions.html)
object (which extends
[`HttpClientOptions`](https://vertx.io/docs/apidocs/io/vertx/core/http/HttpClientOptions.html)).
This gives operators full control over connection pool behavior, timeouts, and
other low-level HTTP client settings without requiring new Druid configuration
fields.
+
+Set properties using the prefix
`druid.indexer.runner.k8sAndWorker.http.vertx.webClientOptions.<propertyName>`,
where `<propertyName>` matches the corresponding setter on `WebClientOptions`
(e.g., `setMaxPoolSize` maps to `maxPoolSize`).
+
+**Tuning connection keep-alive timeouts**
+
+This pass-through is particularly useful for environments where an
intermediate network component (such as an AWS ALB, or service mesh sidecar)
closes idle connections before the client expects it. When this happens, the
client may attempt to reuse a connection that the server has already closed,
resulting in unexpected connection errors.
+
+To mitigate this, set the client-side keep-alive and idle timeouts to a value
**lower** than the shortest server-side timeout in your network path. For
example, if your AWS ALB has a 60 second idle timeout (the default), setting
the client to 30 seconds ensures the client retires idle connections well
before the server does:
+
+```properties
+druid.indexer.runner.k8sAndWorker.http.vertx.webClientOptions.keepAliveTimeout=30
+druid.indexer.runner.k8sAndWorker.http.vertx.webClientOptions.idleTimeout=30
+```
+
+**Commonly useful properties**
+
+|Property|Type|Description|Default|
+|--------|----|-----------|-------|
+|`keepAliveTimeout`|`Integer`|Seconds an idle HTTP keep-alive connection is
retained before the client closes it.|`60`|
+|`idleTimeout`|`Integer`|Seconds a connection can remain idle before being
closed. Acts as a general safety net alongside `keepAliveTimeout`.|`0`
(disabled)|
+|`maxPoolSize`|`Integer`|Maximum number of connections in the pool per
endpoint.|`5`|
+|`connectTimeout`|`Integer`|Milliseconds to wait when establishing a new
connection.|`60000`|
+|`poolCleanerPeriod`|`Integer`|Milliseconds between pool sweeps that evict
connections exceeding the above timeouts.|`1000`|
+
+Any property with a public setter on `WebClientOptions` or its parent classes
(`HttpClientOptions`, `ClientOptionsBase`, `TCPSSLOptions`, `NetworkOptions`)
can be set through this mechanism.
+
#### OkHttp Client
|Property| Possible Values |Description| Default |required|
diff --git a/embedded-tests/pom.xml b/embedded-tests/pom.xml
index 192bfa71b03..75e33d0cae0 100644
--- a/embedded-tests/pom.xml
+++ b/embedded-tests/pom.xml
@@ -572,7 +572,6 @@
<dependency>
<groupId>io.kubernetes</groupId>
<artifactId>client-java-api</artifactId>
- <version>19.0.0</version>
<scope>test</scope>
</dependency>
<dependency>
diff --git a/extensions-core/kubernetes-extensions/pom.xml
b/extensions-core/kubernetes-extensions/pom.xml
index 9b80d9bb129..dc02ef900cd 100644
--- a/extensions-core/kubernetes-extensions/pom.xml
+++ b/extensions-core/kubernetes-extensions/pom.xml
@@ -34,23 +34,6 @@
<relativePath>../../pom.xml</relativePath>
</parent>
- <properties>
- <kubernetes.client.version>19.0.0</kubernetes.client.version>
- </properties>
-
-
- <dependencyManagement>
- <dependencies>
- <!-- This is an indirect dependency of io.kubernetes.client-java
- update to address vulnerability in transitive dependency okio used by
okhttp -->
- <dependency>
- <groupId>com.squareup.okhttp3</groupId>
- <artifactId>okhttp</artifactId>
- <version>4.12.0</version>
- </dependency>
- </dependencies>
- </dependencyManagement>
-
<dependencies>
<dependency>
<groupId>org.apache.druid</groupId>
@@ -68,17 +51,53 @@
<dependency>
<groupId>io.kubernetes</groupId>
<artifactId>client-java</artifactId>
- <version>${kubernetes.client.version}</version>
+ <exclusions>
+ <!-- client-java-proto brings in protobuf-java 4.x which Druid does
not yet support project-wide.
+ The Kubernetes API supports multiple wire formats (JSON, YAML,
Protobuf, etc.) but this extension
+ is not currently using the proto client so exclusion is safe -->
+ <exclusion>
+ <groupId>io.kubernetes</groupId>
+ <artifactId>client-java-proto</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.google.protobuf</groupId>
+ <artifactId>protobuf-java</artifactId>
+ </exclusion>
+ <!-- Azure AD auth library and its transitive OIDC/JWT stack — only
needed for AKS token-based
+ authentication which this extension does not use -->
+ <exclusion>
+ <groupId>com.microsoft.azure</groupId>
+ <artifactId>adal4j</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
<dependency>
<groupId>io.kubernetes</groupId>
<artifactId>client-java-extended</artifactId>
- <version>${kubernetes.client.version}</version>
+ <exclusions>
+ <!-- client-java-proto brings in protobuf-java 4.x which Druid does
not yet support project-wide.
+ The Kubernetes API supports multiple wire formats (JSON, YAML,
Protobuf, etc.) but this extension
+ is not currently using the proto client so exclusion is safe -->
+ <exclusion>
+ <groupId>io.kubernetes</groupId>
+ <artifactId>client-java-proto</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
<dependency>
<groupId>io.kubernetes</groupId>
<artifactId>client-java-api</artifactId>
- <version>${kubernetes.client.version}</version>
+ <exclusions>
+ <!-- jakarta.ws.rs-api 4.0 conflicts with the javax.ws.rs 1.x used
elsewhere in Druid -->
+ <exclusion>
+ <groupId>jakarta.ws.rs</groupId>
+ <artifactId>jakarta.ws.rs-api</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>okhttp-jvm</artifactId>
</dependency>
<!-- Tests -->
@@ -140,7 +159,7 @@
<artifactId>maven-dependency-plugin</artifactId>
<configuration>
<!-- analyze incorrectly flags this dependency as missing when
omitted, and unused when declared -->
-
<ignoredDependencies>io.kubernetes:client-java-api-fluent:jar:19.0.0</ignoredDependencies>
+
<ignoredDependencies>io.kubernetes:client-java-api-fluent:jar:${kubernetes.client.version}</ignoredDependencies>
</configuration>
</plugin>
</plugins>
diff --git
a/extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DefaultK8sApiClient.java
b/extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DefaultK8sApiClient.java
index 03a1b6914a8..adf10a42dd4 100644
---
a/extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DefaultK8sApiClient.java
+++
b/extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DefaultK8sApiClient.java
@@ -30,6 +30,7 @@ import io.kubernetes.client.openapi.ApiException;
import io.kubernetes.client.openapi.apis.CoreV1Api;
import io.kubernetes.client.openapi.models.V1Pod;
import io.kubernetes.client.openapi.models.V1PodList;
+import io.kubernetes.client.util.PatchUtils;
import io.kubernetes.client.util.Watch;
import org.apache.druid.discovery.DiscoveryDruidNode;
import org.apache.druid.discovery.NodeRole;
@@ -65,7 +66,22 @@ public class DefaultK8sApiClient implements K8sApiClient
public void patchPod(String podName, String podNamespace, String
jsonPatchStr)
{
try {
- coreV1Api.patchNamespacedPod(podName, podNamespace, new
V1Patch(jsonPatchStr), "true", null, null, null, null);
+ PatchUtils.patch(
+ V1Pod.class,
+ () -> coreV1Api.patchNamespacedPodCall(
+ podName,
+ podNamespace,
+ new V1Patch(jsonPatchStr),
+ "true",
+ null,
+ null,
+ null,
+ null,
+ null
+ ),
+ V1Patch.PATCH_FORMAT_JSON_PATCH,
+ realK8sClient
+ );
}
catch (ApiException ex) {
throw new RE(ex, "Failed to patch pod[%s/%s], code[%d], error[%s].",
podNamespace, podName, ex.getCode(), ex.getResponseBody());
@@ -80,7 +96,20 @@ public class DefaultK8sApiClient implements K8sApiClient
)
{
try {
- V1PodList podList = coreV1Api.listNamespacedPod(podNamespace, null,
null, null, null, labelSelector, 0, null, null, null, null, null);
+ V1PodList podList = coreV1Api.listNamespacedPod(
+ podNamespace,
+ null,
+ null,
+ null,
+ null,
+ labelSelector,
+ null,
+ null,
+ null,
+ null,
+ null,
+ null
+ );
Preconditions.checkState(podList != null, "WTH: NULL podList");
Map<String, DiscoveryDruidNode> allNodes = new HashMap();
@@ -113,8 +142,20 @@ public class DefaultK8sApiClient implements K8sApiClient
Watch<V1Pod> watch =
Watch.createWatch(
realK8sClient,
- coreV1Api.listNamespacedPodCall(namespace, null, true, null,
null,
- labelSelector, null,
lastKnownResourceVersion, null, null, 0, true, null
+ coreV1Api.listNamespacedPodCall(
+ namespace,
+ null,
+ true,
+ null,
+ null,
+ labelSelector,
+ null,
+ lastKnownResourceVersion,
+ null,
+ null,
+ null,
+ true,
+ null
),
new TypeReference<Watch.Response<V1Pod>>()
{
diff --git a/extensions-core/kubernetes-overlord-extensions/pom.xml
b/extensions-core/kubernetes-overlord-extensions/pom.xml
index 61698bd3c3f..f90d596b58e 100644
--- a/extensions-core/kubernetes-overlord-extensions/pom.xml
+++ b/extensions-core/kubernetes-overlord-extensions/pom.xml
@@ -34,6 +34,9 @@
<relativePath>../../pom.xml</relativePath>
</parent>
+ <properties>
+ <vertx.version>4.5.24</vertx.version>
+ </properties>
<dependencies>
<dependency>
@@ -138,13 +141,17 @@
</dependency>
<dependency>
<groupId>com.squareup.okhttp3</groupId>
- <artifactId>okhttp</artifactId>
- <version>4.12.0</version>
+ <artifactId>okhttp-jvm</artifactId>
</dependency>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-core</artifactId>
- <version>4.5.24</version>
+ <version>${vertx.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.vertx</groupId>
+ <artifactId>vertx-web-client</artifactId>
+ <version>${vertx.version}</version>
</dependency>
<dependency>
<groupId>javax.ws.rs</groupId>
diff --git
a/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesOverlordModule.java
b/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesOverlordModule.java
index b6ba2fdd677..96b4e05db55 100644
---
a/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesOverlordModule.java
+++
b/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesOverlordModule.java
@@ -39,6 +39,7 @@ import org.apache.druid.guice.JsonConfigProvider;
import org.apache.druid.guice.JsonConfigurator;
import org.apache.druid.guice.LazySingleton;
import org.apache.druid.guice.PolyBind;
+import org.apache.druid.guice.annotations.Json;
import org.apache.druid.guice.annotations.LoadScope;
import org.apache.druid.guice.annotations.Self;
import org.apache.druid.guice.annotations.Smile;
@@ -386,17 +387,19 @@ public class KubernetesOverlordModule implements
DruidModule
private static class VertxHttpClientFactoryProvider implements
Provider<DruidKubernetesHttpClientFactory>
{
private DruidKubernetesVertxHttpClientConfig config;
+ private ObjectMapper jsonMapper;
@Inject
- public void inject(DruidKubernetesVertxHttpClientConfig config)
+ public void inject(DruidKubernetesVertxHttpClientConfig config, @Json
ObjectMapper jsonMapper)
{
- this.config = config; // Guice injects the Vertx-specific config
+ this.config = config;
+ this.jsonMapper = jsonMapper;
}
@Override
public DruidKubernetesHttpClientFactory get()
{
- return new DruidKubernetesVertxHttpClientFactory(config);
+ return new DruidKubernetesVertxHttpClientFactory(config, jsonMapper);
}
}
diff --git
a/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/httpclient/vertx/DruidKubernetesVertxHttpClientConfig.java
b/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/httpclient/vertx/DruidKubernetesVertxHttpClientConfig.java
index 14e7a122d40..681423fb300 100644
---
a/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/httpclient/vertx/DruidKubernetesVertxHttpClientConfig.java
+++
b/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/httpclient/vertx/DruidKubernetesVertxHttpClientConfig.java
@@ -22,6 +22,9 @@ package org.apache.druid.k8s.overlord.common.httpclient.vertx;
import com.fasterxml.jackson.annotation.JsonProperty;
import io.vertx.core.VertxOptions;
+import java.util.Collections;
+import java.util.Map;
+
public class DruidKubernetesVertxHttpClientConfig
{
@JsonProperty
@@ -33,6 +36,9 @@ public class DruidKubernetesVertxHttpClientConfig
@JsonProperty
private int internalBlockingPoolSize =
VertxOptions.DEFAULT_INTERNAL_BLOCKING_POOL_SIZE;
+ @JsonProperty
+ private Map<String, Object> webClientOptions = Collections.emptyMap();
+
public int getWorkerPoolSize()
{
return workerPoolSize;
@@ -47,4 +53,9 @@ public class DruidKubernetesVertxHttpClientConfig
{
return internalBlockingPoolSize;
}
+
+ public Map<String, Object> getWebClientOptions()
+ {
+ return webClientOptions;
+ }
}
diff --git
a/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/httpclient/vertx/DruidKubernetesVertxHttpClientFactory.java
b/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/httpclient/vertx/DruidKubernetesVertxHttpClientFactory.java
index f26c9ddcc1c..b33d1d23873 100644
---
a/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/httpclient/vertx/DruidKubernetesVertxHttpClientFactory.java
+++
b/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/httpclient/vertx/DruidKubernetesVertxHttpClientFactory.java
@@ -19,31 +19,55 @@
package org.apache.druid.k8s.overlord.common.httpclient.vertx;
-import io.fabric8.kubernetes.client.vertx.VertxHttpClientBuilder;
+import com.fasterxml.jackson.databind.ObjectMapper;
import io.fabric8.kubernetes.client.vertx.VertxHttpClientFactory;
import io.vertx.core.Vertx;
import io.vertx.core.VertxOptions;
import io.vertx.core.file.FileSystemOptions;
import io.vertx.core.spi.resolver.ResolverProvider;
+import io.vertx.ext.web.client.WebClientOptions;
+import org.apache.druid.java.util.common.logger.Logger;
import
org.apache.druid.k8s.overlord.common.httpclient.DruidKubernetesHttpClientFactory;
/**
* Similar to {@link VertxHttpClientFactory} but allows us to override thread
pool configurations.
*/
-public class DruidKubernetesVertxHttpClientFactory implements
DruidKubernetesHttpClientFactory
+public class DruidKubernetesVertxHttpClientFactory extends
VertxHttpClientFactory implements DruidKubernetesHttpClientFactory
{
public static final String TYPE_NAME = "vertx";
- private final Vertx vertx;
- public DruidKubernetesVertxHttpClientFactory(final
DruidKubernetesVertxHttpClientConfig httpClientConfig)
+ private static final Logger LOG = new
Logger(DruidKubernetesVertxHttpClientFactory.class);
+
+ private final DruidKubernetesVertxHttpClientConfig httpClientConfig;
+ private final ObjectMapper objectMapper;
+
+ public DruidKubernetesVertxHttpClientFactory(
+ DruidKubernetesVertxHttpClientConfig httpClientConfig,
+ ObjectMapper objectMapper
+ )
{
- this.vertx = createVertxInstance(httpClientConfig);
+ super(createVertxInstance(httpClientConfig));
+ this.httpClientConfig = httpClientConfig;
+ this.objectMapper = objectMapper;
}
@Override
- public VertxHttpClientBuilder<DruidKubernetesVertxHttpClientFactory>
newBuilder()
+ protected void additionalConfig(WebClientOptions options)
{
- return new VertxHttpClientBuilder<>(this, vertx);
+ if (!httpClientConfig.getWebClientOptions().isEmpty()) {
+ try {
+ LOG.info("Applying additional WebClientOptions from configuration:
%s", httpClientConfig.getWebClientOptions());
+ objectMapper.updateValue(options,
httpClientConfig.getWebClientOptions());
+ }
+ catch (Exception e) {
+ throw new RuntimeException(
+ "Failed to apply webClientOptions to WebClientOptions. "
+ + "Check that all property names and values are valid. "
+ + "Properties provided: " + httpClientConfig.getWebClientOptions(),
+ e
+ );
+ }
+ }
}
/**
diff --git
a/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/KubernetesTaskRunnerFactoryTest.java
b/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/KubernetesTaskRunnerFactoryTest.java
index 96aec79b68f..f13dc57e8e3 100644
---
a/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/KubernetesTaskRunnerFactoryTest.java
+++
b/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/KubernetesTaskRunnerFactoryTest.java
@@ -67,7 +67,7 @@ public class KubernetesTaskRunnerFactoryTest
Config config = new ConfigBuilder().build();
druidKubernetesClient =
- new DruidKubernetesClient(new
DruidKubernetesVertxHttpClientFactory(new
DruidKubernetesVertxHttpClientConfig()), config);
+ new DruidKubernetesClient(new
DruidKubernetesVertxHttpClientFactory(new
DruidKubernetesVertxHttpClientConfig(), new ObjectMapper()), config);
druidKubernetesCachingClient = null;
taskAdapter = new TestTaskAdapter();
kubernetesTaskRunnerConfig = new
KubernetesTaskRunnerEffectiveConfig(kubernetesTaskRunnerStaticConfig, () ->
null);
diff --git
a/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/common/httpclient/vertx/DruidKubernetesVertxHttpClientConfigTest.java
b/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/common/httpclient/vertx/DruidKubernetesVertxHttpClientConfigTest.java
new file mode 100644
index 00000000000..28946242dc3
--- /dev/null
+++
b/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/common/httpclient/vertx/DruidKubernetesVertxHttpClientConfigTest.java
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.k8s.overlord.common.httpclient.vertx;
+
+import com.google.common.collect.ImmutableList;
+import com.google.inject.Injector;
+import org.apache.druid.guice.GuiceInjectors;
+import org.apache.druid.guice.JsonConfigProvider;
+import org.apache.druid.guice.JsonConfigurator;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.Properties;
+
+class DruidKubernetesVertxHttpClientConfigTest
+{
+ private static final String PROPERTY_PREFIX =
"druid.indexer.runner.k8sAndWorker.http.vertx";
+
+ @Test
+ void testSerdeWebClientOptionsMap()
+ {
+ final Injector injector = createInjector();
+ final JsonConfigProvider<DruidKubernetesVertxHttpClientConfig> provider =
JsonConfigProvider.of(
+ PROPERTY_PREFIX,
+ DruidKubernetesVertxHttpClientConfig.class
+ );
+ final Properties properties = new Properties();
+ properties.put(PROPERTY_PREFIX + ".webClientOptions.maxPoolSize", "10");
+ properties.put(PROPERTY_PREFIX + ".webClientOptions.connectTimeout",
"5000");
+ properties.put(PROPERTY_PREFIX + ".webClientOptions.idleTimeout", "120");
+ provider.inject(properties, injector.getInstance(JsonConfigurator.class));
+ final DruidKubernetesVertxHttpClientConfig config = provider.get();
+ Assertions.assertNotNull(config.getWebClientOptions());
+ Assertions.assertEquals(3, config.getWebClientOptions().size());
+ Assertions.assertEquals("10",
config.getWebClientOptions().get("maxPoolSize"));
+ Assertions.assertEquals("5000",
config.getWebClientOptions().get("connectTimeout"));
+ Assertions.assertEquals("120",
config.getWebClientOptions().get("idleTimeout"));
+ }
+
+ @Test
+ void testSerdeEmptyWebClientOptionsMap()
+ {
+ final Injector injector = createInjector();
+ final JsonConfigProvider<DruidKubernetesVertxHttpClientConfig> provider =
JsonConfigProvider.of(
+ PROPERTY_PREFIX,
+ DruidKubernetesVertxHttpClientConfig.class
+ );
+ final Properties properties = new Properties();
+ provider.inject(properties, injector.getInstance(JsonConfigurator.class));
+ final DruidKubernetesVertxHttpClientConfig config = provider.get();
+ Assertions.assertNotNull(config.getWebClientOptions());
+ Assertions.assertEquals(0, config.getWebClientOptions().size());
+ }
+
+ private Injector createInjector()
+ {
+ return GuiceInjectors.makeStartupInjectorWithModules(
+ ImmutableList.of(
+ binder -> JsonConfigProvider.bind(binder, PROPERTY_PREFIX,
DruidKubernetesVertxHttpClientConfig.class)
+ )
+ );
+ }
+}
diff --git
a/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/common/httpclient/vertx/DruidKubernetesVertxHttpClientFactoryTest.java
b/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/common/httpclient/vertx/DruidKubernetesVertxHttpClientFactoryTest.java
new file mode 100644
index 00000000000..bf9022a4dfe
--- /dev/null
+++
b/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/common/httpclient/vertx/DruidKubernetesVertxHttpClientFactoryTest.java
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.k8s.overlord.common.httpclient.vertx;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.ImmutableList;
+import com.google.inject.Injector;
+import io.vertx.ext.web.client.WebClientOptions;
+import org.apache.druid.guice.GuiceInjectors;
+import org.apache.druid.guice.JsonConfigProvider;
+import org.apache.druid.guice.JsonConfigurator;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.Properties;
+
+class DruidKubernetesVertxHttpClientFactoryTest
+{
+ private static final String PROPERTY_PREFIX =
"druid.indexer.runner.k8sAndWorker.http.vertx";
+
+ @Test
+ void testAdditionalConfigAppliesWebClientOptions()
+ {
+ final Injector injector = createInjector();
+ final JsonConfigProvider<DruidKubernetesVertxHttpClientConfig> provider =
JsonConfigProvider.of(
+ PROPERTY_PREFIX,
+ DruidKubernetesVertxHttpClientConfig.class
+ );
+ final Properties properties = new Properties();
+ properties.put(PROPERTY_PREFIX + ".webClientOptions.maxPoolSize", "10");
+ properties.put(PROPERTY_PREFIX + ".webClientOptions.connectTimeout",
"5000");
+ properties.put(PROPERTY_PREFIX + ".webClientOptions.idleTimeout", "120");
+ provider.inject(properties, injector.getInstance(JsonConfigurator.class));
+ final DruidKubernetesVertxHttpClientConfig config = provider.get();
+
+ DruidKubernetesVertxHttpClientFactory factory = new
DruidKubernetesVertxHttpClientFactory(config, new ObjectMapper());
+
+ WebClientOptions options = new WebClientOptions();
+ factory.additionalConfig(options);
+
+ Assertions.assertEquals(10, options.getMaxPoolSize());
+ Assertions.assertEquals(5000, options.getConnectTimeout());
+ Assertions.assertEquals(120, options.getIdleTimeout());
+ }
+
+ @Test
+ void testAdditionalConfigWithEmptyMapDoesNotModifyOptions()
+ {
+ DruidKubernetesVertxHttpClientConfig config = new
DruidKubernetesVertxHttpClientConfig();
+ DruidKubernetesVertxHttpClientFactory factory = new
DruidKubernetesVertxHttpClientFactory(config, new ObjectMapper());
+
+ WebClientOptions options = new WebClientOptions();
+ int defaultMaxPoolSize = options.getMaxPoolSize();
+ int defaultConnectTimeout = options.getConnectTimeout();
+
+ factory.additionalConfig(options);
+
+ Assertions.assertEquals(defaultMaxPoolSize, options.getMaxPoolSize());
+ Assertions.assertEquals(defaultConnectTimeout,
options.getConnectTimeout());
+ }
+
+ @Test
+ void testAdditionalConfigWithInvalidPropertyThrowsException()
+ {
+ final Injector injector = createInjector();
+ final JsonConfigProvider<DruidKubernetesVertxHttpClientConfig> provider =
JsonConfigProvider.of(
+ PROPERTY_PREFIX,
+ DruidKubernetesVertxHttpClientConfig.class
+ );
+ final Properties properties = new Properties();
+ properties.put(PROPERTY_PREFIX + ".webClientOptions.nonExistentProperty",
"value");
+ provider.inject(properties, injector.getInstance(JsonConfigurator.class));
+ final DruidKubernetesVertxHttpClientConfig config = provider.get();
+
+ DruidKubernetesVertxHttpClientFactory factory = new
DruidKubernetesVertxHttpClientFactory(config, new ObjectMapper());
+
+ WebClientOptions options = new WebClientOptions();
+ RuntimeException exception = Assertions.assertThrows(
+ RuntimeException.class,
+ () -> factory.additionalConfig(options)
+ );
+ Assertions.assertTrue(
+ exception.getMessage().contains("Failed to apply webClientOptions"),
+ exception.getMessage()
+ );
+ }
+
+ private Injector createInjector()
+ {
+ return GuiceInjectors.makeStartupInjectorWithModules(
+ ImmutableList.of(
+ binder -> JsonConfigProvider.bind(binder, PROPERTY_PREFIX,
DruidKubernetesVertxHttpClientConfig.class)
+ )
+ );
+ }
+}
diff --git
a/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/taskadapter/DruidPeonClientIntegrationTest.java
b/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/taskadapter/DruidPeonClientIntegrationTest.java
index 68371261e16..6ad8f3baa6e 100644
---
a/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/taskadapter/DruidPeonClientIntegrationTest.java
+++
b/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/taskadapter/DruidPeonClientIntegrationTest.java
@@ -90,7 +90,7 @@ public class DruidPeonClientIntegrationTest
new NamedType(ParallelIndexTuningConfig.class, "index_parallel"),
new NamedType(IndexTask.IndexTuningConfig.class, "index")
);
- k8sClient = new DruidKubernetesClient(new
DruidKubernetesVertxHttpClientFactory(new
DruidKubernetesVertxHttpClientConfig()), new ConfigBuilder().build());
+ k8sClient = new DruidKubernetesClient(new
DruidKubernetesVertxHttpClientFactory(new
DruidKubernetesVertxHttpClientConfig(), new ObjectMapper()), new
ConfigBuilder().build());
peonClient = new KubernetesPeonClient(k8sClient, "default", null, false,
new NoopServiceEmitter());
druidNode = new DruidNode(
"test",
diff --git a/licenses.yaml b/licenses.yaml
index d32056b3bad..72fd052d3fe 100644
--- a/licenses.yaml
+++ b/licenses.yaml
@@ -191,7 +191,7 @@ name: AWS SDK for Java
license_category: binary
module: java-core
license_name: Apache License version 2.0
-version: 1.12.784
+version: 1.12.793
libraries:
- com.amazonaws: aws-java-sdk-core
- com.amazonaws: aws-java-sdk-ec2
@@ -498,7 +498,7 @@ name: Apache Commons Codec
license_category: binary
module: java-core
license_name: Apache License version 2.0
-version: 1.17.1
+version: 1.20.0
libraries:
- commons-codec: commons-codec
notices:
@@ -667,7 +667,7 @@ name: Apache Commons Lang
license_category: binary
module: java-core
license_name: Apache License version 2.0
-version: 3.19.0
+version: 3.20.0
libraries:
- org.apache.commons: commons-lang3
notices:
@@ -860,7 +860,7 @@ name: kubernetes fabric java client
license_category: binary
module: extensions-core/kubernetes-overlord-extensions
license_name: Apache License version 2.0
-version: 7.5.2
+version: 7.6.0
libraries:
- io.fabric8: kubernetes-client-api
- io.fabric8: kubernetes-model-batch
@@ -900,13 +900,14 @@ libraries:
- io.vertx: vertx-core
- io.vertx: vertx-web-client
- io.vertx: vertx-web-common
+ - io.vertx: vertx-uri-template
---
name: kubernetes official java client
license_category: binary
module: extensions/druid-kubernetes-extensions
license_name: Apache License version 2.0
-version: 19.0.0
+version: 25.0.0-legacy
libraries:
- io.kubernetes: client-java
- io.kubernetes: client-java-api
@@ -927,33 +928,33 @@ libraries:
---
-name: org.apache.commons commons-collections4
+name: Swagger
+version: 1.6.12
license_category: binary
-module: extensions/druid-kubernetes-extensions
+module: extensions/druid-protobuf-extensions
license_name: Apache License version 2.0
-version: 4.4
libraries:
- - org.apache.commons: commons-collections4
+ - io.swagger: swagger-annotations
---
-name: io.sundr builder-annotations
+name: org.apache.commons commons-collections4
license_category: binary
module: extensions/druid-kubernetes-extensions
license_name: Apache License version 2.0
-version: 0.22.0
+version: 4.4
libraries:
- - io.sundr: builder-annotations
+ - org.apache.commons: commons-collections4
---
-name: com.squareup.okio okio
+name: io.sundr builder-annotations
license_category: binary
module: extensions/druid-kubernetes-extensions
license_name: Apache License version 2.0
-version: 1.17.2
+version: 0.22.0
libraries:
- - com.squareup.okio: okio
+ - io.sundr: builder-annotations
---
@@ -961,7 +962,7 @@ name: io.gsonfire gson-fire
license_category: binary
module: extensions/druid-kubernetes-extensions
license_name: Apache License version 2.0
-version: 1.8.5
+version: 1.9.0
libraries:
- io.gsonfire: gson-fire
@@ -971,7 +972,7 @@ name: io.swagger swagger-annotations
license_category: binary
module: extensions/druid-kubernetes-extensions
license_name: Apache License version 2.0
-version: 1.6.12
+version: 1.6.16
libraries:
- io.swagger: swagger-annotations
@@ -1032,19 +1033,22 @@ name: com.squareup.okhttp3 okhttp
license_category: binary
module: extensions/druid-kubernetes-extensions
license_name: Apache License version 2.0
-version: 4.12.0
+version: 5.3.2
libraries:
- com.squareup.okhttp3: okhttp
+ - com.squareup.okhttp3: okhttp-jvm
- com.squareup.okhttp3: logging-interceptor
---
-name: com.squareup.okhttp3 okhttp logging-interceptor
+name: com.squareup.okhttp3 okhttp
license_category: binary
-module: extensions/druid-kubernetes-extensions
+module: extensions-core/kubernetes-overlord-extensions
license_name: Apache License version 2.0
-version: 4.11.0
+version: 5.3.2
libraries:
+ - com.squareup.okhttp3: okhttp
+ - com.squareup.okhttp3: okhttp-jvm
- com.squareup.okhttp3: logging-interceptor
---
@@ -1053,13 +1057,23 @@ name: com.squareup.okio okio
license_category: binary
module: extensions/druid-kubernetes-extensions
license_name: Apache License version 2.0
-version: 3.6.0
+version: 3.16.4
libraries:
- com.squareup.okio: okio
- com.squareup.okio: okio-jvm
---
+name: com.squareup.okio okio
+license_category: binary
+module: extensions-core/kubernetes-overlord-extensions
+license_name: Apache License version 2.0
+version: 3.16.4
+libraries:
+ - com.squareup.okio: okio-jvm
+
+---
+
name: io.prometheus simpleclient
license_category: binary
module: extensions/druid-kubernetes-extensions
@@ -1079,7 +1093,7 @@ name: org.snakeyaml snakeyaml-engine
license_category: binary
module: extensions-core/kubernetes-overlord-extensions
license_name: Apache License version 2.0
-version: "2.10"
+version: 3.0.1
libraries:
- org.snakeyaml: snakeyaml-engine
@@ -1099,7 +1113,7 @@ name: org.yaml snakeyaml
license_category: binary
module: extensions/druid-kubernetes-extensions
license_name: Apache License version 2.0
-version: 2.2
+version: 2.5
libraries:
- org.yaml: snakeyaml
@@ -1109,7 +1123,7 @@ name: com.flipkart.zjsonpatch zjsonpatch
license_category: binary
module: extensions/druid-kubernetes-extensions
license_name: Apache License version 2.0
-version: 0.4.14
+version: 0.4.16
libraries:
- com.flipkart.zjsonpatch: zjsonpatch
@@ -1152,7 +1166,7 @@ name: Jetbrains kotlin-stdlib
license_category: binary
module: extensions/kubernetes-extensions
license_name: Apache License version 2.0
-version: 1.9.25
+version: 2.2.21
libraries:
- org.jetbrains.kotlin: kotlin-stdlib
@@ -2548,7 +2562,7 @@ name: Gson
license_category: binary
module: hadoop-client
license_name: Apache License version 2.0
-version: 2.12.0
+version: 2.13.2
libraries:
- com.google.code.gson: gson
@@ -2977,6 +2991,16 @@ libraries:
---
+name: jakarta.annotation
+version: 3.0.0
+license_category: binary
+module: extensions/druid-kubernetes-extensions
+license_name: Eclipse Public License 2.0
+libraries:
+ - jakarta.annotation: jakarta.annotation-api
+
+---
+
name: jakarta.transaction
version: 1.3.3
license_category: binary
diff --git a/pom.xml b/pom.xml
index d49bfd798c7..390ba0d0617 100644
--- a/pom.xml
+++ b/pom.xml
@@ -80,7 +80,7 @@
and update as needed in extensions-core/druid-ranger-security/pm.xml
-->
<apache.ranger.version>2.7.0</apache.ranger.version>
<antlr4.version>4.5.3</antlr4.version>
- <gson.version>2.12.0</gson.version>
+ <gson.version>2.13.2</gson.version>
<scala.library.version>2.13.14</scala.library.version>
<avatica.version>1.27.0</avatica.version>
<avro.version>1.11.5</avro.version>
@@ -96,7 +96,7 @@
<derby.version>10.14.2.0</derby.version>
<dropwizard.metrics.version>4.2.22</dropwizard.metrics.version>
<errorprone.version>2.35.1</errorprone.version>
- <fabric8.version>7.5.2</fabric8.version>
+ <fabric8.version>7.6.0</fabric8.version>
<fastutil.version>8.5.4</fastutil.version>
<guava.version>32.1.3-jre</guava.version>
<guice.version>6.0.0</guice.version>
@@ -119,12 +119,14 @@
<hadoop.compile.version>3.3.6</hadoop.compile.version>
<graaljs.version>22.3.5</graaljs.version>
<mockito.version>5.14.2</mockito.version>
- <aws.sdk.version>1.12.784</aws.sdk.version>
+ <aws.sdk.version>1.12.793</aws.sdk.version>
<caffeine.version>2.8.0</caffeine.version>
<jacoco.version>0.8.14</jacoco.version>
<testcontainers.version>2.0.3</testcontainers.version>
<hibernate-validator.version>6.2.5.Final</hibernate-validator.version>
<httpclient.version>4.5.13</httpclient.version>
+ <okhttp.version>5.3.2</okhttp.version>
+ <kubernetes.client.version>25.0.0-legacy</kubernetes.client.version>
<!-- When upgrading ZK, edit docs and integration tests as well
(integration-tests/docker-base/setup.sh) -->
<zookeeper.version>3.8.4</zookeeper.version>
<checkerframework.version>3.48.1</checkerframework.version>
@@ -317,7 +319,7 @@
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
- <version>1.17.1</version>
+ <version>1.20.0</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
@@ -347,7 +349,7 @@
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
- <version>3.19.0</version>
+ <version>3.20.0</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
@@ -411,7 +413,7 @@
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib</artifactId>
- <version>1.9.25</version>
+ <version>2.2.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains</groupId>
@@ -441,6 +443,36 @@
<artifactId>commons-configuration2</artifactId>
<version>2.10.1</version>
</dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>okhttp</artifactId>
+ <version>${okhttp.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>okhttp-jvm</artifactId>
+ <version>${okhttp.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>logging-interceptor</artifactId>
+ <version>${okhttp.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.kubernetes</groupId>
+ <artifactId>client-java-api</artifactId>
+ <version>${kubernetes.client.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.kubernetes</groupId>
+ <artifactId>client-java</artifactId>
+ <version>${kubernetes.client.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.kubernetes</groupId>
+ <artifactId>client-java-extended</artifactId>
+ <version>${kubernetes.client.version}</version>
+ </dependency>
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
