This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch 33.0.0
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/33.0.0 by this push:
new 607ff981a3a Add documentation + fix bug for task pods in different
namespace scenario (#17898)
607ff981a3a is described below
commit 607ff981a3a95df43720f6696e0a5422606e812d
Author: Virushade <[email protected]>
AuthorDate: Mon Apr 14 19:56:56 2025 +0800
Add documentation + fix bug for task pods in different namespace scenario
(#17898)
(cherry picked from commit 899cb025e835842898aa89d6348a746bdc7ed40e)
---
docs/development/extensions-core/k8s-jobs.md | 12 +++--
.../k8s/overlord/KubernetesTaskRunnerConfig.java | 6 +--
.../k8s/overlord/common/KubernetesPeonClient.java | 14 ++---
.../taskadapter/PodTemplateTaskAdapter.java | 4 +-
.../overlord/common/KubernetesPeonClientTest.java | 60 +++++++++++++++++++++-
5 files changed, 76 insertions(+), 20 deletions(-)
diff --git a/docs/development/extensions-core/k8s-jobs.md
b/docs/development/extensions-core/k8s-jobs.md
index a544b006f82..3e7b096fc16 100644
--- a/docs/development/extensions-core/k8s-jobs.md
+++ b/docs/development/extensions-core/k8s-jobs.md
@@ -744,9 +744,12 @@ It is possible to run task pods in a different namespace
from the rest of your D
If you are running multiple Druid clusters and would like to have a dedicated
namespace for all your task pods, you can make the following changes to the
runtime properties for your Overlord deployment:
- `druid.indexer.runner.namespace`: The namespace where the task pods will
run. It can be the same as the namespace where your Druid cluster is deployed,
or different from it. In the latter case, you need to define the following
`overlordNamespace`.
-- `druid.indexer.runner.overlordNamespace`: The namespace where the Overlord
resides. This must be defined when tasks are scheduled in different namespace.
When `druid.indexer.runner.overlordNamespace` is not provided, Druid will
assume that the entire cluster operates within a single namespace and will
default to `druid.indexer.runner.namespace`.
+- `druid.indexer.runner.overlordNamespace`: The namespace where the Overlord
resides. This must be defined when tasks are scheduled in different namespace.
+- `druid.indexer.runner.k8sTaskPodNamePrefix` (Optional): Self-defined field
to differentiate which task pods are created from which namespace. More
information [here](#differentiating-task-pods-created-from-multiple-namespaces).
-Druid will tag Kubernetes jobs with a `druid.overlord.namespace` label. This
label helps Druid filter out Kubernetes jobs belonging to other namespaces.
+Warning: When `druid.indexer.runner.overlordNamespace` and
`druid.indexer.runner.k8sTaskPodNamePrefix` is configured, users should ensure
that all running tasks are stopped when changing these values. Failure to do so
will cause the Overlord to lose track of running tasks, and re-launch them.
This may lead to duplicate data and possibly metadata inconsistency issues.
+
+Druid will tag Kubernetes jobs with a `druid.overlord.namespace` label. This
label helps Druid filter out Kubernetes jobs belonging to other namespaces.
Should you need to deploy a Druid cluster on a namespace `N1` that is already
running tasks from another namespace `N2`, take note to set
`druid.indexer.runner.overlordNamespace` to `druid.indexer.runner.namespace`
(which is `N1`). Failure to do so will result in the cluster in `N1` detecting
task pods created from both `N1` and `N2`.
##### Differentiating Task Pods Created From Multiple Namespaces
@@ -758,6 +761,7 @@ When configuring the
`druid.indexer.runner.k8sTaskPodNamePrefix`, you should not
- The prefix will cut off at 30 characters, as the task pod names must respect
a character limit of 63 in Kubernetes.
- Special characters `: - . _` will be ignored.
- The prefix will be converted to lowercase.
+- All running tasks must be stopped during configuration. Failure to do so
will cause the Overlord to lose track of running tasks, and re-launch them.
This may lead to duplicate data and possibly metadata inconsistency issues.
##### Dealing with ZooKeeper Problems
@@ -771,8 +775,8 @@ Should you require the needed permissions for interacting
across Kubernetes name
| Property | Possible Values | Description | Default | Required |
| --- | --- | --- | --- | --- |
| `druid.indexer.runner.namespace` | `String` | If Overlord and task pods are
running in different namespaces, specify the Overlord namespace. | - | Yes |
-| `druid.indexer.runner.overlordNamespace` | `String` | Only applicable when
using Custom Template Pod Adapter. If Overlord and task pods are running in
different namespaces, specify the Overlord namespace. |
`druid.indexer.runner.namespace` | No |
-| `druid.indexer.runner.k8sTaskPodNamePrefix` | `String` | Use this if you
want to change your task name to contain a more human-readable prefix. Maximum
30 characters. Special characters `: - . _` will be ignored. | `""` | No |
+| `druid.indexer.runner.overlordNamespace` | `String` | Only applicable when
using Custom Template Pod Adapter. If Overlord and task pods are running in
different namespaces, specify the Overlord namespace. <br /> Warning: You need
to stop all running tasks in Druid to change this property. Failure to do so
will lead to duplicate data and metadata inconsistencies. | `""` | No |
+| `druid.indexer.runner.k8sTaskPodNamePrefix` | `String` | Use this if you
want to change your task name to contain a more human-readable prefix. Maximum
30 characters. Special characters `: - . _` will be ignored. <br /> Warning:
You need to stop all running tasks in Druid to change this property. Failure to
do so will lead to duplicate data and metadata inconsistencies. | `""` | No |
| `druid.indexer.runner.debugJobs` | `boolean` | Clean up K8s jobs after tasks
complete. | False | No |
| `druid.indexer.runner.sidecarSupport` | `boolean` | Deprecated, specify
adapter type as runtime property `druid.indexer.runner.k8s.adapter.type:
overlordMultiContainer` instead. If your overlord pod has sidecars, this will
attempt to start the task with the same sidecars as the overlord pod. | False |
No |
| `druid.indexer.runner.primaryContainerName` | `String` | If running with
sidecars, the `primaryContainerName` should be that of your druid container
like `druid-overlord`. | First container in `podSpec` list | No |
diff --git
a/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunnerConfig.java
b/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunnerConfig.java
index 44613ffe973..ef4f0740e05 100644
---
a/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunnerConfig.java
+++
b/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunnerConfig.java
@@ -26,7 +26,6 @@ import org.apache.commons.lang3.ObjectUtils;
import org.joda.time.Period;
import javax.annotation.Nonnull;
-import javax.annotation.Nullable;
import javax.validation.constraints.Max;
import javax.validation.constraints.Min;
import javax.validation.constraints.NotNull;
@@ -46,8 +45,7 @@ public class KubernetesTaskRunnerConfig
// For cases where we want task pods to run on different namespace from the
overlord, we need to specify the namespace of the overlord here.
// Else, we can simply leave this field alone.
@JsonProperty
- @Nullable
- private String overlordNamespace;
+ private String overlordNamespace = "";
@JsonProperty
private boolean debugJobs = false;
@@ -243,7 +241,7 @@ public class KubernetesTaskRunnerConfig
public String getOverlordNamespace()
{
- return overlordNamespace == null ? namespace : overlordNamespace;
+ return overlordNamespace;
}
public String getK8sTaskPodNamePrefix()
diff --git
a/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClient.java
b/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClient.java
index cea3e9631f2..6ea5ae4312e 100644
---
a/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClient.java
+++
b/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClient.java
@@ -33,7 +33,6 @@ import org.apache.druid.java.util.emitter.EmittingLogger;
import org.apache.druid.java.util.emitter.service.ServiceEmitter;
import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
-import javax.annotation.Nullable;
import java.io.InputStream;
import java.sql.Timestamp;
import java.util.ArrayList;
@@ -47,7 +46,6 @@ public class KubernetesPeonClient
private final KubernetesClientApi clientApi;
private final String namespace;
- private final boolean overlordMaybeInAnotherNamespace;
private final String overlordNamespace;
private final boolean debugJobs;
private final ServiceEmitter emitter;
@@ -55,7 +53,7 @@ public class KubernetesPeonClient
public KubernetesPeonClient(
KubernetesClientApi clientApi,
String namespace,
- @Nullable String overlordNamespace,
+ String overlordNamespace,
boolean debugJobs,
ServiceEmitter emitter
)
@@ -65,7 +63,6 @@ public class KubernetesPeonClient
this.overlordNamespace = overlordNamespace;
this.debugJobs = debugJobs;
this.emitter = emitter;
- this.overlordMaybeInAnotherNamespace = overlordNamespace != null;
}
public KubernetesPeonClient(
@@ -75,7 +72,7 @@ public class KubernetesPeonClient
ServiceEmitter emitter
)
{
- this(clientApi, namespace, null, debugJobs, emitter);
+ this(clientApi, namespace, "", debugJobs, emitter);
}
public Pod launchPeonJobAndWaitForStart(Job job, Task task, long howLong,
TimeUnit timeUnit) throws IllegalStateException
@@ -198,10 +195,9 @@ public class KubernetesPeonClient
public List<Job> getPeonJobs()
{
- if (overlordMaybeInAnotherNamespace) {
- return getPeonJobsWithOverlordNamespaceKeyLabels();
- }
- return getPeonJobsWithoutOverlordNamespaceKeyLabels();
+ return this.overlordNamespace.isEmpty()
+ ? getPeonJobsWithoutOverlordNamespaceKeyLabels()
+ : getPeonJobsWithOverlordNamespaceKeyLabels();
}
private List<Job> getPeonJobsWithoutOverlordNamespaceKeyLabels()
diff --git
a/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/taskadapter/PodTemplateTaskAdapter.java
b/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/taskadapter/PodTemplateTaskAdapter.java
index e7bf0c13de5..1229a6eb89c 100644
---
a/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/taskadapter/PodTemplateTaskAdapter.java
+++
b/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/taskadapter/PodTemplateTaskAdapter.java
@@ -267,13 +267,13 @@ public class PodTemplateTaskAdapter implements TaskAdapter
private Map<String, String> getJobLabels(KubernetesTaskRunnerConfig config,
Task task)
{
- // Namespace is required, or else getOverlordNamespace() will return null,
and this will not work correctly.
Preconditions.checkNotNull(config.getNamespace(), "When using Custom Pod
Templates, druid.indexer.runner.namespace cannot be null.");
+ String overlordNamespace = config.getOverlordNamespace().isEmpty() ?
config.getNamespace() : config.getOverlordNamespace();
return ImmutableMap.<String, String>builder()
.putAll(config.getLabels())
.put(DruidK8sConstants.LABEL_KEY, "true")
- .put(DruidK8sConstants.OVERLORD_NAMESPACE_KEY,
config.getOverlordNamespace())
+ .put(DruidK8sConstants.OVERLORD_NAMESPACE_KEY, overlordNamespace)
.put(getDruidLabel(DruidK8sConstants.TASK_ID),
KubernetesOverlordUtils.convertTaskIdToK8sLabel(task.getId()))
.put(getDruidLabel(DruidK8sConstants.TASK_TYPE),
KubernetesOverlordUtils.convertStringToK8sLabel(task.getType()))
.put(getDruidLabel(DruidK8sConstants.TASK_GROUP_ID),
KubernetesOverlordUtils.convertTaskIdToK8sLabel(task.getGroupId()))
diff --git
a/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClientTest.java
b/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClientTest.java
index f5e94426989..63e95d9a72f 100644
---
a/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClientTest.java
+++
b/extensions-core/kubernetes-overlord-extensions/src/test/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClientTest.java
@@ -350,7 +350,6 @@ public class KubernetesPeonClientTest
.withNewMetadata()
.withName(KUBERNETES_JOB_NAME)
.addToLabels(DruidK8sConstants.LABEL_KEY, "true")
- .addToLabels(DruidK8sConstants.OVERLORD_NAMESPACE_KEY, NAMESPACE)
.endMetadata()
.build();
@@ -361,6 +360,65 @@ public class KubernetesPeonClientTest
Assertions.assertEquals(1, jobs.size());
}
+ @Test
+ void test_getPeonJobs_withJobInDifferentNamespace_returnsPodList()
+ {
+ instance = new KubernetesPeonClient(clientApi, NAMESPACE, "ns", false,
serviceEmitter);
+
+ Job job = new JobBuilder()
+ .withNewMetadata()
+ .withName(KUBERNETES_JOB_NAME)
+ .addToLabels(DruidK8sConstants.LABEL_KEY, "true")
+ .addToLabels(DruidK8sConstants.OVERLORD_NAMESPACE_KEY, "ns")
+ .endMetadata()
+ .build();
+
+ client.batch().v1().jobs().inNamespace(NAMESPACE).resource(job).create();
+
+ List<Job> jobs = instance.getPeonJobs();
+
+ Assertions.assertEquals(1, jobs.size());
+ }
+
+ @Test
+ void
test_getPeonJobs_withJobInDifferentNamespaceButOverlordNamespaceNotSpecified_doesNotReturnPodList()
+ {
+ instance = new KubernetesPeonClient(clientApi, NAMESPACE, "ns", false,
serviceEmitter);
+
+ Job job = new JobBuilder()
+ .withNewMetadata()
+ .withName(KUBERNETES_JOB_NAME)
+ .addToLabels(DruidK8sConstants.LABEL_KEY, "true")
+ .addToLabels(DruidK8sConstants.OVERLORD_NAMESPACE_KEY,
"someOtherNamespace")
+ .endMetadata()
+ .build();
+
+ client.batch().v1().jobs().inNamespace(NAMESPACE).resource(job).create();
+
+ List<Job> jobs = instance.getPeonJobs();
+
+ Assertions.assertEquals(0, jobs.size());
+ }
+
+ @Test
+ void
test_getPeonJobs_withJobInSameNamespaceWithoutLabels_doesNotReturnPodList()
+ {
+ instance = new KubernetesPeonClient(clientApi, NAMESPACE, NAMESPACE,
false, serviceEmitter);
+
+ Job job = new JobBuilder()
+ .withNewMetadata()
+ .withName(KUBERNETES_JOB_NAME)
+ .addToLabels(DruidK8sConstants.LABEL_KEY, "true")
+ .endMetadata()
+ .build();
+
+ client.batch().v1().jobs().inNamespace(NAMESPACE).resource(job).create();
+
+ List<Job> jobs = instance.getPeonJobs();
+
+ Assertions.assertEquals(0, jobs.size());
+ }
+
@Test
void test_getPeonJobs_withoutJob_returnsEmptyList()
{
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
