This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new c6e173b8 [SPARK-59979] Fix `podLogs` selectors and collect all log
lines in E2E tests
c6e173b8 is described below
commit c6e173b84aaf953117bf31e72caf08eb8e659ef4
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Oct 5 12:23:31 2026 -0700
[SPARK-59979] Fix `podLogs` selectors and collect all log lines in E2E tests
### What changes were proposed in this pull request?
This PR aims to fix the `podLogs` collectors in the `catch` blocks of the
E2E tests.
- Select Spark application pods by
`spark.operator/spark-app-name=<SparkApplication name>`, which the operator
adds to both the driver and executor pods.
- Set `tail: -1` to collect all log lines.
### Why are the changes needed?
When a test fails, most of the Spark and operator logs are missing from the
diagnostic output.
- `spark-app-name` comes from `spark.app.name`, not the `SparkApplication`
name. In these tests, it is `spark` for the driver pod and an
application-defined name like `spark-pi` for the executor pods. So,
`spark-app-name=<SparkApplication name>` (`pi-java25`, `pi-with-comet`,
`pi-with-gluten`) matches no pod, and `spark-app-name=spark` (`python`,
`spark-versions`) matches only the driver pod.
- `kubectl logs` with a selector returns only the last 10 lines per
container unless `--tail` is given.
This has been hidden because `kubectl logs` exits with 0 when no pod
matches, and the `catch` blocks run only on failures.
### Does this PR introduce _any_ user-facing change?
No. This is a test-only change.
### How was this patch tested?
Pass the CIs. Since the CIs don't run the `catch` blocks unless the tests
fail, the following were checked manually.
- On a local K8s cluster, the new selectors match the driver and executor
pods of `pi-java25` and `pi-with-comet`, while the old ones match no pod.
- With `chainsaw test --no-cluster`, the selector expressions of `python`
and `spark-versions` evaluate to their `SparkApplication` names.
- `kubectl logs` with a selector returns all lines with `--tail -1`, and
only 10 lines without it.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #934 from dongjoon-hyun/SPARK-59979.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
tests/e2e/events/chainsaw-test.yaml | 3 +++
tests/e2e/kueue/chainsaw-test.yaml | 2 ++
tests/e2e/pi-java25/chainsaw-test.yaml | 4 +++-
tests/e2e/pi-with-comet/chainsaw-test.yaml | 4 +++-
tests/e2e/pi-with-gluten/chainsaw-test.yaml | 4 +++-
tests/e2e/python/chainsaw-test.yaml | 6 +++++-
tests/e2e/resource-selector/chainsaw-test.yaml | 1 +
tests/e2e/spark-versions/chainsaw-test.yaml | 6 +++++-
tests/e2e/suspend-events/chainsaw-test.yaml | 2 ++
tests/e2e/watched-namespaces-file/chainsaw-test.yaml | 1 +
tests/e2e/watched-namespaces/chainsaw-test.yaml | 2 ++
11 files changed, 30 insertions(+), 5 deletions(-)
diff --git a/tests/e2e/events/chainsaw-test.yaml
b/tests/e2e/events/chainsaw-test.yaml
index 9feb2ae8..790fadd5 100644
--- a/tests/e2e/events/chainsaw-test.yaml
+++ b/tests/e2e/events/chainsaw-test.yaml
@@ -141,6 +141,7 @@ spec:
- podLogs:
selector: app.kubernetes.io/name=spark-kubernetes-operator
namespace: default
+ tail: -1
finally:
- script:
timeout: 30s
@@ -188,6 +189,7 @@ spec:
- podLogs:
selector: app.kubernetes.io/name=spark-kubernetes-operator
namespace: default
+ tail: -1
finally:
- script:
timeout: 30s
@@ -235,6 +237,7 @@ spec:
- podLogs:
selector: app.kubernetes.io/name=spark-kubernetes-operator
namespace: default
+ tail: -1
finally:
- script:
timeout: 30s
diff --git a/tests/e2e/kueue/chainsaw-test.yaml
b/tests/e2e/kueue/chainsaw-test.yaml
index ddd09070..3ae9a944 100644
--- a/tests/e2e/kueue/chainsaw-test.yaml
+++ b/tests/e2e/kueue/chainsaw-test.yaml
@@ -55,6 +55,7 @@ spec:
- podLogs:
selector: spark.operator/spark-app-name=spark-job-kueue-test
namespace: default
+ tail: -1
- describe:
apiVersion: kueue.x-k8s.io/v1beta2
kind: Workload
@@ -105,6 +106,7 @@ spec:
- podLogs:
selector: spark.operator/spark-cluster-name=qa
namespace: default
+ tail: -1
- describe:
apiVersion: kueue.x-k8s.io/v1beta2
kind: Workload
diff --git a/tests/e2e/pi-java25/chainsaw-test.yaml
b/tests/e2e/pi-java25/chainsaw-test.yaml
index 9f8a933c..f98d9875 100644
--- a/tests/e2e/pi-java25/chainsaw-test.yaml
+++ b/tests/e2e/pi-java25/chainsaw-test.yaml
@@ -39,11 +39,13 @@ spec:
kind: SparkApplication
namespace: default
- podLogs:
- selector: spark-app-name=pi-java25
+ selector: spark.operator/spark-app-name=pi-java25
namespace: default
+ tail: -1
- podLogs:
selector: app.kubernetes.io/name=spark-kubernetes-operator
namespace: default
+ tail: -1
finally:
- script:
timeout: 120s
diff --git a/tests/e2e/pi-with-comet/chainsaw-test.yaml
b/tests/e2e/pi-with-comet/chainsaw-test.yaml
index d22ee910..08bfee44 100644
--- a/tests/e2e/pi-with-comet/chainsaw-test.yaml
+++ b/tests/e2e/pi-with-comet/chainsaw-test.yaml
@@ -39,11 +39,13 @@ spec:
kind: SparkApplication
namespace: default
- podLogs:
- selector: spark-app-name=pi-with-comet
+ selector: spark.operator/spark-app-name=pi-with-comet
namespace: default
+ tail: -1
- podLogs:
selector: app.kubernetes.io/name=spark-kubernetes-operator
namespace: default
+ tail: -1
finally:
- script:
timeout: 120s
diff --git a/tests/e2e/pi-with-gluten/chainsaw-test.yaml
b/tests/e2e/pi-with-gluten/chainsaw-test.yaml
index f96abc02..a03b6e89 100644
--- a/tests/e2e/pi-with-gluten/chainsaw-test.yaml
+++ b/tests/e2e/pi-with-gluten/chainsaw-test.yaml
@@ -39,11 +39,13 @@ spec:
kind: SparkApplication
namespace: default
- podLogs:
- selector: spark-app-name=pi-with-gluten
+ selector: spark.operator/spark-app-name=pi-with-gluten
namespace: default
+ tail: -1
- podLogs:
selector: app.kubernetes.io/name=spark-kubernetes-operator
namespace: default
+ tail: -1
finally:
- script:
timeout: 120s
diff --git a/tests/e2e/python/chainsaw-test.yaml
b/tests/e2e/python/chainsaw-test.yaml
index e8be4a51..3d912bc4 100644
--- a/tests/e2e/python/chainsaw-test.yaml
+++ b/tests/e2e/python/chainsaw-test.yaml
@@ -40,6 +40,8 @@ spec:
file: pyspark-example.yaml
- name: verify-common-created-resources
bindings:
+ - name: SPARK_APPLICATION_NAME
+ value: (join('-', ['pyspark', replace_all(($SPARK_VERSION), '.',
'-')]))
- name: SPARK_DRIVER_POD_NAME
value: (join('-', ['pyspark', replace_all(($SPARK_VERSION), '.',
'-'), '0-driver']))
- name: SPARK_EXECUTOR_POD_NAME
@@ -64,9 +66,11 @@ spec:
- podLogs:
selector: app.kubernetes.io/name=spark-kubernetes-operator
namespace: default
+ tail: -1
- podLogs:
- selector: spark-app-name=spark
+ selector: (concat('spark.operator/spark-app-name=',
$SPARK_APPLICATION_NAME))
namespace: default
+ tail: -1
finally:
- script:
bindings:
diff --git a/tests/e2e/resource-selector/chainsaw-test.yaml
b/tests/e2e/resource-selector/chainsaw-test.yaml
index 9782036e..39dca419 100644
--- a/tests/e2e/resource-selector/chainsaw-test.yaml
+++ b/tests/e2e/resource-selector/chainsaw-test.yaml
@@ -104,6 +104,7 @@ spec:
- podLogs:
namespace: default
selector:
app.kubernetes.io/component=operator-deployment,app.kubernetes.io/name=spark-kubernetes-operator
+ tail: -1
- describe:
apiVersion: spark.apache.org/v1
kind: SparkApplication
diff --git a/tests/e2e/spark-versions/chainsaw-test.yaml
b/tests/e2e/spark-versions/chainsaw-test.yaml
index fc838c07..c6c29087 100644
--- a/tests/e2e/spark-versions/chainsaw-test.yaml
+++ b/tests/e2e/spark-versions/chainsaw-test.yaml
@@ -57,6 +57,8 @@ spec:
file: spark-example.yaml
- name: verify-common-created-resources
bindings:
+ - name: SPARK_APPLICATION_NAME
+ value: (join('-', ['spark-streaming', replace_all(($SPARK_VERSION),
'.', '-'), replace_all(($SCALA_VERSION), '.', '-')]))
- name: SPARK_DRIVER_POD_NAME
value: (join('-', ['spark-streaming', replace_all(($SPARK_VERSION),
'.', '-'),
replace_all(($SCALA_VERSION), '.', '-'), '0-driver']))
@@ -84,9 +86,11 @@ spec:
- podLogs:
selector: app.kubernetes.io/name=spark-kubernetes-operator
namespace: default
+ tail: -1
- podLogs:
- selector: spark-app-name=spark
+ selector: (concat('spark.operator/spark-app-name=',
$SPARK_APPLICATION_NAME))
namespace: default
+ tail: -1
finally:
- script:
bindings:
diff --git a/tests/e2e/suspend-events/chainsaw-test.yaml
b/tests/e2e/suspend-events/chainsaw-test.yaml
index b39c50b2..f9176b38 100644
--- a/tests/e2e/suspend-events/chainsaw-test.yaml
+++ b/tests/e2e/suspend-events/chainsaw-test.yaml
@@ -91,6 +91,7 @@ spec:
- podLogs:
selector: app.kubernetes.io/name=spark-kubernetes-operator
namespace: default
+ tail: -1
finally:
- script:
timeout: 30s
@@ -151,6 +152,7 @@ spec:
- podLogs:
selector: app.kubernetes.io/name=spark-kubernetes-operator
namespace: default
+ tail: -1
finally:
- script:
timeout: 30s
diff --git a/tests/e2e/watched-namespaces-file/chainsaw-test.yaml
b/tests/e2e/watched-namespaces-file/chainsaw-test.yaml
index 8220b963..a16d8a30 100644
--- a/tests/e2e/watched-namespaces-file/chainsaw-test.yaml
+++ b/tests/e2e/watched-namespaces-file/chainsaw-test.yaml
@@ -68,6 +68,7 @@ spec:
- podLogs:
namespace: default
selector:
app.kubernetes.io/component=operator-deployment,app.kubernetes.io/name=spark-kubernetes-operator
+ tail: -1
- describe:
apiVersion: spark.apache.org/v1
kind: SparkApplication
diff --git a/tests/e2e/watched-namespaces/chainsaw-test.yaml
b/tests/e2e/watched-namespaces/chainsaw-test.yaml
index f2aa7496..5576a33d 100644
--- a/tests/e2e/watched-namespaces/chainsaw-test.yaml
+++ b/tests/e2e/watched-namespaces/chainsaw-test.yaml
@@ -64,6 +64,7 @@ spec:
- podLogs:
namespace: default
selector:
app.kubernetes.io/component=operator-deployment,app.kubernetes.io/name=spark-kubernetes-operator
+ tail: -1
- describe:
apiVersion: spark.apache.org/v1
kind: SparkApplication
@@ -112,6 +113,7 @@ spec:
- podLogs:
namespace: default-2
selector:
app.kubernetes.io/component=operator-deployment,app.kubernetes.io/name=spark-kubernetes-operator
+ tail: -1
- describe:
apiVersion: spark.apache.org/v1
kind: SparkApplication
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
