This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new b8722ab [SPARK-55499] Update `pi-with-eventlog` to generate multiple
log files
b8722ab is described below
commit b8722abec4d6f12629da0598344d5dcc0adffc83
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Feb 12 09:20:34 2026 -0800
[SPARK-55499] Update `pi-with-eventlog` to generate multiple log files
### What changes were proposed in this pull request?
This PR aims to update `pi-with-eventlog` example to generate multiple log
files.
### Why are the changes needed?
To demonstrate event log v2 to the users.
```bash
$ kubectl exec localstack -- awslocal s3 ls
s3://spark-events/eventlog_v2_pi-with-eventlog-0/
2026-02-12 17:03:07 0
2026-02-12 17:03:17 0 appstatus_pi-with-eventlog-0
2026-02-12 17:03:14 77757 events_1_pi-with-eventlog-0.zstd
2026-02-12 17:03:15 50313 events_2_pi-with-eventlog-0.zstd
2026-02-12 17:03:16 48229 events_3_pi-with-eventlog-0.zstd
2026-02-12 17:03:17 47627 events_4_pi-with-eventlog-0.zstd
2026-02-12 17:03:17 13469 events_5_pi-with-eventlog-0.zstd
```
### Does this PR introduce _any_ user-facing change?
No behavior change.
### How was this patch tested?
Manual review.
```
$ kubectl apply -f examples/localstack.yml
$ kubectl apply -f examples/pi-with-eventlog.yaml
# Check the log after the job finishes.
$ kubectl exec localstack -- awslocal s3 ls
s3://spark-events/eventlog_v2_pi-with-eventlog-0/
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #497 from dongjoon-hyun/SPARK-55499.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
examples/pi-with-eventlog.yaml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/examples/pi-with-eventlog.yaml b/examples/pi-with-eventlog.yaml
index 965b95e..cf2b693 100644
--- a/examples/pi-with-eventlog.yaml
+++ b/examples/pi-with-eventlog.yaml
@@ -19,6 +19,7 @@ metadata:
spec:
mainClass: "org.apache.spark.examples.SparkPi"
jars: "local:///opt/spark/examples/jars/spark-examples.jar"
+ driverArgs: [ "3000" ]
sparkConf:
spark.jars.packages: "org.apache.hadoop:hadoop-aws:3.4.2"
spark.jars.ivy: "/tmp/.ivy2.5.2"
@@ -29,6 +30,7 @@ spec:
spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}-scala"
spark.eventLog.enabled: "true"
spark.eventLog.dir: "s3a://spark-events/"
+ spark.eventLog.rolling.maxFileSize: "2MB"
spark.hadoop.fs.s3a.endpoint: "http://localstack:4566";
spark.hadoop.fs.s3a.path.style.access: "true"
spark.hadoop.fs.s3a.access.key: "test"
@@ -36,5 +38,6 @@ spec:
spark.driver.memory: "2g"
applicationTolerations:
resourceRetainPolicy: OnFailure
+ ttlAfterStopMillis: 10000
runtimeVersions:
sparkVersion: "4.1.1"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
