This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 5a6ab4d1 [SPARK-59903] Upgrade `Iceberg` example to use Iceberg 1.12.0
5a6ab4d1 is described below
commit 5a6ab4d1857ad48a83fafacce055f1ed6b011d56
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Sep 30 14:40:20 2026 -0700
[SPARK-59903] Upgrade `Iceberg` example to use Iceberg 1.12.0
### What changes were proposed in this pull request?
This PR aims to upgrade `Iceberg` to 1.12.0 in the
`spark-connect-server-iceberg.yaml` example.
### Why are the changes needed?
To use the latest Iceberg release.
- https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.12.0
(2026-09-30)
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs and manually tests.
```
$ kubectl apply -f examples/spark-connect-server-iceberg.yaml
sparkapplication.spark.apache.org/spark-connect-server-iceberg created
$ kubectl get sparkapp
NAME AGE CURRENT STATE
spark-connect-server-iceberg 6s DriverReady
$ kubectl get pod
NAME READY STATUS RESTARTS
AGE
localstack 1/1 Running 0
50s
spark-connect-server-iceberg-0-driver 1/1 Running 0
15s
spark-kubernetes-operator-5c4476f5d9-8fflq 1/1 Running 0
5m15s
$ kubectl logs spark-connect-server-iceberg-0-driver | grep
iceberg-spark-runtime | head -n2
org.apache.iceberg#iceberg-spark-runtime-4.1_2.13 added as a dependency
found org.apache.iceberg#iceberg-spark-runtime-4.1_2.13;1.12.0 in
central
$ kubectl port-forward spark-connect-server-iceberg-0-driver 15002
Forwarding from 127.0.0.1:15002 -> 15002
Forwarding from [::1]:15002 -> 15002
$ bin/spark-connect-shell --remote sc://localhost:15002
...
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.1.3
/_/
Type in expressions to have them evaluated.
Spark connect server version 4.1.3.
Spark session available as 'spark'.
scala>
sql("""CREATE TABLE taxis(vendor_id bigint, trip_id bigint, trip_distance
float, fare_amount double, store_and_fwd_flag string) PARTITIONED BY
(vendor_id);""").show()
scala>
sql("""INSERT INTO taxis VALUES (1, 1000371, 1.8, 15.32, 'N'), (2, 1000372,
2.5, 22.15, 'N'), (2, 1000373, 0.9, 9.01, 'N'), (1, 1000374, 8.4, 42.13,
'Y');""").show()
scala> sql("SELECT * FROM taxis").show(false)
+---------+-------+-------------+-----------+------------------+
|vendor_id|trip_id|trip_distance|fare_amount|store_and_fwd_flag|
+---------+-------+-------------+-----------+------------------+
|1 |1000374|8.4 |42.13 |Y |
|1 |1000371|1.8 |15.32 |N |
|2 |1000373|0.9 |9.01 |N |
|2 |1000372|2.5 |22.15 |N |
+---------+-------+-------------+-----------+------------------+
scala> sql("SELECT * FROM taxis.history").show(false)
+-----------------------+-------------------+---------+-------------------+
|made_current_at |snapshot_id |parent_id|is_current_ancestor|
+-----------------------+-------------------+---------+-------------------+
|2026-09-30 21:35:59.331|1296194957924373327|NULL |true |
+-----------------------+-------------------+---------+-------------------+
scala> sql("SELECT * FROM taxis VERSION AS OF
1296194957924373327").show(false)
+---------+-------+-------------+-----------+------------------+
|vendor_id|trip_id|trip_distance|fare_amount|store_and_fwd_flag|
+---------+-------+-------------+-----------+------------------+
|1 |1000374|8.4 |42.13 |Y |
|1 |1000371|1.8 |15.32 |N |
|2 |1000373|0.9 |9.01 |N |
|2 |1000372|2.5 |22.15 |N |
+---------+-------+-------------+-----------+------------------+
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #915 from dongjoon-hyun/SPARK-59903.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
examples/spark-connect-server-iceberg.yaml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/examples/spark-connect-server-iceberg.yaml
b/examples/spark-connect-server-iceberg.yaml
index 0dd2cff4..6c3b326b 100644
--- a/examples/spark-connect-server-iceberg.yaml
+++ b/examples/spark-connect-server-iceberg.yaml
@@ -24,7 +24,7 @@ spec:
spark.hadoop.fs.s3a.path.style.access: "true"
spark.hadoop.fs.s3a.secret.key: "test"
spark.jars.ivy: "/tmp/.ivy2.5.2"
- spark.jars.packages:
"org.apache.hadoop:hadoop-aws:3.4.2,org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0"
+ spark.jars.packages:
"org.apache.hadoop:hadoop-aws:3.4.2,org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.12.0"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
spark.kubernetes.container.image: "apache/spark:4.1.3"
spark.kubernetes.executor.podNamePrefix: "spark-connect-server-iceberg"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
