This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 31e9e6f [SPARK-56985] Upgrade `Iceberg` example to use Iceberg 1.11.0
31e9e6f is described below
commit 31e9e6f56ad025df9cdb51cdf3fc945a97ea337b
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu May 21 09:03:38 2026 -0700
[SPARK-56985] Upgrade `Iceberg` example to use Iceberg 1.11.0
### What changes were proposed in this pull request?
This PR aims to upgrade `Iceberg` to 1.11.0 in the
`spark-connect-server-iceberg.yaml` example.
### Why are the changes needed?
To use the latest Iceberg release.
- https://iceberg.apache.org/releases/#1110-release
- https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.11.0
(2026-05-19)
- https://github.com/apache/iceberg/pull/14400
- https://github.com/apache/iceberg/pull/14099
- https://github.com/apache/iceberg/pull/14155
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual review.
```
$ kubectl apply -f examples/spark-connect-server-iceberg.yaml
sparkapplication.spark.apache.org/spark-connect-server-iceberg created
$ kubectl get sparkapp
NAME AGE CURRENT STATE
spark-connect-server-iceberg 46s RunningHealthy
$ kybectl get pod
NAME READY STATUS RESTARTS
AGE
spark-connect-server-iceberg-0-driver 1/1 Running 0
47s
spark-connect-server-iceberg-exec-1 1/1 Running 0
11s
spark-connect-server-iceberg-exec-2 1/1 Running 0
11s
spark-connect-server-iceberg-exec-3 1/1 Running 0
11s
spark-kubernetes-operator-86bfb88655-w5fs8 1/1 Running 7 (42h ago)
6d19h
$ kubectl logs -f spark-connect-server-iceberg-0-driver
...
org.apache.iceberg#iceberg-spark-runtime-4.0_2.13 added as a dependency
...
found org.apache.iceberg#iceberg-spark-runtime-4.0_2.13;1.11.0 in
central
...
downloading
https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-4.0_2.13/1.11.0/iceberg-spark-runtime-4.0_2.13-1.11.0.jar
...
[SUCCESSFUL ]
org.apache.iceberg#iceberg-spark-runtime-4.0_2.13;1.11.0!iceberg-spark-runtime-4.0_2.13.jar
(834ms)
...
org.apache.iceberg#iceberg-spark-runtime-4.0_2.13;1.11.0 from
central in [default]
...
Files
file:///tmp/.ivy2.5.2/jars/org.apache.iceberg_iceberg-spark-runtime-4.0_2.13-1.11.0.jar
from
/tmp/.ivy2.5.2/jars/org.apache.iceberg_iceberg-spark-runtime-4.0_2.13-1.11.0.jar
to
/opt/spark/work-dir/org.apache.iceberg_iceberg-spark-runtime-4.0_2.13-1.11.0.jar
...
26/05/21 14:37:11 INFO SparkConnectServer: Starting Spark session.
26/05/21 14:37:11 INFO SparkContext: Running Spark version 4.0.2
26/05/21 14:37:11 INFO SparkContext: OS info Linux, 6.6.119-0-virt, aarch64
26/05/21 14:37:11 INFO SparkContext: Java version 17.0.17
...
26/05/21 14:37:17 INFO Utils: Successfully started service
'org.apache.spark.sql.connect.service.SparkConnectService$' on port 15002.
26/05/21 14:37:17 INFO SparkConnectServer: Spark Connect server started at:
0:0:0:0:0:0:0:0:15002
```
```
$ kubectl port-forward spark-connect-server-iceberg-0-driver 15002
```
```
$ bin/spark-connect-shell --remote sc://localhost:15002
...
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.0.2
/_/
Type in expressions to have them evaluated.
Spark connect server version 4.0.2.
Spark session available as 'spark'.
scala>
sql("""CREATE TABLE taxis(vendor_id bigint, trip_id bigint, trip_distance
float, fare_amount double, store_and_fwd_flag string) PARTITIONED BY
(vendor_id);""").show()
Using Spark's default log4j profile:
org/apache/spark/log4j2-defaults.properties
++
||
++
++
scala>
sql("""INSERT INTO taxis VALUES (1, 1000371, 1.8, 15.32, 'N'), (2, 1000372,
2.5, 22.15, 'N'), (2, 1000373, 0.9, 9.01, 'N'), (1, 1000374, 8.4, 42.13,
'Y');""").show()
++
||
++
++
scala> sql("SELECT * FROM taxis").show(false)
+---------+-------+-------------+-----------+------------------+
|vendor_id|trip_id|trip_distance|fare_amount|store_and_fwd_flag|
+---------+-------+-------------+-----------+------------------+
|1 |1000374|8.4 |42.13 |Y |
|1 |1000371|1.8 |15.32 |N |
|2 |1000373|0.9 |9.01 |N |
|2 |1000372|2.5 |22.15 |N |
+---------+-------+-------------+-----------+------------------+
scala> sql("SELECT * FROM taxis.history").show(false)
+-----------------------+-------------------+---------+-------------------+
|made_current_at |snapshot_id |parent_id|is_current_ancestor|
+-----------------------+-------------------+---------+-------------------+
|2026-05-21 14:55:46.629|4314466511351517368|NULL |true |
+-----------------------+-------------------+---------+-------------------+
scala> sql("SELECT * FROM taxis VERSION AS OF
4314466511351517368").show(false)
+---------+-------+-------------+-----------+------------------+
|vendor_id|trip_id|trip_distance|fare_amount|store_and_fwd_flag|
+---------+-------+-------------+-----------+------------------+
|1 |1000374|8.4 |42.13 |Y |
|1 |1000371|1.8 |15.32 |N |
|2 |1000373|0.9 |9.01 |N |
|2 |1000372|2.5 |22.15 |N |
+---------+-------+-------------+-----------+------------------+
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.7
Closes #681 from dongjoon-hyun/SPARK-56985.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
examples/spark-connect-server-iceberg.yaml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/examples/spark-connect-server-iceberg.yaml
b/examples/spark-connect-server-iceberg.yaml
index b6d0e88..657a1e8 100644
--- a/examples/spark-connect-server-iceberg.yaml
+++ b/examples/spark-connect-server-iceberg.yaml
@@ -28,7 +28,7 @@ spec:
spark.hadoop.fs.s3a.path.style.access: "true"
spark.hadoop.fs.s3a.secret.key: "test"
spark.jars.ivy: "/tmp/.ivy2.5.2"
- spark.jars.packages:
"org.apache.hadoop:hadoop-aws:3.4.1,org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.10.1"
+ spark.jars.packages:
"org.apache.hadoop:hadoop-aws:3.4.1,org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.11.0"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
spark.kubernetes.container.image: "apache/spark:4.0.2"
spark.kubernetes.executor.podNamePrefix: "spark-connect-server-iceberg"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
