This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 8d602bc [SPARK-58299] Upgrade `Iceberg` example to use Spark 4.1.2
8d602bc is described below
commit 8d602bcedbd047ee8927993bb1b591a51b60d6ce
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Jul 23 14:17:37 2026 -0700
[SPARK-58299] Upgrade `Iceberg` example to use Spark 4.1.2
### What changes were proposed in this pull request?
This PR aims to upgrade `Iceberg` example to use Spark 4.1.2 in the
`spark-connect-server-iceberg.yaml` example.
### Why are the changes needed?
To provide an up-to-date example.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual review.
```
$ kubectl apply -f examples/spark-connect-server-iceberg.yaml
sparkapplication.spark.apache.org/spark-connect-server-iceberg created
```
```
$ kubectl port-forward spark-connect-server-iceberg-0-driver 15002
```
```
$ bin/spark-connect-shell --remote sc://localhost:15002
...
$ bin/spark-connect-shell --remote sc://localhost:15002
WARNING: Using incubator modules: jdk.incubator.vector
26/07/23 14:09:02 INFO BaseAllocator: Debug mode disabled. Enable with the
VM option -Darrow.memory.debug.allocator=true.
26/07/23 14:09:02 INFO DefaultAllocationManagerOption: allocation manager
type not specified, using netty as the default type
26/07/23 14:09:02 INFO CheckAllocator: Using DefaultAllocationManager at
memory/netty/DefaultAllocationManagerFactory.class
Compiling (synthetic)/ammonite/predef/ArgsPredef.sc
Compiling
/Users/dongjoon/APACHE/spark-releases/spark-4.2.0-bin-hadoop3/(console)
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.2.0
/_/
Type in expressions to have them evaluated.
Spark connect server version 4.1.2.
Spark session available as 'spark'.
scala>
sql("""CREATE TABLE taxis(vendor_id bigint, trip_id bigint, trip_distance
float, fare_amount double, store_and_fwd_flag string) PARTITIONED BY
(vendor_id);""").show()
Using Spark's default log4j profile:
org/apache/spark/log4j2-defaults.properties
++
||
++
++
scala>
scala>
sql("""INSERT INTO taxis VALUES (1, 1000371, 1.8, 15.32, 'N'), (2, 1000372,
2.5, 22.15, 'N'), (2, 1000373, 0.9, 9.01, 'N'), (1, 1000374, 8.4, 42.13,
'Y');""").show()
++
||
++
++
scala> sql("SELECT * FROM taxis").show(false)
+---------+-------+-------------+-----------+------------------+
|vendor_id|trip_id|trip_distance|fare_amount|store_and_fwd_flag|
+---------+-------+-------------+-----------+------------------+
|1 |1000374|8.4 |42.13 |Y |
|1 |1000371|1.8 |15.32 |N |
|2 |1000372|2.5 |22.15 |N |
|2 |1000373|0.9 |9.01 |N |
+---------+-------+-------------+-----------+------------------+
scala> sql("SELECT * FROM taxis.history").show(false)
+-----------------------+-------------------+---------+-------------------+
|made_current_at |snapshot_id |parent_id|is_current_ancestor|
+-----------------------+-------------------+---------+-------------------+
|2026-07-23 21:09:24.528|5673557555333823555|NULL |true |
+-----------------------+-------------------+---------+-------------------+
scala> sql("SELECT * FROM taxis VERSION AS OF
5673557555333823555").show(false)
+---------+-------+-------------+-----------+------------------+
|vendor_id|trip_id|trip_distance|fare_amount|store_and_fwd_flag|
+---------+-------+-------------+-----------+------------------+
|1 |1000374|8.4 |42.13 |Y |
|1 |1000371|1.8 |15.32 |N |
|2 |1000372|2.5 |22.15 |N |
|2 |1000373|0.9 |9.01 |N |
+---------+-------+-------------+-----------+------------------+
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #769 from dongjoon-hyun/SPARK-58299.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
examples/spark-connect-server-iceberg.yaml | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/examples/spark-connect-server-iceberg.yaml
b/examples/spark-connect-server-iceberg.yaml
index c5b6432..76a325f 100644
--- a/examples/spark-connect-server-iceberg.yaml
+++ b/examples/spark-connect-server-iceberg.yaml
@@ -28,9 +28,9 @@ spec:
spark.hadoop.fs.s3a.path.style.access: "true"
spark.hadoop.fs.s3a.secret.key: "test"
spark.jars.ivy: "/tmp/.ivy2.5.2"
- spark.jars.packages:
"org.apache.hadoop:hadoop-aws:3.4.1,org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.11.0"
+ spark.jars.packages:
"org.apache.hadoop:hadoop-aws:3.4.2,org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
- spark.kubernetes.container.image: "apache/spark:4.0.3"
+ spark.kubernetes.container.image: "apache/spark:4.1.2"
spark.kubernetes.executor.podNamePrefix: "spark-connect-server-iceberg"
spark.scheduler.mode: "FAIR"
spark.sql.catalog.s3.type: "hadoop"
@@ -41,4 +41,4 @@ spec:
applicationTolerations:
resourceRetainPolicy: OnFailure
runtimeVersions:
- sparkVersion: "4.0.3"
+ sparkVersion: "4.1.2"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
