dongjoon-hyun opened a new pull request, #769:
URL: https://github.com/apache/spark-kubernetes-operator/pull/769
### What changes were proposed in this pull request?
This PR aims to upgrade `Iceberg` example to use Spark 4.1.2 in the
`spark-connect-server-iceberg.yaml` example.
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual review.
```
$ kubectl apply -f examples/spark-connect-server-iceberg.yaml
sparkapplication.spark.apache.org/spark-connect-server-iceberg created
```
```
$ kubectl port-forward spark-connect-server-iceberg-0-driver 15002
```
```
$ bin/spark-connect-shell --remote sc://localhost:15002
...
$ bin/spark-connect-shell --remote sc://localhost:15002
WARNING: Using incubator modules: jdk.incubator.vector
26/07/23 14:09:02 INFO BaseAllocator: Debug mode disabled. Enable with the
VM option -Darrow.memory.debug.allocator=true.
26/07/23 14:09:02 INFO DefaultAllocationManagerOption: allocation manager
type not specified, using netty as the default type
26/07/23 14:09:02 INFO CheckAllocator: Using DefaultAllocationManager at
memory/netty/DefaultAllocationManagerFactory.class
Compiling (synthetic)/ammonite/predef/ArgsPredef.sc
Compiling
/Users/dongjoon/APACHE/spark-releases/spark-4.2.0-bin-hadoop3/(console)
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.2.0
/_/
Type in expressions to have them evaluated.
Spark connect server version 4.1.2.
Spark session available as 'spark'.
scala>
sql("""CREATE TABLE taxis(vendor_id bigint, trip_id bigint, trip_distance
float, fare_amount double, store_and_fwd_flag string) PARTITIONED BY
(vendor_id);""").show()
Using Spark's default log4j profile:
org/apache/spark/log4j2-defaults.properties
++
||
++
++
scala>
scala>
sql("""INSERT INTO taxis VALUES (1, 1000371, 1.8, 15.32, 'N'), (2, 1000372,
2.5, 22.15, 'N'), (2, 1000373, 0.9, 9.01, 'N'), (1, 1000374, 8.4, 42.13,
'Y');""").show()
++
||
++
++
scala> sql("SELECT * FROM taxis").show(false)
+---------+-------+-------------+-----------+------------------+
|vendor_id|trip_id|trip_distance|fare_amount|store_and_fwd_flag|
+---------+-------+-------------+-----------+------------------+
|1 |1000374|8.4 |42.13 |Y |
|1 |1000371|1.8 |15.32 |N |
|2 |1000372|2.5 |22.15 |N |
|2 |1000373|0.9 |9.01 |N |
+---------+-------+-------------+-----------+------------------+
scala> sql("SELECT * FROM taxis.history").show(false)
+-----------------------+-------------------+---------+-------------------+
|made_current_at |snapshot_id |parent_id|is_current_ancestor|
+-----------------------+-------------------+---------+-------------------+
|2026-07-23 21:09:24.528|5673557555333823555|NULL |true |
+-----------------------+-------------------+---------+-------------------+
scala> sql("SELECT * FROM taxis VERSION AS OF
5673557555333823555").show(false)
+---------+-------+-------------+-----------+------------------+
|vendor_id|trip_id|trip_distance|fare_amount|store_and_fwd_flag|
+---------+-------+-------------+-----------+------------------+
|1 |1000374|8.4 |42.13 |Y |
|1 |1000371|1.8 |15.32 |N |
|2 |1000372|2.5 |22.15 |N |
|2 |1000373|0.9 |9.01 |N |
+---------+-------+-------------+-----------+------------------+
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
