This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f9252a908a03 [SPARK-50177][BUILD] Upgrade `Arrow` to 18.0.0
f9252a908a03 is described below
commit f9252a908a032884f0cb6c2771e47f3d1b6d8b98
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Oct 30 17:47:25 2024 -0700
[SPARK-50177][BUILD] Upgrade `Arrow` to 18.0.0
### What changes were proposed in this pull request?
This PR aims to upgrade `Arrow` to 18.0.0 for Apache Spark 4.0.0.
### Why are the changes needed?
To bring the latest improvements and bug fixes,
- https://arrow.apache.org/release/18.0.0.html (28 October 2024)
Note that `Arrow 18` is the first release who dropped Java 8 like Apache
Spark 4.0.0.
- https://github.com/apache/arrow/issues/38051
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #48708 from dongjoon-hyun/SPARK-50177.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/deps/spark-deps-hadoop-3-hive-2.3 | 10 +++++-----
pom.xml | 2 +-
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 7e9d2edde5f7..c7185538bebb 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -16,11 +16,11 @@ antlr4-runtime/4.13.1//antlr4-runtime-4.13.1.jar
aopalliance-repackaged/3.0.6//aopalliance-repackaged-3.0.6.jar
arpack/3.0.3//arpack-3.0.3.jar
arpack_combined_all/0.1//arpack_combined_all-0.1.jar
-arrow-format/17.0.0//arrow-format-17.0.0.jar
-arrow-memory-core/17.0.0//arrow-memory-core-17.0.0.jar
-arrow-memory-netty-buffer-patch/17.0.0//arrow-memory-netty-buffer-patch-17.0.0.jar
-arrow-memory-netty/17.0.0//arrow-memory-netty-17.0.0.jar
-arrow-vector/17.0.0//arrow-vector-17.0.0.jar
+arrow-format/18.0.0//arrow-format-18.0.0.jar
+arrow-memory-core/18.0.0//arrow-memory-core-18.0.0.jar
+arrow-memory-netty-buffer-patch/18.0.0//arrow-memory-netty-buffer-patch-18.0.0.jar
+arrow-memory-netty/18.0.0//arrow-memory-netty-18.0.0.jar
+arrow-vector/18.0.0//arrow-vector-18.0.0.jar
audience-annotations/0.12.0//audience-annotations-0.12.0.jar
avro-ipc/1.12.0//avro-ipc-1.12.0.jar
avro-mapred/1.12.0//avro-mapred-1.12.0.jar
diff --git a/pom.xml b/pom.xml
index 130adee3d91b..b7bcc9c7c8f0 100644
--- a/pom.xml
+++ b/pom.xml
@@ -226,7 +226,7 @@
./python/pyspark/sql/pandas/utils.py, ./python/packaging/classic/setup.py
and ./python/packaging/connect/setup.py too.
-->
- <arrow.version>17.0.0</arrow.version>
+ <arrow.version>18.0.0</arrow.version>
<ammonite.version>3.0.0</ammonite.version>
<jjwt.version>0.12.6</jjwt.version>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
