This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 76b1c122cb7d [SPARK-44115][BUILD] Upgrade Apache ORC to 2.0.0 76b1c122cb7d is described below commit 76b1c122cb7d77e8f175b25b935b9296a669d5d8 Author: Dongjoon Hyun <dh...@apple.com> AuthorDate: Fri Mar 8 13:31:10 2024 -0800 [SPARK-44115][BUILD] Upgrade Apache ORC to 2.0.0 ### What changes were proposed in this pull request? This PR aims to Upgrade Apache ORC to 2.0.0 for Apache Spark 4.0.0. Apache ORC community has 3-year support policy which is longer than Apache Spark. It's aligned like the following. - Apache ORC 2.0.x <-> Apache Spark 4.0.x - Apache ORC 1.9.x <-> Apache Spark 3.5.x - Apache ORC 1.8.x <-> Apache Spark 3.4.x - Apache ORC 1.7.x (Supported) <-> Apache Spark 3.3.x (End-Of-Support) ### Why are the changes needed? **Release Note** - https://github.com/apache/orc/releases/tag/v2.0.0 **Milestone** - https://github.com/apache/orc/milestone/20?closed=1 - https://github.com/apache/orc/pull/1728 - https://github.com/apache/orc/issues/1801 - https://github.com/apache/orc/issues/1498 - https://github.com/apache/orc/pull/1627 - https://github.com/apache/orc/issues/1497 - https://github.com/apache/orc/pull/1509 - https://github.com/apache/orc/pull/1554 - https://github.com/apache/orc/pull/1708 - https://github.com/apache/orc/pull/1733 - https://github.com/apache/orc/pull/1760 - https://github.com/apache/orc/pull/1743 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45443 from dongjoon-hyun/SPARK-44115. Authored-by: Dongjoon Hyun <dh...@apple.com> Signed-off-by: Dongjoon Hyun <dh...@apple.com> --- dev/deps/spark-deps-hadoop-3-hive-2.3 | 7 ++++--- pom.xml | 17 ++++++++++++++++- sql/core/pom.xml | 5 +++++ 3 files changed, 25 insertions(+), 4 deletions(-) diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index 7e56e8914435..6b357b3e4b70 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 @@ -227,9 +227,10 @@ opencsv/2.3//opencsv-2.3.jar opentracing-api/0.33.0//opentracing-api-0.33.0.jar opentracing-noop/0.33.0//opentracing-noop-0.33.0.jar opentracing-util/0.33.0//opentracing-util-0.33.0.jar -orc-core/1.9.2/shaded-protobuf/orc-core-1.9.2-shaded-protobuf.jar -orc-mapreduce/1.9.2/shaded-protobuf/orc-mapreduce-1.9.2-shaded-protobuf.jar -orc-shims/1.9.2//orc-shims-1.9.2.jar +orc-core/2.0.0/shaded-protobuf/orc-core-2.0.0-shaded-protobuf.jar +orc-format/1.0.0/shaded-protobuf/orc-format-1.0.0-shaded-protobuf.jar +orc-mapreduce/2.0.0/shaded-protobuf/orc-mapreduce-2.0.0-shaded-protobuf.jar +orc-shims/2.0.0//orc-shims-2.0.0.jar oro/2.0.8//oro-2.0.8.jar osgi-resource-locator/1.0.3//osgi-resource-locator-1.0.3.jar paranamer/2.8//paranamer-2.8.jar diff --git a/pom.xml b/pom.xml index 9f1c9ed13f23..404f37be1b5a 100644 --- a/pom.xml +++ b/pom.xml @@ -141,7 +141,7 @@ <!-- After 10.17.1.0, the minimum required version is JDK19 --> <derby.version>10.16.1.1</derby.version> <parquet.version>1.13.1</parquet.version> - <orc.version>1.9.2</orc.version> + <orc.version>2.0.0</orc.version> <orc.classifier>shaded-protobuf</orc.classifier> <jetty.version>11.0.20</jetty.version> <jakartaservlet.version>5.0.0</jakartaservlet.version> @@ -2593,6 +2593,13 @@ </exclusions> </dependency> + <dependency> + <groupId>org.apache.orc</groupId> + <artifactId>orc-format</artifactId> + <version>1.0.0</version> + <classifier>${orc.classifier}</classifier> + <scope>${orc.deps.scope}</scope> + </dependency> <dependency> <groupId>org.apache.orc</groupId> <artifactId>orc-core</artifactId> @@ -2600,6 +2607,14 @@ <classifier>${orc.classifier}</classifier> <scope>${orc.deps.scope}</scope> <exclusions> + <exclusion> + <groupId>org.apache.orc</groupId> + <artifactId>orc-format</artifactId> + </exclusion> + <exclusion> + <groupId>com.aayushatharva.brotli4j</groupId> + <artifactId>brotli4j</artifactId> + </exclusion> <exclusion> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> diff --git a/sql/core/pom.xml b/sql/core/pom.xml index 0ad9e0f690c7..05f906206e5e 100644 --- a/sql/core/pom.xml +++ b/sql/core/pom.xml @@ -93,6 +93,11 @@ <groupId>org.scala-lang.modules</groupId> <artifactId>scala-parallel-collections_${scala.binary.version}</artifactId> </dependency> + <dependency> + <groupId>org.apache.orc</groupId> + <artifactId>orc-format</artifactId> + <classifier>${orc.classifier}</classifier> + </dependency> <dependency> <groupId>org.apache.orc</groupId> <artifactId>orc-core</artifactId> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org