This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 7bdabaae32d [SPARK-46174][BUILD] Upgrade `gcs-connector` to 2.2.18 7bdabaae32d is described below commit 7bdabaae32de3eaee42cab14b465ac64dfc85722 Author: Dongjoon Hyun <dh...@apple.com> AuthorDate: Wed Nov 29 12:36:09 2023 -0800 [SPARK-46174][BUILD] Upgrade `gcs-connector` to 2.2.18 ### What changes were proposed in this pull request? This PR aims to upgrade `gas-connector` to 2.2.18. ### Why are the changes needed? `v2.2.18` upgrades `google-cloud-storage` from `2.25.0` to `2.28.0` to deliver several bug fixes and new features. - https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.18 - https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/1049 - https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/1056 - https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/1068 Here are `google-cloud-storage` release notes. - https://github.com/googleapis/java-storage/releases/tag/v2.26.0 - https://github.com/googleapis/java-storage/releases/tag/v2.26.1 - https://github.com/googleapis/java-storage/releases/tag/v2.27.0 - https://github.com/googleapis/java-storage/releases/tag/v2.27.1 - https://github.com/googleapis/java-storage/releases/tag/v2.28.0 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs and do manual test. ``` $ dev/make-distribution.sh -Phadoop-cloud $ cd dist $ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json $ export EMAIL=$(jq -r '.client_email' < $KEYFILE) $ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE) $ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)" $ bin/spark-shell \ -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \ -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \ -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY" Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT /_/ Using Scala version 2.13.12 (OpenJDK 64-Bit Server VM, Java 17.0.9) Type in expressions to have them evaluated. Type :help for more information. 23/11/29 12:33:39 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://localhost:4040 Spark context available as 'sc' (master = local[*], app id = local-1701290019480). Spark session available as 'spark'. scala> spark.read.text("gs://apache-spark-bucket/README.md").count() val res0: Long = 124 scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc") scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show() +------+--------------+----------------+ | name|favorite_color|favorite_numbers| +------+--------------+----------------+ |Alyssa| NULL| [3, 9, 15, 20]| | Ben| red| []| +------+--------------+----------------+ ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44081 from dongjoon-hyun/SPARK-46174. Authored-by: Dongjoon Hyun <dh...@apple.com> Signed-off-by: Dongjoon Hyun <dh...@apple.com> --- dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +- pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index 3a8c3dc707a..25245d0b74f 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 @@ -62,7 +62,7 @@ datasketches-memory/2.1.0//datasketches-memory-2.1.0.jar derby/10.14.2.0//derby-10.14.2.0.jar dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar flatbuffers-java/1.12.0//flatbuffers-java-1.12.0.jar -gcs-connector/hadoop3-2.2.17/shaded/gcs-connector-hadoop3-2.2.17-shaded.jar +gcs-connector/hadoop3-2.2.18/shaded/gcs-connector-hadoop3-2.2.18-shaded.jar gmetric4j/1.0.10//gmetric4j-1.0.10.jar gson/2.2.4//gson-2.2.4.jar guava/14.0.1//guava-14.0.1.jar diff --git a/pom.xml b/pom.xml index fce9c2b54e0..1e22a3cf092 100644 --- a/pom.xml +++ b/pom.xml @@ -165,7 +165,7 @@ <aws.java.sdk.v2.version>2.20.160</aws.java.sdk.v2.version> <!-- the producer is used in tests --> <aws.kinesis.producer.version>0.12.8</aws.kinesis.producer.version> - <gcs-connector.version>hadoop3-2.2.17</gcs-connector.version> + <gcs-connector.version>hadoop3-2.2.18</gcs-connector.version> <!-- org.apache.httpcomponents/httpclient--> <commons.httpclient.version>4.5.14</commons.httpclient.version> <commons.httpcore.version>4.4.16</commons.httpcore.version> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org