This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 054bef9b30da [SPARK-51024][BUILD] Upgrade `wildfly-openssl` to
2.2.5.Final
054bef9b30da is described below
commit 054bef9b30dadd923d52c7a555d53604f42dd99f
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Jan 28 16:31:17 2025 -0800
[SPARK-51024][BUILD] Upgrade `wildfly-openssl` to 2.2.5.Final
### What changes were proposed in this pull request?
This PR aims to upgrade `wildfly-openssl` to 2.2.5.Final.
### Why are the changes needed?
Like Apache Hadoop 3.4.2 (HADOOP-19262), we need to upgrade to
`wildfly-openssl 2.x` to support the latest openssl in Apache Spark 4.0.0.
- https://github.com/apache/hadoop/pull/7026
As of now, `2.2.5.Final` is the latest one.
-
https://github.com/wildfly-security/wildfly-openssl/releases/tag/2.2.5.Final
### Does this PR introduce _any_ user-facing change?
No behavior change.
### How was this patch tested?
Pass the CIs and do the manual test on `Intel Mac`.
**BEFORE**
```
$ build/sbt package -Phadoop-cloud
$ bin/spark-shell -c
spark.hadoop.fs.s3a.aws.credentials.provider=software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider
-c spark.hadoop.fs.s3a.ssl.channel.mode=openssl
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile:
org/apache/spark/log4j2-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.1.0-SNAPSHOT
/_/
Using Scala version 2.13.15 (OpenJDK 64-Bit Server VM, Java 17.0.1)
Type in expressions to have them evaluated.
Type :help for more information.
25/01/28 15:15:03 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id =
local-1738106104577).
Spark session available as 'spark'.
scala> spark.read.text("s3a://dongjoon/README.md")
25/01/28 15:15:09 WARN MetricsConfig: Cannot locate configuration: tried
hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
25/01/28 15:15:09 WARN FileSystem: Failed to initialize filesystem
s3a://dongjoon/README.md: java.io.IOException:
java.security.NoSuchAlgorithmException: Error constructing implementation
(algorithm: openssl.TLS, provider: openssl, class:
org.wildfly.openssl.OpenSSLContextSPI$OpenSSLTLSContextSpi)
...
Caused by: java.lang.IllegalStateException: Could not load required symbol
from libssl: SSL_get_peer_certificate
at org.wildfly.openssl.SSLImpl.initialize0(Native Method)
at org.wildfly.openssl.SSLImpl.initialize(SSLImpl.java:33)
```
**AFTER**
```
$ build/sbt package -Phadoop-cloud
$ bin/spark-shell -c
spark.hadoop.fs.s3a.aws.credentials.provider=software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider
-c spark.hadoop.fs.s3a.ssl.channel.mode=openssl
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile:
org/apache/spark/log4j2-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.1.0-SNAPSHOT
/_/
Using Scala version 2.13.15 (OpenJDK 64-Bit Server VM, Java 17.0.1)
Type in expressions to have them evaluated.
Type :help for more information.
25/01/28 15:07:41 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id =
local-1738105662294).
Spark session available as 'spark'.
scala> spark.read.text("s3a://dongjoon/README.md")
25/01/28 15:07:47 WARN MetricsConfig: Cannot locate configuration: tried
hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
val res0: org.apache.spark.sql.DataFrame = [value: string]
scala>
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #49716 from dongjoon-hyun/SPARK-51024.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit cb38cd2ead7e86aecca52e67e93d3d18c26487ef)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +-
hadoop-cloud/pom.xml | 12 ++++++++++++
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 88ac18fdde38..96d5f9d47714 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -279,7 +279,7 @@ vertx-auth-common/4.5.11//vertx-auth-common-4.5.11.jar
vertx-core/4.5.11//vertx-core-4.5.11.jar
vertx-web-client/4.5.11//vertx-web-client-4.5.11.jar
vertx-web-common/4.5.11//vertx-web-common-4.5.11.jar
-wildfly-openssl/1.1.3.Final//wildfly-openssl-1.1.3.Final.jar
+wildfly-openssl/2.2.5.Final//wildfly-openssl-2.2.5.Final.jar
xbean-asm9-shaded/4.26//xbean-asm9-shaded-4.26.jar
xmlschema-core/2.3.1//xmlschema-core-2.3.1.jar
xz/1.10//xz-1.10.jar
diff --git a/hadoop-cloud/pom.xml b/hadoop-cloud/pom.xml
index 336108154924..34d0dd1d45d7 100644
--- a/hadoop-cloud/pom.xml
+++ b/hadoop-cloud/pom.xml
@@ -36,6 +36,7 @@
<sbt.project.name>hadoop-cloud</sbt.project.name>
<okhttp.version>3.12.12</okhttp.version>
<okio.version>1.17.6</okio.version>
+ <wildfly-openssl.version>2.2.5.Final</wildfly-openssl.version>
</properties>
<dependencies>
@@ -80,6 +81,17 @@
<artifactId>hadoop-aws</artifactId>
<version>${hadoop.version}</version>
<scope>${hadoop.deps.scope}</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.wildfly.openssl</groupId>
+ <artifactId>wildfly-openssl</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.wildfly.openssl</groupId>
+ <artifactId>wildfly-openssl</artifactId>
+ <version>${wildfly-openssl.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
