dongjoon-hyun opened a new pull request, #56349:
URL: https://github.com/apache/spark/pull/56349
### What changes were proposed in this pull request?
This PR aims to add the `wildfly-openssl-macosx-aarch64` artifact as a
dependency of the `hadoop-cloud` module to support `Apple Silicon` when
`hadoop-cloud` profile is enabled.
### Why are the changes needed?
The main `wildfly-openssl` artifact does not ship the native OpenSSL binding
for macOS on Apple Silicon (`macosx-aarch64`). Without this platform-specific
artifact, `wildfly-openssl` fails to load the native OpenSSL library on Apple
Silicon and falls back to the JSSE implementation. Bundling the
`macosx-aarch64` variant enables native OpenSSL acceleration (e.g., for S3A via
`hadoop-aws`) on Apple Silicon machines.
**BEFORE**
```
$ build/sbt package -Phadoop-cloud
$ bin/spark-shell -c
spark.hadoop.fs.s3a.aws.credentials.provider=software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider
-c spark.hadoop.fs.s3a.ssl.channel.mode=openssl --driver-java-options
"-Dorg.wildfly.openssl.path=/opt/homebrew/opt/openssl@3/lib"
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile:
org/apache/spark/log4j2-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 5.0.0-SNAPSHOT
/_/
Using Scala version 2.13.18 (OpenJDK 64-Bit Server VM, Java 21.0.11)
Type in expressions to have them evaluated.
Type :help for more information.
26/06/05 14:41:20 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id =
local-1780695680644).
Spark session available as 'spark'.
scala> spark.read.text("s3a://dongjoon/README.md")
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
26/06/05 14:41:23 WARN FileSystem: Failed to initialize filesystem
s3a://dongjoon/README.md: java.io.IOException:
java.security.NoSuchAlgorithmException: Error constructing implementation
(algorithm: openssl.TLS, provider: openssl, class:
org.wildfly.openssl.OpenSSLContextSPI$OpenSSLTLSContextSpi)
26/06/05 14:41:23 WARN FileStreamSink: Assume no metadata directory. Error
while looking for metadata directory in the path: s3a://dongjoon/README.md.
java.io.IOException: java.security.NoSuchAlgorithmException: Error
constructing implementation (algorithm: openssl.TLS, provider: openssl, class:
org.wildfly.openssl.OpenSSLContextSPI$OpenSSLTLSContextSpi)
...
```
**AFTER**
```
$ build/sbt package -Phadoop-cloud
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs. This is a dependency-only change; verified that the new
artifact is resolved on the classpath via `${wildfly-openssl.version}`
(`2.3.0.Final`).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
