GitHub user soumilshah1995 created a discussion: Running Apache Gluten locally
on macOS (Apple Silicon) with Spark 3.5 – libgluten.dylib missing
Hi Gluten community,
I’m trying to run Apache Gluten locally on macOS (Apple Silicon / M1) with
Apache Spark 3.5.3, and I’m running into a native library loading issue.
I wanted to check whether local macOS support is expected to work today, or if
Gluten is currently Linux-only for the Velox backend.
OS: macOS (Apple Silicon – aarch64)
Spark: 3.5.3 (spark-3.5.3-bin-hadoop3)
Java: OpenJDK 17
Gluten:
gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.5.0.jar
Backend: Velox
Execution mode: local
```
$SPARK_HOME/bin/spark-shell \
--jars $GLUTEN_JAR \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.gluten.sql.enable=true \
--conf spark.gluten.execution.backend=velox \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=2g \
--conf spark.driver.extraClassPath=$GLUTEN_JAR \
--conf spark.executor.extraClassPath=$GLUTEN_JAR \
--conf spark.sql.columnVector.offheap.enabled=true \
--conf spark.gluten.sql.fallback.enabled=true \
--conf spark.driver.bindAddress=127.0.0.1 \
--conf spark.driver.host=127.0.0.1
```
Error Observed
```
WARN Discovery: Found duplicated component class
org.apache.gluten.component.VeloxPaimonComponent in the classpath, ignoring.
ERROR SparkContext: Error initializing SparkContext.
org.apache.gluten.exception.GlutenException:
java.io.FileNotFoundException: darwin/aarch64/libgluten.dylib
at org.apache.gluten.jni.JniLibLoader.load(JniLibLoader.java:89)
```
Questions
Is running Gluten + Velox locally on macOS supported today?
Are there prebuilt macOS (darwin/aarch64) native binaries for Gluten?
I took jar from here
https://downloads-he-de-2.apache.org/incubator/gluten/1.4.0-incubating/apache-gluten-1.4.0-incubating-bin-spark35.tar.gz
GitHub link: https://github.com/apache/incubator-gluten/discussions/11535
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
