GitHub user ammarchalifah added a comment to the discussion: Deployment Guide
for Spark on EMR
Got it. I'm currently just doing testing and want to see how far Gluten on EMR
goes, and if it breaks, at which point it breaks. Purely for experiment.
The way I'm doing the deployment right now is
- Downloading the JAR `gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar`
and dump it to S3
- During Spark submit, pass extra parameters:
```
--conf spark.yarn.dist.files={s3_jar_path}
--conf spark.plugins=org.apache.gluten.GlutenPlugin
--conf spark.memory.offHeap.enabled=true
--conf spark.memory.offHeap.size=20g
--conf
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
--conf spark.driver.extraClassPath={jar_path}
--conf spark.executor.extraClassPath={jar_path}
```
The Spark application is properly booted up with the plugin
<img width="998" height="218" alt="Screenshot 2025-12-11 at 13 36 10"
src="https://github.com/user-attachments/assets/baaa9d66-0c81-4b91-9c10-599d56f5ea41";
/>
<img width="1087" height="42" alt="Screenshot 2025-12-11 at 13 36 37"
src="https://github.com/user-attachments/assets/314ae5f5-fc37-452e-83b8-caa90a28bf45";
/>
<img width="1674" height="39" alt="Screenshot 2025-12-11 at 13 36 54"
src="https://github.com/user-attachments/assets/c3fea101-5a34-465a-8645-69c387f72973";
/>
GitHub link:
https://github.com/apache/incubator-gluten/discussions/11279#discussioncomment-15230305
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
