zhouyuan commented on PR #52071:
URL: https://github.com/apache/spark/pull/52071#issuecomment-3253896748
Hi @pan3793 @itskals, In most cases, Gluten can work well with the existing
mechanism, the issue I ran into is on the filesystem related configurations.
E.g. for the azure related configurations. Spark now accepts two type of
configurations:
- specify all the configurations in `spark-defaults.conf`, with `spark`
prefix:
`spark.hadoop.fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net
OAuth`
- specify all the configurations in a spark session,
`spark.conf.set("fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net",
"OAuth")`
The 1st one works well with Gluten. However for the 2nd method, if I
understand correctly, Spark will relies on some JVM Hadoop components to do the
job - which is not available and replaced by native libs in Gluten.
My intention for this patch is to to check with community first to see if
this is acceptable. With this approach it will be much easier for impl these
features with native lib. Thanks for the inputs!
thanks, -yuan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
