marin-ma commented on code in PR #5075:
URL: https://github.com/apache/incubator-gluten/pull/5075#discussion_r1535176984
##########
backends-velox/src/main/scala/org/apache/spark/sql/expression/UDFResolver.scala:
##########
@@ -152,34 +143,39 @@ object UDFResolver extends Logging {
// Get the full paths of all libraries.
// If it's a directory, get all files ends with ".so" recursively.
- def getAllLibraries(files: String, sparkConf: SparkConf,
canAccessSparkFiles: Boolean): String = {
+ def getAllLibraries(files: String, sparkConf: SparkConf): String = {
val hadoopConf = SparkHadoopUtil.newConfiguration(sparkConf)
files
.split(",")
.map {
f =>
- val file = new File(f)
- // Relative paths should be uploaded via --files or --archives
- // Use SparkFiles.get to download and unpack
- if (!file.isAbsolute) {
- if (!canAccessSparkFiles) {
- throw new IllegalArgumentException(
- "On yarn-client mode, driver only accepts absolute paths, but
got " + f)
+ val uri = Utils.resolveURI(f)
Review Comment:
> In your example, libmyudf.so is a local file
That doesn't make sense to me. `libmyudf.so` here should refer to the file
uploaded by `--files`. But here it's resolved as a relative path to the runtime
directory.
> The call to Utils.resolveURI() is used to determine whether the file is
local or remote.
It would be better to check whether it's a relative path first. Although
it's resolved as a local file and pass the `if` condition, the path doesn't
even exist.
> When --master=yarn is specified, the file is copied to the working
directory on all nodes (both the driver and executors).
Based on my previous experience, for yarn client mode, the files will be
copied to all executor container + AM container, so they won't be copied to the
driver node. In this case, if we only use the two configurations below
```
--files /path/to/gluten/cpp/build/velox/udf/examples/libmyudf.so
--conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=libmyudf.so
```
the driver will fail to get the actual path for `libmyudf.so`, and that's
the reason for adding
`spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
