Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/22449#discussion_r218498363
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/source/image/ImageFileFormat.scala ---
@@ -85,7 +85,9 @@ private[image] class ImageFileFormat extends FileFormat
with DataSourceRegister
val filteredResult = if (imageSourceOptions.dropInvalid) {
resultOpt.toIterator
} else {
-
Iterator(resultOpt.getOrElse(ImageSchema.invalidImageRow(origin)))
+ val basePath =
Path.getPathWithoutSchemeAndAuthority(path).toString()
--- End diff --
Seems authority got dropped here.
~~~
scala> Path.getPathWithoutSchemeAndAuthority(new
Path("s3://dbc/test/ajdj/dfdfd"))
res10: org.apache.hadoop.fs.Path = /test/ajdj/dfdfd
~~~
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
