Github user jeanlyn commented on a diff in the pull request:
https://github.com/apache/spark/pull/4356#discussion_r24067924
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala ---
@@ -207,13 +219,21 @@ class HadoopTableReader(
* If `filterOpt` is defined, then it will be used to filter files from
`path`. These files are
* returned in a single, comma-separated string.
*/
- private def applyFilterIfNeeded(path: Path, filterOpt:
Option[PathFilter]): String = {
- filterOpt match {
- case Some(filter) =>
- val fs = path.getFileSystem(sc.hiveconf)
- val filteredFiles = fs.listStatus(path,
filter).map(_.getPath.toString)
- filteredFiles.mkString(",")
- case None => path.toString
+ private def applyFilterIfNeeded(path: Path, filterOpt:
Option[PathFilter]): Option[String] = {
+ if (fs.exists(path)) {
--- End diff --
I think we'd better get `fs` from the `path`,because in the `hadoop
namenode federation` we may get some problems like ` Wrong FS` exception if we
use the `FileSystem.get(sc.hiveconf)` to get fs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
