KeiichiHirobe commented on issue #23288: [SPARK-26339][SQL]Throws better exception when reading files that start with underscore URL: https://github.com/apache/spark/pull/23288#issuecomment-450286704 Thank you for your reply. Let me make sure what you mean, I don't know for certain whether I should list filterOuted files/dirs recursively or list files/dirs only directly under specified path. For the example below, when `spark.read.csv("foo")` should we debug log `_a.csv` and `_b.csv` or `_a.csv` only? I noticed below behavior for your information. `bulkListLeafFiles` lists `_a.csv` and `_b.csv`(recursively listed) `spark.read.csv("foo").show` lists only `a.csv`(read only files directly under specified path) foo/ ├ a.csv ├ _a.csv ├ bar/ │ ├ b.csv │ ├ _b.csv
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
