CalvQ commented on code in PR #56374:
URL: https://github.com/apache/spark/pull/56374#discussion_r3455704227
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala:
##########
@@ -154,8 +154,9 @@ object InMemoryFileIndex extends Logging {
parameters: Map[String, String] = Map.empty): Seq[(Path,
Seq[FileStatus])] = {
val fileSystemList =
sparkSession.sessionState.conf.useListFilesFileSystemList.split(",").map(_.trim)
- val ignoreMissingFiles =
- new FileSourceOptions(CaseInsensitiveMap(parameters)).ignoreMissingFiles
+ val fileSourceOptions = new
FileSourceOptions(CaseInsensitiveMap(parameters))
+ val ignoreMissingFiles = fileSourceOptions.ignoreMissingFiles
+ val listHiddenFiles = fileSourceOptions.listHiddenFiles
Review Comment:
Do you mean removing the SQLConf entirely? Existing patterns such as
`ignoreCorruptFiles` and `ignoreMissingFiles` keep both a session conf and a
per-read option. Removing the conf would be cache safe, but to match existing
patterns I think it would make sense to keep both the conf and per-read option
-- this would require the caching bypass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
