[ https://issues.apache.org/jira/browse/SPARK-34526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17290177#comment-17290177 ]
Apache Spark commented on SPARK-34526: -------------------------------------- User 'xuanyuanking' has created a pull request for this issue: https://github.com/apache/spark/pull/31638 > Add a flag to skip checking file sink format and handle glob path > ----------------------------------------------------------------- > > Key: SPARK-34526 > URL: https://issues.apache.org/jira/browse/SPARK-34526 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 3.1.0 > Reporter: Yuanjian Li > Priority: Major > > This ticket fixes the following issues related to file sink format checking > together: > * Some users may use a very long glob path to read and `isDirectory` may > fail when the path is too long. We should ignore the error when the path is a > glob path since the file streaming sink doesn’t support glob paths. > * Checking whether a directory is outputted by File Streaming Sink may fail > for various issues happening in the storage. We should add a flag to allow > users to disable the checking logic and read the directory as a batch output. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org