Github user frreiss commented on a diff in the pull request:
https://github.com/apache/spark/pull/15258#discussion_r80838376
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ListingFileCatalog.scala
---
@@ -50,6 +50,19 @@ class ListingFileCatalog(
refresh()
+ /**
+ * Often HDFS create temporary files while copying to a new directory or
writing new content.
+ * These files are unintentionally picked up by streaming - causing job
failures. This option lets
+ * HDFS skip these files matching the configured regex-patterns from
being picked up by Streaming
+ * Job.
+ */
+ private lazy val excludeFiles: Set[String] = parameters
+ .getOrElse("excludeFiles", ".*._COPYING_,_temporary").split(",").toSet
+
+ private def isExcludedFile(path: Path): Boolean = {
+ excludeFiles.map(path.getName.matches).fold(false)(_ || _)
--- End diff --
Probably better to precompile a regex here, as this method will be called a
lot (100X per second X number of files under the root directory, by default)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
