Github user petermaxlee commented on a diff in the pull request:
https://github.com/apache/spark/pull/14728#discussion_r75795996
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
---
@@ -17,21 +17,18 @@
package org.apache.spark.sql.execution.streaming
-import scala.util.Try
+import scala.collection.JavaConverters._
import org.apache.hadoop.fs.Path
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.Logging
import org.apache.spark.sql.{DataFrame, Dataset, SparkSession}
-import org.apache.spark.sql.execution.datasources.{CaseInsensitiveMap,
DataSource, ListingFileCatalog, LogicalRelation}
+import org.apache.spark.sql.execution.datasources.{DataSource,
ListingFileCatalog, LogicalRelation}
import org.apache.spark.sql.types.StructType
-import org.apache.spark.util.collection.OpenHashSet
/**
- * A very simple source that reads text files from the given directory as
they appear.
- *
- * TODO Clean up the metadata files periodically
--- End diff --
Put it back. Also updated the file with a new test case to make the seen
map more robust.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
