Christopher Highman created SPARK-31962: -------------------------------------------
Summary: Provide option to load files after a specified date when reading from a folder path Key: SPARK-31962 URL: https://issues.apache.org/jira/browse/SPARK-31962 Project: Spark Issue Type: Improvement Components: SQL, Structured Streaming Affects Versions: 3.1.0 Reporter: Christopher Highman When using structured streaming with a FileDataSource, I've encountered a number of occasions where I want to be able to stream from a folder containing any number of historical delta files in CSV format. When I start reading from a folder, however, I might only care about files were created after a certain time. {code:java} spark.readStream .option("header", "true") .option("delimiter", "\t") .format("csv") .load("/mnt/Deltas/") {code} In [https://github.com/apache/spark/blob/f3771c6b47d0b3aef10b86586289a1f675c7cfe2/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala], there is a method, _checkAndGlobPathIfNecessary,_ which appears create an in-memory index of files for a given path. There may a rather clean opportunity to consider options here. Having the ability to provide an option specifying a timestamp by which to begin globbing files would result in quite a bit of less complexity needed on a consumer who leverages the ability to stream from a folder path but does not have an interest in reading what could be thousands of files that are not relevant. One example to could be "createdFileTime" accepting a UTC datetime like below. {code:java} spark.readStream .option("header", "true") .option("delimiter", "\t") .option("createdFileTime", "2020-05-01 00:00:00") .format("csv") .load("/mnt/Deltas/") {code} If this option is specified, the expected behavior would be that files within the _"/mnt/Deltas/"_ path must have a created been created at or later than the specified time in order to be consumed for purposes of reading the files in general or for purposes of structured streaming. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org