With file timestamp, you can actually see the finding new files logic from here <https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala#L172>
Thanks Best Regards On Fri, May 15, 2015 at 2:25 AM, Vadim Bichutskiy < [email protected]> wrote: > How does textFileStream work behind the scenes? How does Spark Streaming > know what files are new and need to be processed? Is it based on time > stamp, file name? > > Thanks, > Vadim > ᐧ >
