Re: textFileStream Question

[Akhil Das]

With file timestamp, you can actually see the finding new files logic from
here
https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala#L172

Thanks
Best Regards

On Fri, May 15, 2015 at 2:25 AM, Vadim Bichutskiy 
vadim.bichuts...@gmail.com wrote:

 How does textFileStream work behind the scenes? How does Spark Streaming
 know what files are new and need to be processed? Is it based on time
 stamp, file name?

 Thanks,
 Vadim
 ᐧ

Re: textFileStream Question

[Vadim Bichutskiy]

This is cool. Thanks Akhil.

ᐧ

On Sun, May 17, 2015 at 11:25 AM, Akhil Das ak...@sigmoidanalytics.com
wrote:

 With file timestamp, you can actually see the finding new files logic
 from here
 https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala#L172

 Thanks
 Best Regards

 On Fri, May 15, 2015 at 2:25 AM, Vadim Bichutskiy 
 vadim.bichuts...@gmail.com wrote:

 How does textFileStream work behind the scenes? How does Spark Streaming
 know what files are new and need to be processed? Is it based on time
 stamp, file name?

 Thanks,
 Vadim
 ᐧ

textFileStream Question

[Vadim Bichutskiy]

How does textFileStream work behind the scenes? How does Spark Streaming
know what files are new and need to be processed? Is it based on time
stamp, file name?

Thanks,
Vadim
ᐧ

回复：textFileStream Question

[董帅阳]

file timestamp




-- 原始邮件 --
发件人: Vadim Bichutskiy;vadim.bichuts...@gmail.com;
发送时间: 2015年5月15日(星期五) 凌晨4:55
收件人: user@spark.apache.orguser@spark.apache.org; 

主题: textFileStream Question



How does textFileStream work behind the scenes? How does Spark Streaming know 
what files are new and need to be processed? Is it based on time stamp, file 
name?

Thanks,
Vadim
ᐧ