[
https://issues.apache.org/jira/browse/SPARK-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348160#comment-14348160
]
Yi Tian commented on SPARK-6061:
--------------------------------
In spark 1.2.1, when you set the {{newFilesOnly}} to {{false}}, it means this
{{FileInputDStream}} would not only handle coming files, but also include files
which came in the past 1 minute (not all the old files). The length of time
defined in {{FileInputDStream.MIN_REMEMBER_DURATION}}.
I think we should make this length of time configurable.
> File source dstream can not include the old file which timestamp is before
> the system time
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-6061
> URL: https://issues.apache.org/jira/browse/SPARK-6061
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.2.1
> Reporter: Jack Hu
> Labels: FileSourceDStream, OlderFiles, Streaming
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> The file source dstream (StreamContext.fileStream) has a properties named
> "newFilesOnly" to include the old files, it worked fine with 1.1.0, and
> broken at 1.2.1, the older files always be ignored no mattern what value is
> set.
> Here is the simple reproduce code:
> https://gist.github.com/jhu-chang/1ee5b0788c7479414eeb
> The reason is that: the "modTimeIgnoreThreshold" in
> FileInputDStream::findNewFiles is set to a time closed to system time (Spark
> Streaming Clock time), so the files old than this time are ignored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
