[
https://issues.apache.org/jira/browse/SPARK-31371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17095286#comment-17095286
]
Prashant Sharma commented on SPARK-31371:
-----------------------------------------
Thank you for responding. As you said, a link to the JIRA or discussion would
be helpful. My hope is something could be done, can we hide this behind a flag?
What happens if a file is changed, which was processed earlier. Can we process
the file as a fresh input, on each time it is updated(or checksum changed) ?
> FileStreamSource: Decide seen files on the checksum, instead of filename.
> -------------------------------------------------------------------------
>
> Key: SPARK-31371
> URL: https://issues.apache.org/jira/browse/SPARK-31371
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 2.4.5, 3.0.0
> Reporter: Prashant Sharma
> Priority: Major
>
> At the moment structured streaming's file source, ignores updates to the same
> file, it has processed earlier. However, for reasons beyond our control, a
> software might update the same file with new data. A case in point can be
> rolling logs, where the latest log file is always e.g. log.txt and the rolled
> logs could be log-1.txt etc...
> So by supporting this, it may not actually be a special casing but supporting
> a genuine use case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
