Michael Armbrust created SPARK-19715:
----------------------------------------
Summary: Option to Strip Paths in FileSource
Key: SPARK-19715
URL: https://issues.apache.org/jira/browse/SPARK-19715
Project: Spark
Issue Type: New Feature
Components: Structured Streaming
Affects Versions: 2.1.0
Reporter: Michael Armbrust
Today, we compare the whole path when deciding if a file is new in the
FileSource for structured streaming. However, this cause cause false negatives
in the case where the path has changed in a cosmetic way (i.e. changing s3n to
s3a). We should add an option {{fileNameOnly}} that causes the new file check
to be based only on the filename (but still store the whole path in the log).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
