[jira] [Commented] (SPARK-19715) Option to Strip Paths in FileSource
[ https://issues.apache.org/jira/browse/SPARK-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890190#comment-15890190 ] Apache Spark commented on SPARK-19715: -- User 'lw-lin' has created a pull request for this issue: https://github.com/apache/spark/pull/17120 > Option to Strip Paths in FileSource > --- > > Key: SPARK-19715 > URL: https://issues.apache.org/jira/browse/SPARK-19715 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 2.1.0 >Reporter: Michael Armbrust > > Today, we compare the whole path when deciding if a file is new in the > FileSource for structured streaming. However, this cause cause false > negatives in the case where the path has changed in a cosmetic way (i.e. > changing s3n to s3a). We should add an option {{fileNameOnly}} that causes > the new file check to be based only on the filename (but still store the > whole path in the log). -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19715) Option to Strip Paths in FileSource
[ https://issues.apache.org/jira/browse/SPARK-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884230#comment-15884230 ] Steve Loughran commented on SPARK-19715: OK. I'd recommend going twith Path.getURI.getPath() to get the full path, though there's the always the risk of >1 s3a bucket referring to the same objects Some filesystems (HDFS, file:) have checksums you can ask for, though S3a doesn't, yet: HADOOP-13282 has discussed serving up etags, primarily to aid distcp updates. If added, you could use that as the differentiator, or at least to identify changed files To be ruthless, it may have been simpler for the user just to edit the fs.s3n.impl binding to point to S3AFileSystem.class & then left the URLs the same > Option to Strip Paths in FileSource > --- > > Key: SPARK-19715 > URL: https://issues.apache.org/jira/browse/SPARK-19715 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 2.1.0 >Reporter: Michael Armbrust > > Today, we compare the whole path when deciding if a file is new in the > FileSource for structured streaming. However, this cause cause false > negatives in the case where the path has changed in a cosmetic way (i.e. > changing s3n to s3a). We should add an option {{fileNameOnly}} that causes > the new file check to be based only on the filename (but still store the > whole path in the log). -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19715) Option to Strip Paths in FileSource
[ https://issues.apache.org/jira/browse/SPARK-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883642#comment-15883642 ] Michael Armbrust commented on SPARK-19715: -- This isn't a hypothetical. A user of structured streaming upgraded to {{s3a}} and was surprised to see duplicate computation in the results. Their files are named with a combination of upload time and a GUID, so I don't think there is any risk for this use case. That said, I would not make this option the default. > Option to Strip Paths in FileSource > --- > > Key: SPARK-19715 > URL: https://issues.apache.org/jira/browse/SPARK-19715 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 2.1.0 >Reporter: Michael Armbrust > > Today, we compare the whole path when deciding if a file is new in the > FileSource for structured streaming. However, this cause cause false > negatives in the case where the path has changed in a cosmetic way (i.e. > changing s3n to s3a). We should add an option {{fileNameOnly}} that causes > the new file check to be based only on the filename (but still store the > whole path in the log). -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19715) Option to Strip Paths in FileSource
[ https://issues.apache.org/jira/browse/SPARK-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882995#comment-15882995 ] Steve Loughran commented on SPARK-19715: This is a silly question, but has the situation " a filesystem schema has changed" ever arisen? Because I can see the risk of that being lower than the risk that a file with the same name is added to > 1 directory included in the same scan > Option to Strip Paths in FileSource > --- > > Key: SPARK-19715 > URL: https://issues.apache.org/jira/browse/SPARK-19715 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 2.1.0 >Reporter: Michael Armbrust > > Today, we compare the whole path when deciding if a file is new in the > FileSource for structured streaming. However, this cause cause false > negatives in the case where the path has changed in a cosmetic way (i.e. > changing s3n to s3a). We should add an option {{fileNameOnly}} that causes > the new file check to be based only on the filename (but still store the > whole path in the log). -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19715) Option to Strip Paths in FileSource
[ https://issues.apache.org/jira/browse/SPARK-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881965#comment-15881965 ] Liwei Lin commented on SPARK-19715: --- I'll work on this. Thanks! > Option to Strip Paths in FileSource > --- > > Key: SPARK-19715 > URL: https://issues.apache.org/jira/browse/SPARK-19715 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 2.1.0 >Reporter: Michael Armbrust > > Today, we compare the whole path when deciding if a file is new in the > FileSource for structured streaming. However, this cause cause false > negatives in the case where the path has changed in a cosmetic way (i.e. > changing s3n to s3a). We should add an option {{fileNameOnly}} that causes > the new file check to be based only on the filename (but still store the > whole path in the log). -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19715) Option to Strip Paths in FileSource
[ https://issues.apache.org/jira/browse/SPARK-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881731#comment-15881731 ] Michael Armbrust commented on SPARK-19715: -- [~lwlin] another file source features you might want to work on. > Option to Strip Paths in FileSource > --- > > Key: SPARK-19715 > URL: https://issues.apache.org/jira/browse/SPARK-19715 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 2.1.0 >Reporter: Michael Armbrust > > Today, we compare the whole path when deciding if a file is new in the > FileSource for structured streaming. However, this cause cause false > negatives in the case where the path has changed in a cosmetic way (i.e. > changing s3n to s3a). We should add an option {{fileNameOnly}} that causes > the new file check to be based only on the filename (but still store the > whole path in the log). -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org