[jira] [Commented] (SPARK-31911) Using S3A staging committer, pending uploads are committed more than once and listed incorrectly in _SUCCESS data

[ https://issues.apache.org/jira/browse/SPARK-31911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504273#comment-17504273 ] Steve Loughran commented on SPARK-31911: I'm going to close as fixed now; the spark changes will

[jira] [Commented] (SPARK-31911) Using S3A staging committer, pending uploads are committed more than once and listed incorrectly in _SUCCESS data

[ https://issues.apache.org/jira/browse/SPARK-31911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17228775#comment-17228775 ] Steve Loughran commented on SPARK-31911: yeah, its the uuid. see SPARK-33230 for the problem: if

[jira] [Commented] (SPARK-31911) Using S3A staging committer, pending uploads are committed more than once and listed incorrectly in _SUCCESS data

[ https://issues.apache.org/jira/browse/SPARK-31911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130164#comment-17130164 ] Brandon commented on SPARK-31911: - Some more notes: My Spark app is configured with

[jira] [Commented] (SPARK-31911) Using S3A staging committer, pending uploads are committed more than once and listed incorrectly in _SUCCESS data

[ https://issues.apache.org/jira/browse/SPARK-31911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126936#comment-17126936 ] Brandon commented on SPARK-31911: - Thanks for looking [~ste...@apache.org], I attached logs to the

[jira] [Commented] (SPARK-31911) Using S3A staging committer, pending uploads are committed more than once and listed incorrectly in _SUCCESS data

[ https://issues.apache.org/jira/browse/SPARK-31911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126609#comment-17126609 ] Steve Loughran commented on SPARK-31911: FYI [~mackrorysd] > Using S3A staging committer,

[jira] [Commented] (SPARK-31911) Using S3A staging committer, pending uploads are committed more than once and listed incorrectly in _SUCCESS data

[ https://issues.apache.org/jira/browse/SPARK-31911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126608#comment-17126608 ] Steve Loughran commented on SPARK-31911: BTW, given we complete the multipart uploads only once,

[jira] [Commented] (SPARK-31911) Using S3A staging committer, pending uploads are committed more than once and listed incorrectly in _SUCCESS data

[ https://issues.apache.org/jira/browse/SPARK-31911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126606#comment-17126606 ] Steve Loughran commented on SPARK-31911: thanks for this. Created HADOOP-17066 I'd like to see

[jira] [Commented] (SPARK-31911) Using S3A staging committer, pending uploads are committed more than once and listed incorrectly in _SUCCESS data

[ https://issues.apache.org/jira/browse/SPARK-31911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126556#comment-17126556 ] Brandon commented on SPARK-31911: - Interesting, it looks like the staging committer supports a