[
https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-13974:
----------------------------------
Attachment: HADOOP-13974.001.patch
Attaching v1 patch.
Some caveats I'd like feedback on:
- The `hadoop s3guard uploads -abort` command is dangerous, especially if there
are MPU commits in flight. I feel like this could use a failsafe "are you sure"
prompt. I'm leaning towards implementing that when the specified age is less
than one day. Presumably this is "safer" if your filter only matches older
upload parts.
- Also think a disclaimer about clock synchronization might be warranted in the
docs. S3's MPU list gives an "initiated time" which I use here. I'm wondering
if this is server side or client side?
- There is a test gap.. I tested the age-based filtering by hand but don't have
an automated test for it. v2 patch should probably have at least a very basic
test of the age filtering.
> S3a CLI to support list/purge of pending multipart commits
> ----------------------------------------------------------
>
> Key: HADOOP-13974
> URL: https://issues.apache.org/jira/browse/HADOOP-13974
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0-beta1
> Reporter: Steve Loughran
> Assignee: Aaron Fabbri
> Attachments: HADOOP-13974.001.patch
>
>
> The S3A CLI will need to be able to list and delete pending multipart
> commits.
> We can do the cleanup already via fs.s3a properties. The CLI will let scripts
> stat for outstanding data (have a different exit code) and permit batch jobs
> to explicitly trigger cleanups.
> This will become critical with the multipart committer, as there's a
> significantly higher likelihood of commits remaining outstanding.
> We may also want to be able to enumerate/cancel all pending commits in the FS
> tree
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
