[
https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-13974:
----------------------------------
Attachment: HADOOP-13974.006.patch
[[email protected]] this is almost ready. I had to add eventually() retries
around multipart list assertions that were flaky. I want to convince myself
this really is eventual consistency before we commit this. If so we need to
consider any implications for the magic committer.. haven't thought through it
yet.
v6 patch:
- Fix NPE in some unit tests: made a sacrifice to the Mockito gods (update S3A
Mock FS to return something from getWriteOperationHelper())).
- Add eventually() around a list MPU test assertion that was failing
occasionally.
- Remove a redundant abortUpload() function: something similar was added for
Committer work so use that instead.
- Cleanup test logging a bit, fix typo in CLI usage.
- Fix a couple test issues from before I added the "-force" flag for "-abort"
option. Note these tests use the S3Guard assumption so they only run with that
profile.
Ran all integration *and* unit tests in us-west-2
> S3a CLI to support list/purge of pending multipart commits
> ----------------------------------------------------------
>
> Key: HADOOP-13974
> URL: https://issues.apache.org/jira/browse/HADOOP-13974
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0-beta1
> Reporter: Steve Loughran
> Assignee: Aaron Fabbri
> Attachments: HADOOP-13974.001.patch, HADOOP-13974.002.patch,
> HADOOP-13974.003.patch, HADOOP-13974.004.patch, HADOOP-13974.005.patch,
> HADOOP-13974.006.patch
>
>
> The S3A CLI will need to be able to list and delete pending multipart
> commits.
> We can do the cleanup already via fs.s3a properties. The CLI will let scripts
> stat for outstanding data (have a different exit code) and permit batch jobs
> to explicitly trigger cleanups.
> This will become critical with the multipart committer, as there's a
> significantly higher likelihood of commits remaining outstanding.
> We may also want to be able to enumerate/cancel all pending commits in the FS
> tree
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
