[
https://issues.apache.org/jira/browse/HDFS-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-13713:
----------------------------------
Resolution: Fixed
Fix Version/s: 3.3.0
Status: Resolved (was: Patch Available)
Committed to trunk; playing with -author so git log shows you as the author;
its GPG signed by me though. Thanks for putting up with me here.
The next thing I'd like us to go through is moving off service loaders
(brittle, inflexible) into making this an API Call off FileSystem and
FileContext, something like
{code}
MultipartUploader createMultipartUploader(path)
{code}
+ some hasPathCapabilities(path) check first for MPU support. S3A, local, hdfs,
webfs => has support, instantiate, viewfs will relay.
BTW, if you look at HADOOP-15229 I'm exploring async completion fo openFile()
calls. What do you think (before we actually ship the MPU stuff) to make the
upload/complete calls async too? For local & hdfs they'd be sync with delayed
failure, in S3A they'd be fully async in the worker queue. This would allow a
single client to upload blocks in parallel without having to do so much work
themselves
> Add specification of Multipart Upload API to FS specification, with contract
> tests
> ----------------------------------------------------------------------------------
>
> Key: HDFS-13713
> URL: https://issues.apache.org/jira/browse/HDFS-13713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs, test
> Affects Versions: 3.2.0
> Reporter: Steve Loughran
> Assignee: Ewan Higgs
> Priority: Blocker
> Fix For: 3.3.0
>
> Attachments: HADOOP-13713-004.patch, HADOOP-13713-004.patch,
> HADOOP-13713-005.patch, HADOOP-13713-006.patch, HADOOP-13713-007.patch,
> HADOOP-13713-009.patch, HDFS-13713.001.patch, HDFS-13713.002.patch,
> HDFS-13713.003.patch, HDFS-13713.008.patch, HDFS-13713.008.patch,
> multipartuploader.md
>
>
> There's nothing in the FS spec covering the new API. Add it in a new .md file
> * add FS model with the notion of a function mapping (uploadID -> Upload),
> the operations (list, commit, abort). The [TLA+
> mode|https://issues.apache.org/jira/secure/attachment/12865161/objectstore.pdf]l
> of HADOOP-13786 shows how to do this.
> * Contract tests of not just the successful path, but all the invalid ones.
> * implementations of the contract tests of all FSs which support the new API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
