[
https://issues.apache.org/jira/browse/HDFS-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-13713:
----------------------------------
Status: Patch Available (was: Open)
Patch 007, which should be me done with it as far as 3.2.0 goes
specify what our concurrency on >1 upload policy must be.
# Either the MPU rejects a second attempt to write to the destination (which
the filesystem one now does)
# Or it permits it, after which the second write to complete will (eventually)
be the observable one.
* Fix up {{FileSystemMultipartUploader}} to detect concurrent uploads and fail
(it just looks for the completion dir & rejects if it exists)
* New test for concurrent uploads and ordering of visibility (eventually the
last upload committed will be come and remain the final file)
* In {{TestHDFSContractMultipartUploader.testUploadEmptyBlock()}} expect a
failure, downgrade to WARN and skip(). This will need fixing in 3.3; for now
just skip that test. See HDFS-13936
I think this now shows up too many problems to get into 3.2.0, specifically
around concurrent uploads on filesystems. Because that rejecting of filesystem
concurrent will fail if there is >1 concurrent upload to the parent dir, which
is also in need of a test, etc. etc.
For this reason, I'm going to propose
For 3.2.0
* tag all interfaces in FileSystem.md as private/unstable
* postpone this patch to 3.3.0
it's too late to worrying about how to fix concurrency, empty files etc, even
though they are core requirements of a functional MPU framework. Rather than
rush to get something broken out, just tag the APIs as "not ready for use" and
we can fix this later.
submitting patch and about to move to a 3.3.0 feature, with a new patch to mark
up the relevant interfaces
> Add specification of Multipart Upload API to FS specification, with contract
> tests
> ----------------------------------------------------------------------------------
>
> Key: HDFS-13713
> URL: https://issues.apache.org/jira/browse/HDFS-13713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs, test
> Affects Versions: 3.2.0
> Reporter: Steve Loughran
> Assignee: Ewan Higgs
> Priority: Blocker
> Attachments: HADOOP-13713-004.patch, HADOOP-13713-004.patch,
> HADOOP-13713-005.patch, HADOOP-13713-006.patch, HADOOP-13713-007.patch,
> HDFS-13713.001.patch, HDFS-13713.002.patch, HDFS-13713.003.patch,
> multipartuploader.md
>
>
> There's nothing in the FS spec covering the new API. Add it in a new .md file
> * add FS model with the notion of a function mapping (uploadID -> Upload),
> the operations (list, commit, abort). The [TLA+
> mode|https://issues.apache.org/jira/secure/attachment/12865161/objectstore.pdf]l
> of HADOOP-13786 shows how to do this.
> * Contract tests of not just the successful path, but all the invalid ones.
> * implementations of the contract tests of all FSs which support the new API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
