[
https://issues.apache.org/jira/browse/HDFS-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16624225#comment-16624225
]
Steve Loughran commented on HDFS-13713:
---------------------------------------
patch 004. Review, with a fair amount of revision of the tests; little bit on
the API
FileSystemMultipartUploader
* switch IOE on complete to PathIsDirectoryException if dest is now a dir
* move to map<> over list. This simplifies testing as it removes the need for
a specific test (duplicates)
* add some reusable helper methods to validate preconditions
S3AMultipartUploader
* Support 0-byte parts. Needed to allow a 0-byte file upload.
AbstractContractMultipartUploaderTest
* MPU create done in test setup
* Created upload handle is *always*
* factored out core put/abort operations
* track the handle and dest file of the upload used in the test, always abort
in test teardown. Some up us get billed by AWS!
* test that 0-byte files can be uploaded
ITestS3AContractMultipartUploader
* switch to scale test timeout, as one of tests (testMultipartUploadAbort
before I changed the block count) was timing out
filesystem.md
* clarify that the ordering of parts is the "natural" ordering; part 1 comes
first.
testing: local FS & HDFS, remote : s3 ireland w/ s3guard enabled. made sure
there weren't any outstanding uploads afterwards.
I'm happy with this: the API is good and with the 0-byte uploads test working,
rounded out functionality.
> Add specification of Multipart Upload API to FS specification, with contract
> tests
> ----------------------------------------------------------------------------------
>
> Key: HDFS-13713
> URL: https://issues.apache.org/jira/browse/HDFS-13713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs, test
> Affects Versions: 3.2.0
> Reporter: Steve Loughran
> Assignee: Ewan Higgs
> Priority: Blocker
> Attachments: HADOOP-13713-004.patch, HDFS-13713.001.patch,
> HDFS-13713.002.patch, HDFS-13713.003.patch, multipartuploader.md
>
>
> There's nothing in the FS spec covering the new API. Add it in a new .md file
> * add FS model with the notion of a function mapping (uploadID -> Upload),
> the operations (list, commit, abort). The [TLA+
> mode|https://issues.apache.org/jira/secure/attachment/12865161/objectstore.pdf]l
> of HADOOP-13786 shows how to do this.
> * Contract tests of not just the successful path, but all the invalid ones.
> * implementations of the contract tests of all FSs which support the new API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
