ehiggs commented on a change in pull request #609: HADOOP-16193. add extra S3A
MPU test to see what happens if a file is created during the MPU
URL: https://github.com/apache/hadoop/pull/609#discussion_r278064407
##########
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMultipartUploader.java
##########
@@ -159,4 +170,47 @@ public void testDirectoryInTheWay() throws Exception {
public void testMultipartUploadReverseOrder() throws Exception {
ContractTestUtils.skip("skipped for speed");
}
+
+ /**
+ * This creates and then deletes a zero-byte file while an upload
+ * is in progress, and verifies that the uploaded file is ultimately
+ * visible.
+ */
+ @Test
+ public void testMultipartOverlapWithTransientFile() throws Throwable {
+ // until there's a way to explicitly ask for a multipart uploader from a
+ // specific FS, explicitly create one bonded to the raw FS.
+ describe("testMultipartOverlapWithTransientFile");
+ S3AFileSystem fs = getFileSystem();
+ Path path = path("testMultipartOverlapWithTransientFile");
+ fs.delete(path, true);
+ MultipartUploader mpu = mpu(1);
+ UploadHandle upload1 = mpu.initialize(path);
+ byte[] dataset = dataset(1024, '0', 10);
+ final Map<Integer, PartHandle> handles = new HashMap<>();
+ LOG.info("Uploading multipart entry");
+ PartHandle value = mpu.putPart(path, new ByteArrayInputStream(dataset), 1,
+ upload1,
+ dataset.length);
+ // upload 1K
+ handles.put(1, value);
+ // confirm the path is absent
+ ContractTestUtils.assertPathDoesNotExist(fs,
+ "path being uploaded", path);
+ // now create an empty file
+ ContractTestUtils.touch(fs, path);
+ final FileStatus touchStatus = fs.getFileStatus(path);
+ LOG.info("0-byte file has been created: {}", touchStatus);
+ fs.delete(path, false);
+ // now complete the upload
+ mpu.complete(path, handles, upload1);
Review comment:
No, I mean mpu.complete returns a **Path**Handle. The part handles are
**Part**Handles. Indeed PartHandles aren't useful out of the context of putPart
(intentionally). But PathHandle is indeed useful and can be used to open the
file to make sure it's the same file.
We use the path later to get the mpuStatus, so maybe this is just moot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
