bgaborg commented on a change in pull request #609: HADOOP-16193. add extra S3A
MPU test to see what happens if a file is created during the MPU
URL: https://github.com/apache/hadoop/pull/609#discussion_r272163931
##########
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMultipartUploader.java
##########
@@ -159,4 +170,47 @@ public void testDirectoryInTheWay() throws Exception {
public void testMultipartUploadReverseOrder() throws Exception {
ContractTestUtils.skip("skipped for speed");
}
+
+ /**
+ * This creates and then deletes a zero-byte file while an upload
+ * is in progress, and verifies that the uploaded file is ultimately
+ * visible.
+ */
+ @Test
+ public void testMultipartOverlapWithTransientFile() throws Throwable {
+ // until there's a way to explicitly ask for a multipart uploader from a
+ // specific FS, explicitly create one bonded to the raw FS.
+ describe("testMultipartOverlapWithTransientFile");
+ S3AFileSystem fs = getFileSystem();
+ Path path = path("testMultipartOverlapWithTransientFile");
+ fs.delete(path, true);
+ MultipartUploader mpu = mpu(1);
+ UploadHandle upload1 = mpu.initialize(path);
+ byte[] dataset = dataset(1024, '0', 10);
+ final Map<Integer, PartHandle> handles = new HashMap<>();
+ LOG.info("Uploading multipart entry");
+ PartHandle value = mpu.putPart(path, new ByteArrayInputStream(dataset), 1,
+ upload1,
+ dataset.length);
+ // upload 1K
+ handles.put(1, value);
+ // confirm the path is absent
+ ContractTestUtils.assertPathDoesNotExist(fs,
+ "path being uploaded", path);
+ // now create an empty file
+ ContractTestUtils.touch(fs, path);
+ final FileStatus touchStatus = fs.getFileStatus(path);
+ LOG.info("0-byte file has been created: {}", touchStatus);
+ fs.delete(path, false);
+ // now complete the upload
+ mpu.complete(path, handles, upload1);
+
+ // wait for the data to arrive
+ eventually(timeToBecomeConsistentMillis(), 500, () -> {
+ FileStatus mpuStatus = fs.getFileStatus(path);
+ assertTrue("File is empty in " + mpuStatus, mpuStatus.getLen() > 0);
+ return mpuStatus;
Review comment:
Once we've checked the file is there can we delete that file? Or is it not
necessary? (Cleanup at the end of the test)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
