>From Wail Alkowaileet <[email protected]>: Wail Alkowaileet has submitted this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/17737 )
Change subject: [ASTERIXDB-3250][STO] Ensure writing indexes' empty files to S3 ...................................................................... [ASTERIXDB-3250][STO] Ensure writing indexes' empty files to S3 - user model changes: no - storage format changes: no - interface changes: yes Details: Empty indexes files are not written to S3. This patch fixes this issue. Interface changes: - Add isEmpty() in ICloudBufferedWriter Change-Id: I740c0d9e82bd0f933a37c4f68c3d95dc715cb46b Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/17737 Integration-Tests: Jenkins <[email protected]> Reviewed-by: Murtadha Al Hubail <[email protected]> Tested-by: Jenkins <[email protected]> --- M asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/CloudResettableInputStream.java M asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3BufferedWriter.java M asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/ICloudBufferedWriter.java 3 files changed, 46 insertions(+), 2 deletions(-) Approvals: Murtadha Al Hubail: Looks good to me, approved Jenkins: Verified; Verified diff --git a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/CloudResettableInputStream.java b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/CloudResettableInputStream.java index 3a7504a..7ba95c5 100644 --- a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/CloudResettableInputStream.java +++ b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/CloudResettableInputStream.java @@ -100,7 +100,13 @@ public void finish() throws HyracksDataException { open(); try { - if (writeBuffer.position() > 0) { + if (writeBuffer.position() > 0 || bufferedWriter.isEmpty()) { + /* + * upload if: + * (1) the writeBuffer is not empty + * OR + * (2) nothing was written to the file at all to ensure writing empty file + */ uploadAndWait(); } bufferedWriter.finish(); diff --git a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/ICloudBufferedWriter.java b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/ICloudBufferedWriter.java index 4f08111..16e79f9 100644 --- a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/ICloudBufferedWriter.java +++ b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/ICloudBufferedWriter.java @@ -34,6 +34,13 @@ int upload(InputStream stream, int length); /** + * Checks whether the writer has not written anything + * + * @return true if nothing was written, false otherwise + */ + boolean isEmpty(); + + /** * Finishes the upload * * @throws HyracksDataException HyracksDataException diff --git a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3BufferedWriter.java b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3BufferedWriter.java index 71246f1..2efd214 100644 --- a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3BufferedWriter.java +++ b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3BufferedWriter.java @@ -73,10 +73,17 @@ } @Override + public boolean isEmpty() { + return uploadId == null; + } + + @Override public void finish() throws HyracksDataException { if (uploadId == null) { - return; + throw new IllegalStateException("Cannot finish without writing any bytes"); } + + // A non-empty files, proceed with completing the multipart upload CompletedMultipartUpload completedMultipartUpload = CompletedMultipartUpload.builder().parts(partQueue).build(); CompleteMultipartUploadRequest completeMultipartUploadRequest = CompleteMultipartUploadRequest.builder() .bucket(bucket).key(path).uploadId(uploadId).multipartUpload(completedMultipartUpload).build(); -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/17737 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I740c0d9e82bd0f933a37c4f68c3d95dc715cb46b Gerrit-Change-Number: 17737 Gerrit-PatchSet: 2 Gerrit-Owner: Wail Alkowaileet <[email protected]> Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-Reviewer: Murtadha Al Hubail <[email protected]> Gerrit-Reviewer: Wail Alkowaileet <[email protected]> Gerrit-MessageType: merged
