>From Hussain Towaileb <[email protected]>: Hussain Towaileb has submitted this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18230 )
Change subject: Avoid writing empty files when COPYing TO S3 ...................................................................... Avoid writing empty files when COPYing TO S3 Change-Id: Iee18cb7458495471fe38e5f6f2f3721d8ce9c12f Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18230 Integration-Tests: Jenkins <[email protected]> Tested-by: Jenkins <[email protected]> Reviewed-by: Hussain Towaileb <[email protected]> --- M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/writer/ExternalFileWriter.java 1 file changed, 21 insertions(+), 2 deletions(-) Approvals: Hussain Towaileb: Looks good to me, approved Jenkins: Verified; Verified diff --git a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/writer/ExternalFileWriter.java b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/writer/ExternalFileWriter.java index f9f98da..95dc962 100644 --- a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/writer/ExternalFileWriter.java +++ b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/writer/ExternalFileWriter.java @@ -57,11 +57,17 @@ // Ignore writing values for unresolvable partition paths return; } - writer.write(value); - tupleCounter++; + + // create a new file only when we reach the maximum tuples and we know a new tuple is incoming + // e.g., if max is 1000, we hit tuple 1001, we will upload and create a new file, if we only have 1000 + // we will stop here, and calling the close/finish will upload whatever is written. This is to avoid + // creating and uploading empty files if (tupleCounter >= maxResultPerFile) { newFile(); } + + writer.write(value); + tupleCounter++; } @Override -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18230 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Iee18cb7458495471fe38e5f6f2f3721d8ce9c12f Gerrit-Change-Number: 18230 Gerrit-PatchSet: 5 Gerrit-Owner: Hussain Towaileb <[email protected]> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hussain Towaileb <[email protected]> Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-Reviewer: Wail Alkowaileet <[email protected]> Gerrit-MessageType: merged
