>From Hussain Towaileb <[email protected]>:
Hussain Towaileb has uploaded this change for review. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18230 )
Change subject: Avoid writing empty files when COPYing TO S3
......................................................................
Avoid writing empty files when COPYing TO S3
Change-Id: Iee18cb7458495471fe38e5f6f2f3721d8ce9c12f
---
M
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/writer/ExternalFileWriter.java
1 file changed, 17 insertions(+), 2 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/30/18230/1
diff --git
a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/writer/ExternalFileWriter.java
b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/writer/ExternalFileWriter.java
index f9f98da..95dc962 100644
---
a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/writer/ExternalFileWriter.java
+++
b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/writer/ExternalFileWriter.java
@@ -57,11 +57,17 @@
// Ignore writing values for unresolvable partition paths
return;
}
- writer.write(value);
- tupleCounter++;
+
+ // create a new file only when we reach the maximum tuples and we know
a new tuple is incoming
+ // e.g., if max is 1000, we hit tuple 1001, we will upload and create
a new file, if we only have 1000
+ // we will stop here, and calling the close/finish will upload
whatever is written. This is to avoid
+ // creating and uploading empty files
if (tupleCounter >= maxResultPerFile) {
newFile();
}
+
+ writer.write(value);
+ tupleCounter++;
}
@Override
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18230
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Iee18cb7458495471fe38e5f6f2f3721d8ce9c12f
Gerrit-Change-Number: 18230
Gerrit-PatchSet: 1
Gerrit-Owner: Hussain Towaileb <[email protected]>
Gerrit-MessageType: newchange
