Taewoo Kim has submitted this change and it was merged. Change subject: ASTERIXDB-1751: Spilled partition files will be cleaned properly. ......................................................................
ASTERIXDB-1751: Spilled partition files will be cleaned properly. - Properly closes the spilled parition file during the build phase of hash join so that the file will be deleted after the hash join finishes. Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5 Reviewed-on: https://asterix-gerrit.ics.uci.edu/1395 Sonar-Qube: Jenkins <[email protected]> Tested-by: Jenkins <[email protected]> BAD: Jenkins <[email protected]> Integration-Tests: Jenkins <[email protected]> Reviewed-by: Ian Maxon <[email protected]> --- M hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/join/OptimizedHybridHashJoin.java 1 file changed, 8 insertions(+), 2 deletions(-) Approvals: Ian Maxon: Looks good to me, approved Jenkins: Verified; No violations found; No violations found; Verified diff --git a/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/join/OptimizedHybridHashJoin.java b/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/join/OptimizedHybridHashJoin.java index 0770784..4f85e57 100644 --- a/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/join/OptimizedHybridHashJoin.java +++ b/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/join/OptimizedHybridHashJoin.java @@ -247,10 +247,15 @@ break; } - for (int pid = spilledStatus.nextSetBit(0); pid >= 0; pid = spilledStatus.nextSetBit(pid + 1)) { + for (int pid = spilledStatus.nextSetBit(0); pid >= 0 + && pid < numOfPartitions; pid = spilledStatus.nextSetBit(pid + 1)) { if (bufferManager.getNumTuples(pid) > 0) { bufferManager.flushPartition(pid, getSpillWriterOrCreateNewOneIfNotExist(pid, whichSide)); bufferManager.clearPartition(pid); + } + // It doesn't matter whether a spilled partition currently holds a tuple in memory or not. + // The file that holds the corresponding spilled partition needs to be closed. + if (runFileWriters[pid] != null) { runFileWriters[pid].close(); } } @@ -299,7 +304,8 @@ } private int selectPartitionsToReload(int freeSpace, int pid) { - for (int i = spilledStatus.nextSetBit(pid); i >= 0; i = spilledStatus.nextSetBit(i + 1)) { + for (int i = spilledStatus.nextSetBit(pid); i >= 0 + && i < numOfPartitions; i = spilledStatus.nextSetBit(i + 1)) { assert buildRFWriters[i].getFileSize() > 0 : "How comes a spilled partition have size 0?"; if (freeSpace >= buildRFWriters[i].getFileSize()) { return i; -- To view, visit https://asterix-gerrit.ics.uci.edu/1395 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Taewoo Kim <[email protected]> Gerrit-Reviewer: Ian Maxon <[email protected]> Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-Reviewer: Taewoo Kim <[email protected]>
