ilooner commented on a change in pull request #1606: DRILL-6845: Semi-Hash-Join
to skip incoming build duplicates, automatically stop skipping if too few
URL: https://github.com/apache/drill/pull/1606#discussion_r248158415
##########
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinSpillControlImpl.java
##########
@@ -18,39 +18,211 @@
package org.apache.drill.exec.physical.impl.join;
+ import org.apache.drill.exec.ExecConstants;
import org.apache.drill.exec.memory.BufferAllocator;
+ import org.apache.drill.exec.ops.FragmentContext;
+ import org.apache.drill.exec.record.RecordBatch;
import org.apache.drill.exec.record.RecordBatchMemoryManager;
- import org.apache.drill.exec.record.RecordBatchSizer;
- import org.apache.drill.exec.record.VectorContainer;
+ import org.slf4j.Logger;
+ import org.slf4j.LoggerFactory;
+
+ import javax.annotation.Nullable;
+ import java.util.Set;
+
+ import static org.apache.drill.exec.record.JoinBatchMemoryManager.LEFT_INDEX;
+ import static
org.apache.drill.exec.record.JoinBatchMemoryManager.RIGHT_INDEX;
/**
* This class is currently used only for Semi-Hash-Join that avoids duplicates
by the use of a hash table
- * The method {@link
HashJoinMemoryCalculator.HashJoinSpillControl#shouldSpill(VectorContainer)}
returns true if the memory available now to the allocator if not enough
+ * The method {@link
HashJoinMemoryCalculator.HashJoinSpillControl#shouldSpill()} returns true if
the memory available now to the allocator if not enough
* to hold (a multiple of, for safety) a new allocated batch
*/
-public class HashJoinSpillControlImpl implements
HashJoinMemoryCalculator.HashJoinSpillControl {
+public class HashJoinSpillControlImpl implements
HashJoinMemoryCalculator.BuildSidePartitioning {
+ private static final Logger logger =
LoggerFactory.getLogger(HashJoinSpillControlImpl.class);
+
private BufferAllocator allocator;
private int recordsPerBatch;
private int minBatchesInAvailableMemory;
private RecordBatchMemoryManager batchMemoryManager;
+ private int initialPartitions;
+ private int numPartitions;
+ private int recordsPerPartitionBatchProbe;
+ private FragmentContext context;
- HashJoinSpillControlImpl(BufferAllocator allocator, int recordsPerBatch, int
minBatchesInAvailableMemory, RecordBatchMemoryManager batchMemoryManager) {
+ HashJoinSpillControlImpl(BufferAllocator allocator, int recordsPerBatch, int
minBatchesInAvailableMemory, RecordBatchMemoryManager batchMemoryManager,
FragmentContext context) {
this.allocator = allocator;
this.recordsPerBatch = recordsPerBatch;
this.minBatchesInAvailableMemory = minBatchesInAvailableMemory;
this.batchMemoryManager = batchMemoryManager;
+ this.context = context;
}
@Override
- public boolean shouldSpill(VectorContainer currentVectorContainer) {
- assert currentVectorContainer.hasRecordCount();
- assert currentVectorContainer.getRecordCount() == recordsPerBatch;
- // Expected new batch size like the current, plus the Hash Value vector (4
bytes per HV)
- long batchSize = new
RecordBatchSizer(currentVectorContainer).getActualSize() + 4 * recordsPerBatch;
+ public boolean shouldSpill() {
+ // Expected new batch size like the current, plus the Hash Values vector
(4 bytes per HV)
+ long batchSize = (
batchMemoryManager.getRecordBatchSizer(RIGHT_INDEX).getRowAllocWidth() + 4 ) *
recordsPerBatch;
long reserveForOutgoing = batchMemoryManager.getOutputBatchSize();
long memoryAvailableNow = allocator.getLimit() -
allocator.getAllocatedMemory() - reserveForOutgoing;
Review comment:
- By *partition batch* I was referring to the small batches that are used to
store the records that are appended to each partition. They are stored in the
HashPartition.tmpBatchesList list. We need to reserve space for one partition
batch for each partition. Otherwise we can run into a scenario where we OOM
when appending a row to a partition and allocate a new partition batch to store
it. I already have code to account for these in my memory calculator. See
**HashJoinMemoryCalculatorImpl lines 361 and 373**.
- Also we should not be using MIN_BATCHES_IN_MEMORY. We can account for the
probe batches exactly when handling a spilled partition. I have already
implemented the logic for this in **HashJoinMemoryCalculatorImpl lines 363 -
365, 378 - 380, and 383 - 389**. It should just be a matter of copying the code
and plugging it in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
