Ben-Zvi commented on a change in pull request #1606: Drill 6845: Semi-Hash-Join
to skip incoming build duplicates, automatically stop skipping if too few
URL: https://github.com/apache/drill/pull/1606#discussion_r247738584
##########
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinSpillControlImpl.java
##########
@@ -18,39 +18,211 @@
package org.apache.drill.exec.physical.impl.join;
+ import org.apache.drill.exec.ExecConstants;
import org.apache.drill.exec.memory.BufferAllocator;
+ import org.apache.drill.exec.ops.FragmentContext;
+ import org.apache.drill.exec.record.RecordBatch;
import org.apache.drill.exec.record.RecordBatchMemoryManager;
- import org.apache.drill.exec.record.RecordBatchSizer;
- import org.apache.drill.exec.record.VectorContainer;
+ import org.slf4j.Logger;
+ import org.slf4j.LoggerFactory;
+
+ import javax.annotation.Nullable;
+ import java.util.Set;
+
+ import static org.apache.drill.exec.record.JoinBatchMemoryManager.LEFT_INDEX;
+ import static
org.apache.drill.exec.record.JoinBatchMemoryManager.RIGHT_INDEX;
/**
* This class is currently used only for Semi-Hash-Join that avoids duplicates
by the use of a hash table
- * The method {@link
HashJoinMemoryCalculator.HashJoinSpillControl#shouldSpill(VectorContainer)}
returns true if the memory available now to the allocator if not enough
+ * The method {@link
HashJoinMemoryCalculator.HashJoinSpillControl#shouldSpill()} returns true if
the memory available now to the allocator if not enough
* to hold (a multiple of, for safety) a new allocated batch
*/
-public class HashJoinSpillControlImpl implements
HashJoinMemoryCalculator.HashJoinSpillControl {
+public class HashJoinSpillControlImpl implements
HashJoinMemoryCalculator.BuildSidePartitioning {
+ private static final Logger logger =
LoggerFactory.getLogger(HashJoinSpillControlImpl.class);
+
private BufferAllocator allocator;
private int recordsPerBatch;
private int minBatchesInAvailableMemory;
private RecordBatchMemoryManager batchMemoryManager;
+ private int initialPartitions;
+ private int numPartitions;
+ private int recordsPerPartitionBatchProbe;
+ private FragmentContext context;
- HashJoinSpillControlImpl(BufferAllocator allocator, int recordsPerBatch, int
minBatchesInAvailableMemory, RecordBatchMemoryManager batchMemoryManager) {
+ HashJoinSpillControlImpl(BufferAllocator allocator, int recordsPerBatch, int
minBatchesInAvailableMemory, RecordBatchMemoryManager batchMemoryManager,
FragmentContext context) {
this.allocator = allocator;
this.recordsPerBatch = recordsPerBatch;
this.minBatchesInAvailableMemory = minBatchesInAvailableMemory;
this.batchMemoryManager = batchMemoryManager;
+ this.context = context;
}
@Override
- public boolean shouldSpill(VectorContainer currentVectorContainer) {
- assert currentVectorContainer.hasRecordCount();
- assert currentVectorContainer.getRecordCount() == recordsPerBatch;
- // Expected new batch size like the current, plus the Hash Value vector (4
bytes per HV)
- long batchSize = new
RecordBatchSizer(currentVectorContainer).getActualSize() + 4 * recordsPerBatch;
+ public boolean shouldSpill() {
+ // Expected new batch size like the current, plus the Hash Values vector
(4 bytes per HV)
+ long batchSize = (
batchMemoryManager.getRecordBatchSizer(RIGHT_INDEX).getRowAllocWidth() + 4 ) *
recordsPerBatch;
long reserveForOutgoing = batchMemoryManager.getOutputBatchSize();
long memoryAvailableNow = allocator.getLimit() -
allocator.getAllocatedMemory() - reserveForOutgoing;
Review comment:
* The *incoming* probe batch is not charged to this HJ allocator.
* The *internal* probe batches needed (one for each spilled partition) are
ignored here, but checked later in the "post build" stage (when the number of
spilled partitions is known). If not enough memory at that "post build" time,
then we spill one (or more) "pristine" (i.e., that never spilled before)
partition.
* The initial allocation of *internal* build batches (plus a hash table for
each) is controlled by reducing the number of partitions as needed to
accommodate all these.
* Indeed the above did not cover the case where the *internal probe* batch
is huge, and even if **all** the partitions are spilled, there will not be
enough memory to hold all these batches. One possible solution here can be to
reduce the number of rows per such batch (it's internal anyway). Another
solution was made: In `calculateMemoryUsage()`, take into account the size of
the *internal probe* batch when calculating the number of partitions. (Yes,
this is based on schema only, not real data).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
