[
https://issues.apache.org/jira/browse/DRILL-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567908#comment-16567908
]
ASF GitHub Bot commented on DRILL-6644:
---------------------------------------
ilooner commented on a change in pull request #1409: DRILL-6644: Don't reserve
space for incoming probe batches unnecessarily during the build phase.
URL: https://github.com/apache/drill/pull/1409#discussion_r207466313
##########
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinMemoryCalculatorImpl.java
##########
@@ -391,7 +391,7 @@ private void calculateMemoryUsage()
// probe batch we sniffed.
// TODO when batch sizing project is complete we won't have to sniff
probe batches since
// they will have a well defined size.
- reservedMemory = incompletePartitionsBatchSizes + maxBuildBatchSize +
maxProbeBatchSize;
+ reservedMemory = incompletePartitionsBatchSizes + maxBuildBatchSize +
probeSizePredictor.getBatchSize();
Review comment:
@sohami explained that it is up to the operator whether or not to transfer
ownership of the incoming vector batch and described the two primary use cases
for transferring or not transferring ownership. Looking at the HashJoin code we
never transfer ownership of the incoming probe batch and copy the probe data
directly to the outgoing batch. This explains why you never got an OOM.
So I think you are right, we should not be reserving space for the incoming
probe batch, since it is never owned by HashJoin's allocator. Thanks for
catching this @Ben-Zvi I will update the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> In Some Cases The HashJoin Memory Calculator Over Reserves Memory For The
> Probe Side During The Build Phase
> -----------------------------------------------------------------------------------------------------------
>
> Key: DRILL-6644
> URL: https://issues.apache.org/jira/browse/DRILL-6644
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: Timothy Farkas
> Assignee: Timothy Farkas
> Priority: Major
>
> There are two cases where the HashJoin Memory calculator over reserves memory:
> 1. It reserves a maximum incoming probe batch size during the build phase.
> This is not really necessary because we will not fetch probe data until the
> probe phase. We only have to account for the data received during
> OK_NEW_SCHEMA.
> 2. https://issues.apache.org/jira/browse/DRILL-6646
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
