[
https://issues.apache.org/jira/browse/DRILL-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16562283#comment-16562283
]
ASF GitHub Bot commented on DRILL-6644:
---------------------------------------
ilooner opened a new pull request #1409: DRILL-6644: Don't reserve space for
incoming probe batches unnecessarily during the build phase.
URL: https://github.com/apache/drill/pull/1409
## Problem
Previously the memory calculator for HashJoin was reserving space for the
worst case sized incoming probe batch. This is actually completely unnecessary
since we won't read probe side data before the build phase anymore after
DRILL-6453, so this effectively causes memory to be underutilized.
## Solution
There are two cases we have to handle:
- We received probe side data when fetching the schema. In this case we only
have to account for the memory consumed by the probe data we received in the
build phase. Not the worst case incoming probe batch size.
- We received NO probe side data when fetching the schema for the probe
side. In this case we don't have to reserve any memory for probe data during
the build phase.
## Prerequisite
The PR for DRILL-6453 must be merged first before this can go in. In this PR
only look at the changes in the last commit.
https://github.com/apache/drill/pull/1408
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> In Some Cases The HashJoin Memory Calculator Over Reserves Memory For The
> Probe Side During The Build Phase
> -----------------------------------------------------------------------------------------------------------
>
> Key: DRILL-6644
> URL: https://issues.apache.org/jira/browse/DRILL-6644
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: Timothy Farkas
> Assignee: Timothy Farkas
> Priority: Major
>
> There are two cases where the HashJoin Memory calculator over reserves memory:
> 1. It reserves a maximum incoming probe batch size during the build phase.
> This is not really necessary because we will not fetch probe data until the
> probe phase. We only have to account for the data received during
> OK_NEW_SCHEMA.
> 2. https://issues.apache.org/jira/browse/DRILL-6646
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
