[
https://issues.apache.org/jira/browse/DRILL-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Farkas updated DRILL-6644:
----------------------------------
Description:
There are two cases where the HashJoin Memory calculator over reserves memory:
1. It reserves a maximum incoming probe batch size during the build phase.
This is not really necessary because we will not fetch probe data until the
probe phase. We only have to account for the data received during OK_NEW_SCHEMA.
2. When preparing for the probe phase it uses the worst case hashtable size
even after we have constructed a hash table. In this case when we already have
a hash table we should not use it's predicted worst case size, we should use
its actual size in the memory calculator.
was:
There are two cases where the HashJoin Memory calculator over reserves memory:
1. It reserves a maximum incoming probe batch size during the build phase.
This is not really necessary because we will not fetch probe data until the
probe phase. We only have to account for the space data received during
OK_NEW_SCHEMA occupied.
2. When preparing for the probe phase it uses the worst case hashtable size
even after we have constructed a hash table. In this case when we already have
a hash table we should not use it's predicted worst case size, we should use
its actual size in the memory calculator.
> In Some Cases The HashJoin Memory Calculator Over Reserves Memory
> -----------------------------------------------------------------
>
> Key: DRILL-6644
> URL: https://issues.apache.org/jira/browse/DRILL-6644
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: Timothy Farkas
> Assignee: Timothy Farkas
> Priority: Major
>
> There are two cases where the HashJoin Memory calculator over reserves memory:
> 1. It reserves a maximum incoming probe batch size during the build phase.
> This is not really necessary because we will not fetch probe data until the
> probe phase. We only have to account for the data received during
> OK_NEW_SCHEMA.
> 2. When preparing for the probe phase it uses the worst case hashtable size
> even after we have constructed a hash table. In this case when we already
> have a hash table we should not use it's predicted worst case size, we should
> use its actual size in the memory calculator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
