Venki Korukanti created DRILL-1993:
--------------------------------------
Summary: Fix allocation issues in HashTable and HashAgg to reduce
memory waste
Key: DRILL-1993
URL: https://issues.apache.org/jira/browse/DRILL-1993
Project: Apache Drill
Issue Type: Bug
Components: Execution - Operators
Reporter: Venki Korukanti
Assignee: Venki Korukanti
Fix For: 0.8.0
Issues found:
+ Key container allocation issue in HashTable
Currently we allocate 2^16 records capacity memory for "hashValues" and "links"
vectors in each BatchHolder, but for "key" holders we use the allocateNew which
by default allocates low capacity memory compared to "hashValues" and "links"
vector capacities (incase of Integer key, capacity is 4096 records). This
causes "key" holders to fill up much sooner even though "hashValues" and
"links" vectors still have lot of free entries. As a result we create more
BatchHolders than required causing wasted space in "links" and "hashValues"
vectors in each BatchHolder. And for each new BatchHolder we create a SV4
vector in HashJoinHelper which is another overhead.
+ Allocation issues in HashAggTemplate
HashAggTemplate has its own BatchHolders which has vectors allocated using
allocateNew (i.e small capacity). Whenever a BatchHolder in HashAggTemplate
reaches its capacity, we add a new BatchHolder in HashTable. This causes the
HashTable BatchHolders to be not space efficient.
+ Update the HashAggTemplate.outputCurrentBatch to consider cases where all
entries in a single BatchHolder are can't be copied over to output vectors in a
single pass (output vectors capacity is lower than the number of records in
BatchHolder)
+ Lazy BatchHolder creation for both HashAgg and HashTable
Don't allocate the BatchHolder until first put request is received. This way we
don't waste space in fragments which don't receive any input records. This is
possible when the group by key has very few distinct values (such as shipmode
in TPCH) only few fragments receive the data. Our current parallelization code
is not considering the distinct values when parallelizing hash exchanges.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
