Paul Rogers created DRILL-7687:
----------------------------------
Summary: Inaccurate memory estimates in hash join
Key: DRILL-7687
URL: https://issues.apache.org/jira/browse/DRILL-7687
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.15.0
Reporter: Paul Rogers
See DRILL-7675. In this ticket, we tried to reproduce an OOM case in the
partition sender. In so doing, we mucked with various parallelization options.
The query has 2 MB of data, but at one point the query would fail to run
because the hash join could not obtain enough memory (on a system with 8 GB of
memory available.)
The problem is that the memory calculator sees a worst-case scenario: a row
with 250+ columns. The hash join estimated it needed something like 650MB of
memory to perform the join. (That is 650 MB per fragment, and there were
multiple fragments.) Since there was insufficient memory, and the
{{drill.exec.hashjoin.fallback.enabled}} option was disabled, the hash join
failed before it even started.
Better would be to at least try the query. In this case, with 2MB of data, the
query succeeds. (Had to enable the magic option to do so.)
Better also would be to use the estimated row counts when estimating memory
use. Maybe better estimates for the amount of memory needed per row. (The data
in question has multiple nested map arrays, causing cardinality estimates to
grow by 5x at each level.)
Perhaps use the "batch sizing" mechanism to detect actual memory use by
analyzing the incoming batch.
There is no obvious answer. However, the goal is clear: the query should
succeed if the actual memory needed fits within that available; we should not
fail proactively based on estimates of needed memory. (This what the
{{drill.exec.hashjoin.fallback.enabled}} option does; perhaps it should be on
by default.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
