Tim Armstrong created IMPALA-5885:
-------------------------------------
Summary: Parquet scanner does not free local allocations in filter
contexts
Key: IMPALA-5885
URL: https://issues.apache.org/jira/browse/IMPALA-5885
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.8.0, Impala 2.6.0, Impala 2.7.0, Impala 2.9.0,
Impala 2.10.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong
This problem can occur if runtime filter expressions that are evaluated in the
scan allocate temporary memory - "local allocations". These accumulate for each
scan range and are only
freed upon scan range completion.
A contrived query that exhibits the problem is the following. If I continue
adding upper() and lower() to the expression the memory consumption of the scan
node will continue to grow - up to 100MB for each extra function call!
{code}
set runtime_filter_wait_time_ms=1000000;
select straight_join count(*) from tpch_parquet.lineitem l1 join
tpch_parquet.lineitem l2 on upper(lower(upper(lower(l1.l_comment)))) =
concat(l2.l_comment, 'foo');
summary;
{code}
I think other conjuncts in the scanner may be affected by the same problem,
e.g. the min_max conjuncts.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
