[
https://issues.apache.org/jira/browse/IMPALA-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17870883#comment-17870883
]
ASF subversion and git services commented on IMPALA-13270:
----------------------------------------------------------
Commit 0918147e69b1625e76886506b065c310c7cc52d1 in impala's branch
refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=0918147e6 ]
IMPALA-13270: Fix IllegalStateException on runtime filter
IMPALA-12800 improved ExprSubstitutionMap to use a HashMap for lookups.
Some methods in ExprSubstitutionMap guard against duplicate entries, but
not creation or adding fields, so cases were added where repeated
expressions would be added to the map. In practice, only the first entry
added would be matched. IMPALA-12800 started removing duplicate entries
from the map to reduce memory use, but missed that one caller -
RuntimeFilterGenerator - was expecting the map size to exactly match the
input expression list.
Fixes the IllegalStateException caused by runtime filters where the same
expression is repeated multiple times by changing the precondition to
verify that each SlotRef has a mapping added. It doesn't verify the
final size, because SlotRefs may be repeated and the map will avoid
adding duplicates.
Removes trim method - added in IMPALA-13270 - as it no longer provides
any benefit when performing lookups with a HashMap, and may actually do
more work during the trim. test_query_compilation.py continues to pass,
and I see no discernible difference in "Single node plan" time; both are
30-40ms on my machine.
Adds a test case that failed with the old precondition. IDE-assisted
search did not find any other cases where ExprSubstitutionMap#size is
compared against a non-zero value.
Change-Id: I23c7bcf33e5185f10a6ae475debb8ab70a2ec5eb
Reviewed-on: http://gerrit.cloudera.org:8080/21638
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Bug when comparing ExprSubstitutionMap.size()
> ---------------------------------------------
>
> Key: IMPALA-13270
> URL: https://issues.apache.org/jira/browse/IMPALA-13270
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 4.5.0
> Reporter: Riza Suminto
> Assignee: Michael Smith
> Priority: Critical
>
> IMPALA-12800 implements deduplication in ExprSubstitutionMap. It is possible
> that ExprSubstitutionMap.size() will return different value after
> IMPALA-12800. We need to revisit ExprSubstitutionMap.size() calls, especially
> within Precondition such as in RuntimeFilterGenerator.computeTargetExpr() and
> ExprSubstitutionMap.trim().
> In particular, the following query causes IllegalStateException
> {code}
> USE functional;
> SELECT alt.int_col, alt.string_col
> FROM alltypes alt
> INNER JOIN (
> SELECT int_col,
> concat('20', substring(date_string_col, 7, 2), '-',
> substring(date_string_col, 4, 2), '-', substring(date_string_col, 1, 2), '
> 00:00:00') ts
> FROM alltypesaggnonulls GROUP BY int_col, date_string_col
> ) altann ON altann.ts = from_unixtime(alt.id, 'yyyy-MM-dd HH:mm:ss');
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
