[
https://issues.apache.org/jira/browse/IMPALA-15150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095118#comment-18095118
]
ASF subversion and git services commented on IMPALA-15150:
----------------------------------------------------------
Commit e81559d8d32d4b5ccdae0224d6ed7ba9550e06ca in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=e81559d8d ]
IMPALA-15150: Adds missing sleep in
test_runtime_filter_annotation_with_hbo_cardinality
HBO stats were written asychronously after the query finishes. The HBO
e2e tests are both in the following pattern:
1. Run some queries that produce HBO stats.
2. Sleep 1s to ensure the stats are written.
3. Run EXPLAIN on test queries.
test_runtime_filter_annotation_with_hbo_cardinality is the only test
that misses the sleep. When the HBO stats of the build side is missing,
its cardinality is higher than the actual value. So the estimated
cardinality of the prode side scan after applying runtime filters (i.e.
filteredCardinality_) is higher than its cardinality from HBO stats and
the test fails.
Testing
- Ran the test locally.
Change-Id: I8c0ccc9def6d14c276d2a23403b5eae7a09849c5
Reviewed-on: http://gerrit.cloudera.org:8080/24560
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> test_runtime_filter_annotation_with_hbo_cardinality is flaky
> ------------------------------------------------------------
>
> Key: IMPALA-15150
> URL: https://issues.apache.org/jira/browse/IMPALA-15150
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
>
> test_runtime_filter_annotation_with_hbo_cardinality could fail due to the HBO
> stats on "01:SCAN HDFS [tpch_parquet.orders]" is missing:
> {noformat}
> AssertionError: Max Per-Host Resource Reservation: Memory=28.88MB Threads=6
> Per-Host Resource Estimates: Memory=409MB
>
> PLAN-ROOT SINK
> |
> 07:AGGREGATE [FINALIZE]
> | output: count:merge(l_orderkey)
> | row-size=8B cardinality=1
> |
> 06:EXCHANGE [UNPARTITIONED]
> |
> 03:AGGREGATE
> | output: count(l_orderkey)
> | row-size=8B cardinality=3
> |
> 02:HASH JOIN [INNER JOIN, PARTITIONED]
> | hash predicates: l_orderkey = o_orderkey
> | runtime filters: RF000 <- o_orderkey
> | row-size=62B cardinality=10.95K
> |
> |--05:EXCHANGE [HASH(o_orderkey)]
> | |
> | 01:SCAN HDFS [tpch_parquet.orders]
> | HDFS partitions=1/1 files=2 size=54.21MB
> | predicates: o_custkey < 1000
> | row-size=16B cardinality=150.00K
> |
> 04:EXCHANGE [HASH(l_orderkey)]
> |
> 00:SCAN HDFS [tpch_parquet.lineitem]
> HDFS partitions=1/1 files=3 size=193.97MB
> predicates: l_discount <= 0.07, l_discount >= 0.05, l_quantity < 24,
> l_shipdate < '1995-01-01', l_shipdate >= '1994-01-01'
> runtime filters: RF000 -> l_orderkey
> row-size=46B cardinality=114.16K (from HBO)
> assert 'cardinality=37.88K(filtered from 114.16K from HBO)' in "Max Per-Host
> Resource Reservation: Memory=28.88MB Threads=6\nPer-Host Resource Estimates:
> Memory=409MB\n\nPLAN-ROOT S...', l_shipdate >= '1994-01-01'\n runtime
> filters: RF000 -> l_orderkey\n row-size=46B cardinality=114.16K (from HBO)"
> + where "Max Per-Host Resource Reservation: Memory=28.88MB
> Threads=6\nPer-Host Resource Estimates: Memory=409MB\n\nPLAN-ROOT S...',
> l_shipdate >= '1994-01-01'\n runtime filters: RF000 -> l_orderkey\n
> row-size=46B cardinality=114.16K (from HBO)" = <built-in method join of str
> object at 0x882a98>(['Max Per-Host Resource Reservation: Memory=28.88MB
> Threads=6', 'Per-Host Resource Estimates: Memory=409MB', '', 'PLAN-ROOT
> SINK', '|', '07:AGGREGATE [FINALIZE]', ...])
> + where <built-in method join of str object at 0x882a98> = '\n'.join
> + and ['Max Per-Host Resource Reservation: Memory=28.88MB Threads=6',
> 'Per-Host Resource Estimates: Memory=409MB', '', 'PLAN-ROOT SINK', '|',
> '07:AGGREGATE [FINALIZE]', ...] =
> <tests.common.impala_connection.ImpylaHS2ResultSet object at
> 0x7f9029583950>.data{noformat}
> The cause is that the test doesn't wait for 1s for previous queries to flush
> HBO stats which is asychronous. The other HBO tests all have such a sleep.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
