Laszlo Gaal created IMPALA-13752:
------------------------------------
Summary: test_parquet_late_materialization fails on runtime filter
not published in exhaustive builds
Key: IMPALA-13752
URL: https://issues.apache.org/jira/browse/IMPALA-13752
Project: IMPALA
Issue Type: Bug
Affects Versions: Impala 4.6.0
Reporter: Laszlo Gaal
EE test test_parquet_late_materialization complains about not finding "1 of 1
Runtime Filter Published" in the profile.
Start of the error backtrace:
{code}
query_test/test_parquet_late_materialization.py:38: in
test_parquet_late_materialization
self.run_test_case('QueryTest/parquet-late-materialization', vector)
common/impala_test_suite.py:908: in run_test_case
update_section=pytest.config.option.update_results)
common/test_result_verifier.py:699: in verify_runtime_profile
actual))
E AssertionError: Did not find matches for lines in runtime profile:
E EXPECTED LINES:
E row_regex: .*1 of 1 Runtime Filter Published.*
E
E ACTUAL PROFILE:
E Query (id=e84f6c5546558a00:1f33e21500000000):
E DEBUG MODE WARNING: Query profile created while running a DEBUG build of
Impala. Use RELEASE builds to measure query performance.
E Summary:
E Session ID: 1240823cf745e766:917a175b542d49a7
E Session Type: BEESWAX
E Start Time: 2025-02-08 15:03:03.540779000
E End Time: 2025-02-08 15:03:04.238968000
E Duration: 698.189ms (698189 us)
E Query Type: QUERY
E Query State: FINISHED
E Impala Query State: FINISHED
E Query Status: OK
E Impala Version: impalad version 4.5.0-SNAPSHOT DEBUG (build
856741b1e8371b983da10f23a166f19aac71f1f8)
E User: jenkins
E Connected User: jenkins
E Delegated User:
E Network Address: 127.0.0.1:38404
E Default Db: functional_parquet
E Sql Statement: select * from tpch_parquet.lineitem l
E join tpch_parquet.orders o on l.l_orderkey = o.o_orderkey
E where o_orderdate='1992-06-22' and o_totalprice = 153827.26
E Coordinator:
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000
E Query Options (set by configuration):
ABORT_ON_ERROR=1,EXEC_SINGLE_NODE_ROWS_THRESHOLD=0,RUNTIME_FILTER_WAIT_TIME_MS=5000,DISABLE_CODEGEN_ROWS_THRESHOLD=0,TIMEZONE=America/Los_Angeles,CLIENT_IDENTIFIER=query_test/test_parquet_late_materialization.py::TestParquetLateMaterialization::()::test_parquet_late_materialization[protocol:beeswax|table_format:parquet/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;,ENABLED_RUNTIME_FILTER_TYPES=MIN_MAX,TEST_REPLAN=1
E Query Options (set by configuration and planner):
ABORT_ON_ERROR=1,EXEC_SINGLE_NODE_ROWS_THRESHOLD=0,RUNTIME_FILTER_WAIT_TIME_MS=5000,MT_DOP=0,DISABLE_CODEGEN_ROWS_THRESHOLD=0,TIMEZONE=America/Los_Angeles,CLIENT_IDENTIFIER=query_test/test_parquet_late_materialization.py::TestParquetLateMaterialization::()::test_parquet_late_materialization[protocol:beeswax|table_format:parquet/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;,ENABLED_RUNTIME_FILTER_TYPES=MIN_MAX,MINMAX_FILTER_THRESHOLD=0.5,MINMAX_FILTERING_LEVEL=PAGE,TEST_REPLAN=1
E Tables Queried: tpch_parquet.lineitem,tpch_parquet.orders
E Select Columns:
tpch_parquet.lineitem.l_comment,tpch_parquet.lineitem.l_commitdate,tpch_parquet.lineitem.l_discount,tpch_parquet.lineitem.l_extendedprice,tpch_parquet.lineitem.l_linenumber,tpch_parquet.lineitem.l_linestatus,tpch_parquet.lineitem.l_orderkey,tpch_parquet.lineitem.l_partkey,tpch_parquet.lineitem.l_quantity,tpch_parquet.lineitem.l_receiptdate,tpch_parquet.lineitem.l_returnflag,tpch_parquet.lineitem.l_shipdate,tpch_parquet.lineitem.l_shipinstruct,tpch_parquet.lineitem.l_shipmode,tpch_parquet.lineitem.l_suppkey,tpch_parquet.lineitem.l_tax,tpch_parquet.orders.o_clerk,tpch_parquet.orders.o_comment,tpch_parquet.orders.o_custkey,tpch_parquet.orders.o_orderdate,tpch_parquet.orders.o_orderkey,tpch_parquet.orders.o_orderpriority,tpch_parquet.orders.o_orderstatus,tpch_parquet.orders.o_shippriority,tpch_parquet.orders.o_totalprice
E Where Columns:
tpch_parquet.orders.o_orderdate,tpch_parquet.orders.o_totalprice
E Join Columns:
tpch_parquet.lineitem.l_orderkey,tpch_parquet.orders.o_orderkey
E Plan:
E ----------------
E Max Per-Host Resource Reservation: Memory=69.94MB Threads=5
E Per-Host Resource Estimates: Memory=1.61GB
E Analyzed query: SELECT * FROM tpch_parquet.lineitem l INNER JOIN
E tpch_parquet.orders o ON l.l_orderkey = o.o_orderkey WHERE o_orderdate =
E '1992-06-22' AND o_totalprice = CAST(153827.26 AS DECIMAL(8,2))
E
E F02:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
E | Per-Host Resources: mem-estimate=4.02MB mem-reservation=4.00MB
thread-reservation=1
E PLAN-ROOT SINK
E | output exprs: l.l_orderkey, l.l_partkey, l.l_suppkey, l.l_linenumber,
l.l_quantity, l.l_extendedprice, l.l_discount, l.l_tax, l.l_returnflag,
l.l_linestatus, l.l_shipdate, l.l_commitdate, l.l_receiptdate,
l.l_shipinstruct, l.l_shipmode, l.l_comment, o.o_orderkey, o.o_custkey,
o.o_orderstatus, o.o_totalprice, o.o_orderdate, o.o_orderpriority, o.o_clerk,
o.o_shippriority, o.o_comment
E | mem-estimate=4.00MB mem-reservation=4.00MB spill-buffer=2.00MB
thread-reservation=0
E |
E 04:EXCHANGE [UNPARTITIONED]
E | mem-estimate=16.00KB mem-reservation=0B thread-reservation=0
E | tuple-ids=0,1 row-size=402B cardinality=4
E | in pipelines: 00(GETNEXT)
E |
E F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
E Per-Host Resources: mem-estimate=1.25GB mem-reservation=41.94MB
thread-reservation=2
E 02:HASH JOIN [INNER JOIN, BROADCAST]
E | hash predicates: l.l_orderkey = o.o_orderkey
E | fk/pk conjuncts: l.l_orderkey = o.o_orderkey
E | mem-estimate=1.94MB mem-reservation=1.94MB spill-buffer=64.00KB
thread-reservation=0
E | tuple-ids=0,1 row-size=402B cardinality=4
E | in pipelines: 00(GETNEXT), 01(OPEN)
E |
E |--03:EXCHANGE [BROADCAST]
E | | mem-estimate=16.00KB mem-reservation=0B thread-reservation=0
E | | tuple-ids=1 row-size=171B cardinality=1
E | | in pipelines: 01(GETNEXT)
E | |
E | F01:PLAN FRAGMENT [RANDOM] hosts=2 instances=2
E | Per-Host Resources: mem-estimate=360.68MB mem-reservation=24.00MB
thread-reservation=2
E | 01:SCAN HDFS [tpch_parquet.orders o, RANDOM]
E | HDFS partitions=1/1 files=2 size=54.21MB
E | predicates: o_totalprice = CAST(153827.26 AS DECIMAL(8,2)),
o_orderdate = '1992-06-22'
E | stored statistics:
E | table: rows=1.50M size=54.21MB
E | columns: all
E | extrapolated-rows=disabled max-scan-range-rows=1.18M
E | parquet statistics predicates: o_totalprice = CAST(153827.26 AS
DECIMAL(8,2)), o_orderdate = '1992-06-22'
E | parquet dictionary predicates: o_totalprice = CAST(153827.26 AS
DECIMAL(8,2)), o_orderdate = '1992-06-22'
E | mem-estimate=360.00MB mem-reservation=24.00MB thread-reservation=1
E | tuple-ids=1 row-size=171B cardinality=1
E | in pipelines: 01(GETNEXT)
E |
E 00:SCAN HDFS [tpch_parquet.lineitem l, RANDOM]
E HDFS partitions=1/1 files=3 size=193.98MB
E stored statistics:
E table: rows=6.00M size=193.98MB
E columns: all
E extrapolated-rows=disabled max-scan-range-rows=2.14M
E mem-estimate=1.25GB mem-reservation=40.00MB thread-reservation=1
E tuple-ids=0 row-size=231B cardinality=6.00M
E in pipelines: 00(GETNEXT)
E ----------------
E Estimated Per-Host Mem: 1728301492
E Request Pool: default-pool
E Per Host Min Memory Reservation:
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(41.94 MB)
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(65.94 MB)
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(69.94 MB)
E Per Host Number of Fragment Instances:
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(1)
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(2)
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(3)
E Admission result: Admitted immediately
E Cluster Memory Admitted: 4.83 GB
E Executor Group: default
E Number of running queries in designated executor group when admitted: 2
E ExecSummary:
E Operator #Hosts #Inst Avg Time Max Time #Rows Est.
#Rows Peak Mem Est. Peak Mem Detail
E
-----------------------------------------------------------------------------------------------------------------------------------
E F02:ROOT 1 1 66.994us 66.994us
4.02 MB 4.00 MB
E 04:EXCHANGE 1 1 21.143us 21.143us 7
4 16.00 KB 16.00 KB UNPARTITIONED
E F00:EXCHANGE SENDER 3 3 62.542us 94.405us
1.45 KB 1.60 MB
E 02:HASH JOIN 3 3 71.838ms 78.429ms 7
4 3.25 MB 1.94 MB INNER JOIN, BROADCAST
E |--03:EXCHANGE 3 3 15.379us 17.153us 1
1 16.00 KB 16.00 KB BROADCAST
E | F01:EXCHANGE SENDER 2 2 79.154us 95.129us
274.00 B 699.66 KB
E | 01:SCAN HDFS 2 2 56.545ms 82.494ms 1
1 58.81 MB 360.00 MB tpch_parquet.orders o
E 00:SCAN HDFS 3 3 334.413ms 377.219ms 6.00M
6.00M 102.31 MB 1.25 GB tpch_parquet.lineitem l
E Errors:
E Query Compilation: 5.948ms
E - Metadata of all 2 tables cached: 306.727us (306.727us)
E - Analysis finished: 784.681us (477.954us)
E - Authorization finished (noop): 847.926us (63.245us)
E - Value transfer graph computed: 947.236us (99.310us)
E - Single node plan created: 1.751ms (804.277us)
E - Runtime filters computed: 1.851ms (99.812us)
E - Distributed plan created: 1.985ms (134.647us)
E - Metadata of all 2 tables cached: 3.589ms (1.604ms)
E - Analysis finished: 3.937ms (347.946us)
E - Value transfer graph computed: 4.022ms (84.630us)
E - Single node plan created: 4.622ms (599.503us)
E - Runtime filters computed: 4.670ms (48.314us)
E - Distributed plan created: 4.685ms (15.126us)
E - Planning finished: 5.948ms (1.262ms)
E Query Timeline: 699.594ms
E - Query submitted: 39.603us (39.603us)
E - Planning finished: 6.541ms (6.501ms)
E - Submit for admission: 6.700ms (159.291us)
E - Completed admission: 6.915ms (214.544us)
E - Ready to start on 3 backends: 7.055ms (140.234us)
E - All 3 execution backends (6 fragment instances) started: 11.594ms
(4.538ms)
E - Rows available: 228.188ms (216.593ms)
E - First row fetched: 695.390ms (467.202ms)
E - Last row fetched: 695.406ms (15.857us)
E - Released admission control resources: 698.187ms (2.781ms)
E - AdmissionControlTimeSinceLastUpdate: 89.000ms
E - ComputeScanRangeAssignmentTimer: 66.681us
E Frontend:
E Referenced Tables: tpch_parquet.lineitem, tpch_parquet.orders
E Original Table Versions: tpch_parquet.lineitem, 64866, 1739051164042,
Sat Feb 08 13:46:04 PST 2025
E tpch_parquet.orders, 49695, 1739043890578, Sat Feb 08 11:44:50 PST 2025
E - ExecutorGroupsConsidered: 2 (2)
E Executor group 1 (small):
E Verdict: not enough per-host memory
E - MemoryAsk: 4.83 GB (5184904476)
E - MemoryMax: 192.00 MB (201326592)
E Executor group 2 (large):
E Verdict: Match
E - MemoryAsk: 4.83 GB (5184904476)
E - MemoryMax: 8589934592.00 GB (9223372036854775807)
E PlannerInfo:
E PlannerType: OriginalPlanner
E ImpalaServer:
E - ClientFetchWaitTimeStats: (Avg: 36.451ms ; Min: 36.451ms ; Max:
36.451ms ; Number of samples: 1)
E - GetInFlightProfileTimeStats: 0.000ns (Number of samples: 0)
E - ClientFetchLockWaitTimer: 167.000ns
E - ClientFetchWaitTimer: 36.451ms
E - NumRowsFetched: 7 (7)
E - NumRowsFetchedFromCache: 0 (0)
E - RPCCount: 10 (10)
E - RPCReadTimer: 15.000us
E - RPCWriteTimer: 171.000us
E - RowMaterializationRate: 16.00 /sec
E - RowMaterializationTimer: 433.733ms
E Execution Profile e84f6c5546558a00:1f33e21500000000:(Total: 654.871ms,
non-child: 0.000ns, % non-child: 0.00%)
E Number of filters: 0
E Filter routing table:
E ID Src. Node Tgt. Node(s) Target type Partition filter Pending
(Expected) First arrived Completed Enabled Bloom Size Est fpp Min value
Max value In-list size
E
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
E Backend startup latencies: Count: 3, sum: 6ms, min / max: 1ms / 4ms,
25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 4ms, 95th %-ile:
4ms, 99.9th %-ile: 4ms
E Slowest backend to start up:
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000
E Per Node Peak Memory Usage:
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(116.45 MB)
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(105.26 MB)
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(94.88 MB)
E Per Node Bytes Read:
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(111.86 MB)
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(80.70 MB)
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(55.82 MB)
E Per Node User Time:
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(706.147ms)
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(591.702ms)
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(479.486ms)
E Per Node System Time:
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(215.383ms)
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(146.823ms)
impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(124.292ms)
E - ExchangeScanRatio: 0.00
[....snipped for brevity...]
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
