[
https://issues.apache.org/jira/browse/IMPALA-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riza Suminto resolved IMPALA-13752.
-----------------------------------
Fix Version/s: Impala 4.6.0
Target Version: Impala 4.6.0
Resolution: Fixed
> test_parquet_late_materialization fails on runtime filter not published in
> exhaustive builds
> --------------------------------------------------------------------------------------------
>
> Key: IMPALA-13752
> URL: https://issues.apache.org/jira/browse/IMPALA-13752
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.6.0
> Reporter: Laszlo Gaal
> Assignee: Riza Suminto
> Priority: Blocker
> Labels: broken-build
> Fix For: Impala 4.6.0
>
>
> EE test test_parquet_late_materialization complains about not finding "1 of 1
> Runtime Filter Published" in the profile.
> Start of the error backtrace:
> {code}
> query_test/test_parquet_late_materialization.py:38: in
> test_parquet_late_materialization
> self.run_test_case('QueryTest/parquet-late-materialization', vector)
> common/impala_test_suite.py:908: in run_test_case
> update_section=pytest.config.option.update_results)
> common/test_result_verifier.py:699: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*1 of 1 Runtime Filter Published.*
> E
> E ACTUAL PROFILE:
> E Query (id=e84f6c5546558a00:1f33e21500000000):
> E DEBUG MODE WARNING: Query profile created while running a DEBUG build
> of Impala. Use RELEASE builds to measure query performance.
> E Summary:
> E Session ID: 1240823cf745e766:917a175b542d49a7
> E Session Type: BEESWAX
> E Start Time: 2025-02-08 15:03:03.540779000
> E End Time: 2025-02-08 15:03:04.238968000
> E Duration: 698.189ms (698189 us)
> E Query Type: QUERY
> E Query State: FINISHED
> E Impala Query State: FINISHED
> E Query Status: OK
> E Impala Version: impalad version 4.5.0-SNAPSHOT DEBUG (build
> 856741b1e8371b983da10f23a166f19aac71f1f8)
> E User: jenkins
> E Connected User: jenkins
> E Delegated User:
> E Network Address: 127.0.0.1:38404
> E Default Db: functional_parquet
> E Sql Statement: select * from tpch_parquet.lineitem l
> E join tpch_parquet.orders o on l.l_orderkey = o.o_orderkey
> E where o_orderdate='1992-06-22' and o_totalprice = 153827.26
> E Coordinator:
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000
> E Query Options (set by configuration):
> ABORT_ON_ERROR=1,EXEC_SINGLE_NODE_ROWS_THRESHOLD=0,RUNTIME_FILTER_WAIT_TIME_MS=5000,DISABLE_CODEGEN_ROWS_THRESHOLD=0,TIMEZONE=America/Los_Angeles,CLIENT_IDENTIFIER=query_test/test_parquet_late_materialization.py::TestParquetLateMaterialization::()::test_parquet_late_materialization[protocol:beeswax|table_format:parquet/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;,ENABLED_RUNTIME_FILTER_TYPES=MIN_MAX,TEST_REPLAN=1
> E Query Options (set by configuration and planner):
> ABORT_ON_ERROR=1,EXEC_SINGLE_NODE_ROWS_THRESHOLD=0,RUNTIME_FILTER_WAIT_TIME_MS=5000,MT_DOP=0,DISABLE_CODEGEN_ROWS_THRESHOLD=0,TIMEZONE=America/Los_Angeles,CLIENT_IDENTIFIER=query_test/test_parquet_late_materialization.py::TestParquetLateMaterialization::()::test_parquet_late_materialization[protocol:beeswax|table_format:parquet/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;,ENABLED_RUNTIME_FILTER_TYPES=MIN_MAX,MINMAX_FILTER_THRESHOLD=0.5,MINMAX_FILTERING_LEVEL=PAGE,TEST_REPLAN=1
> E Tables Queried: tpch_parquet.lineitem,tpch_parquet.orders
> E Select Columns:
> tpch_parquet.lineitem.l_comment,tpch_parquet.lineitem.l_commitdate,tpch_parquet.lineitem.l_discount,tpch_parquet.lineitem.l_extendedprice,tpch_parquet.lineitem.l_linenumber,tpch_parquet.lineitem.l_linestatus,tpch_parquet.lineitem.l_orderkey,tpch_parquet.lineitem.l_partkey,tpch_parquet.lineitem.l_quantity,tpch_parquet.lineitem.l_receiptdate,tpch_parquet.lineitem.l_returnflag,tpch_parquet.lineitem.l_shipdate,tpch_parquet.lineitem.l_shipinstruct,tpch_parquet.lineitem.l_shipmode,tpch_parquet.lineitem.l_suppkey,tpch_parquet.lineitem.l_tax,tpch_parquet.orders.o_clerk,tpch_parquet.orders.o_comment,tpch_parquet.orders.o_custkey,tpch_parquet.orders.o_orderdate,tpch_parquet.orders.o_orderkey,tpch_parquet.orders.o_orderpriority,tpch_parquet.orders.o_orderstatus,tpch_parquet.orders.o_shippriority,tpch_parquet.orders.o_totalprice
> E Where Columns:
> tpch_parquet.orders.o_orderdate,tpch_parquet.orders.o_totalprice
> E Join Columns:
> tpch_parquet.lineitem.l_orderkey,tpch_parquet.orders.o_orderkey
> E Plan:
> E ----------------
> E Max Per-Host Resource Reservation: Memory=69.94MB Threads=5
> E Per-Host Resource Estimates: Memory=1.61GB
> E Analyzed query: SELECT * FROM tpch_parquet.lineitem l INNER JOIN
> E tpch_parquet.orders o ON l.l_orderkey = o.o_orderkey WHERE o_orderdate =
> E '1992-06-22' AND o_totalprice = CAST(153827.26 AS DECIMAL(8,2))
> E
> E F02:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> E | Per-Host Resources: mem-estimate=4.02MB mem-reservation=4.00MB
> thread-reservation=1
> E PLAN-ROOT SINK
> E | output exprs: l.l_orderkey, l.l_partkey, l.l_suppkey, l.l_linenumber,
> l.l_quantity, l.l_extendedprice, l.l_discount, l.l_tax, l.l_returnflag,
> l.l_linestatus, l.l_shipdate, l.l_commitdate, l.l_receiptdate,
> l.l_shipinstruct, l.l_shipmode, l.l_comment, o.o_orderkey, o.o_custkey,
> o.o_orderstatus, o.o_totalprice, o.o_orderdate, o.o_orderpriority, o.o_clerk,
> o.o_shippriority, o.o_comment
> E | mem-estimate=4.00MB mem-reservation=4.00MB spill-buffer=2.00MB
> thread-reservation=0
> E |
> E 04:EXCHANGE [UNPARTITIONED]
> E | mem-estimate=16.00KB mem-reservation=0B thread-reservation=0
> E | tuple-ids=0,1 row-size=402B cardinality=4
> E | in pipelines: 00(GETNEXT)
> E |
> E F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
> E Per-Host Resources: mem-estimate=1.25GB mem-reservation=41.94MB
> thread-reservation=2
> E 02:HASH JOIN [INNER JOIN, BROADCAST]
> E | hash predicates: l.l_orderkey = o.o_orderkey
> E | fk/pk conjuncts: l.l_orderkey = o.o_orderkey
> E | mem-estimate=1.94MB mem-reservation=1.94MB spill-buffer=64.00KB
> thread-reservation=0
> E | tuple-ids=0,1 row-size=402B cardinality=4
> E | in pipelines: 00(GETNEXT), 01(OPEN)
> E |
> E |--03:EXCHANGE [BROADCAST]
> E | | mem-estimate=16.00KB mem-reservation=0B thread-reservation=0
> E | | tuple-ids=1 row-size=171B cardinality=1
> E | | in pipelines: 01(GETNEXT)
> E | |
> E | F01:PLAN FRAGMENT [RANDOM] hosts=2 instances=2
> E | Per-Host Resources: mem-estimate=360.68MB mem-reservation=24.00MB
> thread-reservation=2
> E | 01:SCAN HDFS [tpch_parquet.orders o, RANDOM]
> E | HDFS partitions=1/1 files=2 size=54.21MB
> E | predicates: o_totalprice = CAST(153827.26 AS DECIMAL(8,2)),
> o_orderdate = '1992-06-22'
> E | stored statistics:
> E | table: rows=1.50M size=54.21MB
> E | columns: all
> E | extrapolated-rows=disabled max-scan-range-rows=1.18M
> E | parquet statistics predicates: o_totalprice = CAST(153827.26 AS
> DECIMAL(8,2)), o_orderdate = '1992-06-22'
> E | parquet dictionary predicates: o_totalprice = CAST(153827.26 AS
> DECIMAL(8,2)), o_orderdate = '1992-06-22'
> E | mem-estimate=360.00MB mem-reservation=24.00MB thread-reservation=1
> E | tuple-ids=1 row-size=171B cardinality=1
> E | in pipelines: 01(GETNEXT)
> E |
> E 00:SCAN HDFS [tpch_parquet.lineitem l, RANDOM]
> E HDFS partitions=1/1 files=3 size=193.98MB
> E stored statistics:
> E table: rows=6.00M size=193.98MB
> E columns: all
> E extrapolated-rows=disabled max-scan-range-rows=2.14M
> E mem-estimate=1.25GB mem-reservation=40.00MB thread-reservation=1
> E tuple-ids=0 row-size=231B cardinality=6.00M
> E in pipelines: 00(GETNEXT)
> E ----------------
> E Estimated Per-Host Mem: 1728301492
> E Request Pool: default-pool
> E Per Host Min Memory Reservation:
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(41.94
> MB)
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(65.94
> MB)
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(69.94 MB)
> E Per Host Number of Fragment Instances:
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(1)
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(2)
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(3)
> E Admission result: Admitted immediately
> E Cluster Memory Admitted: 4.83 GB
> E Executor Group: default
> E Number of running queries in designated executor group when admitted:
> 2
> E ExecSummary:
> E Operator #Hosts #Inst Avg Time Max Time #Rows Est.
> #Rows Peak Mem Est. Peak Mem Detail
> E
> -----------------------------------------------------------------------------------------------------------------------------------
> E F02:ROOT 1 1 66.994us 66.994us
> 4.02 MB 4.00 MB
> E 04:EXCHANGE 1 1 21.143us 21.143us 7
> 4 16.00 KB 16.00 KB UNPARTITIONED
> E F00:EXCHANGE SENDER 3 3 62.542us 94.405us
> 1.45 KB 1.60 MB
> E 02:HASH JOIN 3 3 71.838ms 78.429ms 7
> 4 3.25 MB 1.94 MB INNER JOIN, BROADCAST
> E |--03:EXCHANGE 3 3 15.379us 17.153us 1
> 1 16.00 KB 16.00 KB BROADCAST
> E | F01:EXCHANGE SENDER 2 2 79.154us 95.129us
> 274.00 B 699.66 KB
> E | 01:SCAN HDFS 2 2 56.545ms 82.494ms 1
> 1 58.81 MB 360.00 MB tpch_parquet.orders o
> E 00:SCAN HDFS 3 3 334.413ms 377.219ms 6.00M
> 6.00M 102.31 MB 1.25 GB tpch_parquet.lineitem l
> E Errors:
> E Query Compilation: 5.948ms
> E - Metadata of all 2 tables cached: 306.727us (306.727us)
> E - Analysis finished: 784.681us (477.954us)
> E - Authorization finished (noop): 847.926us (63.245us)
> E - Value transfer graph computed: 947.236us (99.310us)
> E - Single node plan created: 1.751ms (804.277us)
> E - Runtime filters computed: 1.851ms (99.812us)
> E - Distributed plan created: 1.985ms (134.647us)
> E - Metadata of all 2 tables cached: 3.589ms (1.604ms)
> E - Analysis finished: 3.937ms (347.946us)
> E - Value transfer graph computed: 4.022ms (84.630us)
> E - Single node plan created: 4.622ms (599.503us)
> E - Runtime filters computed: 4.670ms (48.314us)
> E - Distributed plan created: 4.685ms (15.126us)
> E - Planning finished: 5.948ms (1.262ms)
> E Query Timeline: 699.594ms
> E - Query submitted: 39.603us (39.603us)
> E - Planning finished: 6.541ms (6.501ms)
> E - Submit for admission: 6.700ms (159.291us)
> E - Completed admission: 6.915ms (214.544us)
> E - Ready to start on 3 backends: 7.055ms (140.234us)
> E - All 3 execution backends (6 fragment instances) started:
> 11.594ms (4.538ms)
> E - Rows available: 228.188ms (216.593ms)
> E - First row fetched: 695.390ms (467.202ms)
> E - Last row fetched: 695.406ms (15.857us)
> E - Released admission control resources: 698.187ms (2.781ms)
> E - AdmissionControlTimeSinceLastUpdate: 89.000ms
> E - ComputeScanRangeAssignmentTimer: 66.681us
> E Frontend:
> E Referenced Tables: tpch_parquet.lineitem, tpch_parquet.orders
> E Original Table Versions: tpch_parquet.lineitem, 64866,
> 1739051164042, Sat Feb 08 13:46:04 PST 2025
> E tpch_parquet.orders, 49695, 1739043890578, Sat Feb 08 11:44:50 PST 2025
> E - ExecutorGroupsConsidered: 2 (2)
> E Executor group 1 (small):
> E Verdict: not enough per-host memory
> E - MemoryAsk: 4.83 GB (5184904476)
> E - MemoryMax: 192.00 MB (201326592)
> E Executor group 2 (large):
> E Verdict: Match
> E - MemoryAsk: 4.83 GB (5184904476)
> E - MemoryMax: 8589934592.00 GB (9223372036854775807)
> E PlannerInfo:
> E PlannerType: OriginalPlanner
> E ImpalaServer:
> E - ClientFetchWaitTimeStats: (Avg: 36.451ms ; Min: 36.451ms ; Max:
> 36.451ms ; Number of samples: 1)
> E - GetInFlightProfileTimeStats: 0.000ns (Number of samples: 0)
> E - ClientFetchLockWaitTimer: 167.000ns
> E - ClientFetchWaitTimer: 36.451ms
> E - NumRowsFetched: 7 (7)
> E - NumRowsFetchedFromCache: 0 (0)
> E - RPCCount: 10 (10)
> E - RPCReadTimer: 15.000us
> E - RPCWriteTimer: 171.000us
> E - RowMaterializationRate: 16.00 /sec
> E - RowMaterializationTimer: 433.733ms
> E Execution Profile e84f6c5546558a00:1f33e21500000000:(Total: 654.871ms,
> non-child: 0.000ns, % non-child: 0.00%)
> E Number of filters: 0
> E Filter routing table:
> E ID Src. Node Tgt. Node(s) Target type Partition filter Pending
> (Expected) First arrived Completed Enabled Bloom Size Est fpp Min value
> Max value In-list size
> E
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> E Backend startup latencies: Count: 3, sum: 6ms, min / max: 1ms / 4ms,
> 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 4ms, 95th
> %-ile: 4ms, 99.9th %-ile: 4ms
> E Slowest backend to start up:
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000
> E Per Node Peak Memory Usage:
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(116.45
> MB)
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(105.26
> MB)
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(94.88 MB)
> E Per Node Bytes Read:
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(111.86
> MB)
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(80.70
> MB)
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(55.82 MB)
> E Per Node User Time:
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(706.147ms)
>
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(591.702ms)
>
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(479.486ms)
> E Per Node System Time:
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(215.383ms)
>
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(146.823ms)
>
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(124.292ms)
> E - ExchangeScanRatio: 0.00
> [....snipped for brevity...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
