[
https://issues.apache.org/jira/browse/IMPALA-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928688#comment-17928688
]
ASF subversion and git services commented on IMPALA-13752:
----------------------------------------------------------
Commit 9cb9bae84e8888203e4bdfd3e20ee3e61c6059fe in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=9cb9bae84 ]
IMPALA-13758: Use context manager in ImpalaTestSuite.change_database
ImpalaTestSuite.change_database is responsible to point impala client to
database under test. However, it left client pointing to that database
after the test without reverting them back to default database. This
patch does the reversal by changing ImpalaTestSuite.change_database to
use context manager.
This patch change the behavior of execute_query_using_client() and
execute_query_async_using_client(). They used to change database
according to the given vector parameter, but not anymore after this
patch. In practice, this behavior change does not affect many tests
because most queries going through these functions already use fully
qualified table name. Going forward, querying through function other
than run_test_case() should try to use fully qualified table name as
much as possible.
Retain behavior of ImpalaTestSuite._get_table_location() since there are
considerable number of tests relies on it (changing database when
called).
Removed unused test fixtures and fixed several flake8 issues in modified
test files.
Testing:
- Moved nested-types-subplan-single-node.test. This allows the test
framework to point to the right tpch_nested* database.
- Pass exhaustive test except IMPALA-13752 and IMPALA-13761. They will
be fixed in separate patch.
Change-Id: I75bec7403cc302728a630efe3f95e852a84594e2
Reviewed-on: http://gerrit.cloudera.org:8080/22487
Reviewed-by: Csaba Ringhofer <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> test_parquet_late_materialization fails on runtime filter not published in
> exhaustive builds
> --------------------------------------------------------------------------------------------
>
> Key: IMPALA-13752
> URL: https://issues.apache.org/jira/browse/IMPALA-13752
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.6.0
> Reporter: Laszlo Gaal
> Assignee: Riza Suminto
> Priority: Blocker
> Labels: broken-build
> Fix For: Impala 4.6.0
>
>
> EE test test_parquet_late_materialization complains about not finding "1 of 1
> Runtime Filter Published" in the profile.
> Start of the error backtrace:
> {code}
> query_test/test_parquet_late_materialization.py:38: in
> test_parquet_late_materialization
> self.run_test_case('QueryTest/parquet-late-materialization', vector)
> common/impala_test_suite.py:908: in run_test_case
> update_section=pytest.config.option.update_results)
> common/test_result_verifier.py:699: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*1 of 1 Runtime Filter Published.*
> E
> E ACTUAL PROFILE:
> E Query (id=e84f6c5546558a00:1f33e21500000000):
> E DEBUG MODE WARNING: Query profile created while running a DEBUG build
> of Impala. Use RELEASE builds to measure query performance.
> E Summary:
> E Session ID: 1240823cf745e766:917a175b542d49a7
> E Session Type: BEESWAX
> E Start Time: 2025-02-08 15:03:03.540779000
> E End Time: 2025-02-08 15:03:04.238968000
> E Duration: 698.189ms (698189 us)
> E Query Type: QUERY
> E Query State: FINISHED
> E Impala Query State: FINISHED
> E Query Status: OK
> E Impala Version: impalad version 4.5.0-SNAPSHOT DEBUG (build
> 856741b1e8371b983da10f23a166f19aac71f1f8)
> E User: jenkins
> E Connected User: jenkins
> E Delegated User:
> E Network Address: 127.0.0.1:38404
> E Default Db: functional_parquet
> E Sql Statement: select * from tpch_parquet.lineitem l
> E join tpch_parquet.orders o on l.l_orderkey = o.o_orderkey
> E where o_orderdate='1992-06-22' and o_totalprice = 153827.26
> E Coordinator:
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000
> E Query Options (set by configuration):
> ABORT_ON_ERROR=1,EXEC_SINGLE_NODE_ROWS_THRESHOLD=0,RUNTIME_FILTER_WAIT_TIME_MS=5000,DISABLE_CODEGEN_ROWS_THRESHOLD=0,TIMEZONE=America/Los_Angeles,CLIENT_IDENTIFIER=query_test/test_parquet_late_materialization.py::TestParquetLateMaterialization::()::test_parquet_late_materialization[protocol:beeswax|table_format:parquet/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;,ENABLED_RUNTIME_FILTER_TYPES=MIN_MAX,TEST_REPLAN=1
> E Query Options (set by configuration and planner):
> ABORT_ON_ERROR=1,EXEC_SINGLE_NODE_ROWS_THRESHOLD=0,RUNTIME_FILTER_WAIT_TIME_MS=5000,MT_DOP=0,DISABLE_CODEGEN_ROWS_THRESHOLD=0,TIMEZONE=America/Los_Angeles,CLIENT_IDENTIFIER=query_test/test_parquet_late_materialization.py::TestParquetLateMaterialization::()::test_parquet_late_materialization[protocol:beeswax|table_format:parquet/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;,ENABLED_RUNTIME_FILTER_TYPES=MIN_MAX,MINMAX_FILTER_THRESHOLD=0.5,MINMAX_FILTERING_LEVEL=PAGE,TEST_REPLAN=1
> E Tables Queried: tpch_parquet.lineitem,tpch_parquet.orders
> E Select Columns:
> tpch_parquet.lineitem.l_comment,tpch_parquet.lineitem.l_commitdate,tpch_parquet.lineitem.l_discount,tpch_parquet.lineitem.l_extendedprice,tpch_parquet.lineitem.l_linenumber,tpch_parquet.lineitem.l_linestatus,tpch_parquet.lineitem.l_orderkey,tpch_parquet.lineitem.l_partkey,tpch_parquet.lineitem.l_quantity,tpch_parquet.lineitem.l_receiptdate,tpch_parquet.lineitem.l_returnflag,tpch_parquet.lineitem.l_shipdate,tpch_parquet.lineitem.l_shipinstruct,tpch_parquet.lineitem.l_shipmode,tpch_parquet.lineitem.l_suppkey,tpch_parquet.lineitem.l_tax,tpch_parquet.orders.o_clerk,tpch_parquet.orders.o_comment,tpch_parquet.orders.o_custkey,tpch_parquet.orders.o_orderdate,tpch_parquet.orders.o_orderkey,tpch_parquet.orders.o_orderpriority,tpch_parquet.orders.o_orderstatus,tpch_parquet.orders.o_shippriority,tpch_parquet.orders.o_totalprice
> E Where Columns:
> tpch_parquet.orders.o_orderdate,tpch_parquet.orders.o_totalprice
> E Join Columns:
> tpch_parquet.lineitem.l_orderkey,tpch_parquet.orders.o_orderkey
> E Plan:
> E ----------------
> E Max Per-Host Resource Reservation: Memory=69.94MB Threads=5
> E Per-Host Resource Estimates: Memory=1.61GB
> E Analyzed query: SELECT * FROM tpch_parquet.lineitem l INNER JOIN
> E tpch_parquet.orders o ON l.l_orderkey = o.o_orderkey WHERE o_orderdate =
> E '1992-06-22' AND o_totalprice = CAST(153827.26 AS DECIMAL(8,2))
> E
> E F02:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> E | Per-Host Resources: mem-estimate=4.02MB mem-reservation=4.00MB
> thread-reservation=1
> E PLAN-ROOT SINK
> E | output exprs: l.l_orderkey, l.l_partkey, l.l_suppkey, l.l_linenumber,
> l.l_quantity, l.l_extendedprice, l.l_discount, l.l_tax, l.l_returnflag,
> l.l_linestatus, l.l_shipdate, l.l_commitdate, l.l_receiptdate,
> l.l_shipinstruct, l.l_shipmode, l.l_comment, o.o_orderkey, o.o_custkey,
> o.o_orderstatus, o.o_totalprice, o.o_orderdate, o.o_orderpriority, o.o_clerk,
> o.o_shippriority, o.o_comment
> E | mem-estimate=4.00MB mem-reservation=4.00MB spill-buffer=2.00MB
> thread-reservation=0
> E |
> E 04:EXCHANGE [UNPARTITIONED]
> E | mem-estimate=16.00KB mem-reservation=0B thread-reservation=0
> E | tuple-ids=0,1 row-size=402B cardinality=4
> E | in pipelines: 00(GETNEXT)
> E |
> E F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
> E Per-Host Resources: mem-estimate=1.25GB mem-reservation=41.94MB
> thread-reservation=2
> E 02:HASH JOIN [INNER JOIN, BROADCAST]
> E | hash predicates: l.l_orderkey = o.o_orderkey
> E | fk/pk conjuncts: l.l_orderkey = o.o_orderkey
> E | mem-estimate=1.94MB mem-reservation=1.94MB spill-buffer=64.00KB
> thread-reservation=0
> E | tuple-ids=0,1 row-size=402B cardinality=4
> E | in pipelines: 00(GETNEXT), 01(OPEN)
> E |
> E |--03:EXCHANGE [BROADCAST]
> E | | mem-estimate=16.00KB mem-reservation=0B thread-reservation=0
> E | | tuple-ids=1 row-size=171B cardinality=1
> E | | in pipelines: 01(GETNEXT)
> E | |
> E | F01:PLAN FRAGMENT [RANDOM] hosts=2 instances=2
> E | Per-Host Resources: mem-estimate=360.68MB mem-reservation=24.00MB
> thread-reservation=2
> E | 01:SCAN HDFS [tpch_parquet.orders o, RANDOM]
> E | HDFS partitions=1/1 files=2 size=54.21MB
> E | predicates: o_totalprice = CAST(153827.26 AS DECIMAL(8,2)),
> o_orderdate = '1992-06-22'
> E | stored statistics:
> E | table: rows=1.50M size=54.21MB
> E | columns: all
> E | extrapolated-rows=disabled max-scan-range-rows=1.18M
> E | parquet statistics predicates: o_totalprice = CAST(153827.26 AS
> DECIMAL(8,2)), o_orderdate = '1992-06-22'
> E | parquet dictionary predicates: o_totalprice = CAST(153827.26 AS
> DECIMAL(8,2)), o_orderdate = '1992-06-22'
> E | mem-estimate=360.00MB mem-reservation=24.00MB thread-reservation=1
> E | tuple-ids=1 row-size=171B cardinality=1
> E | in pipelines: 01(GETNEXT)
> E |
> E 00:SCAN HDFS [tpch_parquet.lineitem l, RANDOM]
> E HDFS partitions=1/1 files=3 size=193.98MB
> E stored statistics:
> E table: rows=6.00M size=193.98MB
> E columns: all
> E extrapolated-rows=disabled max-scan-range-rows=2.14M
> E mem-estimate=1.25GB mem-reservation=40.00MB thread-reservation=1
> E tuple-ids=0 row-size=231B cardinality=6.00M
> E in pipelines: 00(GETNEXT)
> E ----------------
> E Estimated Per-Host Mem: 1728301492
> E Request Pool: default-pool
> E Per Host Min Memory Reservation:
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(41.94
> MB)
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(65.94
> MB)
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(69.94 MB)
> E Per Host Number of Fragment Instances:
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(1)
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(2)
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(3)
> E Admission result: Admitted immediately
> E Cluster Memory Admitted: 4.83 GB
> E Executor Group: default
> E Number of running queries in designated executor group when admitted:
> 2
> E ExecSummary:
> E Operator #Hosts #Inst Avg Time Max Time #Rows Est.
> #Rows Peak Mem Est. Peak Mem Detail
> E
> -----------------------------------------------------------------------------------------------------------------------------------
> E F02:ROOT 1 1 66.994us 66.994us
> 4.02 MB 4.00 MB
> E 04:EXCHANGE 1 1 21.143us 21.143us 7
> 4 16.00 KB 16.00 KB UNPARTITIONED
> E F00:EXCHANGE SENDER 3 3 62.542us 94.405us
> 1.45 KB 1.60 MB
> E 02:HASH JOIN 3 3 71.838ms 78.429ms 7
> 4 3.25 MB 1.94 MB INNER JOIN, BROADCAST
> E |--03:EXCHANGE 3 3 15.379us 17.153us 1
> 1 16.00 KB 16.00 KB BROADCAST
> E | F01:EXCHANGE SENDER 2 2 79.154us 95.129us
> 274.00 B 699.66 KB
> E | 01:SCAN HDFS 2 2 56.545ms 82.494ms 1
> 1 58.81 MB 360.00 MB tpch_parquet.orders o
> E 00:SCAN HDFS 3 3 334.413ms 377.219ms 6.00M
> 6.00M 102.31 MB 1.25 GB tpch_parquet.lineitem l
> E Errors:
> E Query Compilation: 5.948ms
> E - Metadata of all 2 tables cached: 306.727us (306.727us)
> E - Analysis finished: 784.681us (477.954us)
> E - Authorization finished (noop): 847.926us (63.245us)
> E - Value transfer graph computed: 947.236us (99.310us)
> E - Single node plan created: 1.751ms (804.277us)
> E - Runtime filters computed: 1.851ms (99.812us)
> E - Distributed plan created: 1.985ms (134.647us)
> E - Metadata of all 2 tables cached: 3.589ms (1.604ms)
> E - Analysis finished: 3.937ms (347.946us)
> E - Value transfer graph computed: 4.022ms (84.630us)
> E - Single node plan created: 4.622ms (599.503us)
> E - Runtime filters computed: 4.670ms (48.314us)
> E - Distributed plan created: 4.685ms (15.126us)
> E - Planning finished: 5.948ms (1.262ms)
> E Query Timeline: 699.594ms
> E - Query submitted: 39.603us (39.603us)
> E - Planning finished: 6.541ms (6.501ms)
> E - Submit for admission: 6.700ms (159.291us)
> E - Completed admission: 6.915ms (214.544us)
> E - Ready to start on 3 backends: 7.055ms (140.234us)
> E - All 3 execution backends (6 fragment instances) started:
> 11.594ms (4.538ms)
> E - Rows available: 228.188ms (216.593ms)
> E - First row fetched: 695.390ms (467.202ms)
> E - Last row fetched: 695.406ms (15.857us)
> E - Released admission control resources: 698.187ms (2.781ms)
> E - AdmissionControlTimeSinceLastUpdate: 89.000ms
> E - ComputeScanRangeAssignmentTimer: 66.681us
> E Frontend:
> E Referenced Tables: tpch_parquet.lineitem, tpch_parquet.orders
> E Original Table Versions: tpch_parquet.lineitem, 64866,
> 1739051164042, Sat Feb 08 13:46:04 PST 2025
> E tpch_parquet.orders, 49695, 1739043890578, Sat Feb 08 11:44:50 PST 2025
> E - ExecutorGroupsConsidered: 2 (2)
> E Executor group 1 (small):
> E Verdict: not enough per-host memory
> E - MemoryAsk: 4.83 GB (5184904476)
> E - MemoryMax: 192.00 MB (201326592)
> E Executor group 2 (large):
> E Verdict: Match
> E - MemoryAsk: 4.83 GB (5184904476)
> E - MemoryMax: 8589934592.00 GB (9223372036854775807)
> E PlannerInfo:
> E PlannerType: OriginalPlanner
> E ImpalaServer:
> E - ClientFetchWaitTimeStats: (Avg: 36.451ms ; Min: 36.451ms ; Max:
> 36.451ms ; Number of samples: 1)
> E - GetInFlightProfileTimeStats: 0.000ns (Number of samples: 0)
> E - ClientFetchLockWaitTimer: 167.000ns
> E - ClientFetchWaitTimer: 36.451ms
> E - NumRowsFetched: 7 (7)
> E - NumRowsFetchedFromCache: 0 (0)
> E - RPCCount: 10 (10)
> E - RPCReadTimer: 15.000us
> E - RPCWriteTimer: 171.000us
> E - RowMaterializationRate: 16.00 /sec
> E - RowMaterializationTimer: 433.733ms
> E Execution Profile e84f6c5546558a00:1f33e21500000000:(Total: 654.871ms,
> non-child: 0.000ns, % non-child: 0.00%)
> E Number of filters: 0
> E Filter routing table:
> E ID Src. Node Tgt. Node(s) Target type Partition filter Pending
> (Expected) First arrived Completed Enabled Bloom Size Est fpp Min value
> Max value In-list size
> E
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> E Backend startup latencies: Count: 3, sum: 6ms, min / max: 1ms / 4ms,
> 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 4ms, 95th
> %-ile: 4ms, 99.9th %-ile: 4ms
> E Slowest backend to start up:
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000
> E Per Node Peak Memory Usage:
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(116.45
> MB)
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(105.26
> MB)
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(94.88 MB)
> E Per Node Bytes Read:
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(111.86
> MB)
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(80.70
> MB)
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(55.82 MB)
> E Per Node User Time:
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(706.147ms)
>
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(591.702ms)
>
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(479.486ms)
> E Per Node System Time:
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27000(215.383ms)
>
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27001(146.823ms)
>
> impala-ec2-redhat86-m6i-4xlarge-ondemand-059c.vpc.cloudera.com:27002(124.292ms)
> E - ExchangeScanRatio: 0.00
> [....snipped for brevity...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
