This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
The following commit(s) were added to refs/heads/master by this push:
new 4290c5e IMPALA-9355:
TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory
limit
4290c5e is described below
commit 4290c5e2971b056f5249716655ef2f704465f383
Author: Qifan Chen <[email protected]>
AuthorDate: Fri Jun 11 14:15:36 2021 -0400
IMPALA-9355: TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't
hit the memory limit
This patch restores the previous testing enviroment in which exchange
node 5 in the test query shown below reliablly reports memory
exceeding limit error.
set mem_limit=171520k;
set num_scanner_threads=1;
select *
from tpch_parquet.lineitem l1
join tpch_parquet.lineitem l2 on l1.l_orderkey = l2.l_orderkey and
l1.l_partkey = l2.l_partkey and l1.l_suppkey = l2.l_suppkey
and l1.l_linenumber = l2.l_linenumber
order by l1.l_orderkey desc, l1.l_partkey, l1.l_suppkey, l1.l_linenumber
limit 5
In the patch, the memory limit for the entire query is lowered to 164Mb,
less than 171520k, the minimum for the query to be accepted after
the result spooling is enabled by default. This is achieved by
disabling result spooling by setting spool_query_results to false.
Testing:
1). Ran exchange-mem-scaling.test
Change-Id: Ia4ad4508028645b67de419cfdfa2327d2847cfc4
Reviewed-on: http://gerrit.cloudera.org:8080/17586
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
---
.../functional-query/queries/QueryTest/exchange-mem-scaling.test | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git
a/testdata/workloads/functional-query/queries/QueryTest/exchange-mem-scaling.test
b/testdata/workloads/functional-query/queries/QueryTest/exchange-mem-scaling.test
index d80cd8f..10c7a95 100644
---
a/testdata/workloads/functional-query/queries/QueryTest/exchange-mem-scaling.test
+++
b/testdata/workloads/functional-query/queries/QueryTest/exchange-mem-scaling.test
@@ -4,7 +4,13 @@
# it hits the memory limit in the exchange node when allocating receiver-side
# buffers. It's also possible but less likely that this will hit a memory limit
# in the scan nodes.
-set mem_limit=171520k;
+# IMPALA-9355: An exchange node does not set a memory limit for buffer pool. To
+# hit the memory limit exceeding error reliably, the memory limit for the
entire query
+# has to be set to a value less than 171520k (167.5Mb) which is the minimum
for the query
+# to be accepted after the result spooling is enabled by default. Disable
result spooling
+# here by setting spool_query_results to false, and set the query memory limit
to 164Mb.
+set mem_limit=164m;
+set spool_query_results=false;
set num_scanner_threads=1;
select *
from tpch_parquet.lineitem l1
