IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster. We've observed empirically that giving Impala 80% of system memory doesn't leave enough room for the minicluster and ASAN overhead, leading to the OOM killer striking during test runs (sometimes). This commit reduces the threshold to 70%.
This commit also reduces the memory usage of semi-joins-exhaustive.test by roughly halving the number of records it deals with. This was necessary for tests to pass on a machine with 32GB of RAM. Testing: I've run the ASAN build (more) happily with this change. I've run exhaustive tests on a 32GB machine. Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7 Reviewed-on: http://gerrit.cloudera.org:8080/9395 Reviewed-by: Tim Armstrong <tarmstr...@cloudera.com> Tested-by: Impala Public Jenkins Project: http://git-wip-us.apache.org/repos/asf/impala/repo Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/f90e937a Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/f90e937a Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/f90e937a Branch: refs/heads/2.x Commit: f90e937aa1141dfc5e26cd3bb2526f945f6e62c3 Parents: 6f430bd Author: Philip Zeyliger <phi...@cloudera.com> Authored: Wed Feb 21 10:45:24 2018 -0800 Committer: Impala Public Jenkins <impala-public-jenk...@gerrit.cloudera.org> Committed: Tue Mar 13 01:52:24 2018 +0000 ---------------------------------------------------------------------- bin/start-impala-cluster.py | 6 +++++- .../queries/QueryTest/semi-joins-exhaustive.test | 10 ++++++++-- 2 files changed, 13 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/impala/blob/f90e937a/bin/start-impala-cluster.py ---------------------------------------------------------------------- diff --git a/bin/start-impala-cluster.py b/bin/start-impala-cluster.py index a4326f5..bfe71d5 100755 --- a/bin/start-impala-cluster.py +++ b/bin/start-impala-cluster.py @@ -220,7 +220,11 @@ def start_impalad_instances(cluster_size, num_coordinators, use_exclusive_coordi # mini-cluster with 3 impalads that means 240%. Since having an impalad be OOM killed # is very annoying, the mem limit will be reduced. This can be overridden using the # --impalad_args flag. virtual_memory().total returns the total physical memory. - mem_limit = int(0.8 * psutil.virtual_memory().total / cluster_size) + # The exact ratio to use is somewhat arbitrary. Peak memory usage during + # tests depends on the concurrency of parallel tests as well as their ordering. + # At a ratio of 0.8, on 8-core, 68GB machines, ASAN builds can trigger the OOM + # killer, so this ratio is currently set to 0.7. + mem_limit = int(0.7 * psutil.virtual_memory().total / cluster_size) delay_list = [] if options.catalog_init_delays != "": http://git-wip-us.apache.org/repos/asf/impala/blob/f90e937a/testdata/workloads/functional-query/queries/QueryTest/semi-joins-exhaustive.test ---------------------------------------------------------------------- diff --git a/testdata/workloads/functional-query/queries/QueryTest/semi-joins-exhaustive.test b/testdata/workloads/functional-query/queries/QueryTest/semi-joins-exhaustive.test index da1cbd2..a73b0ec 100644 --- a/testdata/workloads/functional-query/queries/QueryTest/semi-joins-exhaustive.test +++ b/testdata/workloads/functional-query/queries/QueryTest/semi-joins-exhaustive.test @@ -1,12 +1,18 @@ ==== ---- QUERY # Regression test for IMPALA-2256. Join whose right side has very high -# cardinality (121M) and zero materialized slots. +# cardinality (60M) and zero materialized slots. +# Because the right side of the join here is always +# the same key, this query can run out of memory and fail to spill; see +# IMPALA-4857. The cardinality (60M) is chosen so that the test +# runs when impalad has a 7.8GB memlimit. (The peak memory usage +# of the relevant fragment is 3.6GB when tested.) SELECT straight_join COUNT(*) FROM alltypesagg t1 WHERE t1.int_col IN ( SELECT 1 FROM alltypesagg t1 - CROSS JOIN alltypesagg t2) + CROSS JOIN alltypesagg t2 + WHERE t1.int_col < 500) ---- RESULTS 10 ---- TYPES
