IMPALA-6341, IMPALA-5917: Reduce mem-limit for start-impala-cluster.

We've observed empirically that giving Impala 80% of system memory
doesn't leave enough room for the minicluster and ASAN overhead, leading
to the OOM killer striking during test runs (sometimes). This commit
reduces the threshold to 70%.

This commit also reduces the memory usage of semi-joins-exhaustive.test
by roughly halving the number of records it deals with. This was
necessary for tests to pass on a machine with 32GB of RAM.

Testing: I've run the ASAN build (more) happily with this change.
I've run exhaustive tests on a 32GB machine.

Change-Id: Iabca7a95560bd27c2de2b0a147ee9a3c45199db7
Reviewed-by: Tim Armstrong <>
Tested-by: Impala Public Jenkins


Branch: refs/heads/2.x
Commit: f90e937aa1141dfc5e26cd3bb2526f945f6e62c3
Parents: 6f430bd
Author: Philip Zeyliger <>
Authored: Wed Feb 21 10:45:24 2018 -0800
Committer: Impala Public Jenkins <>
Committed: Tue Mar 13 01:52:24 2018 +0000

 bin/                               |  6 +++++-
 .../queries/QueryTest/semi-joins-exhaustive.test          | 10 ++++++++--
 2 files changed, 13 insertions(+), 3 deletions(-)
diff --git a/bin/ b/bin/
index a4326f5..bfe71d5 100755
--- a/bin/
+++ b/bin/
@@ -220,7 +220,11 @@ def start_impalad_instances(cluster_size, 
num_coordinators, use_exclusive_coordi
   # mini-cluster with 3 impalads that means 240%. Since having an impalad be 
OOM killed
   # is very annoying, the mem limit will be reduced. This can be overridden 
using the
   # --impalad_args flag. virtual_memory().total returns the total physical 
-  mem_limit = int(0.8 * psutil.virtual_memory().total / cluster_size)
+  # The exact ratio to use is somewhat arbitrary. Peak memory usage during
+  # tests depends on the concurrency of parallel tests as well as their 
+  # At a ratio of 0.8, on 8-core, 68GB machines, ASAN builds can trigger the 
+  # killer, so this ratio is currently set to 0.7.
+  mem_limit = int(0.7 * psutil.virtual_memory().total / cluster_size)
   delay_list = []
   if options.catalog_init_delays != "":
diff --git 
index da1cbd2..a73b0ec 100644
@@ -1,12 +1,18 @@
 ---- QUERY
 # Regression test for IMPALA-2256. Join whose right side has very high
-# cardinality (121M) and zero materialized slots.
+# cardinality (60M) and zero materialized slots.
+# Because the right side of the join here is always
+# the same key, this query can run out of memory and fail to spill; see
+# IMPALA-4857. The cardinality (60M) is chosen so that the test
+# runs when impalad has a 7.8GB memlimit. (The peak memory usage
+# of the relevant fragment is 3.6GB when tested.)
 SELECT straight_join
 COUNT(*) FROM alltypesagg t1
 WHERE t1.int_col IN (
  SELECT 1 FROM alltypesagg t1
- CROSS JOIN alltypesagg t2)
+ CROSS JOIN alltypesagg t2
+ WHERE t1.int_col < 500)
 ---- TYPES

Reply via email to