This is an automated email from the ASF dual-hosted git repository. tmarshall pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/impala.git
commit 801efdcf097e82c79a077141c753d6d4d44d83e8 Author: Riza Suminto <[email protected]> AuthorDate: Fri Feb 26 07:39:17 2021 -0800 IMPALA-10492: Lower default MAX_CNF_EXPRS query option MAX_CNF_EXPRS was set to unlimited by default. The CNF rewrite can lead to significant frontend memory usage and eventually OutOfMemory for a complex query that contain many predicates. We need to lower the default value to avoid this memory problem while maintaining performance for our TPC-DS and TPC-H workloads. We investigate the maximum number of CNF expressions in TPC-DS and TPC-H by printing out the final value of 'numCnfExprs_' from ConvertToCNFRule.java to the query profile. We found 5 queries that applies CNF rewrite rules as follow: | Query | numCnfExprs_ | |-----------+--------------| | TPCDS-Q13 | 168 | | TPCDS-Q85 | 100 | | TPCDS-Q48 | 34 | | TPCH-Q19 | 124 | | TPCH-Q7 | 3 | This patch lower the default value from unlimited to 200 based on the result above. Testing: - Manually verify that MAX_CNF_EXPRS 200 is enough for our TPC-DS and TPC-H worloads. - Pass core tests. Change-Id: I7ca3d0e094ac01c24a046c25d6a1b56bf134faa8 Reviewed-on: http://gerrit.cloudera.org:8080/17132 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- common/thrift/ImpalaInternalService.thrift | 2 +- common/thrift/ImpalaService.thrift | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/common/thrift/ImpalaInternalService.thrift b/common/thrift/ImpalaInternalService.thrift index fb5e628..2983d12 100644 --- a/common/thrift/ImpalaInternalService.thrift +++ b/common/thrift/ImpalaInternalService.thrift @@ -417,7 +417,7 @@ struct TQueryOptions { 100: optional bool enable_cnf_rewrites = true; // See comment in ImpalaService.thrift - 101: optional i32 max_cnf_exprs = 0; + 101: optional i32 max_cnf_exprs = 200; // See comment in ImpalaService.thrift 102: optional i64 kudu_snapshot_read_timestamp_micros = 0; diff --git a/common/thrift/ImpalaService.thrift b/common/thrift/ImpalaService.thrift index 66a8335..0a48f80 100644 --- a/common/thrift/ImpalaService.thrift +++ b/common/thrift/ImpalaService.thrift @@ -514,7 +514,7 @@ enum TImpalaQueryOptions { // The max number of conjunctive normal form (CNF) exprs to create when converting // a disjunctive expression to CNF. Each AND counts as 1 expression. A value of - // -1 or 0 means no limit. Default is 0 (unlimited). + // -1 or 0 means no limit. Default is 200. MAX_CNF_EXPRS = 100 // Set the timestamp for Kudu snapshot reads in Unix time micros. Only valid if
