Todd Lipcon has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10584
Change subject: IMPALA-7077. Add a configuration for the max number of partitions to load ...................................................................... IMPALA-7077. Add a configuration for the max number of partitions to load This adds a new flag --max_partitions_per_table which prevents catalogd from attempting to load any table with more than some maximum number of partitions. If set to 0 (the default) then no limit is enforced. The purpose of this change is to act as a "guard rail" for users in clusters where there may exist some tables with many hundreds of thousands of partitions that are known to cause catalogd/impalad OOMs. The behavior of this patch is slightly suboptimal in that, if the addition of partitions during a DML pushes a table over the limit, those partitions will be created but then the table will no longer load following the change. However, given that this is meant to be a 'last defense' safeguard, I didn't think it was worth the complexity to try to somehow limit the addition of partitions during a DML. Change-Id: Ife99a97a891ed14675303ea472abb2932a72cb51 --- M be/src/catalog/catalog.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java A testdata/workloads/functional-query/queries/QueryTest/max-partitions-to-load.test A tests/custom_cluster/test_max_partitions.py 12 files changed, 172 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/10584/1 -- To view, visit http://gerrit.cloudera.org:8080/10584 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ife99a97a891ed14675303ea472abb2932a72cb51 Gerrit-Change-Number: 10584 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon <t...@apache.org>
