Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15312
Change subject: IMPALA-9363: Add support for skipping given table types ...................................................................... IMPALA-9363: Add support for skipping given table types This patch allows one to provide Impala with a list of blacklisted table types when Impala is started so that the tables of the types on the list would not be loaded when CatalogServiceCatalog is retrieving the metadata from the Hive metastore. The list of comma-separated blacklisted table types is passed to Impala via '--catalogd_args=--blacklisted_table_types=<list_of_blacklisted_types>'. Five table types are supported, namely, 'hdfs', 'hbase', 'view', 'data_source', and 'kudu'. Current limitations: This patch does not deal with the case in which a user would like to blacklist the views created on top of the specified table types. For instance, even though a user puts 'kudu' on the list of blacklisted table types, the metadata of a view created on top of a Kudu table such as 'functional_kudu.alltypesagg' would still be loaded. More importantly, the costs of invalidating the metadata after applying this patch is prohibitively high. A primitive investigation shows that it takes around 173 seconds to invalidate the metadata of a database consisting of 40,000 tables, whereas it only takes 0.8 second before the patch is applied. Testing: - Added an E2E test in test_blacklisted_dbs_and_tables.py. Change-Id: Ib2e2850f6d642417aec1e3cc5c6439b29bc0d1da --- M be/src/catalog/catalog.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java M tests/custom_cluster/test_blacklisted_dbs_and_tables.py 7 files changed, 145 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/15312/1 -- To view, visit http://gerrit.cloudera.org:8080/15312 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib2e2850f6d642417aec1e3cc5c6439b29bc0d1da Gerrit-Change-Number: 15312 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao <[email protected]> Gerrit-Reviewer: Fang-Yu Rao <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]>
