Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16308
Change subject: IMPALA-4364: Query option to refresh updated partitions ...................................................................... IMPALA-4364: Query option to refresh updated partitions This patch introduces a new boolean query option ENABLE_REFRESH_UPDATED_PARTITIONS. When this query option is set the refresh table command reloads the partitions which have been modified in HMS in addition to adding [removing] the new [removed] partitions. In order to do this the refresh table command needs to fetch all the partitions instead of the just the partition names which can cause the performance of refresh table to degrade when the query option is set. However, for certain use-cases there is no way currently the refresh table doesn't detect changed partitions. For instance, if certain partition locations have been changed, a refresh table will not update those partitions. Testing: 1. Added a new test which sets the query option and makes sure that the updated partitions from hive are reloaded after refresh table command. 2. [In-progress] Running exhaustive tests with the patch. Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af --- M be/src/service/client-request-state.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/CatalogService.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/View.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/metadata/test_reset_metadata.py 20 files changed, 443 insertions(+), 125 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/16308/1 -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
