Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16308 )
Change subject: IMPALA-4364: Query option to refresh updated HMS partitions ...................................................................... IMPALA-4364: Query option to refresh updated HMS partitions This patch introduces a new boolean query option REFRESH_UPDATED_HMS_PARTITIONS. When this query option is set the refresh table command reloads the partitions which have been modified in HMS in addition to adding [removing] the new [removed] partitions. In order to do this the refresh table command needs to fetch all the partitions instead of the just the partition names which can cause the performance of refresh table to degrade when the query option is set. However for certain use-cases currently there is no way to detect changed partitions using refresh table command. For instance, if certain partition locations have been changed, a refresh table will not update those partitions. Testing: 1. Added a new test which sets the query option and makes sure that the updated partitions from hive are reloaded after refresh table command. 2. Ran exhaustive tests with the patch. Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Reviewed-on: http://gerrit.cloudera.org:8080/16308 Reviewed-by: Quanlong Huang <huangquanl...@gmail.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/CatalogService.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/metadata/test_reset_metadata.py 11 files changed, 464 insertions(+), 106 deletions(-) Approvals: Quanlong Huang: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>