Vihang Karajgaonkar has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16081 )
Change subject: IMPALA-7533: Cache partitions across table versions in LocalCatalog ...................................................................... IMPALA-7533: Cache partitions across table versions in LocalCatalog In LocalCatlaog cache, partition metadata is cached with a composed key of table name, table version and partition id. Whenever the table version bumps, e.g. due to comments being changed, all the cached partition values will be unreachable. Following queries have to reload the partitions and cache them with the new table version. Actually, the partition id is an unique id across the whole catalog. It's sufficient to identify the partition. However, there are no partition level invalidations if the partition is modified in-place in catalogd. So we have to include the table version and depend on it. After IMPALA-9778, there are no in-place modifications on partition metadata. We can safely reuse partition meta across table versions in LocalCatalog cache. This patch removes the table name and version in the partition cache key. So metadata of unchanged partitions can be reused when table version bumps. Tests: - Add tests in test_local_catalog.py to verify the partition metadata is reused based on profile metrics. Change-Id: I512f735b596bc51d553e6d395d108f49727619ed Reviewed-on: http://gerrit.cloudera.org:8080/16081 Tested-by: Impala Public Jenkins <[email protected]> Reviewed-by: Vihang Karajgaonkar <[email protected]> --- M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java M tests/custom_cluster/test_local_catalog.py 4 files changed, 72 insertions(+), 37 deletions(-) Approvals: Impala Public Jenkins: Verified Vihang Karajgaonkar: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/16081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I512f735b596bc51d553e6d395d108f49727619ed Gerrit-Change-Number: 16081 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Anurag Mantripragada <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]>
