Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16081
Change subject: IMPALA-7533: Cache partitions across table versions in LocalCatalog ...................................................................... IMPALA-7533: Cache partitions across table versions in LocalCatalog In LocalCatlaog cache, partition metadata is cached with a composed key of table name, table version and partition id. Whenever the table version bumps, e.g. due to comments being changed, all the cached partition values will be unreachable. Following queries have to reload the partitions and cache them with the new table version. Actually, the partition id is an unique id across the whole catalog. It's sufficient to identify the partition. However, there are no partition level invalidations if the partition is modified in-place in catalogd. So we have to include the table version and depend on it. After IMPALA-9778, there are no in-place modifications on partition metadata. We can safely reuse partition meta across table versions in LocalCatalog cache. This patch removes the table name and version in the partition cache key. So metadata of unchanged partitions can be reused when table version bumps. Tests: - Add tests in test_local_catalog.py to verify the partition metadata is reused based on profile metrics. Change-Id: I512f735b596bc51d553e6d395d108f49727619ed --- M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java M tests/custom_cluster/test_local_catalog.py 4 files changed, 72 insertions(+), 37 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/16081/1 -- To view, visit http://gerrit.cloudera.org:8080/16081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I512f735b596bc51d553e6d395d108f49727619ed Gerrit-Change-Number: 16081 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <[email protected]>
