Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17298
to look at the new patch set (#6).
Change subject: IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis
which modify tables and partitions.
......................................................................
IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis
which modify tables and partitions.
For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external tables,
we invalidate the table metadata from cache if HMS DDL apis like alter/drop
table/partition are accessed from catalogd's metastore server. As a result
of which, any subsequent get table request fetches the table from HMS and
loads it in cache. This ensures that any get_table/get_partition requests
after DDL operations on the same table return updated table metadata. This
behavior has a performance penalty since metadata loading in cache takes
time specially for large tables. The change is behind catalogd server's
flag: invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.
Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M
fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 511 insertions(+), 44 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/6
--
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 6
Gerrit-Owner: Sourabh Goyal <soura...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <soura...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>
