Yu-Wen Lai created IMPALA-10801:
-----------------------------------
Summary: Check the latest compaction Id before serving request
Key: IMPALA-10801
URL: https://issues.apache.org/jira/browse/IMPALA-10801
Project: IMPALA
Issue Type: Improvement
Components: Catalog
Reporter: Yu-Wen Lai
Assignee: Yu-Wen Lai
Cache compaction Id for a given table/file-metadata in CatalogD.
Whenever there is a read request to CatalogD, get the latest compaction event
Id from HMS, compare it with what is cached in CatalogD, and based on that
decide whether to serve the data from cache or to refresh it from the
filesystem. This can avoid notification based cache invalidation.
Also, since there will be an open txn for the current long running query which
is being served from CatalogD, we can be sure that current file-metadata being
served is not already deleted by the cleaner.
This proposal will use a new HMS API
(https://issues.apache.org/jira/browse/HIVE-24828) to get the latest compaction
id for a table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
