Quanlong Huang created IMPALA-14548:
---------------------------------------
Summary: Table property to configure staleness of metadata
Key: IMPALA-14548
URL: https://issues.apache.org/jira/browse/IMPALA-14548
Project: IMPALA
Issue Type: New Feature
Components: Catalog
Reporter: Quanlong Huang
For tables that are rarely used in Impala but have frequent external updates,
it's a waste to keep reloading their metadata for each HMS notification events,
e.g. if the table is only used once a hour but has external updates every
minutes.
Consider to add a table property (e.g. metadataCacheMaxStaleAge) to tell
catalogd that it's OK to use the current metadata (i.e. not consider as stale)
if the HMS events are within a period after the last reload.
For instance, if a table has INSERTs every minute from Hive/SparkSQL,
"metadataCacheMaxStaleAge"="10m" means catalogd will roughly reload table
metadata every 10 minutes.
The current behavior matches to "metadataCacheMaxStaleAge"="0".
In long term, we might be able to let catalogd deduce a good period to be used
for a table based on table usages reported from coordinators.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
