Quanlong Huang created IMPALA-14856:
---------------------------------------
Summary: Unneccessay invalidation for just loaded table meta
Key: IMPALA-14856
URL: https://issues.apache.org/jira/browse/IMPALA-14856
Project: IMPALA
Issue Type: Improvement
Components: Catalog, Frontend
Reporter: Quanlong Huang
Assignee: Quanlong Huang
In local catalog mode, running a query on an unloaded table can't make sure its
metadata is loaded in coordinator side. This is due to a race between two
metadata flows:
# Coordinator uses getPartialCatalogObject RPC to load the metadata from
catalogd. Once coordinator gets the response, the table meta is cached.
# getPartialCatalogObject RPC on the unloaded table actually triggers catalogd
to load metadata of it. When the loading is done, catalogd bumps the table
version. The new table version will then be collected in the statestore update,
and send to all coordinators.
For the coordinator that triggers the request, it will invalidate the table
meta when receiving the statestore update, regradless what the table version is.
{code:java}
private void invalidateCacheForTable(String dbName, String tblName,
List<String> invalidated) {
TableCacheKey key = new TableCacheKey(dbName.toLowerCase(),
tblName.toLowerCase());
if (cache_.asMap().remove(key) != null) {
invalidated.add("table " + dbName + "." + tblName);
}
}{code}
[https://github.com/apache/impala/blob/20220fb9232b94d228383fe693a383d2c71a4733/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java#L1843-L1844]
Ideally the invalidation should check the cached table version and only
invalidate the item when it's older than the statestore update. However, there
are two problems:
* Checking the cached version using getIfPresent() and removing the item is
not atomic. Another thread could put a newer version in between and got
invalidated unintentionally.
* The value could be a CompletableFuture loading the metadata from catalogd.
We should remove it only when it's loading an older version. How to get the
version it's loading is also a problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
