Hello Bharath Vissapragada, Vuk Ercegovac,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/11403
to review the following change.
Change subject: IMPALA-7530, IMPALA-7509. Re-plan queries if they fetch
inconsistent metadata
......................................................................
IMPALA-7530, IMPALA-7509. Re-plan queries if they fetch inconsistent metadata
Given the granular catching in CatalogdMetaProvider, it's possible to
have a situation like the following:
- impalad has cached the table list including some table "t"
- some other daemon has issued deletion of "t"
- we try to access "t" before the invalidation has reached us via the
statestore
In this case, we'll get an error back when we try to fetch the table
"t". This can confuse the planning process since it previously
determined that the table exists. Note that this may occur either when
"t" is first referenced or later during planning (eg when fetching a
specific partition post-pruning), so it wouldn't be possible to simply
convert it into a 'table does not exist' result.
The solution here is to throw InconsistentMetadataFetchException after
invalidating the table list and associated object. This then triggers a
re-plan of the query.
This patch implements such re-planning up to 10 times before eventually
giving up. Given that each attempt invalidates the inconsistent cache we
would normally expect it to succeed on the first such retry. The limit
of 10 retries is just to avoid infinite loops in the case of a bug.
Change-Id: I17389954780fa22d7866224c17e128458fffa545
---
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/custom_cluster/test_local_catalog.py
5 files changed, 170 insertions(+), 62 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/11403/1
--
To view, visit http://gerrit.cloudera.org:8080/11403
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I17389954780fa22d7866224c17e128458fffa545
Gerrit-Change-Number: 11403
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <[email protected]>
Gerrit-Reviewer: Bharath Vissapragada <[email protected]>
Gerrit-Reviewer: Vuk Ercegovac <[email protected]>
