[
https://issues.apache.org/jira/browse/IMPALA-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16774605#comment-16774605
]
ASF subversion and git services commented on IMPALA-6900:
---------------------------------------------------------
Commit 4ed947365447934c55eeec94610a2b3cbd993937 in impala's branch
refs/heads/master from Bharath Vissapragada
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=4ed9473 ]
IMPALA-6900: Fix the min version invariant for invalidate metadata
Current flow of invalidate metadata is as follows.
1. Catalog server records the global catalog version *before* reset()
2. Catalog server invalidates all the metadata.
3. Catalog server returns the version recorded in step (1)
4. Coordinator should wait until the minimum version across all the
local catalog objects is > the version returned in the step (3).
However, the invariant in step 4 on the coordinator side was incorrectly
coded resulting in an early termination of the wait loop.
This breaks the invalidate metadata contract and the subsequent queries
can fail while resolving catalog objects in the local cache.
Testing: I could reproduce this issue reliably on a pristine HMS with
high --statestore_update_frequency_ms (see IMPALA-7605 for details).
With the patch, I'm not able to reproduce it.
Change-Id: I22ace3f5917b6167af63c3dfd2620e69ce8ec1cb
Reviewed-on: http://gerrit.cloudera.org:8080/12547
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Invalidate metadata operation is ignored at a coordinator if catalog is empty
> -----------------------------------------------------------------------------
>
> Key: IMPALA-6900
> URL: https://issues.apache.org/jira/browse/IMPALA-6900
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 3.0, Impala 2.12.0
> Reporter: Dimitris Tsirogiannis
> Priority: Blocker
>
> The following workflow may cause an impalad that issued an invalidate
> metadata to falsely consider that the effect of that operation has taken
> effect, thus causing subsequent queries to fail due to unresolved references
> to tables or databases.
> Steps to reproduce:
> # Start an impala cluster connecting to an empty HMS (no databases).
> # Create a database "db" in HMS outside of Impala (e.g. using Hive).
> # Run INVALIDATE METADATA through Impala.
> # Run "use db" statement in Impala.
>
> The while condition in the code snippet below is cause the
> WaitForMinCatalogUpdate function to prematurely return even though INVALIDATE
> METADATA has not taken effect:
> {code:java}
> void ImpalaServer::WaitForMinCatalogUpdate(..) {
> ...
> VLOG_QUERY << "Waiting for minimum catalog object version: "
> << min_req_catalog_object_version << " current version: "
> << min_catalog_object_version;
> while (catalog_update_info_.min_catalog_object_version <
> min_req_catalog_object_version && catalog_update_info_.catalog_service_id ==
> catalog_service_id) {
> catalog_version_update_cv_.Wait(unique_lock);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
