[
https://issues.apache.org/jira/browse/IMPALA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16467632#comment-16467632
]
ASF subversion and git services commented on IMPALA-6948:
---------------------------------------------------------
Commit ffac1ab48c55084e13cc1dd517c25e37e48adc31 in impala's branch
refs/heads/2.x from [~dtsirogiannis]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=ffac1ab ]
IMPALA-6948: Delete catalog update topic entries upon catalog restart
This commit fixes an issue where the statestore may end up with stale
entries in the catalog update topic that do not correspond to the
catalog objects stored in the catalog. This may occur if the catalog
server restarts and some catalog object (e.g. table) that was known to
the catalog before the restart no longer exists in the Hive Metastore
after the restart.
Fix:
The first update for the catalog update topic that is sent by the catalog
instructs the statestore to clear any entries it may have on this topic
before applying the first update. In that way, we guarantee that the
statestore entries are consistent with the catalog objects stored in the
catalog server. Any coordinator that detects the catalog restart will
receive from the statestore a full topic update that reflects the state
of the catalog server.
Testing:
Added statestore test.
Change-Id: I907509bf92da631ece5efd23c275a613ead00e91
Tmp
Change-Id: I74a8ade8e498ac35cb56d3775d2c67a86988d9b6
Reviewed-on: http://gerrit.cloudera.org:8080/10289
Reviewed-by: Vuk Ercegovac <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Coordinators don't detect the deletion of tables that occurred outside of
> impala after catalog restart
> ------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-6948
> URL: https://issues.apache.org/jira/browse/IMPALA-6948
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 3.0, Impala 2.12.0
> Reporter: Dimitris Tsirogiannis
> Assignee: Dimitris Tsirogiannis
> Priority: Blocker
> Labels: catalog-server
>
> Upon catalog restart the coordinators detect this event and request a full
> topic update from the statestore. In certain cases, the topic update protocol
> executed between the statestore and the catalog fails to detect catalog
> objects that were deleted from the Metastore externally (e.g. via HIVE), thus
> causing these objects to show up again in each coordinator's catalog cache.
> The end result is that the catalog server and the coordinator's cache are out
> of sync and in some cases the only solution is to restart both the catalog
> and the statestore.
> The following sequence can reproduce this issue:
> {code:java}
> impala> create table lala(int a);
> bash> kill -9 `pidof catalogd`
> hive> drop table lala;
> bash> restart catalogd
> impala> show tables;
> --- lala shows up in the list of tables;{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
