[
https://issues.apache.org/jira/browse/IMPALA-12060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17715129#comment-17715129
]
Wenzhe Zhou edited comment on IMPALA-12060 at 4/21/23 6:51 PM:
---------------------------------------------------------------
Currently Statestore server does not distinguish the role of each subscriber,
e.g. it does not distinguish if a subscriber is a catalogd, or impalad, or
admissiond. Each daemon must register itself to statestored with an unique
subscriber-id.
catalogd set its subscriber-id as "catalog-server@<hostname>:<port>".
impalad set its subscriber-id as "impalad@<hostname>:<port>".
admissiond set its subscriber-id as "admissiond@<hostname>:<port>".
If two daemons register to statestore with same subscriber-id, statestore
removes the first one from the subscriber list and adds the second one with new
registration_id. The first one will not receiver heartbeat and topic update
RPCs.
If two daemons with different hostnames but same network addresses, statestore
still accept the registration requests as two different subscribers.
To ensure there could only be a single instance of catalogd in the cluster,
statestore have to distinguish the role of each subscriber so that it can count
the number of instance for catalogd. We can find the role of the subscriber by
checking the prefix of subscriber-id, or adding a new field in
TRegisterSubscriberRequest.
was (Author: wzhou):
Currently Statestore server does not distinguish the role of each subscriber,
e.g. it does not distinguish if a subscriber is a catalogd, or impalad, or
admissiond. Each daemon must register itself to statestored with an unique
subscriber-id.
catalogd set its subscriber-id as "catalog-server@<hostname>:<port>".
impalad set its subscriber-id as "impalad@<hostname>:<port>".
admissiond set its subscriber-id as "admissiond@<hostname>:<port>".
If two daemons register to statestore with same subscriber-id, statestore
removes the first one from the subscriber list and adds the second one with new
registration_id. The first one will not receiver heartbeat and topic update
RPCs.
If two daemons with different hostnames but same network addresses, statestore
still accept the registration requests as two different subscribers.
To ensure there could only be a single instance of catalogd in the cluster,
statestore have to distinguish the role of each subscriber so that it counts
the number of instance for catalogd. We can find the role of the subscriber by
checking the prefix of subscriber-id, or adding a new field in
TRegisterSubscriberRequest.
> statestore should only allow a single catalog instance to be part of cluster
> ----------------------------------------------------------------------------
>
> Key: IMPALA-12060
> URL: https://issues.apache.org/jira/browse/IMPALA-12060
> Project: IMPALA
> Issue Type: Bug
> Reporter: Abhishek Rawat
> Assignee: Wenzhe Zhou
> Priority: Critical
>
> Today Impala doesn't support multiple catalogd instances to be part of the
> cluster. When using external Active Passive HA mechanisms like leader
> election, it may be possible for multiple catalog replicas to be part of the
> cluster, in some cases. This is mainly because catalogds have no idea that
> there are other catalogd instances. One way to address this issue could be
> for statestore to always ensure that there could only be a single instance of
> catalogd in the cluster. Coordinator also has a config `catalog_service_host`
> - both statestore and coordinator should have a consistent view and consider
> the proper catalogd instance to be part of the cluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
