[
https://issues.apache.org/jira/browse/IMPALA-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118257#comment-18118257
]
Petr Fedchenkov commented on IMPALA-9976:
-----------------------------------------
I have posted a patch for review: [https://gerrit.cloudera.org/24925]
The statestore does not have enough state to catch up from, as the
description suggests. The request-queue topic only carries per-host and
per-pool aggregates (POOL:/STAT: entries), which are transient: the
statestore deletes them when the admissiond that published them fails.
The per-query allocations needed to release a query correctly when it
finishes (pool, and slots and memory per backend) are only known to the
admissiond and to the query's coordinator.
So the patch makes the coordinators the source of truth:
- coordinators report their admitted queries, with the resources still
held on each unreleased backend, in the admission heartbeat;
- a restarted admissiond re-registers ("adopts") reported queries it does
not know, so pool and host limits hold and later releases work;
- it admits new queries only once every coordinator in its membership has
reported, or after a grace period (10 s by default, and at least the
heartbeat rpc timeout plus two heartbeat periods);
- coordinators resubmit queued queries whose admission state was lost
instead of failing them;
- admission rpcs get timeouts, and after a timeout use a new KRPC
connection (complements IMPALA-14466).
The last-seen statestore aggregates could later be used as a floor while
adoption is in progress, but that is left out of this patch.
> Implement recovery for the admission control service
> ----------------------------------------------------
>
> Key: IMPALA-9976
> URL: https://issues.apache.org/jira/browse/IMPALA-9976
> Project: IMPALA
> Issue Type: Sub-task
> Components: Distributed Exec
> Reporter: Thomas Tauber-Marshall
> Assignee: Bikramjeet Vig
> Priority: Critical
>
> If the new admission control daemon fails, it would be good to be able to
> recover gracefully.
> Most of the admission control state is already stored in the statestore, so
> it should be possible to launch a new admission control daemon and have it
> catch up by retrieving everything from the statestore. Already running
> queries should not be affected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
