Anubhav Jindal created IMPALA-15449:
---------------------------------------
Summary: Impala User/group quota XML can define unusable pools;
Improve Impala to reject distributed queries at submit when no executor group
matches (not queue timeout)
Key: IMPALA-15449
URL: https://issues.apache.org/jira/browse/IMPALA-15449
Project: IMPALA
Issue Type: Improvement
Reporter: Anubhav Jindal
Assignee: Anubhav Jindal
h2. Summary
With a rich interface for user/group quotas, operators edit fair-scheduler XML
directly. This can lead to the mistake of misconfiguring “bad” request pools
(pools that cannot get matching executor groups)
Impala should reject distributed queries attempting to use these bad pools at
submit time with a clear error instead of current behaviour of queueing them
until pool-queue-timeout
h2. Description
h3. Terminology: distributed query
Use this scope in implementation and acceptance tests.
||*Term*||*Meaning*||
|*Distributed query*|Admission schedules the query on *executor* impalads after
selecting an *executor group* (prefix {*}{resolved_pool}-{*}). Examples:
typical table SELECT, TPCH, QE longsleep. *This Jira applies here.*|
|*Coordinator-only query*|Pool has *<onlyCoordinators>true</onlyCoordinators>*
(or equivalent path); *no* executor-group lookup for {pool}-*. *Do not*
fail-fast for missing executor groups on this path.|
|*Executor group*|Membership name registered by executors (e.g.
root.default-group-000, root.group-set-custom-1-group-000) via
{*}--executor_groups{*}.|
h3. Problem
*User/group quotas* encourage direct edits to *fair-scheduler.xml* (new queues,
userQueryLimit, nested pools). It is easy to add a *valid* xml pool path that
*cannot* receive executors on the VW type in use—for example a sibling of
root.default on {*}non-WAAS{*}, or a WAAS pool name with no matching
{*}groupSetPrefix{*}. Users assume XML-defined pools are runnable; instead,
*distributed queries* often *queue until timeout* with {*}0 running{*}, which
looks like a broken quota or hung query rather than misconfiguration.
When a *distributed query’s* *resolved* REQUEST_POOL exists in XML but *no
executor group* is registered whose name matches the Impala prefix rule
{pool}-, admission currently *queues* the query (e.g. “Waiting for executors to
start”) until *impala.admission-control.pool-queue-timeout-ms.\{pool}* (often
600s). Clients and operators experience a {*}timeout{*}, not a clear
configuration/routing error.
On *non-WAAS* VW, only *root.default-** (e.g. root.default-group-000) is
provisioned. Any other pool path that exists in XML (sibling or nested) hits
this behavior today.
On *WAAS* VW, each *groupSetPrefix* (e.g. root.group-set-custom-1) must align
with fair-scheduler queue paths *and* live executor registration
(root.group-set-custom-1-group-000). A pool defined in XML *without* a matching
group set on the VW—or a *REQUEST_POOL* that resolves to such a pool—has the
same queue-until-timeout failure mode.
This is especially painful for *user/group quota* and pool routing work: tests
and users expect *quota rejection* or immediate “bad pool” feedback, not a long
queue with *0 running* on that pool in /admission?json.
h3. Proposed behavior
*Preserve current behaviour at Impala VW start:* do not fail start when
fair-scheduler XML defines pool paths that have *no matching executor group* on
this VW (typical with user/group quota edits; executor groups also register
after start). *Change query-time behaviour:* when a *distributed query* (see
{*}Terminology{*}) reaches admission, the resolved pool has {*}no eligible
executor group{*}, and the case is *not* a transient scale-up/wake on a pool
that can receive executors on this VW:
# *Reject at submit* (HS2 / impala-shell / JDBC) with an explicit error.
# Message should name {*}resolved pool{*}, *VW type* (non-WAAS vs WAAS) if
helpful, and *actionable guidance* (e.g. use root.default, align fair-scheduler
queue path with {*}groupSetPrefix{*}, or add/register the matching executor
group set on WAAS).
# Distinguish from *user quota* rejection and from *scale-up wait* on
*root.default* (executors temporarily 0 while warehouse wakes).
Apply on *coordinator* and *Global Admission Control (admissiond)* paths
consistently.
h3. Scope
* *In scope:* Permanent “no matching executor group for resolved pool {*}P{*}”
on a *distributed query* → reject at submit (not queue until
pool-queue-timeout-ms).
* *In scope:* Same logic on *WAAS* when *P* is in fair-scheduler but has no
matching *groupSetPrefix* / registered *P-** executor group.
* *Out of scope:* Failing VW *start* when fair-scheduler defines pool paths
with no matching executor group (preserve current start behaviour).
* *Out of scope (separate stories):* Silent placement fallthrough to
root.default when requested pool name does not exist; CDW fallback from
"default" to root.default-group-000; parent-prefix inheritance for nested pools.
h2. Steps to reproduce (non-WAAS)
*Environment:* Standard (non-WAAS) Impala VW, running, at least one healthy
root.default-group-000.
# Apply *nonwaas_with-sibling-user-rule.xml* (coordinator + admissiond if
GAC).
# SET REQUEST_POOL='root.user-rule'; submit a *distributed query* (see
*Terminology* above)—not a coordinator-only pool.
# *Actual today:* Queue; *0 running* on root.user-rule in /admission?json
until pool-queue timeout.
# *Expected:* Immediate rejection naming root.user-rule and non-WAAS executor
groups.
Alternate XML: *nonwaas_with-nested-pool1.xml* with
REQUEST_POOL='root.default.pool1'.
h2. Steps to reproduce (WAAS)
*Environment:* WAAS VW whose expected_executor_group_sets includes small,
custom-1, and large (not custom-2).
# Apply *waas_with-orphan-pool.xml* (adds *root.group-set-custom-2* in XML
only).
# SET REQUEST_POOL='root.group-set-custom-2'; submit a *distributed query*
(see *Terminology* above).
# *Actual today:* Queue until timeout (no root.group-set-custom-2-* executor
group).
# *Expected:* Immediate rejection (pool vs groupSetPrefix / backends).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
