Zoltán Borók-Nagy created IMPALA-15518:
------------------------------------------
Summary: COMPUTE_PROCESSING_COST plans up to 128 instances per
host when only the default executor group exists
Key: IMPALA-15518
URL: https://issues.apache.org/jira/browse/IMPALA-15518
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Zoltán Borók-Nagy
When the cluster has only the default executor group (no executor group sets
configured), Frontend.setupThresholdsForExecutorGroupSets() sets
num_cores_per_executor to Integer.MAX_VALUE (Frontend.java:2252, 2278 and 2287
on master 1f34ea915e). With COMPUTE_PROCESSING_COST=true, per-host parallelism
is then limited only by MAX_FRAGMENT_INSTANCES_PER_NODE (default 128),
regardless of how many cores the executors have. Admission control does not
slot-check the default group either (admission-controller.cc:1462 and 1560,
TODO IMPALA-8757).
*Repro:* 5 executors with 16 vCPUs each, mem_limit=40 GB, TPC-DS SF100 Parquet,
SET COMPUTE_PROCESSING_COST=true.
* EXPLAIN of q72 shows 640 instances (128 per host) for the fragment with the
main join chain, about 404 threads per host, and a per-host memory estimate of
about 100 GB.
* 6 of the 104 TPC-DS queries fail:
** q14_1: thread reservation on backend ... is greater than the
THREAD_RESERVATION_LIMIT query option value: 8916 > 3000.
** q22, q23_1, q23_2, q67a, q72: Failed to get minimum memory reservation of
29.81 / 13.45 / 13.50 / 66.93 / 19.58 GB.
* 30 of the remaining 98 queries are more than 10% slower than with
COMPUTE_PROCESSING_COST=false. Most are small queries, for example q7 0.45 ->
1.37 s, q82 0.39 -> 1.29 s, q63 0.26 -> 1.13 s.
*Workaround:* MAX_FRAGMENT_INSTANCES_PER_NODE=8 gives 0 failures (210.0 s for
the workload, against 374.6 s at default options).
*Proposed fix:* use the executors' real core count for the default group.
* Every impalad already publishes MachineInfoPB.cpu_num_cores
(impala-server.cc:2834).
* TExecutorGroupSet.num_cores_per_executor already exists (Frontend.thrift:944).
* PopulateExecutorMembershipRequest (cluster-membership-mgr.cc:760) could set
it to the minimum over the group's executors, and the FE would use that instead
of Integer.MAX_VALUE.
A prototype that assumes 16 cores per executor gives 0 failures and 211.0 s.
Even with this fix, COMPUTE_PROCESSING_COST is slower than a plain MT_DOP=8 on
short queries in this workload (211.0 vs 187.0 s total, geomean 1.29x). That is
a tuning question for IMPALA-15391.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
