Yida Wu created IMPALA-15586:
--------------------------------
Summary: Auto tune cores per executor for COMPUTE_PROCESSING_COST
Key: IMPALA-15586
URL: https://issues.apache.org/jira/browse/IMPALA-15586
Project: IMPALA
Issue Type: Improvement
Components: Backend
Reporter: Yida Wu
Assignee: Yida Wu
With COMPUTE_PROCESSING_COST=true, Impala decides how many fragment instances
to run on each executor based on the total CPU cores the machine reports. It
ignores container limits (like Docker/Kubernetes) and OS rules that restrict
how many cores Impala can actually use. Because of this, Impala can assign too
much work to restricted executors, which uses more memory, creates unnecessary
threads, and makes queries slower. Today, the only fix is to set the core count
by hand on every executor (--num_cores) or on every query
(MAX_FRAGMENT_INSTANCES_PER_NODE).
We probably can add a new flag that automatically limits the maximum fragment
instances per host during query planning. Instead of using raw hardware cores,
the planner will use the executor's effective cores (which respects cgroup
limits and CPU affinity) or its configured admission slots. This ensures Impala
automatically scales query plans to fit the actual compute resources available
to the container or process.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
