Abhishek Rawat created IMPALA-13163:
---------------------------------------
Summary: Use proper defaults when executor membership snapshot is
empty
Key: IMPALA-13163
URL: https://issues.apache.org/jira/browse/IMPALA-13163
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Abhishek Rawat
During initialization, the executor group membership snapshot may not arrive
(i.e empty executor group membership) at coordinator before queries get
submitted, and in that case coordinator uses some default resources (executor
group size, mem and cpu) for planning the query
https://github.com/apache/impala/blob/branch-4.4.0/fe/src/main/java/org/apache/impala/service/Frontend.java#L2053-L2059
{code:java}
I0610 15:05:27.176682 338152 Frontend.java:2035]
4a4b33e82891e4df:b235a7a900000000] The original executor group sets from
executor membership snapshot: []
I0610 15:05:27.176748 338152 Frontend.java:2056]
4a4b33e82891e4df:b235a7a900000000] A total of 1 executor group sets to be
considered for auto-scaling: [TExecutorGroupSet(curr_num_executors:1,
expected_num_executors:20, exec_group_name_prefix:default-pool,
max_mem_limit:9223372036854775807, num_cores_per_executor:2147483647)] {code}
Using these defaults could result in query getting planned using unrealistic
resources and may become unschedulable and will cause delays for other queries
behind it in the admission queue.
The current defaults are not ideal and in most cases these resources and
cluster size could be provided as flagfile options during startup, since the
cluster size and resources is generally known at startup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
