[
https://issues.apache.org/jira/browse/IMPALA-13163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Rawat updated IMPALA-13163:
------------------------------------
Priority: Critical (was: Major)
> Use proper defaults when executor membership snapshot is empty
> --------------------------------------------------------------
>
> Key: IMPALA-13163
> URL: https://issues.apache.org/jira/browse/IMPALA-13163
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Abhishek Rawat
> Assignee: Abhishek Rawat
> Priority: Critical
>
> During initialization, the executor group membership snapshot may not arrive
> (i.e empty executor group membership) at coordinator before queries get
> submitted, and in that case coordinator uses some default resources (executor
> group size, mem and cpu) for planning the query
> https://github.com/apache/impala/blob/branch-4.4.0/fe/src/main/java/org/apache/impala/service/Frontend.java#L2053-L2059
> {code:java}
> I0610 15:05:27.176682 338152 Frontend.java:2035]
> 4a4b33e82891e4df:b235a7a900000000] The original executor group sets from
> executor membership snapshot: []
> I0610 15:05:27.176748 338152 Frontend.java:2056]
> 4a4b33e82891e4df:b235a7a900000000] A total of 1 executor group sets to be
> considered for auto-scaling: [TExecutorGroupSet(curr_num_executors:1,
> expected_num_executors:20, exec_group_name_prefix:default-pool,
> max_mem_limit:9223372036854775807, num_cores_per_executor:2147483647)] {code}
> Using these defaults could result in query getting planned using unrealistic
> resources and may become unschedulable and will cause delays for other
> queries behind it in the admission queue.
> The current defaults are not ideal and in most cases these resources and
> cluster size could be provided as flagfile options during startup, since the
> cluster size and resources is generally known at startup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
