[
https://issues.apache.org/jira/browse/IMPALA-7496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596341#comment-16596341
]
Adriano commented on IMPALA-7496:
---------------------------------
The Impala version used during the issue haven't -IMPALA-1575- The specific
situation appeared once many users connected directly from their workstation to
the specific impalad (this caused more use of memory, as you guessed).
The query options beside the lack of guarantee are very appreciated.
Thank to include the jira in the backlog.
> Schedule query taking in account the mem available on the impalad nodes
> -----------------------------------------------------------------------
>
> Key: IMPALA-7496
> URL: https://issues.apache.org/jira/browse/IMPALA-7496
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend
> Reporter: Adriano
> Priority: Major
> Labels: admission-control, scheduler
>
> Environment description: cluster scale (50/100/150 nodes and terabyte of ram
> available) - Admission Control enabled.
> Issue description:
> Despite the coordinator chosen (with data and statistics unchanged) a query
> will be planned always in the same way based on the metainfo that the
> coordinator have.
> The query will be scheduled always on the same nodes if the memory
> requirements for the admission are satisfied:
> https://github.com/cloudera/Impala/blob/cdh5-2.7.0_5.9.1/be/src/scheduling/admission-controller.cc#L307-L333
> Equal queries are planned/scheduled always in the same way (to hit always the
> same nodes).
> This often lead to queue the queries that are hitting the same nodes are
> queued (not admitted) as on those nodes there's no more memory available
> within its process limit despite the pool have lot of free memory and the
> overall cluster load is low.
> When the plan is finished and the query can be evaluated to be admitted often
> happen that the admission is denied because one of the node have not enough
> memory to run the query operation (and the query is moved in the pool queue)
> despite the cluster have 50/100/150 nodes and terabyte of ram available.
> Why the scheduler does not take in consideration the memory available on the
> nodes involved in the query before to buid the schedule, (maybe preferring a
> remote read/operation on a free memory node instead to include in the plan
> always the same nodes that will end to be:
> 1- overloaded
> 2- the query will be not immediately admitted, risking to be timedout in the
> pool queue
> Since 2.7 the REPLICA_PREFERENCE can possibly help, but it's not good enough
> as it does not prevent the scheduler to choose busy nodes (with the same
> potential effect: query queued for lack of resource on specific node despite
> there are terabytes of free memory).
> Feature Request:
> It would be good if Impala had an option to execute queries (even with worse
> performance) excluding the nodes overloaded and including different nodes in
> order to get the query immediately admitted and executed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
