[
https://issues.apache.org/jira/browse/IMPALA-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899281#comment-16899281
]
Tim Armstrong commented on IMPALA-8803:
---------------------------------------
Yeah, I think there are some queries that fan-in so backends would go idle
earlier. Our scheduler isn't really very smart about this now, but could be
smarter in future. So I think that's a nice optimisation. I think I agree with
your thought process about the batching, I don't think it needs to be totally
optimal, but having a log() bound on it is nice.
> Coordinator should release admitted memory per-backend rather than per-query
> ----------------------------------------------------------------------------
>
> Key: IMPALA-8803
> URL: https://issues.apache.org/jira/browse/IMPALA-8803
> Project: IMPALA
> Issue Type: Sub-task
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
>
> When {{SPOOL_QUERY_RESULTS}} is true, the coordinator backend may be long
> lived, even though all other backends for the query have completed.
> Currently, the Coordinator only releases admitted memory when the entire
> query has completed (include the coordinator fragment) -
> https://github.com/apache/impala/blob/72c9370856d7436885adbee3e8da7e7d9336df15/be/src/runtime/coordinator.cc#L562
> In order to more aggressively return admitted memory, the coordinator should
> release memory when each backend for a query completes, rather than waiting
> for the entire query to complete.
> Releasing memory per backend should be batched because releasing admitted
> memory in the admission controller requires obtaining a global lock and
> refreshing the internal stats of the admission controller. Batching will help
> mitigate any additional overhead from releasing admitted memory per backend.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
