[
https://issues.apache.org/jira/browse/IMPALA-8483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Volker updated IMPALA-8483:
--------------------------------
Labels: resource-management scalability (was: resource-management)
> Make coordinator fragment lighter-weight
> ----------------------------------------
>
> Key: IMPALA-8483
> URL: https://issues.apache.org/jira/browse/IMPALA-8483
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Tim Armstrong
> Assignee: Bikramjeet Vig
> Priority: Major
> Labels: resource-management, scalability
>
> it's possible to write queries with an arbitrarily expensive coordinator
> fragment. One example is a query with a set of unpartitioned analytic
> functions each with a different ORDER BY, which results in many SORT nodes in
> the coordinator fragment.
> This is a problem for dedicated coordinators because it makes the resource
> consumption unpredictable.
> It would be useful if we could offload that work to executors and guarantee
> that only "lightweight' operators are part of the coordinator fragment, e.g.
> operators that take O(#rows returned * log(#executors)) time and
> O(#executors) memory - regular exchanges, merging exchanges, TOP-N,
> non-grouping aggregation, etc.
> I think this can be done in general by inserting an additional exchange on
> top of the "expensive" part of a coordinator fragment, and then ensuring that
> that fragment gets scheduled on an executor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
