Tim Armstrong created IMPALA-8483:
-------------------------------------
Summary: Make coordinator fragment lighter-weight
Key: IMPALA-8483
URL: https://issues.apache.org/jira/browse/IMPALA-8483
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Tim Armstrong
Assignee: Bikramjeet Vig
it's possible to write queries with an arbitrarily expensive coordinator
fragment. One example is a query with a set of unpartitioned analytic functions
each with a different ORDER BY, which results in many SORT nodes in the
coordinator fragment.
This is a problem for dedicated coordinators because it makes the resource
consumption unpredictable.
It would be useful if we could offload that work to executors and guarantee
that only "lightweight' operators are part of the coordinator fragment, e.g.
operators that take O(#rows returned * log(#executors)) time and O(#executors)
memory - regular exchanges, merging exchanges, TOP-N, non-grouping aggregation,
etc.
I think this can be done in general by inserting an additional exchange on top
of the "expensive" part of a coordinator fragment, and then ensuring that that
fragment gets scheduled on an executor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
