Michael Armbrust created SPARK-4410:
---------------------------------------
Summary: Support for external sort
Key: SPARK-4410
URL: https://issues.apache.org/jira/browse/SPARK-4410
Project: Spark
Issue Type: Improvement
Components: SQL
Reporter: Michael Armbrust
Assignee: Michael Armbrust
Priority: Critical
When any given key is of too high cardinality the current sorting code can tip
over (since it loads a whole partition into memory). I propose we add optional
support for using sparks built in external sorting mechanism. It can be off by
default, but if we determine this code path does not regress performance we can
turn it on by default in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
