Matei Zaharia created SPARK-2047:
------------------------------------
Summary: Use less memory in AppendOnlyMap.destructiveSortedIterator
Key: SPARK-2047
URL: https://issues.apache.org/jira/browse/SPARK-2047
Project: Spark
Issue Type: Improvement
Components: Spark Core
Reporter: Matei Zaharia
This method tries to sort an the key-value pairs in the map in-place but ends
up allocating a Tuple2 object for each one, which allocates a nontrivial amount
of memory (32 or more bytes per entry on a 64-bit JVM). We could instead try to
sort the objects in-place within the "data" array, or allocate an int array
with the indices and sort those using a custom comparator. The latter is
probably easiest to begin with.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
