[ https://issues.apache.org/jira/browse/SPARK-18800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734680#comment-15734680 ]
Apache Spark commented on SPARK-18800: -------------------------------------- User 'viirya' has created a pull request for this issue: https://github.com/apache/spark/pull/16232 > UnsafeInMemorySorter throws exception when used in UnsafeKVExternalSorter > ------------------------------------------------------------------------- > > Key: SPARK-18800 > URL: https://issues.apache.org/jira/browse/SPARK-18800 > Project: Spark > Issue Type: Bug > Components: SQL > Reporter: Liang-Chi Hsieh > > UnsafeKVExternalSorter uses UnsafeInMemorySorter to sort the records of > BytesToBytesMap if it is given a map. > Currently we use the number of keys in BytesToBytesMap to determine if the > array used for sort is enough or not. It should be wrong. Because we can have > multiple values of the same key. Extremely said, you can have > BytesToBytesMap.numKeys() == 1, but BytesToBytesMap.numValues() is a big > number. > In this case, we cannot just use BytesToBytesMap's array to do sorting. > Otherwise, a exception will be thrown like this: > {code} > [info] - SPARK-kv sorting with records already in BytesToBytesMap *** FAILED > *** (61 milliseconds) > [info] java.lang.IllegalStateException: There is no space for new record > [info] at > org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.insertRecord(UnsafeInMemorySorter.jav > a:225) > [info] at > org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:147) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org