GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/16232
[SPARK-18800][SQL] Fix UnsafeKVExternalSorter by correctly allocating array
## What changes were proposed in this pull request?
`UnsafeKVExternalSorter` uses `UnsafeInMemorySorter` to sort the records of
`BytesToBytesMap` if it is given a map.
Currently we use the number of keys in `BytesToBytesMap` to determine if
the array used for sort is enough or not. It should be wrong. Because we can
have multiple values of the same key. Extremely said, you can have
BytesToBytesMap.numKeys() == 1, but BytesToBytesMap.numValues() is a big number.
In this case, we cannot just use `BytesToBytesMap`'s array to do sorting.
Otherwise, a exception will be thrown like this:
[info] - SPARK-18800: pass BytesToBytesMap which contains numValues is
more than numKeys *** FAILED *** (61 milliseconds)
[info] java.lang.IllegalStateException: There is no space for new
record
[info] at
org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.insertRecord(UnsafeInMemorySorter.java:225)
[info] at
org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:147)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1
SPARK-18800-fix-UnsafeKVExternalSorter
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16232.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16232
----
commit 8110a82de280bdb1242a438642c277bee9ab0376
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-12-09T08:15:44Z
Fix UnsafeKVExternalSorter by correctly allocating array.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
