GitHub user maryannxue opened a pull request:
https://github.com/apache/spark/pull/21926
[SPARK-24972][SQL] PivotFirst could not handle pivot columns of complex
types
## What changes were proposed in this pull request?
When the pivot column is of a complex type, the eval() result will be an
UnsafeRow, while the keys of the HashMap for column value matching is a
GenericInternalRow. As a result, there will be no match and the result will
always be empty.
So for a pivot column of complex-types, we should:
1) If the complex-type is not comparable (orderable), throw an Exception.
It cannot be a pivot column.
2) Otherwise, if it goes through the `PivotFirst` code path, `PivotFirst`
should use a TreeMap instead of HashMap for such columns.
## How was this patch tested?
Added UT.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maryannxue/spark pivot_followup
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21926.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21926
----
commit b41a45cb22bd3d49e75711950bcbc3d409bc544a
Author: maryannxue <maryannxue@...>
Date: 2018-07-30T23:15:40Z
[SPARK-24972][SQL] PivotFirst could not handle pivot columns of complex
types
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
