steven-aerts opened a new pull request #35426:
URL: https://github.com/apache/spark/pull/35426
The check in array_sort to check for orderable items can be removed. As
it is possible to pass a lambda to make the items orderable:
Seq((Array[Map[String, Int]](Map("a" -> 1), Map()), "x")).toDF("a", "b")
.selectExpr("array_sort(a, (x,y) -> cardinality(x) - cardinality(y))")
And for the cases where it is relevant the check is never hit as the
LessThan operator has a similar check which is evaluated first:
> due to data type mismatch: LessThan does not support ordering on type
map<string,int>
### What changes were proposed in this pull request?
Remove a check which prevents you to sort an array with non-orderable items,
with a lambda which makes everything orderable.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Extra test was added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
