asugranyes commented on PR #53468:
URL: https://github.com/apache/spark/pull/53468#issuecomment-4566471209
Nice generalization to all types via the comparable wrapper.
It looks like `ArraysOverlap` is left out of this.
It splits on the same `TypeUtils.typeWithProperEquals(elementType)`
predicate, but its `fastEval` uses a plain `java.util.HashSet[Any]` (works only
for proper-equals types per its docstring), so complex types fall back to
`bruteForceEval` — the O(n²) nested loop with `ordering.equiv`.
``` scala
/**
* A slower evaluation which performs a nested loop and supports all the
data types.
*/
private def bruteForceEval(arr1: ArrayData, arr2: ArrayData): Any = {
var hasNull = false
if (arr1.numElements() > 0 && arr2.numElements() > 0) {
arr1.foreach(elementType, (_, v1) =>
if (v1 == null) {
hasNull = true
} else {
arr2.foreach(elementType, (_, v2) =>
if (v2 == null) {
hasNull = true
} else if (ordering.equiv(v1, v2)) {
return true
}
)
})
}
if (hasNull) {
null
} else {
false
}
```
With the comparable wrapper, `fastEval` could handle all types, making the
`typeWithProperEquals` split and `bruteForceEval` unnecessary — O(n) for
complex types instead of O(n²). Worth a follow-up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
