dilipbiswal commented on a change in pull request #24073: [SPARK-27134][SQL]
array_distinct function does not work correctly with columns containing array
of array
URL: https://github.com/apache/spark/pull/24073#discussion_r265271294
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
##########
@@ -3112,29 +3112,30 @@ case class ArrayDistinct(child: Expression)
(data: Array[AnyRef]) => new
GenericArrayData(data.distinct.asInstanceOf[Array[Any]])
} else {
(data: Array[AnyRef]) => {
- var foundNullElement = false
- var pos = 0
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ var alreadyStoredNull = false
+ var found = false
for (i <- 0 until data.length) {
- if (data(i) == null) {
- if (!foundNullElement) {
- foundNullElement = true
- pos = pos + 1
+ if (data(i) != null) {
+ found = false
+ var j = 0;
+ while (!found && j < arrayBuffer.size) {
+ val va = arrayBuffer(j)
+ found = (va != null) && ordering.equiv(va, data(i))
Review comment:
@srowen Thanks. Actually in my understanding, ordering does not seem to
handle nulls. Given that, the proposed condition does not work for case when
va != null and data[i] is null. We get a null pointer exception. We can
probably do something like :
```
if (data(i) != null && va != null) {
found = ordering.equiv(va, data(i))
} else if (data(i) == null && va == null) {
found = true
}
```
Given this, i feel the existing code reads better and i think performs
better if we have many null and non-null values in the array by keeping the
null handling outside the inner loop. Also, i believe, for other collection
operation functions we treat nulls separately. But i will change, if you feel
otherwise. Please let me know.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
