zhengruifeng commented on a change in pull request #25178: [SPARK-28421][ML]
SparseVector.apply performance optimization
URL: https://github.com/apache/spark/pull/25178#discussion_r304713452
##########
File path: mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala
##########
@@ -603,6 +603,19 @@ class SparseVector @Since("2.0.0") (
private[spark] override def asBreeze: BV[Double] = new BSV[Double](indices,
values, size)
+ override def apply(i: Int): Double = {
+ if (i < 0 || i >= size) {
+ throw new IndexOutOfBoundsException(s"Index $i out of bounds [0, $size)")
+ }
+
+ if (indices.isEmpty || i < indices(0) || i > indices(indices.length - 1)) {
Review comment:
1, the impl of `Arrays.binarySearch` does not chek the range:
```
public static int binarySearch(int[] a, int key) {
return binarySearch0(a, 0, a.length, key);
}
// Like public version, but without range checks.
private static int binarySearch0(long[] a, int fromIndex, int toIndex,
long key) {
int low = fromIndex;
int high = toIndex - 1;
while (low <= high) {
int mid = (low + high) >>> 1;
long midVal = a[mid];
if (midVal < key)
low = mid + 1;
else if (midVal > key)
high = mid - 1;
else
return mid; // key found
}
return -(low + 1); // key not found.
}
```
2, in `breeze.collection.mutable.SparseArray`, the `findOffset` function
called in `apply` to perform binary seach, take the special case that the key
is out of range into account
```
if (used == 0) {
// empty list do nothing
-1
} else {
val index = this.index
if (i > index(used - 1)) {
// special case for end of list - this is a big win for growing
sparse arrays
~used
```
so I added those simple checking between binary search.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
