[
https://issues.apache.org/jira/browse/SPARK-59516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59516:
-----------------------------------
Labels: pull-request-available (was: )
> Infinity norm of an all-zero SparseVector raises ValueError with NumPy before
> 2.3
> ---------------------------------------------------------------------------------
>
> Key: SPARK-59516
> URL: https://issues.apache.org/jira/browse/SPARK-59516
> Project: Spark
> Issue Type: Bug
> Components: MLlib, PySpark
> Affects Versions: 5.0.0
> Reporter: Akshay Thorat
> Priority: Major
> Labels: pull-request-available
>
> h2. Problem
> With NumPy versions before 2.3, SparseVector.norm passes an empty values
> array to numpy.linalg.norm, which raises on the maximum reduction. Dense
> vectors and sparse vectors storing an explicit zero return 0.0 for the same
> logical vector. Both pyspark.ml.linalg and pyspark.mllib.linalg are affected.
> h2. Reproduction
> Reproduced on upstream master 39776477a3d (PySpark 5.0.0.dev0), Python
> 3.10.11 and NumPy 2.2.6. No SparkSession is required.
> {code:python}
> from pyspark.ml.linalg import SparseVector
> SparseVector(3, [], []).norm(float("inf"))
> # ValueError: zero-size array to reduction operation maximum which has no
> identity
> {code}
> h2. Expected behavior
> The infinity norm should be 0.0, as it is for DenseVector([0.0, 0.0, 0.0]).
> Supported NumPy versions should give the same result for positive-dimensional
> zero vectors.
> h2. Proposed fix and verification
> Return zero for infinity norm when a positive-dimensional sparse vector has
> no stored entries, in both linalg APIs. Preserve NumPy behavior for
> zero-dimensional vectors and invalid norm orders. Regression tests cover both
> APIs and implicit/explicit zeros.
> The regression fails with NumPy 2.2.6 before the fix and passes afterward.
> NumPy 2.3 independently changed the empty-array norm to return zero; the
> Spark fix also supports older NumPy releases. See
> https://numpy.org/doc/2.3/release/2.3.0-notes.html#changes
> Pull request: https://github.com/apache/spark/pull/58788
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
