[
https://issues.apache.org/jira/browse/SPARK-58897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58897:
-----------------------------------
Labels: pull-request-available (was: )
> vector_norm with degree infinity silently ignores NaN elements
> --------------------------------------------------------------
>
> Key: SPARK-58897
> URL: https://issues.apache.org/jira/browse/SPARK-58897
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: sepuri sai krishna
> Priority: Major
> Labels: pull-request-available
>
> h3. Summary
> The infinity-norm branch of {{vector_norm}} silently drops {{NaN}} elements
> instead of
> propagating them. Every other degree of {{vector_norm}} propagates {{NaN}},
> and every other
> max-like function in Spark treats {{NaN}} as the largest value, so the
> infinity norm is the
> lone outlier.
> The single-element case is the damaging one: a vector consisting of {{NaN}}
> is reported as
> having infinity norm {{0.0}}, making it indistinguishable from the zero
> vector. That in turn
> makes {{vector_normalize}} return {{NULL}} (its zero-norm path) rather than a
> vector of
> {{NaN}}.
> h3. Reproduction
> {code:sql}
> -- Wrong: NaN is ignored
> SELECT vector_norm(array(float('nan')), float('inf')); -- 0.0
> expected NaN
> SELECT vector_norm(array(float('nan'), 5.0F), float('inf')); -- 5.0
> expected NaN
> SELECT vector_norm(array(5.0F, float('nan')), float('inf')); -- 5.0
> expected NaN
> SELECT vector_norm(array(float('nan'), float('inf')), float('inf')); --
> Infinity expected NaN
> -- Correct: every other degree propagates NaN
> SELECT vector_norm(array(float('nan')), 1.0F); -- NaN
> SELECT vector_norm(array(float('nan')), 2.0F); -- NaN
> SELECT vector_norm(array(float('nan'), 5.0F), 2.0F); -- NaN
> -- Knock-on effect in vector_normalize
> SELECT vector_normalize(array(float('nan')), float('inf')); --
> NULL expected [NaN]
> SELECT vector_normalize(array(float('nan')), 2.0F); --
> [NaN]
> {code}
> h3. Why this is a bug and not a design choice
> Spark's established convention is that {{NaN}} compares as larger than any
> other value. All of
> the following agree, and disagree with the infinity norm:
> {code:sql}
> SELECT greatest(float('nan'), 5.0F); -- NaN
> SELECT array_max(array(float('nan'), 5.0F)); -- NaN
> SELECT max(v) FROM values (float('nan')), (5.0F) AS t(v); -- NaN
> SELECT sort_array(array(float('nan'), 5.0F)); -- [5.0, NaN]
> {code}
> Since the infinity norm is defined as the maximum absolute value, it should
> follow the same
> convention that {{max}} and {{array_max}} do.
> h3. Root cause
> {{VectorFunctionImplUtils.vectorInfNorm}}
> (sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java)
> hand-rolls the maximum with a {{>}} comparison seeded at zero:
> {code:java}
> float maxAbs = 0.0f;
> for (int i = 0; i < len; i++) {
> if (vec.isNullAt(i)) {
> return null;
> }
> float absVal = Math.abs(vec.getFloat(i));
> if (absVal > maxAbs) {
> maxAbs = absVal;
> }
> }
> return (double) maxAbs;
> {code}
> Any comparison involving {{NaN}} is false under IEEE 754, so {{maxAbs}} is
> never updated to
> {{NaN}} and the {{NaN}} never reaches the result. {{Math.max}}, which returns
> {{NaN}} when
> either argument is {{NaN}}, gives the expected behaviour.
> h3. Scope
> This is distinct from SPARK-58544, which fixed float overflow in the
> intermediate sums of the
> same file. That issue explicitly scoped its "correct as-is" cases to empty
> vectors, NULL
> elements, and genuinely zero magnitude; {{NaN}} input was not covered. The
> existing behaviour
> for empty vectors ({{0.0}}) and for vectors containing NULL ({{NULL}}) is
> unchanged by this
> report.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
