sepuri sai krishna created SPARK-58897:
------------------------------------------
Summary: vector_norm with degree infinity silently ignores NaN
elements
Key: SPARK-58897
URL: https://issues.apache.org/jira/browse/SPARK-58897
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: sepuri sai krishna
h3. Summary
The infinity-norm branch of {{vector_norm}} silently drops {{NaN}} elements
instead of
propagating them. Every other degree of {{vector_norm}} propagates {{NaN}}, and
every other
max-like function in Spark treats {{NaN}} as the largest value, so the infinity
norm is the
lone outlier.
The single-element case is the damaging one: a vector consisting of {{NaN}} is
reported as
having infinity norm {{0.0}}, making it indistinguishable from the zero vector.
That in turn
makes {{vector_normalize}} return {{NULL}} (its zero-norm path) rather than a
vector of
{{NaN}}.
h3. Reproduction
{code:sql}
-- Wrong: NaN is ignored
SELECT vector_norm(array(float('nan')), float('inf')); -- 0.0
expected NaN
SELECT vector_norm(array(float('nan'), 5.0F), float('inf')); -- 5.0
expected NaN
SELECT vector_norm(array(5.0F, float('nan')), float('inf')); -- 5.0
expected NaN
SELECT vector_norm(array(float('nan'), float('inf')), float('inf')); --
Infinity expected NaN
-- Correct: every other degree propagates NaN
SELECT vector_norm(array(float('nan')), 1.0F); -- NaN
SELECT vector_norm(array(float('nan')), 2.0F); -- NaN
SELECT vector_norm(array(float('nan'), 5.0F), 2.0F); -- NaN
-- Knock-on effect in vector_normalize
SELECT vector_normalize(array(float('nan')), float('inf')); -- NULL
expected [NaN]
SELECT vector_normalize(array(float('nan')), 2.0F); -- [NaN]
{code}
h3. Why this is a bug and not a design choice
Spark's established convention is that {{NaN}} compares as larger than any
other value. All of
the following agree, and disagree with the infinity norm:
{code:sql}
SELECT greatest(float('nan'), 5.0F); -- NaN
SELECT array_max(array(float('nan'), 5.0F)); -- NaN
SELECT max(v) FROM values (float('nan')), (5.0F) AS t(v); -- NaN
SELECT sort_array(array(float('nan'), 5.0F)); -- [5.0, NaN]
{code}
Since the infinity norm is defined as the maximum absolute value, it should
follow the same
convention that {{max}} and {{array_max}} do.
h3. Root cause
{{VectorFunctionImplUtils.vectorInfNorm}}
(sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java)
hand-rolls the maximum with a {{>}} comparison seeded at zero:
{code:java}
float maxAbs = 0.0f;
for (int i = 0; i < len; i++) {
if (vec.isNullAt(i)) {
return null;
}
float absVal = Math.abs(vec.getFloat(i));
if (absVal > maxAbs) {
maxAbs = absVal;
}
}
return (double) maxAbs;
{code}
Any comparison involving {{NaN}} is false under IEEE 754, so {{maxAbs}} is
never updated to
{{NaN}} and the {{NaN}} never reaches the result. {{Math.max}}, which returns
{{NaN}} when
either argument is {{NaN}}, gives the expected behaviour.
h3. Scope
This is distinct from SPARK-58544, which fixed float overflow in the
intermediate sums of the
same file. That issue explicitly scoped its "correct as-is" cases to empty
vectors, NULL
elements, and genuinely zero magnitude; {{NaN}} input was not covered. The
existing behaviour
for empty vectors ({{0.0}}) and for vectors containing NULL ({{NULL}}) is
unchanged by this
report.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
