sepuri sai krishna created SPARK-58544:
------------------------------------------
Summary: Vector distance and norm functions return NaN, Infinity
or NULL due to float overflow in intermediate sums
Key: SPARK-58544
URL: https://issues.apache.org/jira/browse/SPARK-58544
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: sepuri sai krishna
The vector distance and norm functions added in 4.2.0 accumulate dot products
and sums of squares in single precision. Those intermediate quantities are
quadratic in the input values, so they overflow to infinity (or underflow to
zero) long before the final result leaves the float range, and the function
silently returns a wrong value.
{code:sql}
SELECT vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(3.0e19F,
4.0e19F)); -- NaN, expected 1.0
SELECT vector_cosine_similarity(array(1.0e-23F, 0.0F), array(1.0e-23F, 0.0F));
-- NULL, expected 1.0
SELECT vector_l2_distance(array(3.0e19F, 4.0e19F), array(0.0F, 0.0F));
-- Infinity, expected 5.0E19
SELECT vector_norm(array(3.0e19F, 4.0e19F), 2.0F);
-- Infinity, expected 5.0E19
SELECT vector_normalize(array(3.0e19F, 4.0e19F), 2.0F);
-- [0.0,0.0], expected [0.6,0.8]
SELECT vector_normalize(array(1.0e-23F, 0.0F), 2.0F);
-- NULL, expected [1.0,0.0]
{code}
||Query||Actual||Expected||
|vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(3.0e19F,
4.0e19F))|NaN|1.0|
|vector_cosine_similarity(array(1.0e-23F, 0.0F), array(1.0e-23F,
0.0F))|NULL|1.0|
|vector_l2_distance(array(3.0e19F, 4.0e19F), array(0.0F, 0.0F))|Infinity|5.0E19|
|vector_norm(array(3.0e19F, 4.0e19F), 2.0F)|Infinity|5.0E19|
|vector_normalize(array(3.0e19F, 4.0e19F), 2.0F)|[0.0, 0.0]|[0.6, 0.8]|
|vector_normalize(array(1.0e-23F, 0.0F), 2.0F)|NULL|[1.0, 0.0]|
In every case above the correct result is an ordinary float; only the
intermediate sum leaves the float range: {{(3e19)^2 + (4e19)^2 = 2.5e39}}
exceeds {{Float.MAX_VALUE}}, and {{(1e-23)^2 = 1e-46}} is below the smallest
positive float. Cosine similarity and normalization are scale invariant by
definition, so rescaling the input must not change the result, but today it
does. The wrong values are returned silently rather than failing, so they
propagate into similarity search results and rankings.
Affected functions: vector_cosine_similarity, vector_inner_product,
vector_l2_distance, vector_norm, vector_normalize.
Root cause is in {{VectorFunctionImplUtils}}: the accumulators, and the
per-element locals inside the 8-way unrolled blocks, are all {{float}}, so both
the individual products and the running sums overflow. {{vector_normalize}}
additionally rounds the norm to a float before dividing, which turns a norm
outside the float range into infinity and yields an all-zero vector.
The fix is to accumulate in {{double}} and round only the final result back to
{{float}}, and to keep the norm in {{double}} inside {{vector_normalize}}. This
also makes the ordinary case correctly rounded:
vector_cosine_similarity(array(1.0F, 2.0F, 3.0F), array(4.0F, 5.0F, 6.0F))
changes from 0.9746319 to 0.97463185, the correctly rounded float of
32/sqrt(1078) = 0.9746318461970763.
Not affected, and correct as-is: 0.0 for empty vectors, NULL for NULL inputs or
vectors containing NULL elements, and NULL for a genuinely zero-magnitude
vector. The problem is that a vector of tiny but non-zero magnitude is
currently treated as if it were zero.
Introduced by SPARK-54713 and SPARK-55030.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
