[
https://issues.apache.org/jira/browse/SPARK-58544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58544:
-----------------------------------
Labels: pull-request-available (was: )
> Vector distance and norm functions return NaN, Infinity or NULL due to float
> overflow in intermediate sums
> ----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58544
> URL: https://issues.apache.org/jira/browse/SPARK-58544
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: sepuri sai krishna
> Priority: Minor
> Labels: pull-request-available
>
> The vector distance and norm functions added in 4.2.0 accumulate dot products
> and sums of squares in single precision. Those intermediate quantities are
> quadratic in the input values, so they overflow to infinity (or underflow to
> zero) long before the final result leaves the float range, and the function
> silently returns a wrong value.
> {code:sql}
> SELECT vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(3.0e19F,
> 4.0e19F)); -- NaN, expected 1.0
> SELECT vector_cosine_similarity(array(1.0e-23F, 0.0F), array(1.0e-23F,
> 0.0F)); -- NULL, expected 1.0
> SELECT vector_l2_distance(array(3.0e19F, 4.0e19F), array(0.0F, 0.0F));
> -- Infinity, expected 5.0E19
> SELECT vector_norm(array(3.0e19F, 4.0e19F), 2.0F);
> -- Infinity, expected 5.0E19
> SELECT vector_normalize(array(3.0e19F, 4.0e19F), 2.0F);
> -- [0.0,0.0], expected [0.6,0.8]
> SELECT vector_normalize(array(1.0e-23F, 0.0F), 2.0F);
> -- NULL, expected [1.0,0.0]
> {code}
> ||Query||Actual||Expected||
> |vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(3.0e19F,
> 4.0e19F))|NaN|1.0|
> |vector_cosine_similarity(array(1.0e-23F, 0.0F), array(1.0e-23F,
> 0.0F))|NULL|1.0|
> |vector_l2_distance(array(3.0e19F, 4.0e19F), array(0.0F,
> 0.0F))|Infinity|5.0E19|
> |vector_norm(array(3.0e19F, 4.0e19F), 2.0F)|Infinity|5.0E19|
> |vector_normalize(array(3.0e19F, 4.0e19F), 2.0F)|[0.0, 0.0]|[0.6, 0.8]|
> |vector_normalize(array(1.0e-23F, 0.0F), 2.0F)|NULL|[1.0, 0.0]|
> In every case above the correct result is an ordinary float; only the
> intermediate sum leaves the float range: {{(3e19)^2 + (4e19)^2 = 2.5e39}}
> exceeds {{Float.MAX_VALUE}}, and {{(1e-23)^2 = 1e-46}} is below the smallest
> positive float. Cosine similarity and normalization are scale invariant by
> definition, so rescaling the input must not change the result, but today it
> does. The wrong values are returned silently rather than failing, so they
> propagate into similarity search results and rankings.
> Affected functions: vector_cosine_similarity, vector_inner_product,
> vector_l2_distance, vector_norm, vector_normalize.
> Root cause is in {{VectorFunctionImplUtils}}: the accumulators, and the
> per-element locals inside the 8-way unrolled blocks, are all {{float}}, so
> both the individual products and the running sums overflow.
> {{vector_normalize}} additionally rounds the norm to a float before dividing,
> which turns a norm outside the float range into infinity and yields an
> all-zero vector.
> The fix is to accumulate in {{double}} and round only the final result back
> to {{float}}, and to keep the norm in {{double}} inside {{vector_normalize}}.
> This also makes the ordinary case correctly rounded:
> vector_cosine_similarity(array(1.0F, 2.0F, 3.0F), array(4.0F, 5.0F, 6.0F))
> changes from 0.9746319 to 0.97463185, the correctly rounded float of
> 32/sqrt(1078) = 0.9746318461970763.
> Not affected, and correct as-is: 0.0 for empty vectors, NULL for NULL inputs
> or vectors containing NULL elements, and NULL for a genuinely zero-magnitude
> vector. The problem is that a vector of tiny but non-zero magnitude is
> currently treated as if it were zero.
> Introduced by SPARK-54713 and SPARK-55030.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
