goutamadwant opened a new issue, #12336:
URL: https://github.com/apache/seatunnel/issues/12336
### Search before asking
- [x] I searched existing issues and pull requests for vector overflow,
underflow, norm and cosine precision. I found no matching fix for the
intermediate arithmetic in the SQL and Calcite vector functions.
### What happened
The vector functions already return or accumulate doubles, but several
products and differences are evaluated as floats first. Valid finite float
inputs can therefore overflow to infinity or underflow to zero before widening
to double. Widening the result afterward cannot recover the lost value.
The affected functions are `VECTOR_NORM`, `INNER_PRODUCT`,
`COSINE_DISTANCE`, `L1_DISTANCE`, `L2_DISTANCE` and `VECTOR_NORMALIZE`. The
same evaluation order appears in SQL's `VectorFunction` and the separate
Calcite transform's six equivalent UDFs.
This matters for unnormalized numerical features and very small vectors; it
is not a claim that ordinary normalized embedding outputs routinely approach
the float limits.
### SeaTunnel Version
Reproduced against `dev` commit `1325a44b2f4013198fdb2c12f9df345de1cde4bc`
(3.0.0-SNAPSHOT). Released versions were not separately tested.
### Reproduction
For the existing SQL implementation:
```java
VectorFunction.vectorNorm(
Collections.<Object>singletonList(new Float[] {Float.MAX_VALUE}));
// Actual: Infinity
// Expected: 3.4028234663852886E38
VectorFunction.vectorNorm(
Collections.<Object>singletonList(new Float[] {Float.MIN_VALUE}));
// Actual: 0.0
// Expected: 1.401298464324817E-45
```
Other regression cases cover opposing large values in L1/L2 distances,
cancelling large dot products, and tiny nonzero vectors incorrectly taking the
zero-norm cosine branch. Expected products use exact BigDecimal representations
of float inputs; reference norms use `Math.hypot`.
These baseline tests use Float arrays so they do not depend on the separate
input-buffer consumption fix.
### SeaTunnel Config
The complete [SQL vector
fixture](https://github.com/goutamadwant/seatunnel/blob/c35c58455d30c3d7122440abed13752179088a14/seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2/src/test/resources/sql_transform/func_vector.conf)
uses FakeSource arrays:
```text
large = [3.0e38, -3.0e38]
opposite = [-3.0e38, 3.0e38]
tiny = [1.0e-30, -1.0e-30]
```
It checks ten numeric bounds through SQL CASE expressions and an Assert
sink. The [Calcite
fixture](https://github.com/goutamadwant/seatunnel/blob/c35c58455d30c3d7122440abed13752179088a14/seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2/src/test/resources/calcite_transform/calcite_builtin_functions.conf)
covers the equivalent separate transform.
### Running Command
With FakeSource, Assert and the transform plugin installed, save a linked
fixture as `vector-arithmetic.conf` and run:
```sh
sh bin/seatunnel.sh -c vector-arithmetic.conf -e local
```
Regression tests on the fix branch:
```sh
./mvnw -B -pl seatunnel-transforms-v2 -am package \
'-Dtest=VectorFunctionTest,SQLVectorFunctionTest,VectorUdfTest,CalciteSQLEngineTest'
\
-Dsurefire.failIfNoSpecifiedTests=false -Dskip.ui=true
```
### Error Exception
The arithmetic normally returns an incorrect value rather than throwing. In
the unchanged-production baseline, the SQL regression group had 7 failures out
of 28 tests, and the Calcite UDF group had 7 failures out of 40 tests, on both
Java 8 and Java 11.
In an actual Zeta runtime check, all ten arithmetic checks failed with the
original transform JAR. The Assert sink rejected the incorrect results. Both
SQL and Calcite checks passed with the fixed combined JAR.
### Proposed fix, before and after
Widen an operand before each affected multiplication or subtraction, for
example `(double) value * value` and `(double) left - right`. Keep the existing
double accumulators and control flow.
- Before: finite inputs can produce incorrect infinity, NaN, zero norms or
zero normalized elements because an intermediate float operation overflows or
underflows.
- After: the affected products and differences use double precision; the
existing return types can represent the finite results across the float input
domain.
- Apply the same correction to SQL and Calcite, without adding a separate
algorithm, dependency or vector API.
- Extend existing test classes, both E2E fixtures, EN/ZH function docs and
the incompatible-changes pages.
### Validation and advantages
Full transform-reactor builds pass on both Java 8 and Java 11, with 1,177
transform tests on each and no failures, errors or skips. Tests include float
maximum, normal/subnormal minimum, ordinary `0.1f` precision, cancellation,
NULL, actual zero and nonfinite inputs. SQL entry-point tests use fresh buffers
independently of the read-side fix.
The combined changes also pass the affected suite on both JDKs and isolated
Java 11 Zeta runtime checks. Runtime validation used an earlier 3.0.0-SNAPSHOT
distribution with the current transform JAR overlaid; it is not the complete
Flink/Spark/Testcontainers matrix.
The benefit is correct distance, norm and normalization results for accepted
finite data. No throughput improvement is claimed.
### Breaking changes
This is a numerical behavior correction. Existing results, including
low-order digits for ordinary inputs, can change. Review thresholds and
previously computed metrics or normalized vectors before replaying data.
No API, option, schema, dependency or state-format changes. Actual
zero-vector and NULL handling remain in place. Normalized output remains float,
so output rounding or underflow is still possible. No nonfinite validation
policy is introduced, but fixing finite intermediates can affect mixed
nonfinite results: cosine for a tiny finite vector and an infinite vector
becomes NaN instead of the accidental zero-norm result 1. Random projection is
outside scope.
### Zeta or Flink or Spark Version
Zeta 3.0.0-SNAPSHOT for targeted runtime checks of SQL and Calcite
transforms. Flink/Spark integration execution is not claimed.
### Java or Scala Version
Java 8u172 and Temurin 11.0.19 for reproduction and full transform tests.
Java 11 for the isolated Zeta runtime checks.
### Are you willing to submit PR?
- [x] Yes, the fix and regression coverage are prepared.
### Code of Conduct
- [x] I agree to follow the project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
