atharv1205 opened a new pull request, #789:
URL: https://github.com/apache/commons-text/pull/789
## What
`CosineSimilarity.cosineSimilarity` can return values slightly outside [0.0,
1.0] for identical inputs, so `CosineDistance` returns tiny non-zero, even
negative, distances:
| Input (identical on both sides) | Before | After |
|---|---|---|
| similarity, 2 tokens | 0.9999999999999998 | 1.0 |
| similarity, 3 / 6 / 12 tokens | 1.0000000000000002 | 1.0 |
| `CosineDistance.apply("a b", "a b")` | 2.220446049250313E-16 | 0.0 |
| `CosineDistance.apply("a b c", "a b c")` | -2.220446049250313E-16 | 0.0 |
| `CosineDistance.apply("a b c d e f", "a b c d e f")` |
-2.220446049250313E-16 | 0.0 |
Over 10,000 random identical token maps, 2093 returned a similarity above
1.0 before this change.
## Why
The denominator was `Math.sqrt(d1) * Math.sqrt(d2)`. Each square root is
rounded separately and their product can be off by one ulp in either direction,
so `dot / norm` misses 1.0 for identical vectors.
## How
Compute `dotProduct / Math.sqrt(d1 * d2)`. For identical vectors `d1 * d2`
is an exact square, and `Math.sqrt` is correctly rounded, so the result is
exactly 1.0. The result is also clamped with `Math.min(1.0, ...)` so rounding
on other inputs cannot exceed 1.0. The `d1 <= 0 || d2 <= 0` branch is unchanged.
## Tests
- `CosineSimilarityTest`: identical vectors of several sizes give exactly
1.0; similarity stays within [0.0, 1.0] over generated identical and
overlapping vectors.
- `CosineDistanceTest`: identical inputs ("a", "a b", "a b c", "a b c d e
f", ...) give exactly 0.0; distance stays within [0.0, 1.0] over generated
inputs.
- The new tests fail without the runtime change (10 of 18 fail) and pass
with it. Existing tests are unchanged.
---
- [x] Read the [contribution guidelines](CONTRIBUTING.md) for this project.
- [x] Read the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) if you use
Artificial Intelligence (AI).
- [x] I used AI to create any part of, or all of, this pull request. Claude
Code assisted with investigation, tests and the fix; I reviewed and verified
every line.
- [x] Run a successful build using the default
[Maven](https://maven.apache.org/) goal with `mvn`; that's `mvn` on the command
line by itself.
- [x] Write unit tests that match behavioral changes, where the tests fail
if the changes to the runtime are not applied.
- [x] Write a pull request description that is detailed enough to understand
what the pull request does, how, and why.
- [x] Each commit in the pull request should have a meaningful subject line
and body.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
