goutamadwant opened a new issue, #12362:
URL: https://github.com/apache/seatunnel/issues/12362
### Search before asking
- [x] Searched existing issues and open/closed PRs for ARRAY_MAX, ARRAY_MIN
and array precision. No matching precision fix was found.
### What happened
SQL `ARRAY_MAX` and `ARRAY_MIN` compare every numeric value through
`Number.doubleValue()`. Distinct BIGINT or DECIMAL values can therefore compare
equal, causing the function to return the wrong original element. A job can
complete successfully with incorrect output.
This affects pipelines using extrema over integral arrays with values beyond
exact double precision, or DECIMAL arrays whose distinct values round to the
same double. This is a reproduced production-code defect, not a report of a
particular customer incident.
### SeaTunnel Version
Reproduced on `dev`, commit `b37af3a9bf634735cabe4908014c2d390b288558`
(3.0.0-SNAPSHOT). Released versions were not separately tested.
### Reproduction
The following assertions can be added to the existing `ArrayFunctionTest`;
they exercise the production functions directly:
```java
Object[] longs = {9007199254740992L, 9007199254740993L};
Assertions.assertEquals(
9007199254740993L,
ArrayFunction.arrayMax(Collections.singletonList((Object) longs)));
Object[] reversed = {9007199254740993L, 9007199254740992L};
Assertions.assertEquals(
9007199254740992L,
ArrayFunction.arrayMin(Collections.singletonList((Object)
reversed)));
BigDecimal smaller = new BigDecimal("1.00000000000000000001");
BigDecimal larger = new BigDecimal("1.00000000000000000002");
Assertions.assertEquals(
larger,
ArrayFunction.arrayMax(
Collections.singletonList((Object) new Object[] {smaller,
larger})));
```
Imports are `java.math.BigDecimal`, `java.util.Collections`, and JUnit
`Assertions`.
### SeaTunnel Config
The implementation extends the existing
`sql_transform/func_array_max_min.conf` E2E fixture with two reversed BIGINT
arrays and DECIMAL values. Its SQL data path is:
1. FakeSource supplies `array<bigint>` and two scalar `decimal(38,20)`
fields.
2. A first SQL transform materializes `ARRAY(first_decimal, second_decimal)`.
3. A second SQL transform evaluates ARRAY_MAX/MIN on those array fields.
4. Assert checks exact extrema, the original integer/string cases, and
exactly two output rows.
DECIMAL arrays are deliberately built from scalar fields; native
`array<decimal(...)>` text-schema parsing is not supported by the current
parser and is not part of this fix.
### Running Command
After adding the regression assertions to the unchanged source:
```sh
./mvnw -B -pl seatunnel-transforms-v2 -am package \
-Dtest=ArrayFunctionTest,SQLNumericFunctionsTest \
-Dsurefire.failIfNoSpecifiedTests=false -Dskip.ui=true
```
Run once with Java 8 and once with Java 11 selected through JAVA_HOME.
### Error Exception / before and after
No exception is required for the defect: it is a wrong result.
- Before: maximum of `[9007199254740992, 9007199254740993]` is
`9007199254740992`; expected `9007199254740993`.
- Before: maximum of the two DECIMAL values above is
`1.00000000000000000001`; expected `1.00000000000000000002`.
- After: both extrema select the correct original element, independently of
input order.
The unchanged baseline failed three regression tests in the 38-test focused
suite on each of Java 8 and Java 11. A real Zeta negative control also failed
the exact-value assertion with the unchanged transform JAR.
### Proposed fix and advantages
- Use exact `long` comparisons when all non-null elements are integral
wrappers.
- Use `BigDecimal.compareTo` when all non-null elements are BigDecimal.
- Preserve the existing double ordering for floating-point and mixed numeric
representations, choosing one comparator for the entire array.
- Return the selected original object; preserve stable ties, null/empty
behavior and string comparisons.
This removes silent precision loss without a new dependency or a change to
ARRAY construction/type inference. Classification is one additional linear pass
with constant extra space; no measured speedup is claimed.
### Breaking changes / compatibility
No configuration, public API, declared output-type or checkpoint-format
changes. Corrected results can differ from previously rounded results. DECIMAL
values are not rescaled, but selecting a different correct original element may
also select a different original scale. FLOAT/DOUBLE NaN, infinity and
signed-zero behavior, and mixed-category fallback, remain unchanged. EN/ZH
function and compatibility documentation are included.
### Validation and Java or Scala Version
- Baseline and fixed unit coverage: Java 8u172 and Java 11.0.19.
- Fixed full transform suites: 1,164 tests on each JDK, zero
failures/errors/skips.
- Final fixture passed on a freshly built distribution with Zeta Java 8/11
and Spark 3.3 / Java 8u352.
- Combined validation with the separate Replace fix passed the 310-module
Java 11 build with tests skipped. This is not a claim that every repository
test or engine/version combination ran.
### Are you willing to submit PR?
Yes, the implementation and regression coverage are ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
