felipepessoto opened a new pull request, #12783:
URL: https://github.com/apache/gluten/pull/12783
## What changes are proposed in this pull request?
**This is a diagnostic, not a fix.** #12377 stays open.
The native Delta DV bitmap aggregator intermittently aborts during a MERGE
that writes
deletion vectors, with one of two messages (both currently quarantined in
`flaky-error-patterns.txt`):
```
Delta RoaringBitmapArray row index 9223372036854775807 exceeds max
representable value 9223372030412324864
Delta bitmap row index cannot be negative: -6254810385378525259
```
Neither value can be a row index Delta produced. Delta's own JVM
`RoaringBitmapArray.add`
enforces the identical bound, so vanilla Delta would abort the same way:
```
Delta JVM MAX_REPRESENTABLE_VALUE = 9223372030412324864
Gluten native kMaxRepresentableValue = 9223372030412324864 identical = true
add(9223372036854775807) -> IllegalArgumentException: requirement failed
add(-6254810385378525259) -> IllegalArgumentException: requirement failed
```
and `BitmapAggregator.update` only skips nulls. `-6254810385378525259` is
arbitrary rather
than a sentinel, so the column feeding the aggregator is carrying garbage,
not a real row
index.
Today the failure reports a single number, which is not enough to tell
**how** the column is
broken. This attaches the input column to the error:
```
Delta RoaringBitmapArray row index 9223372036854775807 exceeds max
representable value
9223372030412324864. Delta bitmap aggregator input column: type=BIGINT
encoding=FLAT size=16
selected=16/16 mayHaveNulls=false decodedBase=FLAT identityMapping=true
constantMapping=false
nulls=0 inRange=15 outOfRange=1 inRangeMin=0 inRangeMax=15
ascendingRuns=13/14
badValues=[(row=9 value=9223372036854775807 hex=0x7fffffffffffffff)]
```
The **ascending-run ratio** is the discriminator: a genuine row-index column
with one corrupt
slot (a lost null flag or an unwritten value slot) stays almost perfectly
ascending; an
entirely wrong column does not. `encoding` / `decodedBase` / `nulls` /
`mayHaveNulls` cover the
rest of the hypothesis space.
`SimpleAggregateAdapter`'s per-row callbacks only see a single decoded
value, so the
column-level context is attached in a thin subclass that still has the raw
input vector.
Unrelated exceptions are rethrown untouched, and a failure while building
the description can
never mask the original error. **It runs only on the failure path**, so
there is no steady-state
cost.
### Why the workflow file is touched
The change is native-only under `cpp/**`, which `delta_spark_ut.yml`'s
`paths:` filter
deliberately does not cover — so without touching it this PR would not run
the Delta suite,
which is the only place the failure occurs. That edit is marked
`TEMPORARY -- REMOVE BEFORE MERGE`.
### What we already ruled out
Gluten vs vanilla `_metadata.row_index` was byte-identical on Spark
3.5/Delta 3.3 **and**
Spark 4.0/Delta 4.0 across: plain DV reads; split Parquet files; DVs wiping
whole read batches
(`maxBatchSize` 32→4096); `useMetadataRowIndex` on and off;
DELETE/UPDATE/MERGE with DVs,
partitioned and not, first-DV and repeated-over-DV; `WHEN NOT MATCHED BY
SOURCE` (rightOuter DV
join); broadcast on/off; off-heap 64m→2g. A local Delta v4.2.0 + Spark 4.1
run of
`*MergeIntoExtendedSyntax*DVs*` (200 tests, includes the reported suite)
also passed. Hence the
diagnostic rather than a speculative fix.
## How was this patch tested?
New unit tests in `DeltaBitmapAggregatorTest` covering both reported
signatures; the whole
suite passes:
```
[==========] Running 7 tests from 1 test suite.
[ PASSED ] 7 tests.
```
Both changed translation units compile clean under the project's `-Werror`
flags, and
`dev/format-cpp-code.sh` (clang-format 15.0.7) is clean.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
