malinjawi commented on issue #12377:
URL: https://github.com/apache/gluten/issues/12377#issuecomment-4979758668
@felipepessoto took me a while to get back to this, sorry — and thanks for
the error-signature quarantine in #12388, that was the right interim call.
Fixing this properly is what lets us delete `flaky-error-patterns.txt`, so I
want to get it done.
I went through the recently merged DV work (#12389, #12390, #12395) to see
if any of it explains or fixes this — none of them touch
`DeltaBitmapAggregator`, `RoaringBitmapArray::addSafe`, or the native row-index
generation, so this is still live on current main.
On the fix: I agree with your option 1. `9223372036854775807` is
`Long.MAX_VALUE` leaking out of our row-index materialization and it should
never reach the aggregator. Teaching `addInput`/`addRowIndex` to skip it like a
NULL would be worse than the abort we have
today — if the sentinel is standing in for a real matched row, that row
never makes it into the DV and quietly resurrects on the next read. Fail-fast
is right until we find the source.
One observation from your #12388 description that actually narrows it down:
the failure landing on a *different* `*DVs*Suite` MERGE test each run means it
can't be about any particular table's data — the tests share nothing except
MERGE writing DVs through the
offloaded metadata-row-index scan. That smells like a boundary condition
(split edge, batch edge, or an uninitialized row-index slot) in the native
scan's row-index column, which would also explain why a byte-identical bundle
passes one run and fails the next. I'll start there.
Two more practical notes:
- #12390 restructured the post-transform rules that shape these MERGE target
plans, so the repro conditions may have moved since it merged. Worth one run of
the `MergeIntoExtendedSyntax*DVs*` suites against current main before trusting
older
bisects.
- I'm reviving #12215, which keeps the DML row-index scan on Spark's reader.
That takes the suspected native materialization out of the path entirely — a
mitigation for the flake (and your CI) while the root cause gets fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
