JingsongLi commented on PR #1352:
URL: https://github.com/apache/paimon-rust/pull/1352#issuecomment-6785213948
Requirement fit: SUPPORTED. The earlier NaN/count-limit issue and quadratic
late-update regression are fixed. I ran 108 aggregator unit tests, 130 related
integration tests, a real write/commit/compaction NaN regression, and an
optimized 20,000-update probe (about 2.5–2.8 ms on this revision versus 625–655
ms on 99a2bbed).
**[P2] Resolve the normal Java write-buffer compatibility boundary for
nested maps** (`crates/paimon/src/table/aggregator/collect.rs:409-413,
446-458`). There is still a reproducible cross-engine difference for a collect
column of type `ARRAY<ARRAY<MAP<INT,INT>>>` with `distinct=true` and
`count-limit=2`. Write three rows for one key, whose collect inputs contain
respectively A=`[{1:10,2:20}]`, B=`[{2:20,1:10}]`, and C=`[{3:30}]`. Rust's
real table write/flush/read path persists A,C. Current Java
`SortBufferWriteBuffer` + `AggregateMergeFunction` persists A,B. Its buffer
serializes the originally generic inputs, and the generated equaliser compares
binary nested arrays by bytes when they contain no floating fields; the new
recursive unordered-map equality collapses A/B instead. This can change the
capped persisted value according to which engine writes the same input.
Java's direct generic-array path also returns A,C, so this is a
representation-dependent Java shortcut, not a request to blindly copy that
behavior. Before treating nested collect as production Java parity, please
establish the intended compatibility contract, cover the actual Java
buffered/persisted path in the cross-engine regression, and either resolve the
upstream equaliser inconsistency with a clear version boundary or document and
scope this divergence explicitly.
Java reference used: current master `FieldCollectAgg` blob
`0620f3103136df1b255d83e686952044216b4181`; generic, binary and actual
write-buffer probes were compared. The other NaN/signed-zero/map semantic cases
passed. Strict MkDocs build also passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]