ErikBPF commented on PR #5786: URL: https://github.com/apache/datafusion-comet/pull/5786#issuecomment-5655314997
The validator now preserves nested projection information and reuses the reader's structural-narrowing check. The regression reads the unique `other` field beside duplicate `dup` siblings in both case-sensitivity modes, compares Spark results, and checks the exact rows. Selected ambiguity and full-subtree decoding still fail before decoding. Embedded Arrow schema hints and synthesized Spark variant schemas retain full nested validation because they can change the decoded schema. The nested-projection regression failed before the fix. A second regression with a real dictionary-encoded Arrow schema hint failed before the conservative hint guard. Final Orion verification passed: four focused Rust tests and the full `CometNativeReaderSuite` with 71 succeeded, 0 failed, and one existing NullType cancellation (#4199 / SPARK-54220). The native build and formatting checks also passed. For merge order, I suggest landing this decoder safety guard before #5654, then rebasing #5654 and preserving rejection until its last-wins path has evidence that ambiguous leaves decode correctly. #5845 is also still open; whichever lands second needs to propagate fallible name folding through the validator, projection decision, and shared structural-narrowing helper without `unwrap`. Please coordinate that order before merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
