JingsongLi commented on PR #975: URL: https://github.com/apache/paimon-rust/pull/975#issuecomment-5935100040
[P2] Preserve CSV physical column positions before truncating old rows The unconditional prefix truncation in `crates/paimon/src/arrow/format/text.rs:860` can turn a previously visible schema incompatibility into incorrect successful reads. CSV Format Tables use the latest schema for existing files, while SQL currently permits dropping non-key columns without a positional-format guard. I reproduced this through a real filesystem Catalog and SQLContext: ```sql -- CSV Format Table columns: id BIGINT, label VARCHAR, extra VARCHAR -- Existing file contains: 1,wrong,present SELECT extra FROM events; -- 'present' ALTER TABLE events DROP COLUMN label; -- succeeds SELECT extra FROM events; -- 'wrong' on this head SELECT id FROM events WHERE extra = 'present'; -- returns no rows ``` After the middle column is removed, the new two-column decoder truncates the old three-column row to `(1,wrong)` and assigns the deleted column's value to the surviving `extra` column. Restoring `text.rs` to the exact baseline makes the post-ALTER read reject `CSV row has 3 fields, expected 2` instead of silently returning the wrong value. Please guard schema changes that shift CSV physical positions, or preserve/map the original positions before allowing truncation. The trailing ADD COLUMN padding can remain supported. Java also uses positional-prefix decoding, so this finding concerns the demonstrated Rust regression, not a claim of different Java output. Validation on head `d874f91bf87e84801c7258ec044cc6f53a273977`: 8 text-format tests passed, and the real CSV ADD COLUMN/short-row/extra-field SQL scenario passed alongside the existing partition-filter test. The additional accepted DROP-middle scenario fails the correctness probe on head; the baseline rejects the unsafe read. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
