jackylee-ch commented on PR #975: URL: https://github.com/apache/paimon-rust/pull/975#issuecomment-5944425018
Addressed by guarding the schema change, keeping the permissive read for genuinely-ragged files. Permissive truncation is correct when a row has an extra *trailing* field, but a CSV format table reads files positionally, so dropping a non-trailing column shrinks the schema below the old files' width and the reader truncates the wrong field — exactly the silent mis-decode you reproduced. `ALTER TABLE events DROP COLUMN label` now fails up front with "it shifts the physical column positions of existing files" instead of making `SELECT extra` return `wrong`. Dropping the trailing column stays aligned and is still allowed, and the trailing ADD COLUMN padding is unaffected. Regression `test_csv_format_table_rejects_dropping_non_trailing_column`: on a CSV format table `(id, label, extra)`, dropping `label` (non-trailing) is rejected while dropping `extra` (trailing) succeeds. I verified non-vacuity: disabling the guard lets the middle-column drop succeed and the test fails; the guard rejects it. The guard lives in `apply_changes`' `DropColumn` arm, which create and alter share. Rebased onto current main. The schema tests pass; `clippy -p paimon --all-targets -D warnings` is clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
