JingsongLi opened a new pull request, #10407: URL: https://github.com/apache/paimon/pull/10407
### Purpose Honor Java's `scan.ignore-lost-files` and `scan.ignore-corrupt-files` for Paimon Parquet and ROW reads, and verify the same behavior through PyPaimon Native. Depends on https://github.com/apache/paimon-rust/pull/1049. The Rust Native CI job continues to install Rust `main`; merge the Rust change before this PR's Native job can pass. - Add the Java option definitions and a small file-scoped recovery wrapper around format reader creation and decoding. - At creation, distinguish missing files from present unreadable files using the actual file's existence. During decoding, only the corrupt-files option may end the current file. - Align Data Evolution with Java: end the group when an active column reader ends rather than manufacture NULLs for unavailable committed values. - Normalize invalid compressed ROW blocks as file-decoding errors, while propagating memory/capacity, unsupported-operation and other non-file failures. - Validate `read.batch-size` before recovery or Native fallback so invalid configuration is not silently treated as an empty scan. - Keep both options disabled by default. The read policy never deletes any file. ### Tests - New Native/Python parity tests cover append, Data Evolution and PK reads; missing/corrupt files; strict and opposite options; LIMIT; damaged update columns; protected errors; ROW payload corruption; and continued reading after an already emitted batch. - Affected read regression suite with Native plan/read/write/update/commit enabled: 1,272 passed, 31 skipped, 88 subtests passed. Includes schema evolution, deletion vectors, ROW sidecars, BLOB collections and updates. All five Native update APIs and Native REST commits were exercised. - Existing `pypaimon/tests/py36` regression suite: 79 passed on the local Python 3.13 runtime. - Flake8 with `dev/cfg.ini` for changed Python files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
