xlb3055 opened a new pull request, #10853: URL: https://github.com/apache/paimon/pull/10853
### Purpose Fixes #10739. Python `LocalFileIO` did not sync data or temporary snapshot/schema files. On a delayed-allocation filesystem, a rename can publish a snapshot filename before its contents reach disk; after a power loss the latest snapshot may be empty and table reads fail. This change hardens the local write paths: - `try_to_write_atomic`: flush+fsync the temp file before rename; on fsync failure return `False` and do not publish; after successful rename, best-effort fsync the parent directory - `new_output_stream`: close flushes+fsyncs file data first; fsync failure is raised after the fd is closed; parent dir fsync only after successful file fsync - Parquet/ORC/Avro/Mosaic writers use `new_output_stream` so they share one file-sync path Directory fsync is best effort where a platform does not support it. No physical power-loss test was run on this host. ### Tests - `local_file_io_fsync_test.py`: 5 passed; checks sync order, fsync failure without publication, stream close behavior, and Parquet writing. - Existing `file_io_test.py` and `resolving_file_io_test.py`: 56 passed. ORC-focused shared-shredding tests: 6 passed. - Local `CatalogFactory` warehouse commit and read: 2 rows returned; 13 fsync calls during the second commit. - `git diff --check`: passed. One broader shared-shredding test could not run because `zstandard` is absent on this host. The full Python CI matrix was not run locally. AI assistance was used for investigation, implementation, tests, review, and this description. No separate human code review is claimed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
