JingsongLi opened a new pull request, #974: URL: https://github.com/apache/paimon-rust/pull/974
### Purpose Support `data-file.path-directory` throughout Paimon table IO so PyPaimon can use Native planning, reading, writing, updating, and committing for tables with a configured data directory. Previously, Rust built bucket paths directly under the table root. Using Native IO could therefore fail to find existing files or write files into a different layout from Java/Python. ### Brief change log - Resolve relative, normalized, absolute, and URI data directories using Java `Path` semantics, including Windows drives and UNC paths. - Use the shared data root for append/PK/changelog/Blob writers, scans, managed Blob packs, update/delete paths, abort cleanup, dynamic-bucket hash indexes, and global-index source reads. - Preserve the explicit relative bucket path for external deletion vectors instead of assuming every bucket lives under the table root. - Make physical/referenced file statistics aware of relocated data and sidecar files, including directories outside the table root. - Reject an empty directory and preserve the stored layout during table copies and catalog option changes. Catalog alterations use the existing index-layout immutability rule, including before the first snapshot, to avoid stranding an already-created writer. - Add end-to-end coverage for relocated files, updates/upserts, repeated deletes, history, cleanup, Blob payloads, dynamic buckets, and sidecar statistics. ### Tests - 35 integration tests passed: `data_file_directory_test`, `table_update_paths_test`, and `table_update_test`. - 76 focused unit tests passed: partition/path utilities, directory immutability, external paths, referenced/physical statistics, dynamic bucket assignment, and index paths. - `cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings` passed. - `cargo fmt --all --check` passed. - Built the Python binding locally and ran the companion PyPaimon Native regression selection with all five Native CI flags: 1368 passed, 2 skipped, and 62 subtests passed (PyArrow 18.1). ### API and Format Adds a `CoreOptions` accessor and options-aware file-statistics entry points. Existing statistics entry points retain their default-directory behavior. No file-format or manifest-format changes; schema, snapshot, manifest, and table-level index metadata remain rooted at the table location. ### Documentation Includes API documentation and Java-derived path expectations in regression tests. The option already exists in Java/PyPaimon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
