JingsongLi commented on PR #387: URL: https://github.com/apache/paimon-rust/pull/387#issuecomment-4714393531
Thanks for catching this. I updated the Vortex writer so it now tracks the queued RecordBatch memory estimate while arrays are staged before close. `num_bytes()` now reports that pre-close estimate (or the final written size after close), and `in_progress_size()` also reflects the staged bytes, so `target-file-size` rolling can observe progress again. I also added a Vortex-specific regression test for `target-file-size = 1b`; it failed with 1 data file before the fix and now produces 2 as expected. Local verification: - `cargo test -p paimon --features fulltext,vortex table::table_write::tests::test_vortex_write_rolling_on_target_file_size -- --nocapture` - `cargo test -p paimon --features fulltext,vortex arrow::format::vortex::tests:: -- --test-threads=8 --nocapture` - `cargo test -p paimon --all-targets --features fulltext,vortex` - `cargo clippy --all-targets --workspace --features fulltext,vortex -- -D warnings` - `cargo fmt --all -- --check` - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
