JingsongLi commented on code in PR #1329: URL: https://github.com/apache/paimon-rust/pull/1329#discussion_r4478750976
########## docs/src/c-binding.md: ########## @@ -802,6 +802,12 @@ batches, file indexes, caller-owned Arrow input, and allocation overhead are not fully charged. A reservation failure does not automatically flush or retry a writer; handle the error as a failed write operation. +For formats that support flushing, a data file also flushes when its own +retained-input reservation reaches `write-buffer-size`, even if encoding (for Review Comment: [P2] Document the buffer option that controls this flush The public table writer gets the threshold from `CoreOptions::write_parquet_buffer_size()`, which reads `write.parquet-buffer-size` and defaults to 256 MiB. The documented `write-buffer-size` key is read by the sorted global-index builder and leaves this file-writer threshold at its default. I reproduced the difference through `Table::new_write_builder().with_resources(...)` with a 6 MiB budget and distinct 4 MiB dictionary-friendly batches. With the documented key set to 1 MiB, the second write fails with `ResourceExhausted`. With `write.parquet-buffer-size=1 MB`, all four batches write, commit, and scan back as 4,096 rows with the expected payload. Please use `write.parquet-buffer-size` here and cover the option through the public table writer so the test also exercises configuration wiring. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
