thswlsqls opened a new issue, #9559: URL: https://github.com/apache/paimon/issues/9559
**Search before asking** - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. **Paimon version** master @ 2788fe596 (2.1-SNAPSHOT) **Compute Engine** Engine-agnostic (core) — any engine writing `file.format = vortex` **Minimal reproduce step** `VortexFileFormat.createWriterFactory()` builds `new ArrowFormatCWriter(type, formatContext.writeBatchSize(), true)`, leaving `memoryUsedMaxInVSR` null, so `ArrowFormatWriter.write()` skips the `write.batch-memory` check and only `write.batch-size` bounds a batch. Is dropping that argument intended, or an oversight of the JNI refactor? Three observations: - #7543 introduced the writer as `new ArrowFormatWriter(type, writeBatchSize(), true, writeBatchMemory().getBytes())`; #8040 swapped in `ArrowFormatCWriter` and the limit was not carried over. - The five-argument `ArrowFormatCWriter(RowType, int, boolean, Long, RowType)` constructor already existed at that commit, so no API was missing. - Among the Arrow-VSR-backed native writers, `OrcFileFormat`, `LanceFileFormat` and `MosaicRecordsWriter` all consume `formatContext.writeBatchMemory()`; vortex is the only one that does not. **What doesn't meet your expectations?** Lowering `write.batch-memory` bounds Arrow batch memory for the other three writers but is silently ignored for vortex. Defaults (128 MB) behave the same either way. **Anything else?** N/A **Are you willing to submit a PR?** - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
