Smith-Cruise opened a new issue, #1230: URL: https://github.com/apache/paimon-rust/issues/1230
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar. ### Motivation I noticed low CPU utilization when inserting data into append-only Paimon tables through DataFusion. The current write path appears to limit parallelism: DataSinkExec requires a single input partition, and PaimonDataSink::write_all() creates one TableWrite and awaits each batch write sequentially. TableWrite::write_arrow_batch() also writes partition/bucket groups sequentially. Although file closing can run in background tasks, Parquet encoding runs synchronously within the writer task, limiting CPU parallelism during ingestion. Increasing target_partitions alone does not create additional Paimon writers. An initial implementation maybe can focus on append-only INSERT INTO, leaving primary-key tables and other write modes unchanged. This should improve multicore utilization in data loading. ### Solution _No response_ ### Anything else? _No response_ ### Willingness to contribute - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
