TheR1sing3un opened a new pull request, #8637: URL: https://github.com/apache/paimon/pull/8637
### Purpose `read.parallelism` defaulted to 1 (serial), the bottleneck for multi-split `to_arrow`/`to_pandas`/`to_duckdb` reads done directly through pypaimon. This adds an opt-in `"auto"` value that resolves to `min(number of splits, CPU count)`, accepted both as the `read.parallelism` option and as the runtime `parallelism` argument. The global default stays 1 so distributed engines (Ray/Daft), which already parallelize across splits at the outer layer, do not double-parallelize. `READ_PARALLELISM` becomes a string option to carry `"auto"`; on-disk table options are already stored as strings, so existing int values keep working and cross-engine schemas are unaffected. ### Tests New unit tests for parallelism parsing (`auto`, integer strings, invalid values, case/whitespace) plus end-to-end `auto`-vs-serial parity tests via both the runtime argument and the table option in `reader_parallel_test.py`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
