XiaoHongbo-Hope opened a new pull request, #9357: URL: https://github.com/apache/paimon/pull/9357
## What changed - Preserve an explicitly configured `num_partitions` unchanged. - When input byte-size metadata is available, size the default from Ray's `DataContext.target_max_block_size`, capped at the existing `max(1, cluster_cpus * 2)` default. - For general MERGE, estimate the source plus the largest relevant target join projection. For `update_by_row_id` and `read_by_row_id`, estimate the source input. - Never execute a lazy Dataset just to choose parallelism. Use guarded logical-plan metadata and retain the previous CPU-based default whenever size metadata is unavailable. ## Why The previous default always used twice the cluster CPU count. On a 320-CPU cluster this creates 640 shuffle partitions even for relatively small inputs, where scheduling overhead can dominate useful work. Size-aware defaults reduce small-job overhead while retaining the existing cap and fallback for large or unknown inputs. Self-merge keeps the previous default because its file-oriented path already caps work by logical file-group count and builds its filtered scan plan later. ## Validation - Relevant Ray merge/update/read suites: 165 passed, 18 subtests passed. - Partition sizing suite: 9 passed on Ray 2.50, 2.54, and 2.57. - Python 3.6 `py_compile`, flake8, and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
