gavin9402 opened a new pull request, #10457:
URL: https://github.com/apache/paimon/pull/10457
## Change made
While investigating a Spark query over a Paimon data-evolution table
containing Blob columns, we observed that the scan stage launched only one task
despite reading a large amount of data.
Data-evolution split planning preserves row-id groups. Files belonging to
the same group can therefore remain in a single large split, which Spark-side
bin packing cannot divide into independent scan partitions. Operators pipelined
with that scan also run with limited parallelism.
This PR adds an optional shuffle immediately after the scan so downstream
operators can run across more partitions. It preserves the source split
boundaries required by the reader.
## Changes
Add the `RepartitionLargePaimonScan` optimizer rule:
- Run after V2 scan pushdown, when the Paimon scan and its input partitions
are available.
- Reuse `BinPackingSplits.computeMaxSplitBytes` to calculate the target
partition size.
- Insert `Repartition(..., shuffle = true)` when any input partition exceeds
that size.
- Calculate the partition count as:
```text
ceil(total bytes across all input partitions / target partition size)
```
- Skip streaming scans and preserve an existing shuffle directly above the
scan.
- Avoid inserting duplicate repartitions when the rule runs repeatedly.
Add a connector option, **disabled by default**:
```sql
SET spark.paimon.read.repartition-large-scan.enabled = true;
```
When disabled, the rule returns the original plan without planning scan
splits or input partitions.
## Scope
This change increases **downstream parallelism**; it does not increase the
number of source scan tasks or split row-id groups.
Partition sizes come from file metadata, including Blob file sizes. They do
not represent decoded row sizes or measured shuffle bytes.
## Tests
All 11 Paimon unit tests pass, covering:
- Default and explicit disablement without scan planning.
- Oversized-partition detection and exact-threshold behavior.
- Total-byte accounting and rounding up.
- Bin-packing configuration precedence and file-open costs.
- Existing shuffles, empty scans, repeated application, and partition-count
overflow.
- Rule registration after scan pushdown and unchanged non-Paimon plans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
