edgarRd opened a new pull request, #18904:
URL: https://github.com/apache/iceberg/pull/18904
Closes #18901
`SparkBatch.planInputPartitions()` broadcasts the table and the `FileIO` on
every call, and `SparkScan.toBatch()` returns a new `SparkBatch` each time.
Spark plans input partitions for every copy of a scan node, during planning.
When dynamic partition pruning filters nest over a chain of joins on a
partition column, the number of copies is exponential and each leaves two
broadcasts in the driver until the plan is released. A production query with 10
such joins created 264,382 broadcasts and exhausted driver memory before
starting a stage. This PR creates the broadcasts once per scan and shares them
between its batches. It does not change how many subqueries Spark plans.
Backports to Spark 3.5, 4.0 and 4.1 will follow once this change is merged.
Test plan (`TestScanBroadcastReuse`):
- Planning input partitions again for the same scan creates no new
broadcasts. This is the behavior the change adds.
- A six-way left join on the partition column with DPP enabled stays below
the number of nested filters (2^6). This checks that Spark shares one scan
between the plan copies, which the fix depends on. Without the change the query
creates more than 400 broadcasts.
```
┌─────────────────────────────────────────────────────┬───────────────────┬─────────────────────────────┐
│ Test (Spark 4.2, upstream main) │ Before │
After │
├─────────────────────────────────────────────────────┼───────────────────┼─────────────────────────────┤
│ 10 repeated planInputPartitions() calls on one scan │ 20 new broadcasts │
0 │
├─────────────────────────────────────────────────────┼───────────────────┼─────────────────────────────┤
│ Six-way left join with nested DPP filters │ 429 broadcasts │
passes (under the 70 bound) │
└─────────────────────────────────────────────────────┴───────────────────┴─────────────────────────────┘
```
---
**AI Disclosure**
- Model: Claude Fable 5.2
- Platform/Tool: Claude Code
- Human Oversight: fully reviewed
- Prompt Summary: Investigate a Spark driver OOM caused by repeated table
and FileIO broadcasts during planning with nested dynamic partition pruning
filters, then implement and test a fix that reuses the broadcasts per scan. The
code changes and tests are generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
