rekbun commented on PR #59522:
URL: https://github.com/apache/spark/pull/59522#issuecomment-6586802889
Thanks @peter-toth. All points are addressed in 3231cced and 67187986, with
follow-ups in 06a5b46e, 56e153c4 and a020f22f:
1. **Cached and checkpointed plans.** Packing is now in `CacheManager`'s
list of configs turned off for cached plans, unless
`canChangeCachedPlanOutputPartitioning` is set. For checkpoints,
`withRegularShuffle` only re-plans when local pipelined shuffle applies, so
turning packing off there would not always take effect. Instead,
`QueryExecution.forCheckpoint` plans with packing off, and `Dataset.checkpoint`
passes that execution to `LogicalRDD.fromDataset`, so a checkpoint's
statistics, partitioning and ordering all come from the execution that produced
its RDD. The three SPARK-53322 checkpoint tests are back to master's
expectations and pass in the packing re-run suite. New tests cover your
cached-scan join, also with `canChangeCachedPlanOutputPartitioning` enabled,
and `checkpoint()` and `localCheckpoint()` directly. The tuning-doc paragraph
about turning packing off before materializing is replaced, and the
`canChangeCachedPlanOutputPartitioning` doc now lists scan packing.
2. **Description.** Rewritten to describe the change against master; the
measurements are in the comment above.
3. **Separate rule.** `PackV2ScanPartitions` is now its own rule, right
after `StampUnionDecisions` in both `QueryExecution.preparations` and AQE's
`queryStagePreparationRules`; AQE passes `requiredDistribution`.
`EnsureRequirements` is unchanged from master, so the shadowed `finalPlan` is
gone. The repeated-planning test runs the new rule after `EnsureRequirements`.
4. **Test tags.** Added `@ExtendedSQLTest` to the three re-run suites.
5. **Migration note.** Removed here and moved to a separate docs PR for
`master` and `branch-4.2`: #59970 (SPARK-60763).
7. **Typed accessors.** Added for the five new configs.
8. **JIRA.** SPARK-60265 now describes the size-based packing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
