peter-toth opened a new pull request, #57502: URL: https://github.com/apache/spark/pull/57502
### What changes were proposed in this pull request? `AliasAwareQueryOutputOrdering.outputOrdering` and `PartitioningPreservingUnaryExecNode.outputPartitioning` (and `BroadcastHashJoinExec`'s output-partitioning expansion) build their results with `multiTransform`, which returns a `LazyList`, and store the bounded result unforced. This forces them into strict collections with `.toList`, right after the existing `.take(...)` so the lazy short-circuit during candidate generation is preserved. ### Why are the changes needed? Each plan node re-wraps the child ordering/partitioning's unforced `LazyList`, so across a deep projection chain the deferred nesting grows unbounded and overflows the stack when the ordering/partitioning is later serialized or deeply traversed. Concretely, task serialization of a `SortedMergeCoalescedRDD` (SPARK-55715) that captures the child `outputOrdering` was observed to fail on the driver with a `StackOverflowError` at `DAGScheduler.submitMissingTasks` on a large bucketed `MERGE`. The lazy `LazyList` in `SortOrder.sameOrderExpressions` (and in the projected `PartitioningCollection`) is the root cause; this has existed since the `multiTransform` path was introduced in SPARK-42049. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New test in `ProjectedOrderingAndPartitioningSuite` asserting `sameOrderExpressions` and the projected `PartitioningCollection.partitionings` are strict (not `LazyList`); it fails before this change (they were `LazyList`) and passes after. Existing `ProjectedOrderingAndPartitioningSuite` / `PlannerSuite` / `EnsureRequirementsSuite` pass. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
