peter-toth opened a new pull request, #57533: URL: https://github.com/apache/spark/pull/57533
### What changes were proposed in this pull request? Backport of SPARK-58323 (#57502) to branch-3.5. `AliasAwareQueryOutputOrdering.outputOrdering`, `PartitioningPreservingUnaryExecNode.outputPartitioning` and `BroadcastHashJoinExec`'s output-partitioning expansion build their results with `multiTransform`, which returns a `Stream` (Scala 2.12), and store the bounded result unforced. This forces them into strict collections with `.toList`, right after the existing `.take(...)` so the lazy short-circuit during candidate generation is preserved. Tailored for branch-3.5 (differs from the master PR): - Scala 2.12: the lazy collection here is `Stream`, not `LazyList` (its 2.13 successor). - `PartitioningPreservingUnaryExecNode.outputPartitioning` IS fixed on 3.5, unlike the 4.0/4.1 backports: on Scala 2.12 the outer `Iterator.toSeq` there is lazy (a `Stream`), so the projected `PartitioningCollection` stored an unforced `Stream`. On Scala 2.13 the same `Iterator.toSeq` is strict (`List`), so 4.0/4.1 need no change there. - The `BroadcastJoinSuite` single-element assertion change is omitted: on branch-3.5 `expandOutputPartitioning` always returns a `PartitioningCollection` (no `case p :: Nil` unwrap), so the single-element output shape is unchanged. ### Why are the changes needed? Each plan node re-wraps the child ordering/partitioning's unforced `Stream`, so across a deep projection chain the deferred nesting grows unbounded and overflows the stack when the ordering/partitioning is later serialized or deeply traversed. On master this was observed as a driver `StackOverflowError` at `DAGScheduler.submitMissingTasks` during task serialization of a `SortedMergeCoalescedRDD` (SPARK-55715) on a large bucketed `MERGE`. That RDD and code path do not exist on branch-3.5, so this backport is defensive hardening of the same latent lazy-collection-in-output-ordering/partitioning hazard, which has existed since the `multiTransform` path was introduced in SPARK-42049. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New test in `ProjectedOrderingAndPartitioningSuite` asserting both `sameOrderExpressions` and the projected `PartitioningCollection.partitionings` are strict (not a `Stream`); it fails before this change and passes after. Existing `ProjectedOrderingAndPartitioningSuite` / `BroadcastJoinSuite` / `BroadcastJoinSuiteAE` pass. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
