wangyum opened a new issue, #12555: URL: https://github.com/apache/gluten/issues/12555
## Description `ColumnarToColumnarExec` (in `gluten-core`) is the abstract base for operators that only reshape/resize/convert columnar batches without touching row distribution or order. Concrete subclasses include: - `VeloxResizeBatchesExec` (batch size resizing) - `ArrowColumnarToVeloxColumnarExec`, `OffloadArrowDataExec`, `LoadArrowDataExec` (Arrow <-> native format conversion) - `GpuResizeBufferColumnarBatchExec` These operators rely on the default `outputPartitioning` / `outputOrdering` inherited from `SparkPlan` (via `UnaryExecNode`), which resolve to `UnknownPartitioning(numColumns)` and `Nil`. This is incorrect: the wrapper is transparent to the physical layout and should propagate the child's partitioning and ordering. ## Impact Because the planner sees `UnknownPartitioning` / `Nil` after such a wrapper: 1. **Storage-partitioned joins (SPJ) / v2 bucketing** are not recognized when a batch-resize or format-convert node sits between a `BatchScanExec` (reporting `KeyGroupedPartitioning`) and the join. The planner inserts a **redundant shuffle**, defeating the partitioning that the scan reported. 2. **SortMergeJoin** plans lose the child's sort order across the wrapper and insert an **extra `Sort`**. 3. More generally, any downstream rule that relies on `outputPartitioning` / `outputOrdering` (exchange reuse, AQE coalescing, ordering-based optimizations) makes suboptimal decisions when one of these wrappers is present. This was found while adapting Gluten to a Spark fork that backports the SPARK-55535 `GroupPartitionsExec` SPJ path, where the wrapper silently regressed SPJ plans that contain a columnar reshape between scan and join. ## Proposed fix Propagate the child's physical properties in `ColumnarToColumnarExec`: ```scala override def outputPartitioning: Partitioning = child.outputPartitioning override def outputOrdering: Seq[SortOrder] = child.outputOrdering ``` with the required imports (`Partitioning`, `SortOrder`). This mirrors how vanilla Spark's `ColumnarToRowExec` / `RowToColumnarExec` and Gluten's own `ColumnarInputAdapter` treat partitioning/ordering transparency. This is a correctness/performance fix that applies to all supported Spark versions, not a specific backend. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
