marin-ma opened a new pull request, #12511: URL: https://github.com/apache/gluten/pull/12511
### Background When GPU execution is enabled, a Spark stage can run entirely on the GPU only when every operator in that stage supports cuDF. If any operator falls back, the whole stage must use the CPU execution path. However, the mapper and reducer sides of a shuffle are separate stages and may use different execution modes. For example: * A scan stage may fall back and execute with the CPU pipeline. * A downstream join stage may still be fully supported by cuDF and execute with the GPU pipeline. Previously, this CPU-to-GPU transition could fail. The `ColumnarShuffleExchangeExec` was configured based on the mapper-side plan and was not marked for GPU execution when the scan stage fell back. The downstream reducer stage could nevertheless require the GPU shuffle reader, resulting in inconsistent shuffle execution modes. ### Changes This PR introduces explicit execution modes for Spark stages and determines the mapper and reducer modes independently at each shuffle boundary. For the example above: * The scan stages are marked as `CPUStageMode` and use the CPU pipeline and CPU shuffle writer. * The downstream join stage is marked as `GPUStageMode` and uses the GPU shuffle reader and GPU pipeline. The selected execution modes are propagated through the shuffle exchange so that the shuffle writer, shuffle reader, and batch-resizing logic use the correct implementation for their respective stages. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
