marin-ma commented on issue #11524: URL: https://github.com/apache/incubator-gluten/issues/11524#issuecomment-3823620725
For data exchange across GPU and CPU clusters, we need to support shuffling data from CPU to GPU and vice versa: - (mapper stage on CPU cluster) CPU shuffle writer -> (reducer stage on GPU cluster) GPU shuffle reader - (mapper stage on GPU cluster) GPU shuffle writer -> (reducer stage on CPU cluster) CPU shuffle reader Currently, we only support cpu shuffle write + cpu shuffle reader, and gpu shuffle writer + gpu shuffle reader, although the output data are in the same serialisation format for cpu and gpu hash shuffle writer. And also there isn't an entry point to tell the shuffle reader whether to output velox rowvector or cudf vector, because the creation of `ColumnarShuffleExchangeExec` is in the mapper stage, but the decision of scheduling to which cluster is made in the reducer stage. One feasible approach is to separate the `ColumnarShuffleExchangeExec` into two operators `ColumnarShuffleWriteExec` and `ColumnarShuffleReadExec`, and move the deserialisation from `ColumnarShuffleReader.read` into `ColumnarShuffleReadExec.doExecuteColumnar`, then we can set which deserialiser to use in `ColumnarShuffleReadExec`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
