marin-ma commented on PR #6671: URL: https://github.com/apache/incubator-gluten/pull/6671#issuecomment-2314625022
@kerwin-zk Could you help to check whether this patch could work? Besides, I noticed it requires some extra configurations to be set for celeborn. Here's what I found and please help to check if there are anything else needed to fully enable sort-based shuffle with celeborn. Thanks! ``` --conf spark.shuffle.manager=celeborn \ --conf spark.celeborn.client.spark.shuffle.writer=sort \ --conf spark.celeborn.client.shuffle.compression.codec=zstd ``` cc: @FelixYBW -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
