r7raul1984 opened a new issue, #13027:
URL: https://github.com/apache/gluten/issues/13027
### Backend
VL (Velox)
### Bug description
Bug description:
[Expected behavior] When
spark.celeborn.client.spark.shuffle.getReducerFileGroup.broadcast.enabled=true,
Gluten's CelebornShuffleManager should successfully look up
ControlMessages$GetReducerFileGroupResponse via reflection and
register the serialization callback, allowing executors to load shuffle
file groups via broadcast.
[Actual behavior] CelebornUtils.GET_REDUCER_FILE_GROUP_RESPONSE_NAME uses
. instead of $ as the inner class separator:
// Wrong
"org.apache.celeborn.common.protocol.message.ControlMessages.GetReducerFileGroupResponse"
// Correct
"org.apache.celeborn.common.protocol.message.ControlMessages$GetReducerFileGroupResponse"
Class.forName() throws ClassNotFoundException, falls back to Object.class,
and SparkUtils.getMethod("serializeGetReducerFileGroupResponse", Integer.class,
Object.class) throws NoSuchMethodException. This propagates as a
RuntimeException that prevents the LifecycleManager from broadcasting file
group info, causing all executors to hit CelebornIOException: Failed to load
file group → FetchFailedException → stage fails 4 times → job aborted.
Trigger conditions (all three must be true):
1. spark.shuffle.manager =
org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager
2. Celeborn client ≥ 0.6.0
3.
spark.celeborn.client.spark.shuffle.getReducerFileGroup.broadcast.enabled = true
Workaround: Set
spark.celeborn.client.spark.shuffle.getReducerFileGroup.broadcast.enabled=false.
Spark version: Spark-3.5.x
Relevant logs:
Caused by: org.apache.spark.shuffle.FetchFailedException:
Celeborn FetchFailure appShuffleId/shuffleId: 3/7
Caused by: org.apache.celeborn.common.exception.CelebornIOException:
Failed to load file group of shuffle 7 partition 139!
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.spark.shuffle.celeborn.SparkUtils
.serializeGetReducerFileGroupResponse(java.lang.Integer,
java.lang.Object)
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
