laxmanmamididd commented on issue #3836: URL: https://github.com/apache/celeborn/issues/3836#issuecomment-5497791487
Found the actual root cause on our end — this isn't a Celeborn bug, it's a config gap in our own deployment. `celeborn.storage.availableTypes` defaults to `HDD` (per `CelebornConf.ACTIVE_STORAGE_TYPES`, category applies to master/worker/**client**) if left unset. Our worker/master config correctly set it to `SSD,S3`, but we never set the matching client-side Spark conf (`spark.celeborn.storage.availableTypes`) — so the client fell back to Celeborn's `HDD` default, and that mismatch is what produced a revived `PartitionLocation` hinted `HDD` that our worker's `SSD,S3` create-file-order list doesn't contain. Setting `spark.celeborn.storage.availableTypes=SSD,S3` on the client resolved it — confirmed with a full ~200GB shuffle job completing successfully end to end, where every previous attempt (without the fix) crashed the same way. Sorry for the noise, and thanks for Celeborn! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
