tgravescs commented on pull request #31936: URL: https://github.com/apache/spark/pull/31936#issuecomment-804902211
this was tried once before under https://github.com/apache/spark/pull/26000 so just putting it here for reference. I think the main thing is if we are going to support that we make sure its truly configurable and not just half done. But it sounds like you are trying to address the concerns brought up there but I need to do a more detailed review. One thing doing a quick skim I don't see if in YarnShuffleService.java its still registering the AuxiliaryService with the name spark_shuffle, Is that not really used by the yarn auxiliary service and it uses what you configure in the node manager? In the very least that needs to be documented in the code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
