tgravescs commented on issue #26682: [SPARK-29306][CORE] Stage Level Sched: Executors need to track what ResourceProfile they are created with URL: https://github.com/apache/spark/pull/26682#issuecomment-569810603 I put up a pull request that has most of the changes for the stage level scheduling features. you can use it as reference: https://github.com/apache/spark/pull/27053 I updated this pr based on that. I did find a few issues with the changes we made to the ResourceProfile, ExecutorResourceRequest(s), and TaskResourceRequest(s) so I included those changes here. The main issue is that the things the ResourceProfile and requests are not long immutable. That meant that someone could change them after calling RDD.withResources and weird things would happen. To prevent that I created a new ImmutableResourceProfile class with the intention that the user can manipulate the REsourceProfile and requests in it all they want. Once they call rdd.withResources with that ResourceProfile spark internally creates the ImmutableResourceProfile from it and that is used internally so we don't have to worry about it changing. The other thing I discovered was that having the user specify "resource.gpu" for the custom resources caused a lot of problems when comparing it to just the application level resources where it was just tracked as "gpu". So I ended up removing that requirement and since we have the ExecutorResourceREquests class where you can use .cores, .memory, .resources... I think that should be fine because its an obvious api. @squito, if you have ideas on better way to split this up into smaller prs let me know. For instance if it would make more sense to do the executor side with the dynamic allocation manager I could change it to have both.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
