tgravescs commented on issue #26284: [SPARK-29415][Core]Stage Level Sched: Add base ResourceProfile and Request classes URL: https://github.com/apache/spark/pull/26284#issuecomment-548545056 thanks for reviewing. I get that its hard to see all the uses. I have the executor pieces ready - meaning you have to keep track of what resource profile the executors are launched with so the allocation manager can track them appropriately and the executor uses the profile at startup to discovery resources (discoveryScript comes in) . I'm working on the allocation manager official pieces now, probably be a few more days to finish up with that. Would it be easier if I combined this pr into one of the other pieces then? Or perhaps just a more detailed design doc? I hadn't written a detailed design doc just for the fact I didn't know if anyone would use it, if its useful I'll be happy to write it up. I could push up my original prototype - but its pretty ugly and things have changed a bit since then. just a note, the vendor is just matching the accelerator aware scheduling feature already committed. The parameter here mirror those, with the exception of the addition of the unit parameter, which is needed for memory, that isn't in the accelerator aware scheduling since pre-existed in spark. https://github.com/apache/spark/blob/master/docs/configuration.md#custom-resource-scheduling-and-configuration-overview I added another comment to the jira about this: https://issues.apache.org/jira/browse/SPARK-29415?focusedCommentId=16964360&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16964360 And honestly the vendor we could actually leave off since k8s doesn't support dynamic allocation yet, I put it in hoping that would be ready for 3.0 and adding it now would be less churn on the api. I can remove though if you prefer. Just let me know which way you think would be best.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
