squito commented on issue #26284: [SPARK-29415][Core]Stage Level Sched: Add base ResourceProfile and Request classes URL: https://github.com/apache/spark/pull/26284#issuecomment-553468948 > Note this current implementation also is not smart about comparing ResourceProfiles. I hadn't really thought about this part before -- but this seems like a pretty confusing part of the api, and I'm not totally sure how to work around it. Imagine you put a resource profile on one stage requesting 8 GB of memory. Later on, somebody adds a resource profile on an earlier stage requesting 9 GB of memory. You wouldn't really want to tear down the executors w/ 9 GB of memory to get some with 8 GB of memory, would you? OTOH, if do re-use the earlier executors, but then you need to request more executors after that, you'd end up with a mix of executors with different resources. It should work fine, but we'd probably want to run some tests. And it doesn't have to be two resource profiles -- it could be that there is only one resource profile in the code, but then somebody bumps up the command line args to 9GB executors for the entire app. Though in other cases it could be a problem if you are way over-allocating. Eg. one stage requested 100 GB of memory -- I guess you would want to tear those down to get containers closer to the requested size. maybe we can't do anything better here. It is asking the user to be pretty smart about their use of ResourceProfiles across their entire application, they can't really think about them in isolation.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
