Ngone51 commented on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841#issuecomment-505930056 > how to avoid launch multiple workers on the same worker node (which will cause resources conflicts)? The current plan is to let Master be the role of coordinator to avoid resources conflicts. So, multiple workers are still allowed to be launched on the same node. But if the available resources can't meet the worker's requested amount, it will fail to launch. >If the discovery script is the dynamic allocation script for the launching worker, then should we limit the script to allocate exactly the same amount resources with requested amount ? Allocate more amount than requested will be wasted because we're possible to launch multiple workers on the same node ? If Master is a coordinator, then the superfluous resources could be allocated to other workers launched on the same node. If no more worker launches, then, those resources could be wasted. Another possibility is, setting the requested amount to the size of resources found by script if user do not config the resources amount explicitly(similar like cpu, memory do). In this way, once a worker launches on the node, then, no more other worker(using the same script) could launch on the same node(as the first worker has already grabbed all the resources). This may could achieve the goal of issue 1.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
