zuston commented on PR #7121: URL: https://github.com/apache/hadoop/pull/7121#issuecomment-2507191200
> @zuston Thanks for the feedback. > > > Oh, I will we encoutered the similar problems about multi node placement about different resource spec. And I found some bugs about this features, please refer to: https://zuston.vercel.app/publish/hadoop-yarn/ > > You are right, I have proposed to fix those bugs in [YARN-9598](https://issues.apache.org/jira/browse/YARN-9598), but it was in dispute after some discussions, and part of it were merged into community in [YARN-11573](https://issues.apache.org/jira/browse/YARN-11573) which you mentioned in your article. FYI, when the scheduler found another node can place the pending request, reserved container for this request can be unreserved before assigning, you can see the details in RegularContainerAllocator#assignContainer. > > > BTW, I think the node sorting policy could be extended by ourself, there is no necessary to change the default policy. > > This PR doesn't change the default policy, just add a new policy can be configured to use. After reading the `RegularContainerAllocator#assignContainer`, I think some bugs still exist like described in my article or [YARN-11728](https://issues.apache.org/jira/browse/YARN-11728). In the below refered code, it just will pickup reserved container from another node, but obviously it don't unreserve this in the commit phase, that's the root cause of this bug. From my sight, this bug also exist in the case of multi node placement disable, which is just covered up by the normal node round-robin strategy. https://github.com/apache/hadoop/blob/964e0897d4ad828834bbb81c9f82ba92834d994c/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java#L581-L621 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
