[
https://issues.apache.org/jira/browse/YARN-9195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752008#comment-16752008
]
Shengyang Sha commented on YARN-9195:
-------------------------------------
[~leftnoteasy] I've been occupied by some urgent things previously, so sorry
for the late~
The patch based on the latest trunk is submitted in the attachment while there
is one remaining issue I'd like to ask your advice.
Once AM registerApplicationMaster,
AbstractYarnScheduler#getTransferredContainers() in RM will return all the
alive containers (except AM container) as containers from previous attempt.
Notice that in case of RM failover, RM will still return all the alive
containers while AM register itself. With respect to unmanaged AM, it's
necessary for RM to return all the alive containers to AM because RM is not
responsible for starting its AM container. But the other types of AMs don't
need to get the containers their already known when RM failover, and will thus
decrease outstanding requests in AMRMClient which is unexpected.
My question is whether we should return different result in
AbstractYarnScheduler#getTransferredContainers() based on whether it is an
unmanaged AM.
Pros:
We can prevent this case just by update RM with this bugfix and don't require
applications to update its yarn client jars.
Cons:
It will make the sematics of
RegisterApplicationMasterResponse#getContainersFromPreviousAttempts() ambiguous
between unmanaged AM and the other types.
> RM Queue's pending container number might get decreased unexpectedly or even
> become negative once RM failover
> -------------------------------------------------------------------------------------------------------------
>
> Key: YARN-9195
> URL: https://issues.apache.org/jira/browse/YARN-9195
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
> Affects Versions: 3.1.0
> Reporter: Shengyang Sha
> Assignee: Shengyang Sha
> Priority: Critical
> Attachments:
> cases_to_recreate_negative_pending_requests_scenario.diff,
> patch.YARN-9195.diff
>
>
> Hi, all:
> Previously we have encountered a serious problem in ResourceManager, we found
> that pending container number of one RM queue became negative after RM failed
> over. Since queues in RM are managed in hierarchical structure, the root
> queue's pending containers became negative at last, thus the scheduling
> process of the whole cluster became affected.
> The version of both our RM server and AMRM client in our application are
> based on yarn 3.1, and we uses AMRMClientAsync#addSchedulingRequests() method
> in our application to request resources from RM.
> After investigation, we found that the direct cause was numAllocations of
> some AMs' requests became negative after RM failed over. And there are at
> lease three necessary conditions:
> (1) Use schedulingRequests in AMRM client, and the application set zero to
> the numAllocations for a schedulingRequest. In our batch job scenario, the
> numAllocations of a schedulingRequest could turn to zero because
> theoretically we can run a full batch job using only one container.
> (2) RM failovers.
> (3) Before AM reregisters itself to RM after RM restarts, RM has already
> recovered some of the application's containers assigned before.
> Here are some more details about the implementation:
> (1) After RM recovers, RM will send all alive containers to AM once it
> re-register itself through
> RegisterApplicationMasterResponse#getContainersFromPreviousAttempts.
> (2) During registerApplicationMaster, AMRMClientImpl will
> removeFromOutstandingSchedulingRequests once AM gets
> ContainersFromPreviousAttempts without checking whether these containers have
> been assigned before. As a consequence, its outstanding requests might be
> decreased unexpectedly even if it may not become negative.
> (3) There is no sanity check in RM to validate requests from AMs.
> For better illustrating this case, I've written a test case based on the
> latest hadoop trunk, posted in the attachment. You may try case
> testAMRMClientWithNegativePendingRequestsOnRMRestart and
> testAMRMClientOnUnexpectedlyDecreasedPendingRequestsOnRMRestart .
> To solve this issue, I propose to filter allocated containers before
> removeFromOutstandingSchedulingRequests in AMRMClientImpl during
> registerApplicationMaster, and some sanity checks are also needed to prevent
> things from getting worse.
> More comments and suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
