[
https://issues.apache.org/jira/browse/FLINK-32133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17724829#comment-17724829
]
Liu commented on FLINK-32133:
-----------------------------
[~huwh] Thanks for the detail analysis. Although the problem is caused by yarn,
we should try to reduce the effect by all means. In our situation, the excess
containers may cause serious resource balance problem. For example, we need
3000 containers but received 5000 containers. The 5000 containers are balanced
among the cluster. After returning 2000 containers which may located at
certain hosts, these hosts may be free while others are busy.
The solution are try-best but not perfect:
# For the batch requests, we can compute the required resources at start for
streaming mode(this is our situation). For the batch mode, I think we can
change the incremental request to batch request when scheduling each phase.
After loop calling resourceManagerClient.addContainerRequest, call
resourceManagerClient.setHeartbeatInterval(containerRequestHeartbeatIntervalMillis).
This may alleviate the problem greatly especially when there are a lots of
requests.
# For removing the container requests, flink removes the requests gradually
every time some containers are allocated in method onContainersAllocated. This
process may repeat multi times which may cause excess container allocated. We
can remove container requests only when all the resources are satisfied to
avoid the problem.
This issue aims to try best to alleviate the problem. After the modification in
our company, we can hardly see the excess problem.
> Batch requests and remove requests in the end to reduce YarnResourceManager's
> excess containers
> -----------------------------------------------------------------------------------------------
>
> Key: FLINK-32133
> URL: https://issues.apache.org/jira/browse/FLINK-32133
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
> Affects Versions: 1.18.0
> Reporter: Liu
> Priority: Major
>
> h1. Problem
> When the initial job requests many containers from yarn, it is easy to get
> more than needed containers for that the YARN AM-RM protocol is not a delta
> protocol (please see YARN-1902). For example, we are needing 3000 containers.
> Consider the following case.
> *Case one:*
> # The job requests 2000 containers firstly and then the yarn client has 2000
> requests.
> # {color:#FF0000}The yarn heartbeat happens and the yarn client +request
> 2000 containers+ to yarn rm.{color}
> # The job requests another 1000 containers and the the yarn client has 3000
> requests.
> # {color:#FF0000}The yarn heartbeat happens and the yarn client +request
> 3000 containers+ to yarn rm.{color}
> # On heartbeat finish, yarn rm {+}returns 2000 containers{+}. After the
> callback the method onContainersAllocated and removeContainerRequest, yarn
> client has 1000 requests.
> # {color:#FF0000}The yarn heartbeat happens and the yarn client +request
> 1000 containers+ to yarn rm. {color}
> # On heartbeat finish, yarn rm {+}returns 3000 containers{+}. After the
> callback the method onContainersAllocated and removeContainerRequest, yarn
> client has 0 requests.
> # {color:#FF0000}The yarn heartbeat happens.{color}
> # On heartbeat finish, yarn rm +returns 1000 containers+{color:#FF0000}
> {color:#172b4d}which are excess since the last client request number is
> 1000.{color}{color}
> {color:#172b4d}In the end, the yarn may allocate 2000 + 3000 + 1000 = 6000
> containers. But we only need 3000 containers and should return 3000
> containers.{color}
> *{color:#172b4d}Case two:{color}*
> # {color:#172b4d}The job requests 3000 containers firstly and the the yarn
> client has 3000 requests.{color}
> # {color:#FF0000}The yarn heartbeat happens and the yarn client +request
> 3000 containers+ to yarn rm.{color}
> # On heartbeat finish, yarn rm {+}returns 1000 containers({+}2000
> allocating{+}){+}. After the callback the method onContainersAllocated and
> removeContainerRequest, yarn client has 2000 requests.
> # {color:#FF0000}The yarn heartbeat happens and the yarn client +request
> 2000 containers+ to yarn rm.{color}
> # On heartbeat finish, yarn rm {+}returns 2000 containers{+}. After the
> callback the method onContainersAllocated and removeContainerRequest, yarn
> client has 0 requests.
> # {color:#FF0000}The yarn heartbeat happens.{color}
> # On heartbeat finish, yarn rm +returns 2000 containers+{color:#FF0000}
> {color:#172b4d}which are excess since the last client request number is
> 2000.{color}{color}
> {color:#172b4d}In the end, the yarn may allocate 1000 + 2000 + 2000 = 5000
> containers. But we only need 3000 containers and should return 2000
> containers.{color}
> {color:#172b4d}The reason is that any update to the yarn client's requests
> may produce undesired behavior. {color}
> h1. {color:#172b4d}Solution{color}
> {color:#172b4d}In our inner flink version, we use two ways to resolve the
> problem as following:{color}
> # {color:#172b4d}{color:#172b4d}Compute the total resource requests at start
> and request by batch{color}{color}{color:#172b4d} to avoid being interrupted
> by yarn heartbeat. Note that we {color}{color:#172b4d}{color:#172b4d}loop
> {color}{color}resourceManagerClient.addContainerRequest(containerRequest){color:#172b4d})
> to simulate batch-request quickly.{color}
> # {color:#172b4d}Remove the yarn client's container requests after receiving
> enough resources to avoid request update.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
