[
https://issues.apache.org/jira/browse/YARN-8428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16514236#comment-16514236
]
Eric Yang commented on YARN-8428:
---------------------------------
YARN service AM requires a ZooKeeper connection to update YARN registry
regarding the service state. This requires to keep one ZooKeeper connection
open per AM. Instead of maintaining persistent ZooKeeper connection, code was
changed to create connection on demand and close connection as soon as the task
is completed. However, using dynamic connections, the number of ZooKeeper
sockets jump from 1 per AM to 8 per AM due to activities to check Service
registry states. Unless YARN registry design is drastically changed,
otherwise, we can not reduce YARN service AM ZooKeeper connection further in
3.x.
This JIRA will focus on second problem in deleting DNS record from ZooKeeper.
> YARN service has ZooKeeper connection leaks
> -------------------------------------------
>
> Key: YARN-8428
> URL: https://issues.apache.org/jira/browse/YARN-8428
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
> Affects Versions: 3.1.0
> Reporter: Eric Yang
> Priority: Critical
>
> There are two leaks discovered during stress test. First, YARN service
> application master uses RegistryDNS to create DNS record for YARN service.
> Data is first persisted on ZooKeeper, and RegistryDNS will lookup data in
> ZooKeeper to create DNS entries. YARN service AM does not close ZooKeeper
> connection after the record has been created. Second, when deleting DNS
> record from ZooKeeper, ZooKeeper connection is left open.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
