[
https://issues.apache.org/jira/browse/YARN-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16868310#comment-16868310
]
kyungwan nam commented on YARN-9521:
------------------------------------
{code:java}
2019-06-18 18:47:38,634 INFO nodelabels.CommonNodeLabelsManager
(CommonNodeLabelsManager.java:internalUpdateLabelsOnNodes(664)) - REPLACE
labels on nodes:
2019-06-18 18:47:38,634 INFO nodelabels.CommonNodeLabelsManager
(CommonNodeLabelsManager.java:internalUpdateLabelsOnNodes(666)) -
NM=test.nm1.com:0, labels=[test]
2019-06-18 18:47:38,635 INFO allocator.AbstractContainerAllocator
(AbstractContainerAllocator.java:getCSAssignmentFromAllocateResult(129)) -
assignedContainer application attempt=appattempt_1560841031202_0111_000001
container=null queue=dev clusterResource=<memory:41158656, vCores:10440>
type=OFF_SWITCH requestedPartition=
2019-06-18 18:47:38,635 INFO allocator.AbstractContainerAllocator
(AbstractContainerAllocator.java:getCSAssignmentFromAllocateResult(129)) -
assignedContainer application attempt=appattempt_1560841031202_0111_000001
container=null queue=dev clusterResource=<memory:41158656, vCores:10440>
type=OFF_SWITCH requestedPartition=
2019-06-18 18:47:38,635 INFO allocator.AbstractContainerAllocator
(AbstractContainerAllocator.java:getCSAssignmentFromAllocateResult(129)) -
assignedContainer application attempt=appattempt_1560841031202_0111_000001
container=null queue=dev clusterResource=<memory:41158656, vCores:10440>
type=OFF_SWITCH requestedPartition=
2019-06-18 18:47:38,635 INFO allocator.AbstractContainerAllocator
(AbstractContainerAllocator.java:getCSAssignmentFromAllocateResult(129)) -
assignedContainer application attempt=appattempt_1560841031202_0111_000001
container=null queue=dev clusterResource=<memory:41158656, vCores:10440>
type=OFF_SWITCH requestedPartition=
2019-06-18 18:47:38,636 INFO rmcontainer.RMContainerImpl
(RMContainerImpl.java:handle(480)) - container_e48_1560841031202_0111_01_002020
Container Transitioned from NEW to ALLOCATED
2019-06-18 18:47:38,636 ERROR nodelabels.CommonNodeLabelsManager
(CommonNodeLabelsManager.java:handleStoreEvent(201)) - Failed to store label
modification to storage
2019-06-18 18:47:38,637 INFO fica.FiCaSchedulerNode
(FiCaSchedulerNode.java:allocateContainer(169)) - Assigned container
container_e48_1560841031202_0111_01_002020 of capacity <memory:8192, vCores:1>
on host test.nm3.com:8454, which has 3 containers, <memory:24576, vCores:3>
used and <memory:133120, vCores:37> available after allocation
2019-06-18 18:47:38,637 FATAL event.AsyncDispatcher
(AsyncDispatcher.java:dispatch(203)) - Error in dispatcher thread
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
Filesystem closed
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.handleStoreEvent(CommonNodeLabelsManager.java:202)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager$ForwardingEventHandler.handle(CommonNodeLabelsManager.java:174)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager$ForwardingEventHandler.handle(CommonNodeLabelsManager.java:169)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:473)
at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1412)
at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1383)
at
org.apache.hadoop.hdfs.DistributedFileSystem$5.doCall(DistributedFileSystem.java:427)
at
org.apache.hadoop.hdfs.DistributedFileSystem$5.doCall(DistributedFileSystem.java:423)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:435)
at
org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:404)
at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1379)
at
org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore.ensureAppendEditlogFile(FileSystemNodeLabelsStore.java:107)
at
org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore.updateNodeToLabelsMappings(FileSystemNodeLabelsStore.java:118)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.handleStoreEvent(CommonNodeLabelsManager.java:196)
... 5 more
2019-06-18 18:47:38,637 INFO capacity.ParentQueue
(ParentQueue.java:apply(1340)) - assignedContainer queue=root
usedCapacity=0.08724866 absoluteUsedCapacity=0.08724866 used=<memory:3563520,
vCores:548> cluster=<memory:41158656, vCores:10440>
2019-06-18 18:47:38,637 INFO capacity.CapacityScheduler
(CapacityScheduler.java:tryCommit(2894)) - Allocation proposal accepted
2019-06-18 18:47:38,637 INFO capacity.CapacityScheduler
(CapacityScheduler.java:tryCommit(2900)) - Failed to accept allocation proposal
2019-06-18 18:47:38,637 INFO capacity.CapacityScheduler
(CapacityScheduler.java:tryCommit(2900)) - Failed to accept allocation proposal
2019-06-18 18:47:38,637 INFO capacity.CapacityScheduler
(CapacityScheduler.java:tryCommit(2900)) - Failed to accept allocation proposal
2019-06-18 18:47:38,638 INFO event.AsyncDispatcher
(AsyncDispatcher.java:run(307)) - Exiting, bbye..
2019-06-18 18:47:38,644 INFO handler.ContextHandler
(ContextHandler.java:doStop(910)) - Stopped
o.e.j.w.WebAppContext@52aa7742{/,null,UNAVAILABLE}{/cluster}
{code}
I’ve met another situation that RM is shutdown when I try to replace node-label
for a node (yarn rmadmin -replaceLabelsOnNode)
I believe it also due to FileSystem cache.
In the current RM, cached FileSystem can be used in SystemServiceManagerImpl,
FileSystemNodeLabelsStore, ServiceClient.
ServiceClient instance would be created and closed whenever yarn-service API
is requested to RM. at this time, cached FileSystem can be closed by closing
ServiceClient instance.
FileSystem cache should be disabled to avoid this issue.
Please, I would like to hear any thought or comment.
> RM failed to start due to system services
> -----------------------------------------
>
> Key: YARN-9521
> URL: https://issues.apache.org/jira/browse/YARN-9521
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 3.1.2
> Reporter: kyungwan nam
> Priority: Major
> Attachments: YARN-9521.001.patch
>
>
> when starting RM, listing system services directory has failed as follows.
> {code}
> 2019-04-30 17:18:25,441 INFO client.SystemServiceManagerImpl
> (SystemServiceManagerImpl.java:serviceInit(114)) - System Service Directory
> is configured to /services
> 2019-04-30 17:18:25,467 INFO client.SystemServiceManagerImpl
> (SystemServiceManagerImpl.java:serviceInit(120)) - UserGroupInformation
> initialized to yarn (auth:SIMPLE)
> 2019-04-30 17:18:25,467 INFO service.AbstractService
> (AbstractService.java:noteFailure(267)) - Service ResourceManager failed in
> state STARTED
> org.apache.hadoop.service.ServiceStateException: java.io.IOException:
> Filesystem closed
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:203)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:869)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1228)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1269)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1265)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1265)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1316)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1501)
> Caused by: java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:473)
> at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1639)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1217)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1233)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1200)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1179)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1175)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusIterator(DistributedFileSystem.java:1187)
> at
> org.apache.hadoop.yarn.service.client.SystemServiceManagerImpl.list(SystemServiceManagerImpl.java:375)
> at
> org.apache.hadoop.yarn.service.client.SystemServiceManagerImpl.scanForUserServices(SystemServiceManagerImpl.java:282)
> at
> org.apache.hadoop.yarn.service.client.SystemServiceManagerImpl.serviceStart(SystemServiceManagerImpl.java:126)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> ... 13 more
> {code}
> it looks like due to the usage of filesystem cache.
> this issue does not happen, when I add "fs.hdfs.impl.disable.cache=true" to
> yarn-site
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
