Github user zhaoyunjiong commented on the issue:
https://github.com/apache/spark/pull/14887
@SaintBacchus Please check below logs:
2016-08-30 10:16:24,982 INFO
org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Disk(s)
failed. 3/12 local-dirs turned bad:
**/hadoop/1/scratch/local,**/hadoop/7/scratch/local,/hadoop/10/scratch/local;3/12
log-dirs turned bad:
/hadoop/1/scratch/logs,/hadoop/7/scratch/logs,/hadoop/10/scratch/logs
...
2016-08-30 10:16:38,008 INFO
org.apache.spark.network.yarn.YarnShuffleService: Initializing YARN shuffle
service for Spark
2016-08-30 10:16:38,008 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Adding
auxiliary service spark_shuffle, "spark_shuffle"
2016-08-30 10:16:38,260 ERROR
org.apache.spark.network.shuffle.ExternalShuffleBlockResolver: error opening
leveldb file **/hadoop/1/scratch/local/registeredExecutors.ldb**. Creating new
file, will not be able to recover state for existing applications
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
/hadoop/1/scratch/local/registeredExecutors.ldb/LOCK: No such file or directory
at
org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
at
org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
at
org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
at
org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.<init>(ExternalShuffleBlockResolver.java:100)
at
org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.<init>(ExternalShuffleBlockResolver.java:81)
at
org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.<init>(ExternalShuffleBlockHandler.java:56)
at
org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:129)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:122)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:220)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:186)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:357)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:404)
2016-08-30 10:16:38,262 WARN
org.apache.spark.network.shuffle.ExternalShuffleBlockResolver: error deleting
/hadoop/1/scratch/local/registeredExecutors.ldb
2016-08-30 10:16:38,262 ERROR
org.apache.spark.network.yarn.YarnShuffleService: Failed to initialize external
shuffle service
java.io.IOException: Unable to create state store
at
org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.<init>(ExternalShuffleBlockResolver.java:129)
at
org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.<init>(ExternalShuffleBlockResolver.java:81)
at
org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.<init>(ExternalShuffleBlockHandler.java:56)
at
org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:129)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:122)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:220)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:186)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:357)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:404)
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO
error: /hadoop/1/scratch/local/registeredExecutors.ldb/LOCK: No such file or
directory
at
org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
at
org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
at
org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
at
org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.<init>(ExternalShuffleBlockResolver.java:127)
... 14 more
2016-08-30 10:16:38,456 INFO
org.apache.spark.network.yarn.YarnShuffleService: Started YARN shuffle service
for Spark on port 7337. Authentication is not enabled. Registered executor
file is /hadoop/1/scratch/local/registeredExecutors.ldb
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
