i has seen that the "tempfs" is full, how i can clear this? 2015-07-23 13:41 GMT+02:00 Pa Rö <paul.roewer1...@googlemail.com>:
> hello spark community, > > i have build an application with geomesa, accumulo and spark. > if it run on spark local mode, it is working, but on spark > cluster not. in short it says: No space left on device. Asked to remove > non-existent executor XY. > I´m confused, because there were many GB´s of free space. do i need to > change my configuration or what else can i do? thanks in advance. > > here is the complete exception: > > og4j:WARN No appenders could be found for logger > (org.apache.accumulo.fate.zookeeper.ZooSession). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for > more info. > Using Spark's default log4j profile: > org/apache/spark/log4j-defaults.properties > 15/07/23 13:26:39 INFO SparkContext: Running Spark version 1.3.0 > 15/07/23 13:26:39 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 15/07/23 13:26:39 INFO SecurityManager: Changing view acls to: marcel > 15/07/23 13:26:39 INFO SecurityManager: Changing modify acls to: marcel > 15/07/23 13:26:39 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(marcel); users > with modify permissions: Set(marcel) > 15/07/23 13:26:39 INFO Slf4jLogger: Slf4jLogger started > 15/07/23 13:26:40 INFO Remoting: Starting remoting > 15/07/23 13:26:40 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkDriver@node1-scads02:52478] > 15/07/23 13:26:40 INFO Utils: Successfully started service 'sparkDriver' > on port 52478. > 15/07/23 13:26:40 INFO SparkEnv: Registering MapOutputTracker > 15/07/23 13:26:40 INFO SparkEnv: Registering BlockManagerMaster > 15/07/23 13:26:40 INFO DiskBlockManager: Created local directory at > /tmp/spark-ca9319d4-68a2-4add-a21a-48b13ae9cf81/blockmgr-cbf8af23-e113-4732-8c2c-7413ad237b3b > 15/07/23 13:26:40 INFO MemoryStore: MemoryStore started with capacity > 1916.2 MB > 15/07/23 13:26:40 INFO HttpFileServer: HTTP File server directory is > /tmp/spark-9d4a04d5-3535-49e0-a859-d278a0cc7bf8/httpd-1882aafc-45fe-4490-803d-c04fc67510a2 > 15/07/23 13:26:40 INFO HttpServer: Starting HTTP Server > 15/07/23 13:26:40 INFO Server: jetty-8.y.z-SNAPSHOT > 15/07/23 13:26:40 INFO AbstractConnector: Started > SocketConnector@0.0.0.0:56499 > 15/07/23 13:26:40 INFO Utils: Successfully started service 'HTTP file > server' on port 56499. > 15/07/23 13:26:40 INFO SparkEnv: Registering OutputCommitCoordinator > 15/07/23 13:26:40 INFO Server: jetty-8.y.z-SNAPSHOT > 15/07/23 13:26:40 INFO AbstractConnector: Started > SelectChannelConnector@0.0.0.0:4040 > 15/07/23 13:26:40 INFO Utils: Successfully started service 'SparkUI' on > port 4040. > 15/07/23 13:26:40 INFO SparkUI: Started SparkUI at > http://node1-scads02:4040 > 15/07/23 13:26:40 INFO AppClient$ClientActor: Connecting to master > akka.tcp://sparkMaster@node1-scads02:7077/user/Master... > 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Connected to Spark > cluster with app ID app-20150723132640-0000 > 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: > app-20150723132640-0000/0 on worker-20150723132524-node3-scads06-7078 > (node3-scads06:7078) with 8 cores > 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20150723132640-0000/0 on hostPort node3-scads06:7078 with 8 cores, > 512.0 MB RAM > 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: > app-20150723132640-0000/1 on worker-20150723132513-node2-scads05-7078 > (node2-scads05:7078) with 8 cores > 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20150723132640-0000/1 on hostPort node2-scads05:7078 with 8 cores, > 512.0 MB RAM > 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: > app-20150723132640-0000/0 is now RUNNING > 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: > app-20150723132640-0000/1 is now RUNNING > 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: > app-20150723132640-0000/0 is now LOADING > 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: > app-20150723132640-0000/1 is now LOADING > 15/07/23 13:26:40 INFO NettyBlockTransferService: Server created on 45786 > 15/07/23 13:26:40 INFO BlockManagerMaster: Trying to register BlockManager > 15/07/23 13:26:40 INFO BlockManagerMasterActor: Registering block manager > node1-scads02:45786 with 1916.2 MB RAM, BlockManagerId(<driver>, > node1-scads02, 45786) > 15/07/23 13:26:40 INFO BlockManagerMaster: Registered BlockManager > 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: > app-20150723132640-0000/0 is now FAILED (java.io.IOException: No space left > on device) > 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Executor > app-20150723132640-0000/0 removed: java.io.IOException: No space left on > device > 15/07/23 13:26:40 ERROR SparkDeploySchedulerBackend: Asked to remove > non-existent executor 0 > 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: > app-20150723132640-0000/2 on worker-20150723132524-node3-scads06-7078 > (node3-scads06:7078) with 8 cores > 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20150723132640-0000/2 on hostPort node3-scads06:7078 with 8 cores, > 512.0 MB RAM > 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: > app-20150723132640-0000/1 is now FAILED (java.io.IOException: No space left > on device) > 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Executor > app-20150723132640-0000/1 removed: java.io.IOException: No space left on > device > 15/07/23 13:26:40 ERROR SparkDeploySchedulerBackend: Asked to remove > non-existent executor 1 > 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: > app-20150723132640-0000/3 on worker-20150723132513-node2-scads05-7078 > (node2-scads05:7078) with 8 cores > 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20150723132640-0000/3 on hostPort node2-scads05:7078 with 8 cores, > 512.0 MB RAM > 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: > app-20150723132640-0000/2 is now LOADING > 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: > app-20150723132640-0000/3 is now LOADING > 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: > app-20150723132640-0000/2 is now RUNNING > 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: SchedulerBackend is > ready for scheduling beginning after reached minRegisteredResourcesRatio: > 0.0 > 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: > app-20150723132640-0000/3 is now RUNNING > 15/07/23 13:26:41 INFO AppClient$ClientActor: Executor updated: > app-20150723132640-0000/2 is now FAILED (java.io.IOException: No space left > on device) > 15/07/23 13:26:41 INFO SparkDeploySchedulerBackend: Executor > app-20150723132640-0000/2 removed: java.io.IOException: No space left on > device > 15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove > non-existent executor 2... >
