If I read the code correctly, that error message came from CoarseGrainedSchedulerBackend.
There may be existing / future error messages, other than the one cited below, which are useful. Maybe change the log level of this message to DEBUG ? Cheers On Sun, Jul 26, 2015 at 3:28 PM, Mridul Muralidharan <mri...@gmail.com> wrote: > Simply customize your log4j confit instead of modifying code if you don't > want messages from that class. > > > Regards > Mridul > > On Sunday, July 26, 2015, Sea <261810...@qq.com> wrote: > >> This exception is so ugly!!! The screen is full of these information >> when the program runs a long time, and they will not fail the job. >> >> I comment it in the source code. I think this information is useless >> because the executor is already removed and I don't know what does the >> executor id mean. >> >> Should we remove this information forever? >> >> >> >> 15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove >> non-existent executor 2... >> >> 15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove >> non-existent executor 2... >> >> >> >> >> >> >> >> ------------------ 原始邮件 ------------------ >> *发件人:* "Ted Yu";<yuzhih...@gmail.com>; >> *发送时间:* 2015年7月26日(星期天) 晚上10:51 >> *收件人:* "Pa Rö"<paul.roewer1...@googlemail.com>; >> *抄送:* "user"<user@spark.apache.org>; >> *主题:* Re: Asked to remove non-existent executor exception >> >> You can list the files in tmpfs in reverse chronological order and remove >> the oldest until you have enough space. >> >> Cheers >> >> On Sun, Jul 26, 2015 at 12:43 AM, Pa Rö <paul.roewer1...@googlemail.com> >> wrote: >> >>> i has seen that the "tempfs" is full, how i can clear this? >>> >>> 2015-07-23 13:41 GMT+02:00 Pa Rö <paul.roewer1...@googlemail.com>: >>> >>>> hello spark community, >>>> >>>> i have build an application with geomesa, accumulo and spark. >>>> if it run on spark local mode, it is working, but on spark >>>> cluster not. in short it says: No space left on device. Asked to remove >>>> non-existent executor XY. >>>> I´m confused, because there were many GB´s of free space. do i need to >>>> change my configuration or what else can i do? thanks in advance. >>>> >>>> here is the complete exception: >>>> >>>> og4j:WARN No appenders could be found for logger >>>> (org.apache.accumulo.fate.zookeeper.ZooSession). >>>> log4j:WARN Please initialize the log4j system properly. >>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig >>>> for more info. >>>> Using Spark's default log4j profile: >>>> org/apache/spark/log4j-defaults.properties >>>> 15/07/23 13:26:39 INFO SparkContext: Running Spark version 1.3.0 >>>> 15/07/23 13:26:39 WARN NativeCodeLoader: Unable to load native-hadoop >>>> library for your platform... using builtin-java classes where applicable >>>> 15/07/23 13:26:39 INFO SecurityManager: Changing view acls to: marcel >>>> 15/07/23 13:26:39 INFO SecurityManager: Changing modify acls to: marcel >>>> 15/07/23 13:26:39 INFO SecurityManager: SecurityManager: authentication >>>> disabled; ui acls disabled; users with view permissions: Set(marcel); users >>>> with modify permissions: Set(marcel) >>>> 15/07/23 13:26:39 INFO Slf4jLogger: Slf4jLogger started >>>> 15/07/23 13:26:40 INFO Remoting: Starting remoting >>>> 15/07/23 13:26:40 INFO Remoting: Remoting started; listening on >>>> addresses :[akka.tcp://sparkDriver@node1-scads02:52478] >>>> 15/07/23 13:26:40 INFO Utils: Successfully started service >>>> 'sparkDriver' on port 52478. >>>> 15/07/23 13:26:40 INFO SparkEnv: Registering MapOutputTracker >>>> 15/07/23 13:26:40 INFO SparkEnv: Registering BlockManagerMaster >>>> 15/07/23 13:26:40 INFO DiskBlockManager: Created local directory at >>>> /tmp/spark-ca9319d4-68a2-4add-a21a-48b13ae9cf81/blockmgr-cbf8af23-e113-4732-8c2c-7413ad237b3b >>>> 15/07/23 13:26:40 INFO MemoryStore: MemoryStore started with capacity >>>> 1916.2 MB >>>> 15/07/23 13:26:40 INFO HttpFileServer: HTTP File server directory is >>>> /tmp/spark-9d4a04d5-3535-49e0-a859-d278a0cc7bf8/httpd-1882aafc-45fe-4490-803d-c04fc67510a2 >>>> 15/07/23 13:26:40 INFO HttpServer: Starting HTTP Server >>>> 15/07/23 13:26:40 INFO Server: jetty-8.y.z-SNAPSHOT >>>> 15/07/23 13:26:40 INFO AbstractConnector: Started >>>> SocketConnector@0.0.0.0:56499 >>>> 15/07/23 13:26:40 INFO Utils: Successfully started service 'HTTP file >>>> server' on port 56499. >>>> 15/07/23 13:26:40 INFO SparkEnv: Registering OutputCommitCoordinator >>>> 15/07/23 13:26:40 INFO Server: jetty-8.y.z-SNAPSHOT >>>> 15/07/23 13:26:40 INFO AbstractConnector: Started >>>> SelectChannelConnector@0.0.0.0:4040 >>>> 15/07/23 13:26:40 INFO Utils: Successfully started service 'SparkUI' on >>>> port 4040. >>>> 15/07/23 13:26:40 INFO SparkUI: Started SparkUI at >>>> http://node1-scads02:4040 >>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Connecting to master >>>> akka.tcp://sparkMaster@node1-scads02:7077/user/Master... >>>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Connected to Spark >>>> cluster with app ID app-20150723132640-0000 >>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: >>>> app-20150723132640-0000/0 on worker-20150723132524-node3-scads06-7078 >>>> (node3-scads06:7078) with 8 cores >>>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID >>>> app-20150723132640-0000/0 on hostPort node3-scads06:7078 with 8 cores, >>>> 512.0 MB RAM >>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: >>>> app-20150723132640-0000/1 on worker-20150723132513-node2-scads05-7078 >>>> (node2-scads05:7078) with 8 cores >>>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID >>>> app-20150723132640-0000/1 on hostPort node2-scads05:7078 with 8 cores, >>>> 512.0 MB RAM >>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>>> app-20150723132640-0000/0 is now RUNNING >>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>>> app-20150723132640-0000/1 is now RUNNING >>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>>> app-20150723132640-0000/0 is now LOADING >>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>>> app-20150723132640-0000/1 is now LOADING >>>> 15/07/23 13:26:40 INFO NettyBlockTransferService: Server created on >>>> 45786 >>>> 15/07/23 13:26:40 INFO BlockManagerMaster: Trying to register >>>> BlockManager >>>> 15/07/23 13:26:40 INFO BlockManagerMasterActor: Registering block >>>> manager node1-scads02:45786 with 1916.2 MB RAM, BlockManagerId(<driver>, >>>> node1-scads02, 45786) >>>> 15/07/23 13:26:40 INFO BlockManagerMaster: Registered BlockManager >>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>>> app-20150723132640-0000/0 is now FAILED (java.io.IOException: No space left >>>> on device) >>>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Executor >>>> app-20150723132640-0000/0 removed: java.io.IOException: No space left on >>>> device >>>> 15/07/23 13:26:40 ERROR SparkDeploySchedulerBackend: Asked to remove >>>> non-existent executor 0 >>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: >>>> app-20150723132640-0000/2 on worker-20150723132524-node3-scads06-7078 >>>> (node3-scads06:7078) with 8 cores >>>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID >>>> app-20150723132640-0000/2 on hostPort node3-scads06:7078 with 8 cores, >>>> 512.0 MB RAM >>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>>> app-20150723132640-0000/1 is now FAILED (java.io.IOException: No space left >>>> on device) >>>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Executor >>>> app-20150723132640-0000/1 removed: java.io.IOException: No space left on >>>> device >>>> 15/07/23 13:26:40 ERROR SparkDeploySchedulerBackend: Asked to remove >>>> non-existent executor 1 >>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: >>>> app-20150723132640-0000/3 on worker-20150723132513-node2-scads05-7078 >>>> (node2-scads05:7078) with 8 cores >>>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID >>>> app-20150723132640-0000/3 on hostPort node2-scads05:7078 with 8 cores, >>>> 512.0 MB RAM >>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>>> app-20150723132640-0000/2 is now LOADING >>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>>> app-20150723132640-0000/3 is now LOADING >>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>>> app-20150723132640-0000/2 is now RUNNING >>>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: SchedulerBackend is >>>> ready for scheduling beginning after reached minRegisteredResourcesRatio: >>>> 0.0 >>>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>>> app-20150723132640-0000/3 is now RUNNING >>>> 15/07/23 13:26:41 INFO AppClient$ClientActor: Executor updated: >>>> app-20150723132640-0000/2 is now FAILED (java.io.IOException: No space left >>>> on device) >>>> 15/07/23 13:26:41 INFO SparkDeploySchedulerBackend: Executor >>>> app-20150723132640-0000/2 removed: java.io.IOException: No space left on >>>> device >>>> 15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove >>>> non-existent executor 2... >>>> >>> >>> >>
