Spark Configuration of spark.worker.cleanup.appDataTtl
Hi guys: I added a parameter spark.worker.cleanup.appDataTtl 3 * 24 * 3600 in my conf/spark-default.conf, then I start my spark cluster. However I got an exception: 15/06/16 14:25:14 INFO util.Utils: Successfully started service 'sparkWorker' on port 43344. 15/06/16 14:25:14 ERROR actor.OneForOneStrategy: exception during creation akka.actor.ActorInitializationException: exception during creation at akka.actor.ActorInitializationException$.apply(Actor.scala:164) at akka.actor.ActorCell.create(ActorCell.scala:596) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at akka.util.Reflect$.instantiate(Reflect.scala:66) at akka.actor.ArgsReflectConstructor.produce(Props.scala:352) at akka.actor.Props.newActor(Props.scala:252) at akka.actor.ActorCell.newActor(ActorCell.scala:552) at akka.actor.ActorCell.create(ActorCell.scala:578) ... 9 more Caused by: java.lang.NumberFormatException: For input string: 3 * 24 * 3600 at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:589) at java.lang.Long.parseLong(Long.java:631) at scala.collection.immutable.StringLike$class.toLong(StringLike.scala:230) at scala.collection.immutable.StringOps.toLong(StringOps.scala:31) at org.apache.spark.SparkConf$$anonfun$getLong$2.apply(SparkConf.scala:194) at org.apache.spark.SparkConf$$anonfun$getLong$2.apply(SparkConf.scala:194) at scala.Option.map(Option.scala:145) at org.apache.spark.SparkConf.getLong(SparkConf.scala:194) at org.apache.spark.deploy.worker.Worker.init(Worker.scala:89) ... 18 more How to set this parameter correctly? BTW, I searched this property in http://spark.apache.org/docs/latest/configuration.html and got no match. This property was found in http://spark.apache.org/docs/latest/spark-standalone.html with a default value 7 * 24 * 3600 (7 days), which I also tried but also failed.Thanks Thanksamp;Best regards! San.Luo
Re: Spark Configuration of spark.worker.cleanup.appDataTtl
I think you have to using 604800 instead of 7 * 24 * 3600, obviously SparkConf will not do multiplication for you.. The exception is quite obvious: Caused by: java.lang.NumberFormatException: For input string: 3 * 24 * 3600 2015-06-16 14:52 GMT+08:00 luohui20...@sina.com: Hi guys: I added a parameter spark.worker.cleanup.appDataTtl 3 * 24 * 3600 in my conf/spark-default.conf, then I start my spark cluster. However I got an exception: 15/06/16 14:25:14 INFO util.Utils: Successfully started service 'sparkWorker' on port 43344. 15/06/16 14:25:14 ERROR actor.OneForOneStrategy: exception during creation akka.actor.ActorInitializationException: exception during creation at akka.actor.ActorInitializationException$.apply(Actor.scala:164) at akka.actor.ActorCell.create(ActorCell.scala:596) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at akka.util.Reflect$.instantiate(Reflect.scala:66) at akka.actor.ArgsReflectConstructor.produce(Props.scala:352) at akka.actor.Props.newActor(Props.scala:252) at akka.actor.ActorCell.newActor(ActorCell.scala:552) at akka.actor.ActorCell.create(ActorCell.scala:578) ... 9 more Caused by: java.lang.NumberFormatException: For input string: 3 * 24 * 3600 at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:589) at java.lang.Long.parseLong(Long.java:631) at scala.collection.immutable.StringLike$class.toLong(StringLike.scala:230) at scala.collection.immutable.StringOps.toLong(StringOps.scala:31) at org.apache.spark.SparkConf$$anonfun$getLong$2.apply(SparkConf.scala:194) at org.apache.spark.SparkConf$$anonfun$getLong$2.apply(SparkConf.scala:194) at scala.Option.map(Option.scala:145) at org.apache.spark.SparkConf.getLong(SparkConf.scala:194) at org.apache.spark.deploy.worker.Worker.init(Worker.scala:89) ... 18 more How to set this parameter correctly? BTW, I searched this property in http://spark.apache.org/docs/latest/configuration.html and got no match. This property was found in http://spark.apache.org/docs/latest/spark-standalone.html with a default value 7 * 24 * 3600 (7 days), which I also tried but also failed. Thanks Thanksamp;Best regards! San.Luo
回复：Re: Spark Configuration of spark.worker.cleanup.appDataTtl
thanks saisai,I should try more times. I thought it will be caculated automatically as the default. Thanksamp;Best regards! San.Luo - 原始邮件 - 发件人：Saisai Shao sai.sai.s...@gmail.com 收件人：罗辉 luohui20...@sina.com 抄送人：user user@spark.apache.org 主题：Re: Spark Configuration of spark.worker.cleanup.appDataTtl 日期：2015年06月16日 15点00分 I think you have to using 604800 instead of 7 * 24 * 3600, obviously SparkConf will not do multiplication for you.. The exception is quite obvious: Caused by: java.lang.NumberFormatException: For input string: 3 * 24 * 3600 2015-06-16 14:52 GMT+08:00 luohui20...@sina.com: Hi guys: I added a parameter spark.worker.cleanup.appDataTtl 3 * 24 * 3600 in my conf/spark-default.conf, then I start my spark cluster. However I got an exception: 15/06/16 14:25:14 INFO util.Utils: Successfully started service 'sparkWorker' on port 43344. 15/06/16 14:25:14 ERROR actor.OneForOneStrategy: exception during creation akka.actor.ActorInitializationException: exception during creation at akka.actor.ActorInitializationException$.apply(Actor.scala:164) at akka.actor.ActorCell.create(ActorCell.scala:596) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at akka.util.Reflect$.instantiate(Reflect.scala:66) at akka.actor.ArgsReflectConstructor.produce(Props.scala:352) at akka.actor.Props.newActor(Props.scala:252) at akka.actor.ActorCell.newActor(ActorCell.scala:552) at akka.actor.ActorCell.create(ActorCell.scala:578) ... 9 more Caused by: java.lang.NumberFormatException: For input string: 3 * 24 * 3600 at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:589) at java.lang.Long.parseLong(Long.java:631) at scala.collection.immutable.StringLike$class.toLong(StringLike.scala:230) at scala.collection.immutable.StringOps.toLong(StringOps.scala:31) at org.apache.spark.SparkConf$$anonfun$getLong$2.apply(SparkConf.scala:194) at org.apache.spark.SparkConf$$anonfun$getLong$2.apply(SparkConf.scala:194) at scala.Option.map(Option.scala:145) at org.apache.spark.SparkConf.getLong(SparkConf.scala:194) at org.apache.spark.deploy.worker.Worker.init(Worker.scala:89) ... 18 more How to set this parameter correctly? BTW, I searched this property in http://spark.apache.org/docs/latest/configuration.html and got no match. This property was found in http://spark.apache.org/docs/latest/spark-standalone.html with a default value 7 * 24 * 3600 (7 days), which I also tried but also failed.Thanks Thanksamp;Best regards! San.Luo
