Well the error states <https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks002.html>
Exception in thread thread_name: java.lang.OutOfMemoryError: GC Overhead limit exceeded Cause: The detail message "GC overhead limit exceeded" indicates that the garbage collector is running all the time and Java program is making very slow progress. After a garbage collection, if the Java process is spending more than approximately 98% of its time doing garbage collection and if it is recovering less than 2% of the heap and has been doing so far the last 5 (compile time constant) consecutive garbage collections, then a java.lang.OutOfMemoryError is thrown. This exception is typically thrown because the amount of live data barely fits into the Java heap having little free space for new allocations. Action: Increase the heap size. The java.lang.OutOfMemoryError exception for *GC Overhead limit exceeded* can be turned off with the command line flag -XX:-UseGCOverheadLimit. We still don't know what the code is doing. You have not provided that info. Are you running Spark on Yarn?. Have you checked yarn logs? HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 4 August 2016 at 10:49, Vasu Devan <[email protected]> wrote: > Hi Team, > > My Spark job fails with below error : > > Could you please advice me what is the problem with my job. > > Below is my error stack: > > 16/08/04 05:11:06 ERROR ActorSystemImpl: Uncaught fatal error from thread > [sparkDriver-akka.actor.default-dispatcher-14] shutting down ActorSystem > [sparkDriver] > java.lang.OutOfMemoryError: GC overhead limit exceeded > at sun.reflect.ByteVectorImpl.trim(ByteVectorImpl.java:70) > at > sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:388) > at > sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:77) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:46) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136) > at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) > at > akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136) > at > akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104) > at scala.util.Try$.apply(Try.scala:161) > 16/08/04 05:11:06 INFO RemoteActorRefProvider$RemotingTerminator: Shutting > down remote daemon. > 16/08/04 05:11:07 INFO RemoteActorRefProvider$RemotingTerminator: Remote > daemon shut down; proceeding with flushing remote transports. > 16/08/04 05:11:07 INFO TaskSetManager: Finished task 18540.0 in stage > 148.0 (TID 153058) in 190291 ms on lhrrhegapq005.enterprisenet.org > (18536/32768) > 16/08/04 05:11:07 INFO TaskSetManager: Finished task 18529.0 in stage > 148.0 (TID 153044) in 190300 ms on lhrrhegapq008.enterprisenet.org > (18537/32768) > 16/08/04 05:11:07 INFO TaskSetManager: Finished task 18530.0 in stage > 148.0 (TID 153049) in 190297 ms on lhrrhegapq005.enterprisenet.org > (18538/32768) > 16/08/04 05:11:07 INFO TaskSetManager: Finished task 18541.0 in stage > 148.0 (TID 153062) in 190291 ms on lhrrhegapq006.enterprisenet.org > (18539/32768) > 16/08/04 05:11:09 INFO TaskSetManager: Finished task 18537.0 in stage > 148.0 (TID 153057) in 191648 ms on lhrrhegapq003.enterprisenet.org > (18540/32768) > 16/08/04 05:11:10 INFO TaskSetManager: Finished task 18557.0 in stage > 148.0 (TID 153073) in 193193 ms on lhrrhegapq003.enterprisenet.org > (18541/32768) > 16/08/04 05:11:10 INFO TaskSetManager: Finished task 18528.0 in stage > 148.0 (TID 153045) in 193206 ms on lhrrhegapq007.enterprisenet.org > (18542/32768) > 16/08/04 05:11:10 INFO TaskSetManager: Finished task 18555.0 in stage > 148.0 (TID 153072) in 193195 ms on lhrrhegapq002.enterprisenet.org > (18543/32768) > 16/08/04 05:11:10 ERROR YarnClientSchedulerBackend: Yarn application has > already exited with state FINISHED! > 16/08/04 05:11:13 WARN QueuedThreadPool: 9 threads could not be stopped > 16/08/04 05:11:13 INFO SparkUI: Stopped Spark web UI at > http://10.90.50.64:4043 > 16/08/04 05:11:15 INFO DAGScheduler: Stopping DAGScheduler > 16/08/04 05:11:16 INFO DAGScheduler: Job 94 failed: save at > ndx_scala_util.scala:1264, took 232.788303 s > 16/08/04 05:11:16 ERROR InsertIntoHadoopFsRelation: Aborting job. > org.apache.spark.SparkException: Job cancelled because SparkContext was > shut down > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:703) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:702) > at scala.collection.mutable.HashSet.foreach(HashSet.scala:79) > at > org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:702) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onStop(DAGScheduler.scala:1525) > at org.apache.spark.util.EventLoop.stop(EventLoop.scala:84) > at > org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1449) > at > org.apache.spark.SparkContext$$anonfun$stop$7.apply$mcV$sp(SparkContext.scala:1724) > at > org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1184) > at org.apache.spark.SparkContext.stop(SparkContext.scala:1723) > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:146) > at > org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1914) > at > org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:150) > at > org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108) > at > org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108) > at > org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56) > at > org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:108) > at > org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57) > at > org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:57) > at > org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:69) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:140) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:138) > at > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) > at > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:138) > at > org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:933) > at > org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:933) > at > org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:197) > at > org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:146) > at > org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:137) > at org.apache.spark.sql.DataFrame.save(DataFrame.scala:1808) > at > com.acnielsen.madras.utils.ndx_scala_util$.newHiveTableData(ndx_scala_util.scala:1264) > at > com.acnielsen.madras.utils.ndx_scala_util$.UPDATE(ndx_scala_util.scala:238) > at > com.acnielsen.madras.pkgews_panel_extract$$anonfun$p_signed_rank_yago$1.apply(pkgews_panel_extract.scala:658) > at > com.acnielsen.madras.pkgews_panel_extract$$anonfun$p_signed_rank_yago$1.apply(pkgews_panel_extract.scala:652) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at > scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) > at > com.acnielsen.madras.pkgews_panel_extract$.p_signed_rank_yago(pkgews_panel_extract.scala:652) > at > com.acnielsen.madras.pkgews_panel_extract$.p_main(pkgews_panel_extract.scala:4844) > at > com.acnielsen.madras.pkgews_panel_extract$.main(pkgews_panel_extract.scala:4655) > at > com.acnielsen.madras.pkgews_panel_extract.main(pkgews_panel_extract.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) > at > org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > 16/08/04 05:11:17 INFO TaskSetManager: Finished task 18544.0 in stage > 148.0 (TID 153064) in 200319 ms on lhrrhegapq005.enterprisenet.org > (18544/32768) > 16/08/04 05:11:17 INFO DAGScheduler: ShuffleMapStage 148 (save at > ndx_scala_util.scala:1264) failed in 233.531 s > 16/08/04 05:11:17 INFO TaskSetManager: Finished task 18552.0 in stage > 148.0 (TID 153067) in 200319 ms on lhrrhegapq008.enterprisenet.org > (18545/32768) > 16/08/04 05:11:17 INFO DAGScheduler: ShuffleMapStage 149 (save at > ndx_scala_util.scala:1264) failed in 233.277 s > 16/08/04 05:11:17 INFO TaskSetManager: Finished task 18549.0 in stage > 148.0 (TID 153056) in 200323 ms on lhrrhegapq002.enterprisenet.org > (18546/32768) > 16/08/04 05:11:17 INFO TaskSetManager: Finished task 18539.0 in stage > 148.0 (TID 153061) in 200321 ms on lhrrhegapq006.enterprisenet.org > (18547/32768) > 16/08/04 05:11:17 INFO YarnClientSchedulerBackend: Shutting down all > executors > 16/08/04 05:11:17 INFO Remoting: Remoting shut down > 16/08/04 05:11:17 INFO RemoteActorRefProvider$RemotingTerminator: Remoting > shut down. > 16/08/04 05:11:17 WARN AkkaRpcEndpointRef: Error sending message [message > = StopExecutors] in 1 attempts > org.apache.spark.rpc.RpcTimeoutException: > Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#-1206596405]] > had already been terminated.. This timeout is controlled by > spark.rpc.askTimeout > at org.apache.spark.rpc.RpcTimeout.org > $apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:214) > at > org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:229) > at > org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:225) > at > scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) > at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185) > at scala.util.Try$.apply(Try.scala:161) > at scala.util.Failure.recover(Try.scala:185) > at > scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324) > at > scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) > at > org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293) > at > scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) > at > scala.concurrent.impl.Promise$DefaultPromise.scala$concurrent$impl$Promise$DefaultPromise$$dispatchOrAddCallback(Promise.scala:280) > at > scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:270) > at scala.concurrent.Future$class.recover(Future.scala:324) > at > scala.concurrent.impl.Promise$DefaultPromise.recover(Promise.scala:153) > at > org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:319) > at > org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100) > at > org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77) > at > org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stopExecutors(CoarseGrainedSchedulerBackend.scala:274) > at > org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stop(CoarseGrainedSchedulerBackend.scala:283) > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:180) > at > org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:439) > at > org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1450) > at > org.apache.spark.SparkContext$$anonfun$stop$7.apply$mcV$sp(SparkContext.scala:1724) > at > org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1184) > at org.apache.spark.SparkContext.stop(SparkContext.scala:1723) > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:146) > Caused by: akka.pattern.AskTimeoutException: > Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#-1206596405]] > had already been terminated. > at > akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134) > at > org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:307) > ... 11 more > 16/08/04 05:11:17 ERROR DefaultWriterContainer: Job job_201608040506_0000 > aborted. > Failed to create dataset in HDFS location : > /user/hive/warehouse//TEWS_MS_TYPE_BASKET/1470301585850 > 16/08/04 05:11:20 INFO TaskSetManager: Finished task 18533.0 in stage > 148.0 (TID 153053) in 202457 ms on lhrrhegapq004.enterprisenet.org > (18548/32768) > 16/08/04 05:11:20 INFO TaskSetManager: Finished task 18551.0 in stage > 148.0 (TID 153070) in 202448 ms on lhrrhegapq002.enterprisenet.org > (18549/32768) > 16/08/04 05:11:20 INFO TaskSetManager: Finished task 18553.0 in stage > 148.0 (TID 153069) in 202449 ms on lhrrhegapq003.enterprisenet.org > (18550/32768) > 16/08/04 05:11:20 INFO TaskSetManager: Finished task 18534.0 in stage > 148.0 (TID 153047) in 202459 ms on lhrrhegapq007.enterprisenet.org > (18551/32768) > 16/08/04 05:11:20 INFO TaskSetManager: Finished task 18554.0 in stage > 148.0 (TID 153071) in 202958 ms on lhrrhegapq003.enterprisenet.org > (18552/32768) > 16/08/04 05:11:20 INFO TaskSetManager: Finished task 18543.0 in stage > 148.0 (TID 153065) in 202961 ms on lhrrhegapq004.enterprisenet.org > (18553/32768) > 16/08/04 05:11:20 INFO TaskSetManager: Finished task 18538.0 in stage > 148.0 (TID 153060) in 202963 ms on lhrrhegapq007.enterprisenet.org > (18554/32768) > 16/08/04 05:11:20 INFO TaskSetManager: Finished task 18524.0 in stage > 148.0 (TID 153048) in 202970 ms on lhrrhegapq005.enterprisenet.org > (18555/32768) > 16/08/04 05:11:20 INFO TaskSetManager: Finished task 18584.0 in stage > 148.0 (TID 153104) in 202166 ms on lhrrhegapq008.enterprisenet.org > (18556/32768) > 16/08/04 05:11:20 WARN AkkaRpcEndpointRef: Error sending message [message > = StopExecutors] in 2 attempts > org.apache.spark.rpc.RpcTimeoutException: > Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#-1206596405]] > had already been terminated.. This timeout is controlled by > spark.rpc.askTimeout > at org.apache.spark.rpc.RpcTimeout.org > $apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:214) > at > org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:229) > at > org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:225) > at > scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) > at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185) > at scala.util.Try$.apply(Try.scala:161) > at scala.util.Failure.recover(Try.scala:185) > at > scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324) > at > scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) > at > org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293) > at > scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) > at > scala.concurrent.impl.Promise$DefaultPromise.scala$concurrent$impl$Promise$DefaultPromise$$dispatchOrAddCallback(Promise.scala:280) > at > scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:270) > at scala.concurrent.Future$class.recover(Future.scala:324) > at > scala.concurrent.impl.Promise$DefaultPromise.recover(Promise.scala:153) > at > org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:319) > at > org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100) > at > org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77) > at > org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stopExecutors(CoarseGrainedSchedulerBackend.scala:274) > at > org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stop(CoarseGrainedSchedulerBackend.scala:283) > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:180) > at > org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:439) > at > org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1450) > at > org.apache.spark.SparkContext$$anonfun$stop$7.apply$mcV$sp(SparkContext.scala:1724) > at > org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1184) > at org.apache.spark.SparkContext.stop(SparkContext.scala:1723) > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:146) > Caused by: akka.pattern.AskTimeoutException: > Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#-1206596405]] > had already been terminated. > at > akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134) > at > org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:307) > ... 11 more > 16/08/04 05:11:20 INFO TaskSetManager: Finished task 18556.0 in stage > 148.0 (TID 153074) in 203328 ms on lhrrhegapq008.enterprisenet.org > (18557/32768) > 16/08/04 05:11:20 INFO TaskSetManager: Finished task 18571.0 in stage > 148.0 (TID 153087) in 202531 ms on lhrrhegapq003.enterprisenet.org > (18558/32768) > 16/08/04 05:11:20 INFO TaskSetManager: Finished task 18545.0 in stage > 148.0 (TID 153066) in 203332 ms on lhrrhegapq006.enterprisenet.org > (18559/32768) > 16/08/04 05:11:20 INFO TaskSetManager: Finished task 18581.0 in stage > 148.0 (TID 153100) in 202531 ms on lhrrhegapq004.enterprisenet.org > (18560/32768) > Failed to fetch affected records > 16/08/04 05:11:20 INFO pkgews_panel_extract$Log$: LOG, > PKGEWS_PANEL_EXTRACT, P_MAIN, CURRENT SCHEMA - mdr_devi1_supt_test, No of > records merged into TEWS_MS_TYPE_BASKET for msc_yago_trend > 03, LOG, > PKGEWS_PANEL_EXTRACT, p_signed_rank_yago, Signed rank assignment for YAGO > Trend > 16/08/04 05:11:21 INFO TaskSetManager: Finished task 18562.0 in stage > 148.0 (TID 153081) in 203403 ms on lhrrhegapq004.enterprisenet.org > (18561/32768) > 16/08/04 05:11:21 INFO TaskSetManager: Finished task 18579.0 in stage > 148.0 (TID 153084) in 203403 ms on lhrrhegapq008.enterprisenet.org > (18562/32768) > 16/08/04 05:11:21 INFO TaskSetManager: Finished task 18547.0 in stage > 148.0 (TID 153077) in 203405 ms on lhrrhegapq007.enterprisenet.org > (18563/32768) > 16/08/04 05:11:21 INFO TaskSetManager: Finished task 18575.0 in stage > 148.0 (TID 153098) in 203401 ms on lhrrhegapq002.enterprisenet.org > (18564/32768) > 16/08/04 05:11:22 INFO TaskSetManager: Finished task 18559.0 in stage > 148.0 (TID 153078) in 204359 ms on lhrrhegapq004.enterprisenet.org > (18565/32768) > 16/08/04 05:11:22 INFO TaskSetManager: Finished task 18560.0 in stage > 148.0 (TID 153082) in 204359 ms on lhrrhegapq006.enterprisenet.org > (18566/32768) > 16/08/04 05:11:22 INFO TaskSetManager: Finished task 18583.0 in stage > 148.0 (TID 153091) in 204357 ms on lhrrhegapq008.enterprisenet.org > (18567/32768) > 16/08/04 05:11:22 INFO TaskSetManager: Finished task 18558.0 in stage > 148.0 (TID 153075) in 204369 ms on lhrrhegapq008.enterprisenet.org > (18568/32768) > 16/08/04 05:11:22 INFO TaskSetManager: Finished task 18591.0 in stage > 148.0 (TID 153103) in 204358 ms on lhrrhegapq006.enterprisenet.org > (18569/32768) > 16/08/04 05:11:22 INFO TaskSetManager: Finished task 18569.0 in stage > 148.0 (TID 153089) in 204362 ms on lhrrhegapq002.enterprisenet.org > (18570/32768) > 16/08/04 05:11:22 INFO TaskSetManager: Finished task 18561.0 in stage > 148.0 (TID 153076) in 204364 ms on lhrrhegapq002.enterprisenet.org > (18571/32768) > 16/08/04 05:11:22 INFO TaskSetManager: Finished task 18568.0 in stage > 148.0 (TID 153088) in 204363 ms on lhrrhegapq007.enterprisenet.org > (18572/32768) > 16/08/04 05:11:22 INFO TaskSetManager: Finished task 18572.0 in stage > 148.0 (TID 153096) in 204362 ms on lhrrhegapq004.enterprisenet.org > (18573/32768) > 16/08/04 05:11:23 INFO TaskSetManager: Finished task 18574.0 in stage > 148.0 (TID 153094) in 205321 ms on lhrrhegapq006.enterprisenet.org > (18574/32768) > 16/08/04 05:11:23 INFO TaskSetManager: Finished task 18576.0 in stage > 148.0 (TID 153090) in 205323 ms on lhrrhegapq003.enterprisenet.org > (18575/32768) > 16/08/04 05:11:23 INFO TaskSetManager: Finished task 18567.0 in stage > 148.0 (TID 153086) in 205324 ms on lhrrhegapq006.enterprisenet.org > (18576/32768) > 16/08/04 05:11:23 INFO TaskSetManager: Finished task 18570.0 in stage > 148.0 (TID 153092) in 205323 ms on lhrrhegapq002.enterprisenet.org > (18577/32768) > 16/08/04 05:11:23 INFO TaskSetManager: Finished task 18582.0 in stage > 148.0 (TID 153102) in 205324 ms on lhrrhegapq007.enterprisenet.org > (18578/32768) > 16/08/04 05:11:23 INFO TaskSetManager: Finished task 18563.0 in stage > 148.0 (TID 153079) in 205328 ms on lhrrhegapq002.enterprisenet.org > (18579/32768) > 16/08/04 05:11:23 INFO TaskSetManager: Finished task 18578.0 in stage > 148.0 (TID 153097) in 205325 ms on lhrrhegapq007.enterprisenet.org > (18580/32768) > 16/08/04 05:11:23 INFO TaskSetManager: Finished task 18589.0 in stage > 148.0 (TID 153101) in 205326 ms on lhrrhegapq005.enterprisenet.org > (18581/32768) > 16/08/04 05:11:23 INFO TaskSetManager: Finished task 18566.0 in stage > 148.0 (TID 153080) in 205330 ms on lhrrhegapq002.enterprisenet.org > (18582/32768) > 16/08/04 05:11:23 INFO TaskSetManager: Finished task 18577.0 in stage > 148.0 (TID 153095) in 205328 ms on lhrrhegapq003.enterprisenet.org > (18583/32768) > 16/08/04 05:11:23 INFO TaskSetManager: Finished task 18565.0 in stage > 148.0 (TID 153085) in 205332 ms on lhrrhegapq005.enterprisenet.org > (18584/32768) > 16/08/04 05:11:24 WARN AkkaRpcEndpointRef: Error sending message [message > = StopExecutors] in 3 attempts > org.apache.spark.rpc.RpcTimeoutException: > Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#-1206596405]] > had already been terminated.. This timeout is controlled by > spark.rpc.askTimeout > at org.apache.spark.rpc.RpcTimeout.org > $apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:214) > at > org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:229) > at > org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:225) > at > scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) > at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185) > at scala.util.Try$.apply(Try.scala:161) > at scala.util.Failure.recover(Try.scala:185) > at > scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324) > at > scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) > at > org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293) > at > scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) > at > scala.concurrent.impl.Promise$DefaultPromise.scala$concurrent$impl$Promise$DefaultPromise$$dispatchOrAddCallback(Promise.scala:280) > at > scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:270) > at scala.concurrent.Future$class.recover(Future.scala:324) > at > scala.concurrent.impl.Promise$DefaultPromise.recover(Promise.scala:153) > at > org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:319) > at > org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100) > at > org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77) > at > org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stopExecutors(CoarseGrainedSchedulerBackend.scala:274) > at > org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stop(CoarseGrainedSchedulerBackend.scala:283) > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:180) > at > org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:439) > at > org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1450) > at > org.apache.spark.SparkContext$$anonfun$stop$7.apply$mcV$sp(SparkContext.scala:1724) > at > org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1184) > at org.apache.spark.SparkContext.stop(SparkContext.scala:1723) > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:146) > Caused by: akka.pattern.AskTimeoutException: > Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#-1206596405]] > had already been terminated. > at > akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134) > at > org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:307) > ... 11 more > 16/08/04 05:11:24 INFO TaskSetManager: Finished task 18564.0 in stage > 148.0 (TID 153083) in 206426 ms on lhrrhegapq005.enterprisenet.org > (18585/32768) > 16/08/04 05:11:24 ERROR Utils: Uncaught exception in thread Yarn > application state monitor > org.apache.spark.SparkException: Error asking standalone scheduler to shut > down executors > at > org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stopExecutors(CoarseGrainedSchedulerBackend.scala:278) > at > org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stop(CoarseGrainedSchedulerBackend.scala:283) > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:180) > at > org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:439) > at > org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1450) > at > org.apache.spark.SparkContext$$anonfun$stop$7.apply$mcV$sp(SparkContext.scala:1724) > at > org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1184) > at org.apache.spark.SparkContext.stop(SparkContext.scala:1723) > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:146) > Caused by: org.apache.spark.SparkException: Error sending message [message > = StopExecutors] > at > org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:118) > at > org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77) > at > org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stopExecutors(CoarseGrainedSchedulerBackend.scala:274) > ... 8 more > Caused by: org.apache.spark.rpc.RpcTimeoutException: > Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#-1206596405]] > had already been terminated.. This timeout is controlled by > spark.rpc.askTimeout > at org.apache.spark.rpc.RpcTimeout.org > $apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:214) > at > org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:229) > at > org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:225) > at > scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) > at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185) > at scala.util.Try$.apply(Try.scala:161) > at scala.util.Failure.recover(Try.scala:185) > at > scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324) > at > scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) > at > org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293) > at > scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) > at > scala.concurrent.impl.Promise$DefaultPromise.scala$concurrent$impl$Promise$DefaultPromise$$dispatchOrAddCallback(Promise.scala:280) > at > scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:270) > at scala.concurrent.Future$class.recover(Future.scala:324) > at > scala.concurrent.impl.Promise$DefaultPromise.recover(Promise.scala:153) > at > org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:319) > at > org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100) > ... 10 more > Caused by: akka.pattern.AskTimeoutException: > Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#-1206596405]] > had already been terminated. > at > akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134) > at > org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:307) > ... 11 more > 16/08/04 05:11:24 INFO TaskSetManager: Finished task 18573.0 in stage > 148.0 (TID 153093) in 206426 ms on lhrrhegapq005.enterprisenet.org > (18586/32768) > 16/08/04 05:11:24 INFO TaskSetManager: Finished task 18580.0 in stage > 148.0 (TID 153099) in 206425 ms on lhrrhegapq003.enterprisenet.org > (18587/32768) > 16/08/04 05:11:24 INFO TaskSetManager: Finished task 18585.0 in stage > 148.0 (TID 153105) in 206418 ms on lhrrhegapq003.enterprisenet.org > (18588/32768) > 16/08/04 05:11:24 INFO TaskSetManager: Finished task 18586.0 in stage > 148.0 (TID 153107) in 206431 ms on lhrrhegapq004.enterprisenet.org > (18589/32768) > 16/08/04 05:11:25 INFO TaskSetManager: Finished task 18596.0 in stage > 148.0 (TID 153112) in 207515 ms on lhrrhegapq008.enterprisenet.org > (18590/32768) > 16/08/04 05:11:25 INFO TaskSetManager: Finished task 18587.0 in stage > 148.0 (TID 153108) in 207517 ms on lhrrhegapq002.enterprisenet.org > (18591/32768) > 16/08/04 05:11:25 INFO TaskSetManager: Finished task 18588.0 in stage > 148.0 (TID 153106) in 207519 ms on lhrrhegapq008.enterprisenet.org > (18592/32768) > 16/08/04 05:11:25 INFO TaskSetManager: Finished task 18590.0 in stage > 148.0 (TID 153109) in 207519 ms on lhrrhegapq007.enterprisenet.org > (18593/32768) > 16/08/04 05:11:25 INFO TaskSetManager: Finished task 18594.0 in stage > 148.0 (TID 153114) in 207524 ms on lhrrhegapq004.enterprisenet.org > (18594/32768) > 16/08/04 05:11:25 INFO TaskSetManager: Finished task 18593.0 in stage > 148.0 (TID 153111) in 207526 ms on lhrrhegapq004.enterprisenet.org > (18595/32768) > 16/08/04 05:11:25 ERROR pkgews_panel_extract$Log$: PKGEWS_PANEL_EXTRACT, > p_signed_rank_yago, 2, Merge into TEWS_MS_TYPE_BASKET < 0 - NOT OK > java.lang.IllegalStateException: Cannot call methods on a stopped > SparkContext > at org.apache.spark.SparkContext.org > $apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104) > at > org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:2063) > at > org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:2076) > at > org.apache.spark.sql.hive.HadoopTableReader.<init>(TableReader.scala:70) > at > org.apache.spark.sql.hive.execution.HiveTableScan.<init>(HiveTableScan.scala:77) > at > org.apache.spark.sql.hive.HiveStrategies$HiveTableScans$$anonfun$3.apply(HiveStrategies.scala:77) > at > org.apache.spark.sql.hive.HiveStrategies$HiveTableScans$$anonfun$3.apply(HiveStrategies.scala:77) > at > org.apache.spark.sql.SQLContext$SparkPlanner.pruneFilterProject(SQLContext.scala:853) > at > org.apache.spark.sql.hive.HiveStrategies$HiveTableScans$.apply(HiveStrategies.scala:73) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:59) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner.planLater(QueryPlanner.scala:54) > at > org.apache.spark.sql.execution.SparkStrategies$EquiJoinSelection$.makeBroadcastHashJoin(SparkStrategies.scala:92) > at > org.apache.spark.sql.execution.SparkStrategies$EquiJoinSelection$.apply(SparkStrategies.scala:101) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:59) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner.planLater(QueryPlanner.scala:54) > at > org.apache.spark.sql.execution.SparkStrategies$BasicOperators$.apply(SparkStrategies.scala:346) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:59) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner.planLater(QueryPlanner.scala:54) > at > org.apache.spark.sql.execution.SparkStrategies$EquiJoinSelection$.makeBroadcastHashJoin(SparkStrategies.scala:92) > at > org.apache.spark.sql.execution.SparkStrategies$EquiJoinSelection$.apply(SparkStrategies.scala:101) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:59) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner.planLater(QueryPlanner.scala:54) > at > org.apache.spark.sql.execution.SparkStrategies$BasicOperators$.apply(SparkStrategies.scala:346) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:59) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner.planLater(QueryPlanner.scala:54) > at > org.apache.spark.sql.execution.SparkStrategies$Aggregation$.apply(SparkStrategies.scala:235) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) > at > org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:59) > at > org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan$lzycompute(SQLContext.scala:926) > at > org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan(SQLContext.scala:924) > at > org.apache.spark.sql.SQLContext$QueryExecution.executedPlan$lzycompute(SQLContext.scala:930) > at > org.apache.spark.sql.SQLContext$QueryExecution.executedPlan(SQLContext.scala:930) > at > org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:53) > at > org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:1904) > at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1385) > at > com.acnielsen.madras.pkgews_panel_extract$.p_signed_rank_yago(pkgews_panel_extract.scala:685) > at > com.acnielsen.madras.pkgews_panel_extract$.p_main(pkgews_panel_extract.scala:4844) > at > com.acnielsen.madras.pkgews_panel_extract$.main(pkgews_panel_extract.scala:4655) > at > com.acnielsen.madras.pkgews_panel_extract.main(pkgews_panel_extract.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) > at > org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > 16/08/04 05:11:25 INFO TaskSetManager: Finished task 18592.0 in stage > 148.0 (TID 153110) in 207527 ms on lhrrhegapq008.enterprisenet.org > (18596/32768) > 16/08/04 05:11:25 INFO TaskSetManager: Finished task 18603.0 in stage > 148.0 (TID 153116) in 207525 ms on lhrrhegapq006.enterprisenet.org > (18597/32768) > 16/08/04 05:11:25 ERROR pkgews_panel_extract$Log$: PKGEWS_PANEL_EXTRACT, > p_signed_rank_yago, 2, Merge into TEWS_MS_TYPE_BASKET < 0 - NOT OK , > PKGEWS_PANEL_EXTRACT, p_signed_rank_yago, 2, Error in p_signed_rank_yago - > NOT OK > Error >>>>> : null > Exception in thread "main" java.lang.Exception > at > com.acnielsen.madras.pkgews_panel_extract$.p_main(pkgews_panel_extract.scala:4875) > at > com.acnielsen.madras.pkgews_panel_extract$.main(pkgews_panel_extract.scala:4655) > at > com.acnielsen.madras.pkgews_panel_extract.main(pkgews_panel_extract.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) > at > org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > 16/08/04 05:11:25 INFO TaskSetManager: Finished task 18602.0 in stage > 148.0 (TID 153115) in 207526 ms on lhrrhegapq006.enterprisenet.org > (18598/32768) > 16/08/04 05:11:27 WARN QueuedThreadPool: 6 threads could not be stopped > 16/08/04 05:11:27 INFO TaskSetManager: Finished task 18597.0 in stage > 148.0 (TID 153113) in 208704 ms on lhrrhegapq008.enterprisenet.org > (18599/32768) > 16/08/04 05:11:27 INFO DiskBlockManager: Shutdown hook called > 16/08/04 05:11:27 WARN AkkaRpcEndpointRef: Error sending message [message > = StopMapOutputTracker] in 1 attempts > org.apache.spark.rpc.RpcTimeoutException: > Recipient[Actor[akka://sparkDriver/user/MapOutputTracker#1304693884]] had > already been terminated.. This timeout is controlled by spark.rpc.askTimeout > at org.apache.spark.rpc.RpcTimeout.org > $apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:214) > at > org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:229) > at > org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:225) > at > scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) > at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185) > at scala.util.Try$.apply(Try.scala:161) > at scala.util.Failure.recover(Try.scala:185) > at > scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324) > at > scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) > at > org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293) > at > scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) > at > scala.concurrent.impl.Promise$DefaultPromise.scala$concurrent$impl$Promise$DefaultPromise$$dispatchOrAddCallback(Promise.scala:280) > at > scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:270) > at scala.concurrent.Future$class.recover(Future.scala:324) > at > scala.concurrent.impl.Promise$DefaultPromise.recover(Promise.scala:153) > at > org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:319) > at > org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100) > at > org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77) > at > org.apache.spark.MapOutputTracker.askTracker(MapOutputTracker.scala:109) > at > org.apache.spark.MapOutputTracker.sendTracker(MapOutputTracker.scala:119) > at > org.apache.spark.MapOutputTrackerMaster.stop(MapOutputTracker.scala:382) > at org.apache.spark.SparkEnv.stop(SparkEnv.scala:97) > at > org.apache.spark.SparkContext$$anonfun$stop$12.apply$mcV$sp(SparkContext.scala:1749) > at > org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1184) > at org.apache.spark.SparkContext.stop(SparkContext.scala:1748) > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:146) > Caused by: akka.pattern.AskTimeoutException: > Recipient[Actor[akka://sparkDriver/user/MapOutputTracker#1304693884]] had > already been terminated. > at > akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134) > at > org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:307) > ... 10 more > 16/08/04 05:11:27 INFO TaskSetManager: Finished task 18604.0 in stage > 148.0 (TID 153127) in 208703 ms on lhrrhegapq003.enterprisenet.org > (18600/32768) > 16/08/04 05:11:27 INFO ZooKeeper: Session: 0x15590c0c2202d11 closed > 16/08/04 05:11:27 INFO TaskSetManager: Finished task 18598.0 in stage > 148.0 (TID 153117) in 208707 ms on lhrrhegapq002.enterprisenet.org > (18601/32768) > 16/08/04 05:11:27 INFO ClientCnxn: EventThread shut down > 16/08/04 05:11:27 INFO CuratorFrameworkSingleton: Closing ZooKeeper client. > 16/08/04 05:11:27 INFO TaskSetManager: Finished task 18599.0 in stage > 148.0 (TID 153119) in 208708 ms on lhrrhegapq004.enterprisenet.org > (18602/32768) > 16/08/04 05:11:27 INFO ShutdownHookManager: Shutdown hook called > 16/08/04 05:11:27 INFO ShutdownHookManager: Deleting directory > /tmp/spark-419c3d6c-a6d4-49c4-9f77-b79c5d3c8f37 > 16/08/04 05:11:27 INFO ShutdownHookManager: Deleting directory > /tmp/spark-579fc117-08a4-45a6-a2a1-2a6e90c1adcd/userFiles-f739d328-716d-4641-821f-a1b2af709182 > 16/08/04 05:11:27 INFO ShutdownHookManager: Deleting directory > /tmp/spark-579fc117-08a4-45a6-a2a1-2a6e90c1adcd > 16/08/04 05:11:27 INFO TaskSetManager: Finished task 18608.0 in stage > 148.0 (TID 153121) in 208715 ms on lhrrhegapq006.enterprisenet.org > (18603/32768) > > Best regards, > Vasu >
