Indeed it was a problem on the executor side… I have to figure out how to fix it now ;-)
Thanks! Mehdi De : Yana Kadiyska [mailto:[email protected]] Envoyé : mercredi 15 octobre 2014 18:32 À : Mehdi Singer Cc : [email protected] Objet : Re: Problem executing Spark via JBoss application From this line : Removing executor app-20141015142644-0125/0 because it is EXITED I would guess that you need to examine the executor log to see why the executor actually exited. My guess would be that the executor cannot connect back to your driver. But check the log from the executor. It should be in SPARK_HOME/work/app-id/executor_id/stderr on the worker box, I believe. On Wed, Oct 15, 2014 at 8:56 AM, Mehdi Singer <[email protected]<mailto:[email protected]>> wrote: Hi, I have a Spark standalone example application which is working fine. I'm now trying to integrate this application into a J2EE application, deployed on JBoss 7.1.1 and accessed via a web service. The JBoss server is installed on my local machine (Windows 7) and the master Spark is remote (Linux). The example simply executes a count on my RDD. When I call the webservice I'm getting the following error at JBoss side when executing the count: 11:48:10,232 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/el2-etrm-spark].[ws]] (http--127.0.0.1-8082-3) "Servlet.service()" pour la servlet ws a généré une exception: java.lang.RuntimeException: org.apache.cxf.interceptor.Fault: Job cancelled because SparkContext was shut down at org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage(AbstractFaultChainInitiatorObserver.java:116) [cxf-api-2.6.9.jar:2.6.9] at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:322) [cxf-api-2.6.9.jar:2.4.3] at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [cxf-api-2.6.9.jar:2.6.9] at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:211) [cxf-bundle-2.6.2.jar:2.6.2] at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:213) [cxf-bundle-2.6.2.jar:2.6.2] at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:154) [cxf-bundle-2.6.2.jar:2.6.2] at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:130) [cxf-bundle-2.6.2.jar:2.6.2] at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:221) [cxf-bundle-2.6.2.jar:2.6.2] at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:146) [cxf-bundle-2.6.2.jar:2.6.2] at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:197) [cxf-bundle-2.6.2.jar:2.6.2] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:180) [spring-orm-3.2.3.RELEASE.jar:3.2.3.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-3.2.3.RELEASE.jar:3.2.3.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:186) [spring-security-web-3.1.3.RELEASE.jar:3.1.3.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) [spring-security-web-3.1.3.RELEASE.jar:3.1.3.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-3.2.3.RELEASE.jar:3.2.3.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259) [spring-web-3.2.3.RELEASE.jar:3.2.3.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.1.1.Final.jar:7.1.1.Final] at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] Caused by: org.apache.cxf.interceptor.Fault: Job cancelled because SparkContext was shut down at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162) [cxf-api-2.6.9.jar:2.6.9] at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128) [cxf-api-2.6.9.jar:2.6.9] at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:167) [cxf-bundle-2.6.2.jar:2.6.2] at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:94) [cxf-bundle-2.6.2.jar:2.6.2] at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57) [cxf-api-2.6.9.jar:2.6.9] at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93) [cxf-api-2.6.9.jar:2.6.9] at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262) [cxf-api-2.6.9.jar:2.4.3] ... 33 more Caused by: org.apache.spark.SparkException: Job cancelled because SparkContext was shut down at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:694) [spark-core_2.10-1.1.0.jar:1.1.0] at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:693) [spark-core_2.10-1.1.0.jar:1.1.0] at scala.collection.mutable.HashSet.foreach(HashSet.scala:79) [scala-library-2.10.4.jar:] at org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:693) [spark-core_2.10-1.1.0.jar:1.1.0] at org.apache.spark.scheduler.DAGSchedulerEventProcessActor.postStop(DAGScheduler.scala:1399) [spark-core_2.10-1.1.0.jar:1.1.0] at akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:201) [akka-actor_2.10-2.2.3-shaded-protobuf.jar:] at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:163) [akka-actor_2.10-2.2.3-shaded-protobuf.jar:] at akka.actor.ActorCell.terminate(ActorCell.scala:338) [akka-actor_2.10-2.2.3-shaded-protobuf.jar:] at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:431) [akka-actor_2.10-2.2.3-shaded-protobuf.jar:] at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447) [akka-actor_2.10-2.2.3-shaded-protobuf.jar:] at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262) [akka-actor_2.10-2.2.3-shaded-protobuf.jar:] at akka.dispatch.Mailbox.run(Mailbox.scala:218) [akka-actor_2.10-2.2.3-shaded-protobuf.jar:] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) [akka-actor_2.10-2.2.3-shaded-protobuf.jar:] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.10.4.jar:] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.10.4.jar:] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.10.4.jar:] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.10.4.jar:] On Spark master side I'm getting the following output: 14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 got disassociated, removing it. 14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 got disassociated, removing it. 14/10/15 11:48:10 INFO LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.21.6.4%3A51850-649#254460547] was not delivered. [215] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 14/10/15 11:48:10 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@sparkmastermachinename:7077] -> [akka.tcp://sparkDriver@mymachinename:51834]: Error [Association failed with [akka.tcp://sparkDriver@mymachinename:51834]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkDriver@mymachinename:51834] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: mymachinename/10.21.6.4:51834<http://10.21.6.4:51834> ] 14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 got disassociated, removing it. 14/10/15 11:48:10 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@sparkmastermachinename:7077] -> [akka.tcp://sparkDriver@mymachinename:51834]: Error [Association failed with [akka.tcp://sparkDriver@mymachinename:51834]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkDriver@mymachinename:51834] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: mymachinename/10.21.6.4:51834<http://10.21.6.4:51834> ] 14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 got disassociated, removing it. 14/10/15 11:48:10 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@sparkmastermachinename:7077] -> [akka.tcp://sparkDriver@mymachinename:51834]: Error [Association failed with [akka.tcp://sparkDriver@mymachinename:51834]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkDriver@mymachinename:51834] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: mymachinename/10.21.6.4:51834<http://10.21.6.4:51834> ] 14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 got disassociated, removing it. I've excluded the "Connection refused" messages from the possible reasons for my problem as I have the same message when I execute my example as standalone application... For information, here's the Spark master output when I execute the example in standalone: 14/10/15 14:26:44 INFO Master: Registering app ts-aggregation 14/10/15 14:26:44 INFO Master: Registered app ts-aggregation with ID app-20141015142644-0125 14/10/15 14:26:44 INFO Master: Launching executor app-20141015142644-0125/0 on worker worker-20141015100053-sparkmastermachinename.lampiris.biz-39058 14/10/15 14:26:44 INFO Master: Launching executor app-20141015142644-0125/1 on worker worker-20141015100053-sparkmastermachinename.lampiris.biz-46115 14/10/15 14:26:48 INFO Master: Removing executor app-20141015142644-0125/0 because it is EXITED 14/10/15 14:26:48 INFO Master: Removing executor app-20141015142644-0125/1 because it is EXITED 14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 got disassociated, removing it. 14/10/15 14:26:49 INFO Master: Removing app app-20141015142644-0125 14/10/15 14:26:49 INFO LocalActorRef: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.21.6.4%3A58543-653#1905465071] was not delivered. [216] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 14/10/15 14:26:49 INFO LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.21.6.4%3A58543-653#1905465071] was not delivered. [217] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 got disassociated, removing it. 14/10/15 14:26:49 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@sparkmastermachinename:7077] -> [akka.tcp://sparkDriver@mymachinename:58525]: Error [Association failed with [akka.tcp://sparkDriver@mymachinename:58525]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkDriver@mymachinename:58525] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: mymachinename/10.21.6.4:58525<http://10.21.6.4:58525> ] 14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 got disassociated, removing it. 14/10/15 14:26:49 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@sparkmastermachinename:7077] -> [akka.tcp://sparkDriver@mymachinename:58525]: Error [Association failed with [akka.tcp://sparkDriver@mymachinename:58525]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkDriver@mymachinename:58525] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: mymachinename/10.21.6.4:58525<http://10.21.6.4:58525> ] 14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 got disassociated, removing it. 14/10/15 14:26:49 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@sparkmastermachinename:7077] -> [akka.tcp://sparkDriver@mymachinename:58525]: Error [Association failed with [akka.tcp://sparkDriver@mymachinename:58525]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkDriver@mymachinename:58525] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: mymachinename/10.21.6.4:58525<http://10.21.6.4:58525> ] 14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 got disassociated, removing it. Does anyone have any clue on what may cause this error? Thank you for your help ! Regards
