Hi,
I have a Spark standalone example application which is working fine.
I'm now trying to integrate this application into a J2EE application, deployed
on JBoss 7.1.1 and accessed via a web service. The JBoss server is installed on
my local machine (Windows 7) and the master Spark is remote (Linux).
The example simply executes a count on my RDD.
When I call the webservice I'm getting the following error at JBoss side when
executing the count:
11:48:10,232 ERROR
[org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/el2-etrm-spark].[ws]]
(http--127.0.0.1-8082-3) "Servlet.service()" pour la servlet ws a généré
une exception: java.lang.RuntimeException: org.apache.cxf.interceptor.Fault:
Job cancelled because SparkContext was shut down
at
org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage(AbstractFaultChainInitiatorObserver.java:116)
[cxf-api-2.6.9.jar:2.6.9]
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:322)
[cxf-api-2.6.9.jar:2.4.3]
at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
[cxf-api-2.6.9.jar:2.6.9]
at
org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:211)
[cxf-bundle-2.6.2.jar:2.6.2]
at
org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:213)
[cxf-bundle-2.6.2.jar:2.6.2]
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:154)
[cxf-bundle-2.6.2.jar:2.6.2]
at
org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:130)
[cxf-bundle-2.6.2.jar:2.6.2]
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:221)
[cxf-bundle-2.6.2.jar:2.6.2]
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:146)
[cxf-bundle-2.6.2.jar:2.6.2]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
[jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:197)
[cxf-bundle-2.6.2.jar:2.6.2]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
[jbossweb-7.0.13.Final.jar:]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
[jbossweb-7.0.13.Final.jar:]
at
org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:180)
[spring-orm-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
[spring-web-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
[jbossweb-7.0.13.Final.jar:]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
[jbossweb-7.0.13.Final.jar:]
at
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:186)
[spring-security-web-3.1.3.RELEASE.jar:3.1.3.RELEASE]
at
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
[spring-security-web-3.1.3.RELEASE.jar:3.1.3.RELEASE]
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
[spring-web-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
[spring-web-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
[jbossweb-7.0.13.Final.jar:]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
[jbossweb-7.0.13.Final.jar:]
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
[jbossweb-7.0.13.Final.jar:]
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
[jbossweb-7.0.13.Final.jar:]
at
org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
[jboss-as-jpa-7.1.1.Final.jar:7.1.1.Final]
at
org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
[jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
[jbossweb-7.0.13.Final.jar:]
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
[jbossweb-7.0.13.Final.jar:]
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
[jbossweb-7.0.13.Final.jar:]
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
[jbossweb-7.0.13.Final.jar:]
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877)
[jbossweb-7.0.13.Final.jar:]
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671)
[jbossweb-7.0.13.Final.jar:]
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
[jbossweb-7.0.13.Final.jar:]
at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
Caused by: org.apache.cxf.interceptor.Fault: Job cancelled because SparkContext
was shut down
at
org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162)
[cxf-api-2.6.9.jar:2.6.9]
at
org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128)
[cxf-api-2.6.9.jar:2.6.9]
at
org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:167)
[cxf-bundle-2.6.2.jar:2.6.2]
at
org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:94)
[cxf-bundle-2.6.2.jar:2.6.2]
at
org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
[cxf-api-2.6.9.jar:2.6.9]
at
org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)
[cxf-api-2.6.9.jar:2.6.9]
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)
[cxf-api-2.6.9.jar:2.4.3]
... 33 more
Caused by: org.apache.spark.SparkException: Job cancelled because SparkContext
was shut down
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:694)
[spark-core_2.10-1.1.0.jar:1.1.0]
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:693)
[spark-core_2.10-1.1.0.jar:1.1.0]
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
[scala-library-2.10.4.jar:]
at
org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:693)
[spark-core_2.10-1.1.0.jar:1.1.0]
at
org.apache.spark.scheduler.DAGSchedulerEventProcessActor.postStop(DAGScheduler.scala:1399)
[spark-core_2.10-1.1.0.jar:1.1.0]
at
akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:201)
[akka-actor_2.10-2.2.3-shaded-protobuf.jar:]
at
akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:163)
[akka-actor_2.10-2.2.3-shaded-protobuf.jar:]
at akka.actor.ActorCell.terminate(ActorCell.scala:338)
[akka-actor_2.10-2.2.3-shaded-protobuf.jar:]
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:431)
[akka-actor_2.10-2.2.3-shaded-protobuf.jar:]
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447)
[akka-actor_2.10-2.2.3-shaded-protobuf.jar:]
at
akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262)
[akka-actor_2.10-2.2.3-shaded-protobuf.jar:]
at akka.dispatch.Mailbox.run(Mailbox.scala:218)
[akka-actor_2.10-2.2.3-shaded-protobuf.jar:]
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
[akka-actor_2.10-2.2.3-shaded-protobuf.jar:]
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[scala-library-2.10.4.jar:]
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[scala-library-2.10.4.jar:]
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[scala-library-2.10.4.jar:]
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[scala-library-2.10.4.jar:]
On Spark master side I'm getting the following output:
14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 got
disassociated, removing it.
14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 got
disassociated, removing it.
14/10/15 11:48:10 INFO LocalActorRef: Message
[akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from
Actor[akka://sparkMaster/deadLetters] to
Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.21.6.4%3A51850-649#254460547]
was not delivered. [215] dead letters encountered. This logging can be turned
off or adjusted with configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
14/10/15 11:48:10 ERROR EndpointWriter: AssociationError
[akka.tcp://sparkMaster@sparkmastermachinename:7077] ->
[akka.tcp://sparkDriver@mymachinename:51834]: Error [Association failed with
[akka.tcp://sparkDriver@mymachinename:51834]] [
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://sparkDriver@mymachinename:51834]
Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: mymachinename/10.21.6.4:51834
]
14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 got
disassociated, removing it.
14/10/15 11:48:10 ERROR EndpointWriter: AssociationError
[akka.tcp://sparkMaster@sparkmastermachinename:7077] ->
[akka.tcp://sparkDriver@mymachinename:51834]: Error [Association failed with
[akka.tcp://sparkDriver@mymachinename:51834]] [
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://sparkDriver@mymachinename:51834]
Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: mymachinename/10.21.6.4:51834
]
14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 got
disassociated, removing it.
14/10/15 11:48:10 ERROR EndpointWriter: AssociationError
[akka.tcp://sparkMaster@sparkmastermachinename:7077] ->
[akka.tcp://sparkDriver@mymachinename:51834]: Error [Association failed with
[akka.tcp://sparkDriver@mymachinename:51834]] [
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://sparkDriver@mymachinename:51834]
Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: mymachinename/10.21.6.4:51834
]
14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 got
disassociated, removing it.
I've excluded the "Connection refused" messages from the possible reasons for
my problem as I have the same message when I execute my example as standalone
application...
For information, here's the Spark master output when I execute the example in
standalone:
14/10/15 14:26:44 INFO Master: Registering app ts-aggregation
14/10/15 14:26:44 INFO Master: Registered app ts-aggregation with ID
app-20141015142644-0125
14/10/15 14:26:44 INFO Master: Launching executor app-20141015142644-0125/0 on
worker worker-20141015100053-sparkmastermachinename.lampiris.biz-39058
14/10/15 14:26:44 INFO Master: Launching executor app-20141015142644-0125/1 on
worker worker-20141015100053-sparkmastermachinename.lampiris.biz-46115
14/10/15 14:26:48 INFO Master: Removing executor app-20141015142644-0125/0
because it is EXITED
14/10/15 14:26:48 INFO Master: Removing executor app-20141015142644-0125/1
because it is EXITED
14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 got
disassociated, removing it.
14/10/15 14:26:49 INFO Master: Removing app app-20141015142644-0125
14/10/15 14:26:49 INFO LocalActorRef: Message
[akka.remote.transport.AssociationHandle$Disassociated] from
Actor[akka://sparkMaster/deadLetters] to
Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.21.6.4%3A58543-653#1905465071]
was not delivered. [216] dead letters encountered. This logging can be turned
off or adjusted with configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
14/10/15 14:26:49 INFO LocalActorRef: Message
[akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from
Actor[akka://sparkMaster/deadLetters] to
Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.21.6.4%3A58543-653#1905465071]
was not delivered. [217] dead letters encountered. This logging can be turned
off or adjusted with configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 got
disassociated, removing it.
14/10/15 14:26:49 ERROR EndpointWriter: AssociationError
[akka.tcp://sparkMaster@sparkmastermachinename:7077] ->
[akka.tcp://sparkDriver@mymachinename:58525]: Error [Association failed with
[akka.tcp://sparkDriver@mymachinename:58525]] [
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://sparkDriver@mymachinename:58525]
Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: mymachinename/10.21.6.4:58525
]
14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 got
disassociated, removing it.
14/10/15 14:26:49 ERROR EndpointWriter: AssociationError
[akka.tcp://sparkMaster@sparkmastermachinename:7077] ->
[akka.tcp://sparkDriver@mymachinename:58525]: Error [Association failed with
[akka.tcp://sparkDriver@mymachinename:58525]] [
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://sparkDriver@mymachinename:58525]
Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: mymachinename/10.21.6.4:58525
]
14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 got
disassociated, removing it.
14/10/15 14:26:49 ERROR EndpointWriter: AssociationError
[akka.tcp://sparkMaster@sparkmastermachinename:7077] ->
[akka.tcp://sparkDriver@mymachinename:58525]: Error [Association failed with
[akka.tcp://sparkDriver@mymachinename:58525]] [
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://sparkDriver@mymachinename:58525]
Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: mymachinename/10.21.6.4:58525
]
14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 got
disassociated, removing it.
Does anyone have any clue on what may cause this error?
Thank you for your help !
Regards
