milan.b created SPARK-6524:
------------------------------
Summary: Problem connecting JAVA API to Spark Yarn Cluster or yarn
Client
Key: SPARK-6524
URL: https://issues.apache.org/jira/browse/SPARK-6524
Project: Spark
Issue Type: Question
Affects Versions: 1.2.0
Environment: Ubuntu 14.10
java JDK 1.8
Reporter: milan.b
Hi Team,
I am trying to submit a spark job to yarn-cluster or yarn client using
Java API but I was unable to do so. Following is the configuration code
System.setProperty("SPARK_YARN_MODE", "true");
SparkConf sparkYarnConf = new SparkConf().setAppName("Spark yarn")
.setMaster("yarn-cluster")
.set("spark.executor.memory", "258m")
.set("spark.driver.memory", "2588m")
.set("spark.yarn.app.id", "append");
ClientArguments clientArgs =new ClientArguments(args,
sparkYarnConf);
Error :
org.apache.spark.util.Utils] (MSC service thread 1-2) Service 'SparkUI' could
not bind on port 4043. Attempting port 4044.
20:57:41,808 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:41 WARN
Utils: Service 'SparkUI' could not bind on port 4043. Attempting port 4044.
20:57:41,842 INFO [org.apache.spark.util.Utils] (MSC service thread 1-2)
Successfully started service 'SparkUI' on port 4044.
20:57:41,846 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:41 INFO
Utils: Successfully started service 'SparkUI' on port 4044.
20:57:41,850 INFO [org.apache.spark.ui.SparkUI] (MSC service thread 1-2)
Started SparkUI at http://D7271:4044
20:57:41,850 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:41 INFO
SparkUI: Started SparkUI at http://D7271:4044
20:57:41,968 INFO [org.apache.spark.scheduler.cluster.YarnClusterScheduler]
(MSC service thread 1-2) Created YarnClusterScheduler
20:57:41,968 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:41 INFO
YarnClusterScheduler: Created YarnClusterScheduler
20:57:42,243 INFO [org.apache.spark.network.netty.NettyBlockTransferService]
(MSC service thread 1-2) Server created on 45902
20:57:42,243 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:42 INFO
NettyBlockTransferService: Server created on 45902
20:57:42,247 INFO [org.apache.spark.storage.BlockManagerMaster] (MSC service
thread 1-2) Trying to register BlockManager
20:57:42,247 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:42 INFO
BlockManagerMaster: Trying to register BlockManager
20:57:42,264 INFO [org.apache.spark.storage.BlockManagerMasterActor]
(sparkDriver-akka.actor.default-dispatcher-2) Registering block manager
D7271:45902 with 246.0 MB RAM, BlockManagerId(<driver>, D7271, 45902)
20:57:42,265 ERROR [stderr] (sparkDriver-akka.actor.default-dispatcher-2)
15/03/24 20:57:42 INFO BlockManagerMasterActor: Registering block manager
D7271:45902 with 246.0 MB RAM, BlockManagerId(<driver>, D7271, 45902)
20:57:42,272 INFO [org.apache.spark.storage.BlockManagerMaster] (MSC service
thread 1-2) Registered BlockManager
20:57:42,276 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:42 INFO
BlockManagerMaster: Registered BlockManager
20:57:42,330 ERROR [org.springframework.web.context.ContextLoader] (MSC service
thread 1-2) Context initialization failed:
org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'exampleInitBean' defined in ServletContext resource
[/WEB-INF/mvc-dispatcher-servlet.xml]: Invocation of init method failed; nested
exception is java.lang.NullPointerException
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1553)
[spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)
[spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
[spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302)
[spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
[spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298)
[spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
[spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:703)
[spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:760)
[spring-context-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
[spring-context-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at
org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:403)
[spring-web-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at
org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
[spring-web-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at
org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106)
[spring-web-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at
io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:173)
[undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
at
io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:193)
[undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:87)
at
org.wildfly.extension.undertow.deployment.UndertowDeploymentService.start(UndertowDeploymentService.java:72)
at
org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
at
org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[rt.jar:1.8.0_31]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[rt.jar:1.8.0_31]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31]
Caused by: java.lang.NullPointerException
at
org.apache.spark.deploy.yarn.ApplicationMaster$.sparkContextInitialized(ApplicationMaster.scala:534)
[spark-yarn_2.10-1.2.1.jar:1.2.1]
at
org.apache.spark.scheduler.cluster.YarnClusterScheduler.postStartHook(YarnClusterScheduler.scala:46)
[spark-yarn_2.10-1.2.1.jar:1.2.1]
at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
[spark-core_2.10-1.2.1.jar:1.2.1]
at
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
[spark-core_2.10-1.2.1.jar:1.2.1]
at
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:75)
[spark-core_2.10-1.2.1.jar:1.2.1]
at com.prodapt.spark.util.SparkUtil.initSparkContext(SparkUtil.java:52)
[spark-pro-utils-1.0.jar:]
at
com.prodapt.billanalysis.spark.SparkInitializer.initializeSpark(SparkInitializer.java:19)
[classes:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.8.0_31]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_31]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_31]
at java.lang.reflect.Method.invoke(Method.java:483) [rt.jar:1.8.0_31]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1681)
[spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1620)
[spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1549)
[spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
... 21 more
Any help is appreciated
Thanks and Regards
Milan.B
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
