[
https://issues.apache.org/jira/browse/SPARK-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-6524.
------------------------------
Resolution: Not a Problem
Going to go out on a limb and say this looks like an app problem, or at least,
the next step is on user@
> Problem connecting JAVA API to Spark Yarn Cluster or yarn Client
> ----------------------------------------------------------------
>
> Key: SPARK-6524
> URL: https://issues.apache.org/jira/browse/SPARK-6524
> Project: Spark
> Issue Type: Question
> Components: Java API, YARN
> Affects Versions: 1.2.0
> Environment: Ubuntu 14.10
> java JDK 1.8
> Reporter: milan.b
>
> Hi Team,
> I am trying to submit a spark job to yarn-cluster or yarn client
> using Java API but I was unable to do so. Following is the configuration code
>
> System.setProperty("SPARK_YARN_MODE", "true");
> SparkConf sparkYarnConf = new SparkConf().setAppName("Spark yarn")
> .setMaster("yarn-cluster")
> .set("spark.executor.memory", "258m")
> .set("spark.driver.memory", "2588m")
> .set("spark.yarn.app.id", "append");
> ClientArguments clientArgs =new ClientArguments(args,
> sparkYarnConf);
> Error :
> org.apache.spark.util.Utils] (MSC service thread 1-2) Service 'SparkUI' could
> not bind on port 4043. Attempting port 4044.
> 20:57:41,808 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:41 WARN
> Utils: Service 'SparkUI' could not bind on port 4043. Attempting port 4044.
> 20:57:41,842 INFO [org.apache.spark.util.Utils] (MSC service thread 1-2)
> Successfully started service 'SparkUI' on port 4044.
> 20:57:41,846 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:41 INFO
> Utils: Successfully started service 'SparkUI' on port 4044.
> 20:57:41,850 INFO [org.apache.spark.ui.SparkUI] (MSC service thread 1-2)
> Started SparkUI at http://D7271:4044
> 20:57:41,850 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:41 INFO
> SparkUI: Started SparkUI at http://D7271:4044
> 20:57:41,968 INFO [org.apache.spark.scheduler.cluster.YarnClusterScheduler]
> (MSC service thread 1-2) Created YarnClusterScheduler
> 20:57:41,968 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:41 INFO
> YarnClusterScheduler: Created YarnClusterScheduler
> 20:57:42,243 INFO [org.apache.spark.network.netty.NettyBlockTransferService]
> (MSC service thread 1-2) Server created on 45902
> 20:57:42,243 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:42 INFO
> NettyBlockTransferService: Server created on 45902
> 20:57:42,247 INFO [org.apache.spark.storage.BlockManagerMaster] (MSC service
> thread 1-2) Trying to register BlockManager
> 20:57:42,247 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:42 INFO
> BlockManagerMaster: Trying to register BlockManager
> 20:57:42,264 INFO [org.apache.spark.storage.BlockManagerMasterActor]
> (sparkDriver-akka.actor.default-dispatcher-2) Registering block manager
> D7271:45902 with 246.0 MB RAM, BlockManagerId(<driver>, D7271, 45902)
> 20:57:42,265 ERROR [stderr] (sparkDriver-akka.actor.default-dispatcher-2)
> 15/03/24 20:57:42 INFO BlockManagerMasterActor: Registering block manager
> D7271:45902 with 246.0 MB RAM, BlockManagerId(<driver>, D7271, 45902)
> 20:57:42,272 INFO [org.apache.spark.storage.BlockManagerMaster] (MSC service
> thread 1-2) Registered BlockManager
> 20:57:42,276 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:42 INFO
> BlockManagerMaster: Registered BlockManager
> 20:57:42,330 ERROR [org.springframework.web.context.ContextLoader] (MSC
> service thread 1-2) Context initialization failed:
> org.springframework.beans.factory.BeanCreationException: Error creating bean
> with name 'exampleInitBean' defined in ServletContext resource
> [/WEB-INF/mvc-dispatcher-servlet.xml]: Invocation of init method failed;
> nested exception is java.lang.NullPointerException
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1553)
> [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)
> [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
> [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302)
> [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
> [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298)
> [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
> [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:703)
> [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
> at
> org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:760)
> [spring-context-4.0.6.RELEASE.jar:4.0.6.RELEASE]
> at
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
> [spring-context-4.0.6.RELEASE.jar:4.0.6.RELEASE]
> at
> org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:403)
> [spring-web-4.0.6.RELEASE.jar:4.0.6.RELEASE]
> at
> org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
> [spring-web-4.0.6.RELEASE.jar:4.0.6.RELEASE]
> at
> org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106)
> [spring-web-4.0.6.RELEASE.jar:4.0.6.RELEASE]
> at
> io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:173)
> [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
> at
> io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:193)
> [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
> at
> org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:87)
> at
> org.wildfly.extension.undertow.deployment.UndertowDeploymentService.start(UndertowDeploymentService.java:72)
> at
> org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
> at
> org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [rt.jar:1.8.0_31]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [rt.jar:1.8.0_31]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31]
> Caused by: java.lang.NullPointerException
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$.sparkContextInitialized(ApplicationMaster.scala:534)
> [spark-yarn_2.10-1.2.1.jar:1.2.1]
> at
> org.apache.spark.scheduler.cluster.YarnClusterScheduler.postStartHook(YarnClusterScheduler.scala:46)
> [spark-yarn_2.10-1.2.1.jar:1.2.1]
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
> [spark-core_2.10-1.2.1.jar:1.2.1]
> at
> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
> [spark-core_2.10-1.2.1.jar:1.2.1]
> at
> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:75)
> [spark-core_2.10-1.2.1.jar:1.2.1]
> at com.prodapt.spark.util.SparkUtil.initSparkContext(SparkUtil.java:52)
> [spark-pro-utils-1.0.jar:]
> at
> com.prodapt.billanalysis.spark.SparkInitializer.initializeSpark(SparkInitializer.java:19)
> [classes:]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [rt.jar:1.8.0_31]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [rt.jar:1.8.0_31]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [rt.jar:1.8.0_31]
> at java.lang.reflect.Method.invoke(Method.java:483) [rt.jar:1.8.0_31]
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1681)
> [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1620)
> [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1549)
> [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE]
> ... 21 more
> Any help is appreciated
> Thanks and Regards
> Milan.B
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
