milan.b created SPARK-6524: ------------------------------ Summary: Problem connecting JAVA API to Spark Yarn Cluster or yarn Client Key: SPARK-6524 URL: https://issues.apache.org/jira/browse/SPARK-6524 Project: Spark Issue Type: Question Affects Versions: 1.2.0 Environment: Ubuntu 14.10 java JDK 1.8
Reporter: milan.b Hi Team, I am trying to submit a spark job to yarn-cluster or yarn client using Java API but I was unable to do so. Following is the configuration code System.setProperty("SPARK_YARN_MODE", "true"); SparkConf sparkYarnConf = new SparkConf().setAppName("Spark yarn") .setMaster("yarn-cluster") .set("spark.executor.memory", "258m") .set("spark.driver.memory", "2588m") .set("spark.yarn.app.id", "append"); ClientArguments clientArgs =new ClientArguments(args, sparkYarnConf); Error : org.apache.spark.util.Utils] (MSC service thread 1-2) Service 'SparkUI' could not bind on port 4043. Attempting port 4044. 20:57:41,808 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:41 WARN Utils: Service 'SparkUI' could not bind on port 4043. Attempting port 4044. 20:57:41,842 INFO [org.apache.spark.util.Utils] (MSC service thread 1-2) Successfully started service 'SparkUI' on port 4044. 20:57:41,846 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:41 INFO Utils: Successfully started service 'SparkUI' on port 4044. 20:57:41,850 INFO [org.apache.spark.ui.SparkUI] (MSC service thread 1-2) Started SparkUI at http://D7271:4044 20:57:41,850 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:41 INFO SparkUI: Started SparkUI at http://D7271:4044 20:57:41,968 INFO [org.apache.spark.scheduler.cluster.YarnClusterScheduler] (MSC service thread 1-2) Created YarnClusterScheduler 20:57:41,968 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:41 INFO YarnClusterScheduler: Created YarnClusterScheduler 20:57:42,243 INFO [org.apache.spark.network.netty.NettyBlockTransferService] (MSC service thread 1-2) Server created on 45902 20:57:42,243 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:42 INFO NettyBlockTransferService: Server created on 45902 20:57:42,247 INFO [org.apache.spark.storage.BlockManagerMaster] (MSC service thread 1-2) Trying to register BlockManager 20:57:42,247 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:42 INFO BlockManagerMaster: Trying to register BlockManager 20:57:42,264 INFO [org.apache.spark.storage.BlockManagerMasterActor] (sparkDriver-akka.actor.default-dispatcher-2) Registering block manager D7271:45902 with 246.0 MB RAM, BlockManagerId(<driver>, D7271, 45902) 20:57:42,265 ERROR [stderr] (sparkDriver-akka.actor.default-dispatcher-2) 15/03/24 20:57:42 INFO BlockManagerMasterActor: Registering block manager D7271:45902 with 246.0 MB RAM, BlockManagerId(<driver>, D7271, 45902) 20:57:42,272 INFO [org.apache.spark.storage.BlockManagerMaster] (MSC service thread 1-2) Registered BlockManager 20:57:42,276 ERROR [stderr] (MSC service thread 1-2) 15/03/24 20:57:42 INFO BlockManagerMaster: Registered BlockManager 20:57:42,330 ERROR [org.springframework.web.context.ContextLoader] (MSC service thread 1-2) Context initialization failed: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'exampleInitBean' defined in ServletContext resource [/WEB-INF/mvc-dispatcher-servlet.xml]: Invocation of init method failed; nested exception is java.lang.NullPointerException at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1553) [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539) [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475) [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302) [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298) [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:703) [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:760) [spring-context-4.0.6.RELEASE.jar:4.0.6.RELEASE] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482) [spring-context-4.0.6.RELEASE.jar:4.0.6.RELEASE] at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:403) [spring-web-4.0.6.RELEASE.jar:4.0.6.RELEASE] at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306) [spring-web-4.0.6.RELEASE.jar:4.0.6.RELEASE] at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106) [spring-web-4.0.6.RELEASE.jar:4.0.6.RELEASE] at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:173) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:193) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:87) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.start(UndertowDeploymentService.java:72) at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_31] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31] Caused by: java.lang.NullPointerException at org.apache.spark.deploy.yarn.ApplicationMaster$.sparkContextInitialized(ApplicationMaster.scala:534) [spark-yarn_2.10-1.2.1.jar:1.2.1] at org.apache.spark.scheduler.cluster.YarnClusterScheduler.postStartHook(YarnClusterScheduler.scala:46) [spark-yarn_2.10-1.2.1.jar:1.2.1] at org.apache.spark.SparkContext.<init>(SparkContext.scala:509) [spark-core_2.10-1.2.1.jar:1.2.1] at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61) [spark-core_2.10-1.2.1.jar:1.2.1] at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:75) [spark-core_2.10-1.2.1.jar:1.2.1] at com.prodapt.spark.util.SparkUtil.initSparkContext(SparkUtil.java:52) [spark-pro-utils-1.0.jar:] at com.prodapt.billanalysis.spark.SparkInitializer.initializeSpark(SparkInitializer.java:19) [classes:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_31] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_31] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_31] at java.lang.reflect.Method.invoke(Method.java:483) [rt.jar:1.8.0_31] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1681) [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1620) [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1549) [spring-beans-4.0.6.RELEASE.jar:4.0.6.RELEASE] ... 21 more Any help is appreciated Thanks and Regards Milan.B -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org