YARN 2.7.1 (running on the cluster) was built with Java 1.8, I assume. Have you used the following command to retrieve / inspect logs ? yarn logs -applicationId
Cheers On Mon, Oct 5, 2015 at 8:41 AM, mvle <m...@us.ibm.com> wrote: > Hi, > > I have successfully run pyspark on Spark 1.5.1 on YARN 2.7.1 with Java > OpenJDK 1.7. > However, when I run the same test on Java OpenJDK 1.8 (or Oracle Java 1.8), > I cannot start up pyspark. > Has anyone been able to run Spark on YARN with Java 1.8? > > I get ApplicationMaster disassociated messages... > > 15/10/05 09:55:05 INFO cluster.YarnClientSchedulerBackend: Application > application_1444055784612_0003 has started running. > 15/10/05 09:55:05 INFO util.Utils: Successfully started service > 'org.apache.spark.network.netty.NettyBlockTransferService' on port 53518. > 15/10/05 09:55:05 INFO netty.NettyBlockTransferService: Server created on > 53518 > 15/10/05 09:55:05 INFO storage.BlockManagerMaster: Trying to register > BlockManager > 15/10/05 09:55:05 INFO storage.BlockManagerMasterEndpoint: Registering > block > manager xxx.172.232.xx:53518 with 530.0 MB RAM, BlockManagerId(driver, > xxx.172.232.xx, 53518) > 15/10/05 09:55:05 INFO storage.BlockManagerMaster: Registered BlockManager > 15/10/05 09:55:05 INFO scheduler.EventLoggingListener: Logging events to > hdfs://h-all1-001:9000/user/hadoop/sparklogs/application_1444055784612_0003 > 15/10/05 09:55:07 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: > ApplicationMaster has disassociated: xxx.172.232.xx:42628 > 15/10/05 09:55:07 WARN remote.ReliableDeliverySupervisor: Association with > remote system [akka.tcp://sparkyar...@xxx.172.232.xx:42628] has failed, > address is now gated for [5000] ms. Reason: [Disassociated] > 15/10/05 09:55:07 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: > ApplicationMaster has disassociated: xxx.172.232.xx:42628 > 15/10/05 09:55:09 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: > ApplicationMaster registered as > AkkaRpcEndpointRef(Actor[akka.tcp://sparkyar...@xxx.172.232.xx > :60077/user/YarnAM#-560267402]) > 15/10/05 09:55:09 INFO cluster.YarnClientSchedulerBackend: Add WebUI > Filter. > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS > -> h-all1-001, PROXY_URI_BASES -> > http://h-all1-001:8088/proxy/application_1444055784612_0003), > /proxy/application_1444055784612_0003 > 15/10/05 09:55:09 INFO ui.JettyUtils: Adding filter: > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter > 15/10/05 09:55:13 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: > ApplicationMaster has disassociated: xxx.172.232.xx:60077 > 15/10/05 09:55:13 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: > ApplicationMaster has disassociated: xxx.172.232.xx:60077 > 15/10/05 09:55:13 WARN remote.ReliableDeliverySupervisor: Association with > remote system [akka.tcp://sparkyar...@xxx.172.232.xx:60077] has failed, > address is now gated for [5000] ms. Reason: [Disassociated] > 15/10/05 09:55:13 ERROR cluster.YarnClientSchedulerBackend: Yarn > application > has already exited with state FINISHED! > 15/10/05 09:55:13 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/metrics/json,null} > 15/10/05 09:55:13 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/stages/stage/kill,null} > 15/10/05 09:55:13 INFO handler.ContextHandler: stopped > o.s.j.s.ServletContextHandler{/api,null} > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-YARN-using-Java-1-8-fails-tp24925.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
