RE: Problem submitting a spark job using yarn-client as master

2016-11-16 Thread David Robison
] Sent: Tuesday, November 15, 2016 9:27 PM To: David Robison Cc: user@spark.apache.org Subject: Re: Problem submitting a spark job using yarn-client as master you can set hdfs as defaults, sparksession.sparkContext().hadoopConfiguration().set("fs.defaultFS", “hdfs://master_node:8020”

Re: Problem submitting a spark job using yarn-client as master

2016-11-15 Thread Rohit Verma
you can set hdfs as defaults, sparksession.sparkContext().hadoopConfiguration().set("fs.defaultFS", “hdfs://master_node:8020”); Regards Rohit On Nov 16, 2016, at 3:15 AM, David Robison mailto:david.robi...@psgglobal.net>> wrote: I am trying to submit a spark job through the yarn-client master

Problem submitting a spark job using yarn-client as master

2016-11-15 Thread David Robison
I am trying to submit a spark job through the yarn-client master setting. The job gets created and submitted to the clients but immediately errors out. Here is the relevant portion of the log: 15:39:37,385 INFO [org.apache.spark.deploy.yarn.Client] (default task-1) Requesting a new application