Hi Ajay,
Are you trying to save to your local file system or to HDFS?
// This would save to HDFS under "/user/hadoop/counter"
counter.saveAsTextFile("/user/hadoop/counter");
David
On Sun, Aug 30, 2015 at 11:21 AM, Ajay Chander <itsche...@gmail.com> wrote:
> Hi Everyone,
>
> Recently we have installed spark on yarn in hortonworks cluster. Now I am
> trying to run a wordcount program in my eclipse and I
> did setMaster("local") and I see the results that's as expected. Now I want
> to submit the same job to my yarn cluster from my eclipse. In storm
> basically I was doing the same by using StormSubmitter class and by passing
> nimbus & zookeeper host to Config object. I was looking for something
> exactly the same.
>
> When I went through the documentation online, it read that I am suppose to
> "export HADOOP_HOME_DIR=path to the conf dir". So now I copied the conf
> folder from one of sparks gateway node to my local Unix box. Now I did
> export that dir...
>
> export HADOOP_HOME_DIR=/Users/user1/Documents/conf/
>
> And I did the same in .bash_profile too. Now when I do echo
> $HADOOP_HOME_DIR, I see the path getting printed in the command prompt. Now
> my assumption is, in my program when I change setMaster("local") to
> setMaster("yarn-client") my program should pick up the resource mangers i.e
> yarn cluster info from the directory which I have exported and the job
> should get submitted to resolve manager from my eclipse. But somehow it's
> not happening. Please tell me if my assumption is wrong or if I am missing
> anything here.
>
> I have attached the word count program that I was using. Any help is
> highly appreciated.
>
> Thank you,
> Ajay
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
--
### Confidential e-mail, for recipient's (or recipients') eyes only, not
for distribution. ###
