Hi, This is the command I am using for submitting my application, SimpleApp:
./bin/spark-submit --class org.apache.spark.examples.SimpleApp --deploy-mode client --master spark://karthik:7077 $SPARK_HOME/examples/*/scala-*/spark-examples-*.jar /text-data On Thu, Sep 25, 2014 at 6:52 AM, Tobias Pfeiffer <t...@preferred.jp> wrote: > Hi, > > I assume you unintentionally did not reply to the list, so I'm adding it > back to CC. > > How do you submit your job to the cluster? > > Tobias > > > On Thu, Sep 25, 2014 at 2:21 AM, rapelly kartheek <kartheek.m...@gmail.com > > wrote: > >> How do I find out whether a node in the cluster is a master or slave?? >> Till now I was thinking that slaves file under the conf folder makes the >> difference. Also, the MASTER_MASTER_IP in the spark-env.sh file. >> >> what else differentiates a slave from the master?? >> >> On Wed, Sep 24, 2014 at 10:46 PM, rapelly kartheek < >> kartheek.m...@gmail.com> wrote: >> >>> The job execution is taking place perfectly. Previously, all my print >>> statements used to be stored in spark/work/*/stdout file. But, now after >>> doing the rsync, I find that none of the prtint statements are getting >>> reflected in the stdout file under work folder. But, when I go to the code, >>> I find the statements in the code. But, they are not reflected into the >>> stdout file as before. >>> >>> Can you please tell me where I went wrong. All I want is to see my >>> mofication in the code getting relected in output >>> . >>> >>> On Wed, Sep 24, 2014 at 10:22 PM, rapelly kartheek < >>> kartheek.m...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I have a very important and fundamental doubt: I have rsynced the >>>> entire spark folder from the master to all slaves in the cluster. When I >>>> execute a job, its working perfectly. But, when I rsync the entire spark >>>> folder of the master to all the slaves, is it not that I am sending the >>>> master configurations to all the slaves and making the slaves behave like >>>> master?? >>>> >>>> First of all, is it correct to rsync the entire spark folder?? >>>> But, if I change only one file, then how do I rsync it to all?? >>>> >>>> On Fri, Sep 19, 2014 at 8:44 PM, rapelly kartheek < >>>> kartheek.m...@gmail.com> wrote: >>>> >>>>> Thank you Soumya Simantha and Tobias. I've deleted the contents of the >>>>> work folder in all the nodes. >>>>> Now its working perfectly as it was before. >>>>> >>>>> Thank you >>>>> Karthik >>>>> >>>>> On Fri, Sep 19, 2014 at 4:46 PM, Soumya Simanta < >>>>> soumya.sima...@gmail.com> wrote: >>>>> >>>>>> One possible reason is maybe that the checkpointing directory >>>>>> $SPARK_HOME/work is rsynced as well. >>>>>> Try emptying the contents of the work folder on each node and try >>>>>> again. >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Sep 19, 2014 at 4:53 AM, rapelly kartheek < >>>>>> kartheek.m...@gmail.com> wrote: >>>>>> >>>>>>> I >>>>>>> * followed this command:rsync -avL --progress path/to/spark-1.0.0 >>>>>>> username@destinationhostname:* >>>>>>> >>>>>>> >>>>>>> *path/to/destdirectory. Anyway, for now, I did it individually for >>>>>>> each node.* >>>>>>> >>>>>>> I have copied to each node at a time individually using the above >>>>>>> command. So, I guess the copying may not contain any mixture of files. >>>>>>> Also, as of now, I am not facing any MethodNotFound exceptions. But, >>>>>>> there >>>>>>> is no job execution taking place. >>>>>>> >>>>>>> After sometime, one by one, each goes down and the cluster shuts >>>>>>> down. >>>>>>> >>>>>>> On Fri, Sep 19, 2014 at 2:15 PM, Tobias Pfeiffer <t...@preferred.jp> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> On Fri, Sep 19, 2014 at 5:17 PM, rapelly kartheek < >>>>>>>> kartheek.m...@gmail.com> wrote: >>>>>>>> >>>>>>>>> > , >>>>>>>>> >>>>>>>>> * you have copied a lot of files from various hosts to >>>>>>>>> username@slave3:path* >>>>>>>>> only from one node to all the other nodes... >>>>>>>>> >>>>>>>> >>>>>>>> I don't think rsync can do that in one command as you described. My >>>>>>>> guess is that now you have a wild mixture of jar files all across your >>>>>>>> cluster which will lead to fancy exceptions like MethodNotFound etc., >>>>>>>> that's maybe why your cluster is not working correctly. >>>>>>>> >>>>>>>> Tobias >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
