Re: Running JavaBased Implementationof StreamingKmeans
Hi, Can someone please look into this and tell me whats wrong?and why am I not getting any output? Thanks & Regards Biplob Biswas On Sun, Jun 19, 2016 at 1:29 PM, Biplob Biswas <revolutioni...@gmail.com> wrote: > Hi, > > Thanks for that input, I tried doing that but apparently thats not working > as well. I thought i am having problems with my spark installation so I ran > simple word count and that works, so I am not really sure what the problem > is now. > > Is my translation of the scala code correct? I don't understand the syntax > of scala very well thus wrote my own implementation of streaming kmeans in > java, so i am hoping thats correct. > > Thanks & Regards > Biplob Biswas > > On Sun, Jun 19, 2016 at 3:23 AM, Akhil Das <ak...@hacked.work> wrote: > >> SparkStreaming does not pick up old files by default, so you need to >> start your job with master=local[2] (It needs 2 or more working threads, 1 >> to read the files and the other to do your computation) and once the job >> start to run, place your input files in the input directories and you can >> see them being picked up by sparkstreaming. >> >> On Sun, Jun 19, 2016 at 12:37 AM, Biplob Biswas <revolutioni...@gmail.com >> > wrote: >> >>> Hi, >>> >>> I tried local[*] and local[2] and the result is the same. I don't really >>> understand the problem here. >>> How can I confirm that the files are read properly? >>> >>> Thanks & Regards >>> Biplob Biswas >>> >>> On Sat, Jun 18, 2016 at 5:59 PM, Akhil Das <ak...@hacked.work> wrote: >>> >>>> Looks like you need to set your master to local[2] or local[*] >>>> >>>> On Sat, Jun 18, 2016 at 4:54 PM, Biplob Biswas < >>>> revolutioni...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I implemented the streamingKmeans example provided in the spark >>>>> website but >>>>> in Java. >>>>> The full implementation is here, >>>>> >>>>> http://pastebin.com/CJQfWNvk >>>>> >>>>> But i am not getting anything in the output except occasional >>>>> timestamps >>>>> like one below: >>>>> >>>>> --- >>>>> Time: 1466176935000 ms >>>>> --- >>>>> >>>>> Also, i have 2 directories: >>>>> "D:\spark\streaming example\Data Sets\training" >>>>> "D:\spark\streaming example\Data Sets\test" >>>>> >>>>> and inside these directories i have 1 file each >>>>> "samplegpsdata_train.txt" >>>>> and "samplegpsdata_test.txt" with training data having 500 datapoints >>>>> and >>>>> test data with 60 datapoints. >>>>> >>>>> I am very new to the spark systems and any help is highly appreciated. >>>>> >>>>> Thank you so much >>>>> Biplob Biswas >>>>> >>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Running-JavaBased-Implementationof-StreamingKmeans-tp27192.html >>>>> Sent from the Apache Spark User List mailing list archive at >>>>> Nabble.com. >>>>> >>>>> - >>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>> >>>>> >>>> >>>> >>>> -- >>>> Cheers! >>>> >>>> >>> >> >> >> -- >> Cheers! >> >> >
Re: Running JavaBased Implementationof StreamingKmeans
Hi, Thanks for that input, I tried doing that but apparently thats not working as well. I thought i am having problems with my spark installation so I ran simple word count and that works, so I am not really sure what the problem is now. Is my translation of the scala code correct? I don't understand the syntax of scala very well thus wrote my own implementation of streaming kmeans in java, so i am hoping thats correct. Thanks & Regards Biplob Biswas On Sun, Jun 19, 2016 at 3:23 AM, Akhil Das <ak...@hacked.work> wrote: > SparkStreaming does not pick up old files by default, so you need to start > your job with master=local[2] (It needs 2 or more working threads, 1 to > read the files and the other to do your computation) and once the job start > to run, place your input files in the input directories and you can see > them being picked up by sparkstreaming. > > On Sun, Jun 19, 2016 at 12:37 AM, Biplob Biswas <revolutioni...@gmail.com> > wrote: > >> Hi, >> >> I tried local[*] and local[2] and the result is the same. I don't really >> understand the problem here. >> How can I confirm that the files are read properly? >> >> Thanks & Regards >> Biplob Biswas >> >> On Sat, Jun 18, 2016 at 5:59 PM, Akhil Das <ak...@hacked.work> wrote: >> >>> Looks like you need to set your master to local[2] or local[*] >>> >>> On Sat, Jun 18, 2016 at 4:54 PM, Biplob Biswas <revolutioni...@gmail.com >>> > wrote: >>> >>>> Hi, >>>> >>>> I implemented the streamingKmeans example provided in the spark website >>>> but >>>> in Java. >>>> The full implementation is here, >>>> >>>> http://pastebin.com/CJQfWNvk >>>> >>>> But i am not getting anything in the output except occasional timestamps >>>> like one below: >>>> >>>> --- >>>> Time: 1466176935000 ms >>>> --- >>>> >>>> Also, i have 2 directories: >>>> "D:\spark\streaming example\Data Sets\training" >>>> "D:\spark\streaming example\Data Sets\test" >>>> >>>> and inside these directories i have 1 file each >>>> "samplegpsdata_train.txt" >>>> and "samplegpsdata_test.txt" with training data having 500 datapoints >>>> and >>>> test data with 60 datapoints. >>>> >>>> I am very new to the spark systems and any help is highly appreciated. >>>> >>>> Thank you so much >>>> Biplob Biswas >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://apache-spark-user-list.1001560.n3.nabble.com/Running-JavaBased-Implementationof-StreamingKmeans-tp27192.html >>>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>>> >>>> - >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: user-h...@spark.apache.org >>>> >>>> >>> >>> >>> -- >>> Cheers! >>> >>> >> > > > -- > Cheers! > >
Re: Running JavaBased Implementationof StreamingKmeans
SparkStreaming does not pick up old files by default, so you need to start your job with master=local[2] (It needs 2 or more working threads, 1 to read the files and the other to do your computation) and once the job start to run, place your input files in the input directories and you can see them being picked up by sparkstreaming. On Sun, Jun 19, 2016 at 12:37 AM, Biplob Biswas <revolutioni...@gmail.com> wrote: > Hi, > > I tried local[*] and local[2] and the result is the same. I don't really > understand the problem here. > How can I confirm that the files are read properly? > > Thanks & Regards > Biplob Biswas > > On Sat, Jun 18, 2016 at 5:59 PM, Akhil Das <ak...@hacked.work> wrote: > >> Looks like you need to set your master to local[2] or local[*] >> >> On Sat, Jun 18, 2016 at 4:54 PM, Biplob Biswas <revolutioni...@gmail.com> >> wrote: >> >>> Hi, >>> >>> I implemented the streamingKmeans example provided in the spark website >>> but >>> in Java. >>> The full implementation is here, >>> >>> http://pastebin.com/CJQfWNvk >>> >>> But i am not getting anything in the output except occasional timestamps >>> like one below: >>> >>> --- >>> Time: 1466176935000 ms >>> --- >>> >>> Also, i have 2 directories: >>> "D:\spark\streaming example\Data Sets\training" >>> "D:\spark\streaming example\Data Sets\test" >>> >>> and inside these directories i have 1 file each "samplegpsdata_train.txt" >>> and "samplegpsdata_test.txt" with training data having 500 datapoints and >>> test data with 60 datapoints. >>> >>> I am very new to the spark systems and any help is highly appreciated. >>> >>> Thank you so much >>> Biplob Biswas >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Running-JavaBased-Implementationof-StreamingKmeans-tp27192.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> - >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >> >> -- >> Cheers! >> >> > -- Cheers!
Re: Running JavaBased Implementationof StreamingKmeans
Hi, I tried local[*] and local[2] and the result is the same. I don't really understand the problem here. How can I confirm that the files are read properly? Thanks & Regards Biplob Biswas On Sat, Jun 18, 2016 at 5:59 PM, Akhil Das <ak...@hacked.work> wrote: > Looks like you need to set your master to local[2] or local[*] > > On Sat, Jun 18, 2016 at 4:54 PM, Biplob Biswas <revolutioni...@gmail.com> > wrote: > >> Hi, >> >> I implemented the streamingKmeans example provided in the spark website >> but >> in Java. >> The full implementation is here, >> >> http://pastebin.com/CJQfWNvk >> >> But i am not getting anything in the output except occasional timestamps >> like one below: >> >> --- >> Time: 1466176935000 ms >> --- >> >> Also, i have 2 directories: >> "D:\spark\streaming example\Data Sets\training" >> "D:\spark\streaming example\Data Sets\test" >> >> and inside these directories i have 1 file each "samplegpsdata_train.txt" >> and "samplegpsdata_test.txt" with training data having 500 datapoints and >> test data with 60 datapoints. >> >> I am very new to the spark systems and any help is highly appreciated. >> >> Thank you so much >> Biplob Biswas >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Running-JavaBased-Implementationof-StreamingKmeans-tp27192.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > > > -- > Cheers! > >
Re: Running JavaBased Implementationof StreamingKmeans
Looks like you need to set your master to local[2] or local[*] On Sat, Jun 18, 2016 at 4:54 PM, Biplob Biswas <revolutioni...@gmail.com> wrote: > Hi, > > I implemented the streamingKmeans example provided in the spark website but > in Java. > The full implementation is here, > > http://pastebin.com/CJQfWNvk > > But i am not getting anything in the output except occasional timestamps > like one below: > > --- > Time: 1466176935000 ms > --- > > Also, i have 2 directories: > "D:\spark\streaming example\Data Sets\training" > "D:\spark\streaming example\Data Sets\test" > > and inside these directories i have 1 file each "samplegpsdata_train.txt" > and "samplegpsdata_test.txt" with training data having 500 datapoints and > test data with 60 datapoints. > > I am very new to the spark systems and any help is highly appreciated. > > Thank you so much > Biplob Biswas > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Running-JavaBased-Implementationof-StreamingKmeans-tp27192.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- Cheers!
Running JavaBased Implementationof StreamingKmeans
Hi, I implemented the streamingKmeans example provided in the spark website but in Java. The full implementation is here, http://pastebin.com/CJQfWNvk But i am not getting anything in the output except occasional timestamps like one below: --- Time: 1466176935000 ms --- Also, i have 2 directories: "D:\spark\streaming example\Data Sets\training" "D:\spark\streaming example\Data Sets\test" and inside these directories i have 1 file each "samplegpsdata_train.txt" and "samplegpsdata_test.txt" with training data having 500 datapoints and test data with 60 datapoints. I am very new to the spark systems and any help is highly appreciated. Thank you so much Biplob Biswas -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-JavaBased-Implementationof-StreamingKmeans-tp27190.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
