Thank you Tathagata. This had me going for far too long.
On Mon, Jul 14, 2014 at 3:57 PM, Tathagata Das <tathagata.das1...@gmail.com> wrote: > The problem is not really for local[1] or local. The problem arises when > there are more input streams than there are cores. > But I agree, for people who are just beginning to use it by running it > locally, there should be a check addressing this. > > I made a JIRA for this. > https://issues.apache.org/jira/browse/SPARK-2464 > > TD > > > On Sun, Jul 13, 2014 at 4:26 PM, Sean Owen <so...@cloudera.com> wrote: > >> How about a PR that rejects a context configured for local or local[1]? >> As I understand it is not intended to work and has bitten several people. >> On Jul 14, 2014 12:24 AM, "Michael Campbell" <michael.campb...@gmail.com> >> wrote: >> >>> This almost had me not using Spark; I couldn't get any output. It is >>> not at all obvious what's going on here to the layman (and to the best of >>> my knowledge, not documented anywhere), but now you know you'll be able to >>> answer this question for the numerous people that will also have it. >>> >>> >>> On Sun, Jul 13, 2014 at 5:24 PM, Walrus theCat <walrusthe...@gmail.com> >>> wrote: >>> >>>> Great success! >>>> >>>> I was able to get output to the driver console by changing the >>>> construction of the Streaming Spark Context from: >>>> >>>> val ssc = new StreamingContext("local" /**TODO change once a cluster >>>> is up **/, >>>> "AppName", Seconds(1)) >>>> >>>> >>>> to: >>>> >>>> val ssc = new StreamingContext("local[2]" /**TODO change once a cluster >>>> is up **/, >>>> "AppName", Seconds(1)) >>>> >>>> >>>> I found something that tipped me off that this might work by digging >>>> through this mailing list. >>>> >>>> >>>> On Sun, Jul 13, 2014 at 2:00 PM, Walrus theCat <walrusthe...@gmail.com> >>>> wrote: >>>> >>>>> More strange behavior: >>>>> >>>>> lines.foreachRDD(x => println(x.first)) // works >>>>> lines.foreachRDD(x => println((x.count,x.first))) // no output is >>>>> printed to driver console >>>>> >>>>> >>>>> >>>>> >>>>> On Sun, Jul 13, 2014 at 11:47 AM, Walrus theCat < >>>>> walrusthe...@gmail.com> wrote: >>>>> >>>>>> >>>>>> Thanks for your interest. >>>>>> >>>>>> lines.foreachRDD(x => println(x.count)) >>>>>> >>>>>> And I got 0 every once in a while (which I think is strange, because >>>>>> lines.print prints the input I'm giving it over the socket.) >>>>>> >>>>>> >>>>>> When I tried: >>>>>> >>>>>> lines.map(_->1).reduceByKey(_+_).foreachRDD(x => println(x.count)) >>>>>> >>>>>> I got no count. >>>>>> >>>>>> Thanks >>>>>> >>>>>> >>>>>> On Sun, Jul 13, 2014 at 11:34 AM, Tathagata Das < >>>>>> tathagata.das1...@gmail.com> wrote: >>>>>> >>>>>>> Try doing DStream.foreachRDD and then printing the RDD count and >>>>>>> further inspecting the RDD. >>>>>>> On Jul 13, 2014 1:03 AM, "Walrus theCat" <walrusthe...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I have a DStream that works just fine when I say: >>>>>>>> >>>>>>>> dstream.print >>>>>>>> >>>>>>>> If I say: >>>>>>>> >>>>>>>> dstream.map(_,1).print >>>>>>>> >>>>>>>> that works, too. However, if I do the following: >>>>>>>> >>>>>>>> dstream.reduce{case(x,y) => x}.print >>>>>>>> >>>>>>>> I don't get anything on my console. What's going on? >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >
