How about a PR that rejects a context configured for local or local[1]? As
I understand it is not intended to work and has bitten several people.
On Jul 14, 2014 12:24 AM, "Michael Campbell" <michael.campb...@gmail.com>
wrote:
> This almost had me not using Spark; I couldn't get any output. It is not
> at all obvious what's going on here to the layman (and to the best of my
> knowledge, not documented anywhere), but now you know you'll be able to
> answer this question for the numerous people that will also have it.
>
>
> On Sun, Jul 13, 2014 at 5:24 PM, Walrus theCat <walrusthe...@gmail.com>
> wrote:
>
>> Great success!
>>
>> I was able to get output to the driver console by changing the
>> construction of the Streaming Spark Context from:
>>
>> val ssc = new StreamingContext("local" /**TODO change once a cluster is
>> up **/,
>> "AppName", Seconds(1))
>>
>>
>> to:
>>
>> val ssc = new StreamingContext("local[2]" /**TODO change once a cluster
>> is up **/,
>> "AppName", Seconds(1))
>>
>>
>> I found something that tipped me off that this might work by digging
>> through this mailing list.
>>
>>
>> On Sun, Jul 13, 2014 at 2:00 PM, Walrus theCat <walrusthe...@gmail.com>
>> wrote:
>>
>>> More strange behavior:
>>>
>>> lines.foreachRDD(x => println(x.first)) // works
>>> lines.foreachRDD(x => println((x.count,x.first))) // no output is
>>> printed to driver console
>>>
>>>
>>>
>>>
>>> On Sun, Jul 13, 2014 at 11:47 AM, Walrus theCat <walrusthe...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> Thanks for your interest.
>>>>
>>>> lines.foreachRDD(x => println(x.count))
>>>>
>>>> And I got 0 every once in a while (which I think is strange, because
>>>> lines.print prints the input I'm giving it over the socket.)
>>>>
>>>>
>>>> When I tried:
>>>>
>>>> lines.map(_->1).reduceByKey(_+_).foreachRDD(x => println(x.count))
>>>>
>>>> I got no count.
>>>>
>>>> Thanks
>>>>
>>>>
>>>> On Sun, Jul 13, 2014 at 11:34 AM, Tathagata Das <
>>>> tathagata.das1...@gmail.com> wrote:
>>>>
>>>>> Try doing DStream.foreachRDD and then printing the RDD count and
>>>>> further inspecting the RDD.
>>>>> On Jul 13, 2014 1:03 AM, "Walrus theCat" <walrusthe...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have a DStream that works just fine when I say:
>>>>>>
>>>>>> dstream.print
>>>>>>
>>>>>> If I say:
>>>>>>
>>>>>> dstream.map(_,1).print
>>>>>>
>>>>>> that works, too. However, if I do the following:
>>>>>>
>>>>>> dstream.reduce{case(x,y) => x}.print
>>>>>>
>>>>>> I don't get anything on my console. What's going on?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>
>>>>
>>>
>>
>
