Thank you, yes that makes sense.
I was aware of transformations and actions, but did not realize foreach was
an action. I've found the exhaustive list here
http://spark.apache.org/docs/latest/programming-guide.html#actions
and it's clear to me again.
Thank you for your help!
On Wed, Mar 9, 2016 a
Hi Jeff,
> But in our development environment, the returned RDD results were empty and
> b.function(_) was never executed
what do you mean by "the returned RDD results were empty", did you try
running a foreach, collect or any other action on the returned RDD[C]?
Spark provides two kinds of oper
Spark RDDs are lazily computed and hence unless an 'action' is applied
which mandates the computation - there won't be any computation. You can
read more on spark docs.
On Mar 9, 2016 7:11 AM, "Hemminger Jeff" wrote:
>
> I'm currently developing a Spark Streaming application.
>
> I have a functio
I'm currently developing a Spark Streaming application.
I have a function that receives an RDD and an object instance as a
parameter, and returns an RDD:
def doTheThing(a: RDD[A], b: B): RDD[C]
Within the function, I do some processing within a map of the RDD.
Like this:
def doTheThing(a: RD
