Have you looked at: https://spark.apache.org/docs/latest/programming-guide.html#accumulators
On Wed, Jun 15, 2016 at 1:24 PM, Mathieu Longtin <math...@closetwork.org> wrote: > Is there a way to report warnings from the workers back to the driver > process? > > Let's say I have an RDD and do this: > > newrdd = rdd.map(somefunction) > > In *somefunction*, I want to catch when there are invalid values in *rdd *and > either put them in another RDD or send some sort of message back. > > Is that possible? > -- > Mathieu Longtin > 1-514-803-8977 >
