Re: Reporting warnings from workers

It turns out you can easily use a Python set, so I can send back a list of failed files. Thanks. On Wed, Jun 15, 2016 at 4:28 PM Ted Yu wrote: > Have you looked at: > > https://spark.apache.org/docs/latest/programming-guide.html#accumulators > > On Wed, Jun 15, 2016 at 1:24

Re: Reporting warnings from workers

Have you looked at: https://spark.apache.org/docs/latest/programming-guide.html#accumulators On Wed, Jun 15, 2016 at 1:24 PM, Mathieu Longtin wrote: > Is there a way to report warnings from the workers back to the driver > process? > > Let's say I have an RDD and do

Reporting warnings from workers

Is there a way to report warnings from the workers back to the driver process? Let's say I have an RDD and do this: newrdd = rdd.map(somefunction) In *somefunction*, I want to catch when there are invalid values in *rdd *and either put them in another RDD or send some sort of message back. Is