Re: Reporting warnings from workers

[Mathieu Longtin]

It turns out you can easily use a Python set, so I can send back a list of
failed files. Thanks.

On Wed, Jun 15, 2016 at 4:28 PM Ted Yu  wrote:

> Have you looked at:
>
> https://spark.apache.org/docs/latest/programming-guide.html#accumulators
>
> On Wed, Jun 15, 2016 at 1:24 PM, Mathieu Longtin 
> wrote:
>
>> Is there a way to report warnings from the workers back to the driver
>> process?
>>
>> Let's say I have an RDD and do this:
>>
>> newrdd = rdd.map(somefunction)
>>
>> In *somefunction*, I want to catch when there are invalid values in *rdd
>> *and either put them in another RDD or send some sort of message back.
>>
>> Is that possible?
>> --
>> Mathieu Longtin
>> 1-514-803-8977
>>
>
> --
Mathieu Longtin
1-514-803-8977

Re: Reporting warnings from workers

[Ted Yu]

Have you looked at:

https://spark.apache.org/docs/latest/programming-guide.html#accumulators

On Wed, Jun 15, 2016 at 1:24 PM, Mathieu Longtin 
wrote:

> Is there a way to report warnings from the workers back to the driver
> process?
>
> Let's say I have an RDD and do this:
>
> newrdd = rdd.map(somefunction)
>
> In *somefunction*, I want to catch when there are invalid values in *rdd *and
> either put them in another RDD or send some sort of message back.
>
> Is that possible?
> --
> Mathieu Longtin
> 1-514-803-8977
>

Reporting warnings from workers

[Mathieu Longtin]

Is there a way to report warnings from the workers back to the driver
process?

Let's say I have an RDD and do this:

newrdd = rdd.map(somefunction)

In *somefunction*, I want to catch when there are invalid values in *rdd *and
either put them in another RDD or send some sort of message back.

Is that possible?
-- 
Mathieu Longtin
1-514-803-8977