This would still be on a single mapper - not all mappers : which, I
suppose, is the intention ?
I was pretty confused by this requirement (still am actually :) ).
Regards,
Mridul
On Tuesday 03 May 2011 01:04 AM, Dmitriy Ryaboy wrote:
Ok, so you don't want to exit from a UDF, you want to stop processing the
mapper.
What you want is an early EOF from your InputFormat. So I guess the answer
is to have a custom inputFormat that monitors some object that your UDF can
modify, and the input format can report EOF if the condition is satisfied
(or it's actually out of data).
D
On Sun, May 1, 2011 at 11:11 PM, souri datta<[email protected]>wrote:
I think I was not able to explain my problem here.
So, the udf runs in a mapper job which is doing some very expensive i/o
operation. If it processes all the input files, the time consumed will be
huge. So, I want to exit from the mapper on meeting the particular
condition
so that I don't process all the input files.
Thanks,
Souri
On Mon, May 2, 2011 at 1:46 AM, Dmitriy Ryaboy<[email protected]> wrote:
Right. I assume there is a reason you don't want to or are unable to have
your udf check your condition and call return?
-----Original Message-----
From: "souri datta"<[email protected]>
To: [email protected]
Sent: 5/1/2011 8:50 AM
Subject: Re: Early exit from Pig udf
Meaning it should be able to finish quickly(return from the method).
On Fri, Apr 29, 2011 at 9:52 PM, Dmitriy Ryaboy<[email protected]>
wrote:
What do you mean by return?
On Fri, Apr 29, 2011 at 5:01 AM, souri datta<[email protected]
wrote:
Hi,
I have a pig udf.My requirement is , on meeting certain criteria, I
want
to
return from Pig udf.Is there any way I can early exit from Pig udf?
Also, how can it be done in a Map/Reduce job?
Thanks,
Souri
