Hey Alex,
No problem. Please go ahead and close it as duplicate.
Sorry if I came across as being rude (did not mean to)
Regards
-...@nkur
On 2/11/10 3:32 PM, "Alex Parvulescu" <[email protected]> wrote:
Ok, my bad
Should I close this one now?
alex
On Thu, Feb 11, 2010 at 9:41 AM, Ankur C. Goel <[email protected]> wrote:
> There is already a JIRA (with patch) opened for this -
> http://issues.apache.org/jira/browse/PIG-1233
>
> -...@nkur
>
> On 2/11/10 2:01 PM, "Alex Parvulescu" <[email protected]> wrote:
>
> Hello,
>
> thanks Dmitriy!
>
> Wow how could I have missed that one? seems easy enough: AVG( val == null ?
> 0 : val)
> I'll give it a go asap :)
>
> Here is the Jira issue, I hope I got everything in there
> https://issues.apache.org/jira/browse/PIG-1236
>
> thanks,
> Alex
>
> On Tue, Feb 9, 2010 at 6:04 PM, Dmitriy Ryaboy <[email protected]> wrote:
>
> > This is a legit bug, I think, in the new accumulator interface
> > implementation. Nice find, Alex. Can you open a jira?
> >
> > btw, I saw on your blog you had some issues with how pig was ignoring
> > nulls when calculating average values before (this is documented and
> > expected behavior btw), and wound up writing your own. You don't
> > really need to:
> >
> > averages = foreach A generate AVG( val == null ? 0 : val);
> >
> >
> > On Tue, Feb 9, 2010 at 2:57 AM, Mridul Muralidharan
> > <[email protected]> wrote:
> > >
> > > Someone from pig team can answer better if there is any impl issues
> here
> > > with average.
> > > But assuming there are none, if you can treat null's as zeros - you
> could
> > > add additional checks to the statements, to allow it to proceed.
> > >
> > > Something to check for :
> > > a) If A == null, generate 0.
> > > b) If A.v == null, generate 0. (This is a strong possibility too).
> > >
> > >
> > > Regards,
> > > Mridul
> > >
> > > On Tuesday 09 February 2010 04:08 PM, Alex Parvulescu wrote:
> > >>
> > >> hello Mridul,
> > >>
> > >> and thanks for the quick answer!
> > >>
> > >> A itself is not null, just some group by values. I can't drop the
> nulls
> > >> because I also need a count in the group by, even if it's only null
> > >> values.
> > >>
> > >> I just wandered if theres anything to be done about the NPE to make it
> > >> more clear, that's all.
> > >>
> > >> I guess you can see this as an eventual feature / improvement of some
> > >> sort, no problems :)
> > >>
> > >> alex
> > >>
> > >> On Tue, Feb 9, 2010 at 11:35 AM, Mridul Muralidharan
> > >> <[email protected] <mailto:[email protected]>> wrote:
> > >>
> > >>
> > >> On second thought, probably A itself is NULL - in which case you
> > >> will need a null check on A, and not on A.v (which, I think, is
> > >> handled iirc).
> > >>
> > >>
> > >> Regards,
> > >> Mridul
> > >>
> > >>
> > >> On Tuesday 09 February 2010 04:02 PM, Mridul Muralidharan wrote:
> > >>
> > >>
> > >> Without knowing rest of the script, you could do something like
> :
> > >>
> > >> C = FOREACH B {
> > >> X = FILTER A BY v IS NOT NULL;
> > >> GENERATE group, (int)AVG(X) as statsavg;
> > >> };
> > >>
> > >> I am assuming it is cos there are nulls in your bag field.
> > >>
> > >> Regards,
> > >> Mridul
> > >>
> > >>
> > >> On Tuesday 09 February 2010 03:52 PM, Alex Parvulescu wrote:
> > >>
> > >> Hello,
> > >>
> > >> I ran into a NPE today, which seems to be my fault, but I'm
> > >> wondering if
> > >> there anythig that could be done to make the error more
> > clear.
> > >>
> > >> What I did it is:
> > >> 'C = FOREACH B GENERATE group, (int)AVG(A.v) as statsavg;'
> > >> The problem here is the AVG ran into some null values and
> > >> returned null. And
> > >> consequently the cast failed with a NPE.
> > >>
> > >> This is the stacktrace
> > >> 2010-02-09 11:14:36,444 [Thread-85] WARN
> > >> org.apache.hadoop.mapred.LocalJobRunner - job_local_0006
> > >> java.lang.NullPointerException
> > >> at
> > >> org.apache.pig.builtin.IntAvg.getValue(IntAvg.java:282)
> > >> at
> > org.apache.pig.builtin.IntAvg.getValue(IntAvg.java:39)
> > >> at
> > >>
> > >>
> >
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:208)
> > >> at
> > >>
> > >>
> >
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:281)
> > >> at
> > >>
> > >>
> >
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:182)
> > >> at
> > >>
> > >>
> >
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:352)
> > >> at
> > >>
> > >>
> >
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:277)
> > >> at
> > >>
> > >>
> >
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:423)
> > >> at
> > >>
> > >>
> >
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:391)
> > >> at
> > >>
> > >>
> >
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:371)
> > >> at
> > >>
> > >>
> >
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:239)
> > >> at
> > >>
> > >>
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:463)
> > >> at
> > >>
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
> > >> at
> > >>
> > >>
> > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:215)
> > >>
> > >> Now, because I'm not well aware how this works, I did not
> > >> realize that the
> > >> cast throws the NPE and not the computation of the average
> > >> function on null
> > >> values provided by the data set.
> > >> I initially thought this was a bug in Pig.
> > >>
> > >> I know the NPE is all on me, but is there anything you can
> > >> do to improve the
> > >> error message
> > >>
> > >> thanks,
> > >> alex
> > >>
> > >>
> > >>
> > >>
> > >
> > >
> >
>
>
