If I Store the temporary result and then Load, the problem disappears.
Thanks for all your help!
________________________________________
From: Steve Terrell <sterr...@oculus360.us>
Sent: Friday, May 6, 2016 7:36:33 AM
To: user
Subject: Re: BinSedesTuple cannot be cast to or.apache.pig.pen.util.ExampleTuple
Can you reduce the script and input to the bare minimum needed to reproduce
the problem?
On Thu, May 5, 2016 at 11:11 PM, Chloe Huang <chloe.hu...@ussuning.com>
wrote:
> Hi Guys,
>
>
> When I am running my Pig Script, I am doing OK with small data sets. But
> if I use the same script for big data sets, I got this Exception:
>
> ERROR 0: Exception while executing (Name: Kff: Filter[bag] - scope-142
> Operator Key: scope-142):
> org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception
> while executing [PORelationToExprProject (Name:
> RelationToExpressionProject[bag][*] - scope-134 Operator Key: scope-134)
> children: null at []]: java.lang.ClassCastException:
> org.apache.pig.data.BinSedesTuple cannot be cast to
> org.apache.pig.pen.util.ExampleTuple
>
>
> And I am using the following script:
>
> F = LOAD './q' AS (ori_query:chararray, t:chararray, w:chararray,
> wordCnts:long);
>
>
> G = JOIN F by t, E by word;
>
>
> H = FOREACH G GENERATE F::ori_query as ori_query, F::t as word, F::w as
> weight, F::wordCnts as wordCnts, E::proID as proID, E::proScore as proScore;
>
>
> I = FOREACH H GENERATE ori_query, proID, wordCnts,
> que_udfs.getQueryScore(weight, proScore) as queryScore;
>
>
> J = COGROUP I BY (ori_query, proID);
>
>
> K = FOREACH J GENERATE group.ori_query as ori_query, group.proID as proID,
> I.wordCnts as wordCnts, I.queryScore as queryScore, SIZE(I.queryScore) as
> matchCnts;
>
> K_fil = FILTER K BY (ori_query is not NULL) AND (proID is not NULL);
>
>
> Kgetter = FOREACH K_fil {
>
> count = limit wordCnts 1;
>
> GENERATE ori_query, proID, flatten(count) as count, queryScore,
> matchCnts;
>
> }
>
> Kff = FILTER Kgetter BY (count is not NULL) and (matchCnts is not NULL)
> and (count == matchCnts) and (queryScore is not NULL);
>
> E has this data format:
>
> 8801 102588050 0.555555555556
>
> ZERO 102588050 0.0
>
> s 102588050 0.555555555556
>
> zero 102588050 0.555555555556
>
> US 102588050 0.0376254180602
>
> hello 102588050 0.0376254180602
>
> people 102588050 0.0376254180602
>
> any 102588050 0.0376254180602
>
> F has this data format:
>
> 8801 8801s 4.285714285714286 2
>
> hello Media hello 5.714285714285715 2
>
> ZERO mate s ZERO 7.127210931539155 3
>
> ZERO mate s mate 0.6432799241636031 3
>
> ZERO mate s s 2.229509144297241 3
>
> US US 10.0 1
>
> any people any 0.1620255817609147 2
>
> any people people 9.837974418239085 2
>
>
> Does anyone has any idea why this Exception is happening?
>
> Many thanks in advance!
>
> Best Regards,
> Chloe
>
>
>
