[
https://issues.apache.org/jira/browse/PIG-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025723#comment-16025723
]
liyunzhang_intel commented on PIG-5167:
---------------------------------------
[~nkollar]: my suggestion is
1. add a new verify_pig_script to Limit_13
{code}
{
'num' => 13,
'execonly' => 'spark', # Limit_4 failed on
Spark: distinct doesn't do implicit sort like it does in MR
'pig' =>q\a = load
':INPATH:/singlefile/studentnulltab10k';
b = distinct a;
c = limit b 100;
store c into ':OUTPATH:';\,
'verify_pig_script' =>q\a = load
':INPATH:/singlefile/studentnulltab10k';
b = distinct a;
c = limit b 100;
store c into ':OUTPATH:';\,
}
{code}
It is not very good because the script and verify_script are same.
2. If #option1 is not accepted, remove Limit_13 and leave PIG-5167 open
I have tried #option1 but failed because the verify_pig_script is executed by
benchmark mode(mr). so the result is different in spark and mr.
> Limit_4 is failing with spark exec type
> ---------------------------------------
>
> Key: PIG-5167
> URL: https://issues.apache.org/jira/browse/PIG-5167
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Nandor Kollar
> Assignee: Nandor Kollar
> Fix For: spark-branch
>
> Attachments: PIG-5167_2.patch, PIG-5167_3.patch, PIG-5167.patch
>
>
> results are different:
> {code}
> diff <(head -n 5 Limit_4.out/out_sorted) <(head -n 5
> Limit_4_benchmark.out/out_sorted)
> 1,5c1,5
> < 50 3.00
> < 74 2.22
> < alice carson 66 2.42
> < alice quirinius 71 0.03
> < alice van buren 28 2.50
> ---
> > bob allen 0.28
> > bob allen 22 0.92
> > bob allen 25 2.54
> > bob allen 26 2.35
> > bob allen 27 2.17
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
