[
https://issues.apache.org/jira/browse/PIG-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohini Palaniswamy updated PIG-3045:
------------------------------------
Description:
PIG-2782 fixed a number of tests where the parameters passed to the
verification sort was incorrect.
However, there are still problems with the patch in PIG-2782 where it
introduced new errors while checking. E.g. the pig script sorts on column one
and two, but the verification only checks that output is sorted on column one.
was:
PIG-2782 fixed a number of tests where the parameters passed to the
verification sort was incorrect.
However, there are still problems with the patch in PIG-2782. E.g. the pig
script sorts on column one and two, but the verification only checks that
output is sorted on column one.
For file test/e2e/pig/tests/nightly.conf:
===================
@@ -1728,7 +1728,7 @@
'pig' =>q\a = load
':INPATH:/singlefile/studentnulltab10k' as (name:chararray, age:int,
gpa:double);
b = order a by name, age, gpa;
store b into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0', '-1', '+1n', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2n,2n'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2n,3n'],
===================
Similar
@@ -1736,7 +1736,7 @@
'pig' =>q\a = load
':INPATH:/singlefile/studentnulltab10k' as (name:chararray, age:int,
gpa:double);
b = order a by name desc, age desc, gpa desc;
store b into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0r', '-1', '+1nr', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,2nr'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,3nr'],
===================
and
@@ -1752,7 +1752,7 @@
'pig' =>q\a = load
':INPATH:/singlefile/studentnulltab10k' as (name, age:long, gpa:float);
b = order a by name desc, age desc, gpa desc;
store b into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0r', '-1', '+1nr', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,2nr'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,3nr'],
===================
@@ -1847,7 +1847,7 @@
'pig' => q\a = load
':INPATH:/singlefile/studentnulltab10k' as (name:chararray, age:int,
gpa:double);
b = order a by *;
store b into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0', '-1', '+1n', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2n,2n'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2n,3n'],
===================
@@ -1855,7 +1855,7 @@
'pig' => q\a = load
':INPATH:/singlefile/studentnulltab10k' as (name:chararray, age:int,
gpa:double);
b = order a by * desc;
store b into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0r', '-1', '+1nr', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,2nr'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2nr,3nr'],
===================
@@ -1943,7 +1943,7 @@
c = filter b by $0 > 'a'; -- break the sort/limit optimization
d = limit c 100;
store d into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0', '-1'],
+ 'sortArgs' => ['-t', ' ', '-k', '1,1'],
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1,2'],
===================
@@ -1952,7 +1952,7 @@
b = order a by $0, $1;
c = limit b 100;
store c into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0', '-1'],
+ 'sortArgs' => ['-t', ' ', '-k', '1,1'],
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1,2'],
===================
@@ -2222,7 +2222,7 @@
D = order B by age, extra;
store D into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+1n', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '2n,2n'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '2n,2n', '-k', '4,4'],
(This last is decidedly minor, as the 'extra' column is empty, but for sake of
consistency...)
Environment: (was: Mac OS X Lion 10.7.3
Hadoop 1.0.1-SNAPSHOT
Apache Pig version 0.11.0-SNAPSHOT (r1355798))
Affects Version/s: (was: 0.10.1)
(was: 0.10.0)
0.11
Assignee: Rohini Palaniswamy (was: Cheolsoo Park)
Summary: Specifying sorting field(s) at nightly.conf - fix
sortArgs (was: Specifying sorting field(s) at nightly.conf - further changes)
Moving the description from Egil outlining the issues to Comment section as it
is too long. The line numbers in the diff come from applying the patch of
PIG-2782 to branch-0.10.
For file test/e2e/pig/tests/nightly.conf:
===================
@@ -1728,7 +1728,7 @@
'pig' =>q\a = load
':INPATH:/singlefile/studentnulltab10k' as (name:chararray, age:int,
gpa:double);
b = order a by name, age, gpa;
store b into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0', '-1', '+1n', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2n,2n'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2n,3n'],
===================
Similar
@@ -1736,7 +1736,7 @@
'pig' =>q\a = load
':INPATH:/singlefile/studentnulltab10k' as (name:chararray, age:int,
gpa:double);
b = order a by name desc, age desc, gpa desc;
store b into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0r', '-1', '+1nr', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,2nr'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,3nr'],
===================
and
@@ -1752,7 +1752,7 @@
'pig' =>q\a = load
':INPATH:/singlefile/studentnulltab10k' as (name, age:long, gpa:float);
b = order a by name desc, age desc, gpa desc;
store b into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0r', '-1', '+1nr', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,2nr'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,3nr'],
===================
@@ -1847,7 +1847,7 @@
'pig' => q\a = load
':INPATH:/singlefile/studentnulltab10k' as (name:chararray, age:int,
gpa:double);
b = order a by *;
store b into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0', '-1', '+1n', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2n,2n'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2n,3n'],
===================
@@ -1855,7 +1855,7 @@
'pig' => q\a = load
':INPATH:/singlefile/studentnulltab10k' as (name:chararray, age:int,
gpa:double);
b = order a by * desc;
store b into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0r', '-1', '+1nr', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,2nr'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2nr,3nr'],
===================
@@ -1943,7 +1943,7 @@
c = filter b by $0 > 'a'; -- break the sort/limit optimization
d = limit c 100;
store d into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0', '-1'],
+ 'sortArgs' => ['-t', ' ', '-k', '1,1'],
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1,2'],
===================
@@ -1952,7 +1952,7 @@
b = order a by $0, $1;
c = limit b 100;
store c into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0', '-1'],
+ 'sortArgs' => ['-t', ' ', '-k', '1,1'],
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1,2'],
===================
@@ -2222,7 +2222,7 @@
D = order B by age, extra;
store D into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+1n', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '2n,2n'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '2n,2n', '-k', '4,4'],
(This last is decidedly minor, as the 'extra' column is empty, but for sake of
consistency...)
> Specifying sorting field(s) at nightly.conf - fix sortArgs
> ----------------------------------------------------------
>
> Key: PIG-3045
> URL: https://issues.apache.org/jira/browse/PIG-3045
> Project: Pig
> Issue Type: Bug
> Components: e2e harness
> Affects Versions: 0.11
> Reporter: Egil Sorensen
> Assignee: Rohini Palaniswamy
> Labels: test
> Fix For: 0.11
>
>
> PIG-2782 fixed a number of tests where the parameters passed to the
> verification sort was incorrect.
> However, there are still problems with the patch in PIG-2782 where it
> introduced new errors while checking. E.g. the pig script sorts on column one
> and two, but the verification only checks that output is sorted on column one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
