[
https://issues.apache.org/jira/browse/PIG-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Egil Sorensen updated PIG-3045:
-------------------------------
Description:
PIG-2782 fixed a number of tests where the parameters passed to the
verification sort was incorrect.
However, there are still problems with the patch in PIG-2782. E.g. the pig
script sorts on column one and two, but the verification only checks that
output is sorted on column one.
For file test/e2e/pig/tests/nightly.conf:
===================
@@ -1728,7 +1728,7 @@
'pig' =>q\a = load
':INPATH:/singlefile/studentnulltab10k' as (name:chararray, age:int,
gpa:double);
b = order a by name, age, gpa;
store b into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0', '-1', '+1n', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2n,2n'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2n,3n'],
===================
Similar
@@ -1736,7 +1736,7 @@
'pig' =>q\a = load
':INPATH:/singlefile/studentnulltab10k' as (name:chararray, age:int,
gpa:double);
b = order a by name desc, age desc, gpa desc;
store b into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0r', '-1', '+1nr', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,2nr'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,3nr'],
===================
and
@@ -1752,7 +1752,7 @@
'pig' =>q\a = load
':INPATH:/singlefile/studentnulltab10k' as (name, age:long, gpa:float);
b = order a by name desc, age desc, gpa desc;
store b into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0r', '-1', '+1nr', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,2nr'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,3nr'],
===================
@@ -1847,7 +1847,7 @@
'pig' => q\a = load
':INPATH:/singlefile/studentnulltab10k' as (name:chararray, age:int,
gpa:double);
b = order a by *;
store b into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0', '-1', '+1n', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2n,2n'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2n,3n'],
===================
@@ -1855,7 +1855,7 @@
'pig' => q\a = load
':INPATH:/singlefile/studentnulltab10k' as (name:chararray, age:int,
gpa:double);
b = order a by * desc;
store b into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0r', '-1', '+1nr', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,2nr'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2nr,3nr'],
===================
@@ -1943,7 +1943,7 @@
c = filter b by $0 > 'a'; -- break the sort/limit optimization
d = limit c 100;
store d into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0', '-1'],
+ 'sortArgs' => ['-t', ' ', '-k', '1,1'],
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1,2'],
===================
@@ -1952,7 +1952,7 @@
b = order a by $0, $1;
c = limit b 100;
store c into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+0', '-1'],
+ 'sortArgs' => ['-t', ' ', '-k', '1,1'],
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '1,2'],
===================
@@ -2222,7 +2222,7 @@
D = order B by age, extra;
store D into ':OUTPATH:';\,
- 'sortArgs' => ['-t', ' ', '+1n', '-2'],
+ 'sortArgs' => ['-t', ' ', '-k', '2n,2n'],
},
Should have been:
+ 'sortArgs' => ['-t', ' ', '-k', '2n,2n', '-k', '4,4'],
(This last is decidedly minor, as the 'extra' column is empty, but for sake of
consistency...)
was:
After running the Checkin tests, it fails because one of the parameters passed
to the sort is incorrect (instead of +1 -2, on POSIX is -k2,2).
According to this http://ss64.com/bash/sort.html, it was on an old notation.
> Specifying sorting field(s) at nightly.conf - further changes
> -------------------------------------------------------------
>
> Key: PIG-3045
> URL: https://issues.apache.org/jira/browse/PIG-3045
> Project: Pig
> Issue Type: Bug
> Components: e2e harness
> Affects Versions: 0.10.0, 0.10.1
> Environment: Mac OS X Lion 10.7.3
> Hadoop 1.0.1-SNAPSHOT
> Apache Pig version 0.11.0-SNAPSHOT (r1355798)
> Reporter: Egil Sorensen
> Assignee: Cheolsoo Park
> Labels: test
> Fix For: 0.11
>
>
> PIG-2782 fixed a number of tests where the parameters passed to the
> verification sort was incorrect.
> However, there are still problems with the patch in PIG-2782. E.g. the pig
> script sorts on column one and two, but the verification only checks that
> output is sorted on column one.
> For file test/e2e/pig/tests/nightly.conf:
> ===================
> @@ -1728,7 +1728,7 @@
> 'pig' =>q\a = load
> ':INPATH:/singlefile/studentnulltab10k' as (name:chararray, age:int,
> gpa:double);
> b = order a by name, age, gpa;
> store b into ':OUTPATH:';\,
> - 'sortArgs' => ['-t', ' ', '+0', '-1', '+1n', '-2'],
> + 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2n,2n'],
> },
> Should have been:
> + 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2n,3n'],
> ===================
> Similar
> @@ -1736,7 +1736,7 @@
> 'pig' =>q\a = load
> ':INPATH:/singlefile/studentnulltab10k' as (name:chararray, age:int,
> gpa:double);
> b = order a by name desc, age desc, gpa desc;
> store b into ':OUTPATH:';\,
> - 'sortArgs' => ['-t', ' ', '+0r', '-1', '+1nr', '-2'],
> + 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,2nr'],
> },
> Should have been:
> + 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,3nr'],
> ===================
> and
> @@ -1752,7 +1752,7 @@
> 'pig' =>q\a = load
> ':INPATH:/singlefile/studentnulltab10k' as (name, age:long, gpa:float);
> b = order a by name desc, age desc, gpa desc;
> store b into ':OUTPATH:';\,
> - 'sortArgs' => ['-t', ' ', '+0r', '-1', '+1nr', '-2'],
> + 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,2nr'],
> },
> Should have been:
> + 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,3nr'],
> ===================
> @@ -1847,7 +1847,7 @@
> 'pig' => q\a = load
> ':INPATH:/singlefile/studentnulltab10k' as (name:chararray, age:int,
> gpa:double);
> b = order a by *;
> store b into ':OUTPATH:';\,
> - 'sortArgs' => ['-t', ' ', '+0', '-1', '+1n', '-2'],
> + 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2n,2n'],
> },
> Should have been:
> + 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2n,3n'],
> ===================
> @@ -1855,7 +1855,7 @@
> 'pig' => q\a = load
> ':INPATH:/singlefile/studentnulltab10k' as (name:chararray, age:int,
> gpa:double);
> b = order a by * desc;
> store b into ':OUTPATH:';\,
> - 'sortArgs' => ['-t', ' ', '+0r', '-1', '+1nr', '-2'],
> + 'sortArgs' => ['-t', ' ', '-k', '1r,1r', '-k', '2nr,2nr'],
> },
> Should have been:
> + 'sortArgs' => ['-t', ' ', '-k', '1,1', '-k', '2nr,3nr'],
> ===================
> @@ -1943,7 +1943,7 @@
> c = filter b by $0 > 'a'; -- break the sort/limit optimization
> d = limit c 100;
> store d into ':OUTPATH:';\,
> - 'sortArgs' => ['-t', ' ', '+0', '-1'],
> + 'sortArgs' => ['-t', ' ', '-k', '1,1'],
> Should have been:
> + 'sortArgs' => ['-t', ' ', '-k', '1,2'],
> ===================
> @@ -1952,7 +1952,7 @@
> b = order a by $0, $1;
> c = limit b 100;
> store c into ':OUTPATH:';\,
> - 'sortArgs' => ['-t', ' ', '+0', '-1'],
> + 'sortArgs' => ['-t', ' ', '-k', '1,1'],
> Should have been:
> + 'sortArgs' => ['-t', ' ', '-k', '1,2'],
> ===================
> @@ -2222,7 +2222,7 @@
> D = order B by age, extra;
> store D into ':OUTPATH:';\,
> - 'sortArgs' => ['-t', ' ', '+1n', '-2'],
> + 'sortArgs' => ['-t', ' ', '-k', '2n,2n'],
> },
> Should have been:
> + 'sortArgs' => ['-t', ' ', '-k', '2n,2n', '-k',
> '4,4'],
> (This last is decidedly minor, as the 'extra' column is empty, but for sake
> of consistency...)
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
