[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too complex and hard to work with
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895463#action_12895463 ] Hadoop QA commented on PIG-1178: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12451203/PIG-1178-5.patch against trunk revision 982423. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 91 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/375/console This message is automatically generated. > LogicalPlan and Optimizer are too complex and hard to work with > --- > > Key: PIG-1178 > URL: https://issues.apache.org/jira/browse/PIG-1178 > Project: Pig > Issue Type: Improvement >Reporter: Alan Gates >Assignee: Daniel Dai > Fix For: 0.8.0 > > Attachments: expressions-2.patch, expressions.patch, lp.patch, > lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, pig_1178.patch, pig_1178.patch, > PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, > pig_1178_3.4.patch, pig_1178_3.patch > > > The current implementation of the logical plan and the logical optimizer in > Pig has proven to not be easily extensible. Developer feedback has indicated > that adding new rules to the optimizer is quite burdensome. In addition, the > logical plan has been an area of numerous bugs, many of which have been > difficult to fix. Developers also feel that the logical plan is difficult to > understand and maintain. The root cause for these issues is that a number of > design decisions that were made as part of the 0.2 rewrite of the front end > have now proven to be sub-optimal. The heart of this proposal is to revisit a > number of those proposals and rebuild the logical plan with a simpler design > that will make it much easier to maintain the logical plan as well as extend > the logical optimizer. > See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full > details. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1199) help includes obsolete options
[ https://issues.apache.org/jira/browse/PIG-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895460#action_12895460 ] Hadoop QA commented on PIG-1199: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12451182/PIG-1199.patch against trunk revision 981984. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 406 release audit warnings (more than the trunk's current 405 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/374/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/374/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/374/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/374/console This message is automatically generated. > help includes obsolete options > -- > > Key: PIG-1199 > URL: https://issues.apache.org/jira/browse/PIG-1199 > Project: Pig > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Olga Natkovich >Assignee: Olga Natkovich > Fix For: 0.8.0 > > Attachments: PIG-1199.patch > > > This is confusing to users -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1527) No need to deserialize UDFContext on the client side
[ https://issues.apache.org/jira/browse/PIG-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895310#action_12895310 ] Hadoop QA commented on PIG-1527: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12451181/PIG-1527.patch against trunk revision 981984. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 406 release audit warnings (more than the trunk's current 405 warnings). +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/373/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/373/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/373/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/373/console This message is automatically generated. > No need to deserialize UDFContext on the client side > > > Key: PIG-1527 > URL: https://issues.apache.org/jira/browse/PIG-1527 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Richard Ding >Assignee: Richard Ding > Fix For: 0.8.0 > > Attachments: PIG-1527.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1461) support union operation that merges based on column names
[
https://issues.apache.org/jira/browse/PIG-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895212#action_12895212
]
Hadoop QA commented on PIG-1461:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12451175/PIG-1461.1.patch
against trunk revision 981984.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 407 release audit warnings
(more than the trunk's current 405 warnings).
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/372/testReport/
Release audit warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/372/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/372/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/372/console
This message is automatically generated.
> support union operation that merges based on column names
> -
>
> Key: PIG-1461
> URL: https://issues.apache.org/jira/browse/PIG-1461
> Project: Pig
> Issue Type: New Feature
> Components: impl
>Affects Versions: 0.8.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.8.0
>
> Attachments: PIG-1461.1.patch, PIG-1461.patch
>
>
> When the data has schema, it often makes sense to union on column names in
> schema rather than the position of the columns.
> The behavior of existing union operator should remain backward compatible .
> This feature can be supported using either a new operator or extending union
> to support 'using' clause . I am thinking of having a new operator called
> either unionschema or merge . Does anybody have any other suggestions for the
> syntax ?
> example -
> L1 = load 'x' as (a,b);
> L2 = load 'y' as (b,c);
> U = unionschema L1, L2;
> describe U;
> U: {a:bytearray, b:byetarray, c:bytearray}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1533) Compression codec should be a per-store property
[
https://issues.apache.org/jira/browse/PIG-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895139#action_12895139
]
Hadoop QA commented on PIG-1533:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12451140/PIG-1533.patch
against trunk revision 981984.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/371/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/371/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/371/console
This message is automatically generated.
> Compression codec should be a per-store property
>
>
> Key: PIG-1533
> URL: https://issues.apache.org/jira/browse/PIG-1533
> Project: Pig
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Richard Ding
>Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1533.patch
>
>
> The following script with multi-query optimization
> {code}
> a = load 'input';
> store a into 'outout.bz2';
> store a into 'outout2'
> {code}
> generates two .bz files, while only one of them should be compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1461) support union operation that merges based on column names
[
https://issues.apache.org/jira/browse/PIG-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895067#action_12895067
]
Hadoop QA commented on PIG-1461:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12451133/PIG-1461.patch
against trunk revision 980930.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 405 release audit warnings
(more than the trunk's current 403 warnings).
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/370/testReport/
Release audit warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/370/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/370/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/370/console
This message is automatically generated.
> support union operation that merges based on column names
> -
>
> Key: PIG-1461
> URL: https://issues.apache.org/jira/browse/PIG-1461
> Project: Pig
> Issue Type: New Feature
> Components: impl
>Affects Versions: 0.8.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.8.0
>
> Attachments: PIG-1461.patch
>
>
> When the data has schema, it often makes sense to union on column names in
> schema rather than the position of the columns.
> The behavior of existing union operator should remain backward compatible .
> This feature can be supported using either a new operator or extending union
> to support 'using' clause . I am thinking of having a new operator called
> either unionschema or merge . Does anybody have any other suggestions for the
> syntax ?
> example -
> L1 = load 'x' as (a,b);
> L2 = load 'y' as (b,c);
> U = unionschema L1, L2;
> describe U;
> U: {a:bytearray, b:byetarray, c:bytearray}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1526) HiveColumnarLoader Partitioning Support
[ https://issues.apache.org/jira/browse/PIG-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894933#action_12894933 ] Hadoop QA commented on PIG-1526: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12451115/PIG-1526-2.patch against trunk revision 980930. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/369/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/369/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/369/console This message is automatically generated. > HiveColumnarLoader Partitioning Support > --- > > Key: PIG-1526 > URL: https://issues.apache.org/jira/browse/PIG-1526 > Project: Pig > Issue Type: Improvement >Affects Versions: 0.8.0 >Reporter: Gerrit Jansen van Vuuren >Assignee: Gerrit Jansen van Vuuren >Priority: Minor > Fix For: 0.8.0 > > Attachments: PIG-1526-2.patch, PIG-1526.patch > > > I've made allot improvements on the HiveColumnarLoader: > -> Added support for LoadMetadata and data path Partitioning > -> Improved and simplefied column loading > Data Path Partitioning: > Hive stores partitions as folders like to > /mytable/partition1=[value]/partition2=[value]. That is the table mytable > contains 2 partitions [partition1, partition2]. > The HiveColumnarLoader will scan the inputpath /mytable and add to the > PigSchema the columns partition2 and partition2. > These columns can then be used in filtering. > For example: We've got year,month,day,hour partitions in our data uploads. > So a table might look like mytable/year=2010/month=02/day=01. > Loading with the HiveColumnarLoader allows our pig scripts do filter by date > using the standard pig Filter operator. > I've added 2 classes for this: > -> PathPartitioner > -> PathPartitionHelper > These classes are not hive dependent and could be used by any other loader > that wants to support partitioning and helps with implementing the > LoadMetadata interface. > For this reason I though it best to put it into the package > org.apache.pig.piggybank.storage.partition. > What would be nice is in the future have the PigStorage also use these 2 > classes to provide automatic path partitioning support. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1434) Allow casting relations to scalars
[ https://issues.apache.org/jira/browse/PIG-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894837#action_12894837 ] Hadoop QA commented on PIG-1434: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12451096/ScalarImplFinale1.patch against trunk revision 980930. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. -1 release audit. The applied patch generated 409 release audit warnings (more than the trunk's current 403 warnings). +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/368/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/368/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/368/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/368/console This message is automatically generated. > Allow casting relations to scalars > -- > > Key: PIG-1434 > URL: https://issues.apache.org/jira/browse/PIG-1434 > Project: Pig > Issue Type: Improvement >Reporter: Olga Natkovich >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: scalarImpl.patch, ScalarImpl1.patch, ScalarImpl5.patch, > ScalarImplFinale.patch, ScalarImplFinale1.patch > > > This jira is to implement a simplified version of the functionality described > in https://issues.apache.org/jira/browse/PIG-801. > The proposal is to allow casting relations to scalar types in foreach. > Example: > A = load 'data' as (x, y, z); > B = group A all; > C = foreach B generate COUNT(A); > . > X = > Y = foreach X generate $1/(long) C; > Couple of additional comments: > (1) You can only cast relations including a single value or an error will be > reported > (2) Name resolution is needed since relation X might have field named C in > which case that field takes precedence. > (3) Y will look for C closest to it. > Implementation thoughts: > The idea is to store C into a file and then convert it into scalar via a UDF. > I believe we already have a UDF that Ben Reed contributed for this purpose. > Most of the work would be to update the logical plan to > (1) Store C > (2) convert the cast to the UDF -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1526) HiveColumnarLoader Partitioning Support
[ https://issues.apache.org/jira/browse/PIG-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894087#action_12894087 ] Hadoop QA commented on PIG-1526: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450900/PIG-1526.patch against trunk revision 980276. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/367/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/367/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/367/console This message is automatically generated. > HiveColumnarLoader Partitioning Support > --- > > Key: PIG-1526 > URL: https://issues.apache.org/jira/browse/PIG-1526 > Project: Pig > Issue Type: Improvement >Affects Versions: 0.8.0 >Reporter: Gerrit Jansen van Vuuren >Assignee: Gerrit Jansen van Vuuren >Priority: Minor > Fix For: 0.8.0 > > Attachments: PIG-1526.patch > > > I've made allot improvements on the HiveColumnarLoader: > -> Added support for LoadMetadata and data path Partitioning > -> Improved and simplefied column loading > Data Path Partitioning: > Hive stores partitions as folders like to > /mytable/partition1=[value]/partition2=[value]. That is the table mytable > contains 2 partitions [partition1, partition2]. > The HiveColumnarLoader will scan the inputpath /mytable and add to the > PigSchema the columns partition2 and partition2. > These columns can then be used in filtering. > For example: We've got year,month,day,hour partitions in our data uploads. > So a table might look like mytable/year=2010/month=02/day=01. > Loading with the HiveColumnarLoader allows our pig scripts do filter by date > using the standard pig Filter operator. > I've added 2 classes for this: > -> PathPartitioner > -> PathPartitionHelper > These classes are not hive dependent and could be used by any other loader > that wants to support partitioning and helps with implementing the > LoadMetadata interface. > For this reason I though it best to put it into the package > org.apache.pig.piggybank.storage.partition. > What would be nice is in the future have the PigStorage also use these 2 > classes to provide automatic path partitioning support. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1434) Allow casting relations to scalars
[ https://issues.apache.org/jira/browse/PIG-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893939#action_12893939 ] Hadoop QA commented on PIG-1434: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450872/ScalarImplFinale.patch against trunk revision 980276. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. -1 javac. The applied patch generated 146 javac compiler warnings (more than the trunk's current 145 warnings). -1 findbugs. The patch appears to introduce 5 new Findbugs warnings. -1 release audit. The applied patch generated 406 release audit warnings (more than the trunk's current 400 warnings). +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/366/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/366/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/366/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/366/console This message is automatically generated. > Allow casting relations to scalars > -- > > Key: PIG-1434 > URL: https://issues.apache.org/jira/browse/PIG-1434 > Project: Pig > Issue Type: Improvement >Reporter: Olga Natkovich >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: scalarImpl.patch, ScalarImpl1.patch, ScalarImpl5.patch, > ScalarImplFinale.patch > > > This jira is to implement a simplified version of the functionality described > in https://issues.apache.org/jira/browse/PIG-801. > The proposal is to allow casting relations to scalar types in foreach. > Example: > A = load 'data' as (x, y, z); > B = group A all; > C = foreach B generate COUNT(A); > . > X = > Y = foreach X generate $1/(long) C; > Couple of additional comments: > (1) You can only cast relations including a single value or an error will be > reported > (2) Name resolution is needed since relation X might have field named C in > which case that field takes precedence. > (3) Y will look for C closest to it. > Implementation thoughts: > The idea is to store C into a file and then convert it into scalar via a UDF. > I believe we already have a UDF that Ben Reed contributed for this purpose. > Most of the work would be to update the logical plan to > (1) Store C > (2) convert the cast to the UDF -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1452) to remove hadoop20.jar from lib and use hadoop from the apache maven repo.
[ https://issues.apache.org/jira/browse/PIG-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893793#action_12893793 ] Hadoop QA commented on PIG-1452: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450812/PIG-1452V2.PATCH against trunk revision 980276. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/365/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/365/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/365/console This message is automatically generated. > to remove hadoop20.jar from lib and use hadoop from the apache maven repo. > -- > > Key: PIG-1452 > URL: https://issues.apache.org/jira/browse/PIG-1452 > Project: Pig > Issue Type: Improvement > Components: build >Affects Versions: 0.8.0 >Reporter: Giridharan Kesavan >Assignee: Giridharan Kesavan > Fix For: 0.8.0 > > Attachments: PIG-1452.PATCH, PIG-1452V2.PATCH > > > pig use ivy for dependency management. But still it uses hadoop20.jar from > the lib folder. > Now that we have the hadoop-0.20.2 artifacts available in the maven repo, pig > should leverage ivy for resolving/retrieving hadoop artifacts. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1521) explain plan does not show correct Physical operator in MR plan when POSortedDistinct, POPackageLite are used
[ https://issues.apache.org/jira/browse/PIG-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893661#action_12893661 ] Hadoop QA commented on PIG-1521: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450784/PIG-1521.patch against trunk revision 980276. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 409 release audit warnings (more than the trunk's current 406 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/385/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/385/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/385/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/385/console This message is automatically generated. > explain plan does not show correct Physical operator in MR plan when > POSortedDistinct, POPackageLite are used > - > > Key: PIG-1521 > URL: https://issues.apache.org/jira/browse/PIG-1521 > Project: Pig > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Minor > Fix For: 0.8.0 > > Attachments: PIG-1521.patch > > > MR plan in explain shows PODistinct and Package (POPackage), when the > operators POSortedDistinct and PackageLite (POPackageLite) are actually being > used. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1516) finalize in bag implementations causes pig to run out of memory in reduce
[ https://issues.apache.org/jira/browse/PIG-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893589#action_12893589 ] Hadoop QA commented on PIG-1516: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450778/PIG-1516.2.patch against trunk revision 980276. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 402 release audit warnings (more than the trunk's current 400 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/364/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/364/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/364/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/364/console This message is automatically generated. > finalize in bag implementations causes pig to run out of memory in reduce > -- > > Key: PIG-1516 > URL: https://issues.apache.org/jira/browse/PIG-1516 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Fix For: 0.8.0 > > Attachments: PIG-1516.2.patch, PIG-1516.patch > > > *Problem:* > pig bag implementations that are subclasses of DefaultAbstractBag, have > finalize methods implemented. As a result, the garbage collector moves them > to a finalization queue, and the memory used is freed only after the > finalization happens on it. > If the bags are not finalized fast enough, a lot of memory is consumed by the > finalization queue, and pig runs out of memory. This can happen if large > number of small bags are being created. > *Solution:* > The finalize function exists for the purpose of deleting the spill files that > are created when the bag is too large. But if the bags are small enough, no > spill files are created, and there is no use of the finalize function. > A new class that holds a list of files will be introduced (FileList). This > class will have a finalize method that deletes the files. The bags will no > longer have finalize methods, and the bags will use FileList instead of > ArrayList. > *Possible workaround for earlier releases:* > Since the fix is going into 0.8, here is a workaround - > Disabling the combiner will reduce the number of bags getting created, as > there will not be the stage of combining intermediate merge results. But I > would recommend disabling it only if you have this problem as it is likely to > slow down the query . > To disable combiner, set the property: -Dpig.exec.nocombiner=true -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1510) Add `deepCopy` for LogicalExpressions
[
https://issues.apache.org/jira/browse/PIG-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893568#action_12893568
]
Hadoop QA commented on PIG-1510:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12450096/deepCopy.patch
against trunk revision 980276.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 2 new Findbugs warnings.
-1 release audit. The applied patch generated 435 release audit warnings
(more than the trunk's current 406 warnings).
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/384/testReport/
Release audit warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/384/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/384/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/384/console
This message is automatically generated.
> Add `deepCopy` for LogicalExpressions
> -
>
> Key: PIG-1510
> URL: https://issues.apache.org/jira/browse/PIG-1510
> Project: Pig
> Issue Type: New Feature
> Components: data
>Affects Versions: 0.8.0
>Reporter: Swati Jain
>Assignee: Swati Jain
> Fix For: 0.8.0
>
> Attachments: deepCopy.patch
>
>
> It would be useful to have a way to `deepCopy` an expression. `deepCopy` will
> create a new object so that changes made to one object will not reflect in
> the copy. There are 2 reasons why we don't override clone.
> * It may be better to use `deepCopy` since the copy semantics are explicit
> (since deepCopy may be expensive).
> * A second important reason for defining `deepCopy` as a separate routine is
> that it can be passed a plan as an argument which will be updated as the
> expression is copied (through plan.add and plan.connect).
> The usage would look like the following:
> {noformat}
> LogicalExpressionPlan logicalPlan = new LogicalExpressionPlan();
> LogicalExpression copyExpression = origExpression.deepCopy( logicalPlan );
> {noformat}
> An immediate motivation for this would be for constructing the expressions
> that constitute the CNF form of an expression.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1517) Pig needs to support keywords in the package name
[ https://issues.apache.org/jira/browse/PIG-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893519#action_12893519 ] Hadoop QA commented on PIG-1517: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450768/KeywordSupportName.patch against trunk revision 980148. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/363/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/363/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/363/console This message is automatically generated. > Pig needs to support keywords in the package name > - > > Key: PIG-1517 > URL: https://issues.apache.org/jira/browse/PIG-1517 > Project: Pig > Issue Type: Bug > Components: grunt >Reporter: Aniket Mokashi >Assignee: Aniket Mokashi >Priority: Minor > Fix For: 0.8.0 > > Attachments: KeywordSupportName.patch, pigusergroup656.patch > > > Pig needs to support keywords in the package name. Pig supports most of the > keywords as this was fixed in https://issues.apache.org/jira/browse/PIG-656. > There are a few missing tokens like "eq","gt","lt","gte","lte","neq" that > need to be supported. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1500) guava.jar should be removed from the lib folder
[ https://issues.apache.org/jira/browse/PIG-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893437#action_12893437 ] Hadoop QA commented on PIG-1500: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450736/guava.jar.r06_4.patch against trunk revision 979918. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/362/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/362/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/362/console This message is automatically generated. > guava.jar should be removed from the lib folder > --- > > Key: PIG-1500 > URL: https://issues.apache.org/jira/browse/PIG-1500 > Project: Pig > Issue Type: Bug > Components: build >Reporter: Giridharan Kesavan >Assignee: niraj rai > Fix For: 0.8.0 > > Attachments: guava.jar.06.afterjython.patch, guava.jar.r06.patch, > guava.jar.r06_4.patch, removeGuavaJar.patch > > > guava jar is available in the maven repository but still its is checked into > the pig trunk's lib folder. > I ve checked the availability of guava jar in the maven repository. > http://mvnrepository.com/artifact/com.google.guava/guava -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1513) Pig doesn't handle empty input directory
[
https://issues.apache.org/jira/browse/PIG-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893433#action_12893433
]
Hadoop QA commented on PIG-1513:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12450727/PIG-1513.patch
against trunk revision 979918.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/383/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/383/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/383/console
This message is automatically generated.
> Pig doesn't handle empty input directory
>
>
> Key: PIG-1513
> URL: https://issues.apache.org/jira/browse/PIG-1513
> Project: Pig
> Issue Type: Bug
>Reporter: Richard Ding
>Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1513.patch
>
>
> The following script
> {code}
> A = load 'input';
> B = load 'emptydir';
> C = join B by $0, A by $0 using 'skewed';
> store C into 'output';
> {code}
> fails with "ERROR: java.lang.RuntimeException: Empty samples file';
> In this case, the sample job has 0 maps. Pig doesn't expect this and fails .
> For merge join the script
> The merge join script
> {code}
> A = load 'input';
> B = load 'emptydir';
> C = join A by $0, B by $0 using 'merge';
> store C into 'output';
> {code}
> the sample job again has 0 maps and the script fails with " ERROR 2176:
> Error processing right input during merge join".
> But if we change the join order:
> {code}
> A = load 'input';
> B = load 'emptydir';
> C = join B by $0, A by $0 using 'merge';
> store C into 'output';
> {code}
> The second job (merge) now has 0 maps and 0 reduces. And it generates an
> empty 'output' directory.
> Order by on empty directory works fine and generates empty part files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1520) Remove Owl from Pig contrib
[ https://issues.apache.org/jira/browse/PIG-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893149#action_12893149 ] Hadoop QA commented on PIG-1520: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450615/PIG-1520.patch against trunk revision 979918. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 345 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/382/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/382/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/382/console This message is automatically generated. > Remove Owl from Pig contrib > --- > > Key: PIG-1520 > URL: https://issues.apache.org/jira/browse/PIG-1520 > Project: Pig > Issue Type: Task > Components: impl >Affects Versions: 0.8.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.8.0 > > Attachments: PIG-1520.patch > > > Yahoo has transitioned work on Owl to Howl (which will not be a Pig contrib > project). Since no one else is working on Owl and there will be no one to > support it we should remove it from our contrib before releasing 0.8. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1500) guava.jar should be removed from the lib folder
[ https://issues.apache.org/jira/browse/PIG-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893071#action_12893071 ] Hadoop QA commented on PIG-1500: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450607/guava.jar.06.afterjython.patch against trunk revision 979781. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/361/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/361/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/361/console This message is automatically generated. > guava.jar should be removed from the lib folder > --- > > Key: PIG-1500 > URL: https://issues.apache.org/jira/browse/PIG-1500 > Project: Pig > Issue Type: Bug > Components: build >Reporter: Giridharan Kesavan >Assignee: niraj rai > Fix For: 0.8.0 > > Attachments: guava.jar.06.afterjython.patch, guava.jar.r06.patch, > removeGuavaJar.patch > > > guava jar is available in the maven repository but still its is checked into > the pig trunk's lib folder. > I ve checked the availability of guava jar in the maven repository. > http://mvnrepository.com/artifact/com.google.guava/guava -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1288) EvalFunc returnType is wrong for generic subclasses
[ https://issues.apache.org/jira/browse/PIG-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893001#action_12893001 ] Hadoop QA commented on PIG-1288: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450538/PIG-1288-4.patch against trunk revision 979781. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 17 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/381/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/381/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/381/console This message is automatically generated. > EvalFunc returnType is wrong for generic subclasses > --- > > Key: PIG-1288 > URL: https://issues.apache.org/jira/browse/PIG-1288 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.7.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.8.0 > > Attachments: PIG-1288-1.patch, PIG-1288-2.patch, PIG-1288-3.patch, > PIG-1288-4.patch > > > From Garrett Buster Kaminaga: > The EvalFunc constructor has code to determine the return type of the > function. > This walks up the object hierarchy until it encounters EvalFunc, then calls > getActualTypeArguments and extracts type > param 0. > However, if the user class is itself a generic extension of EvalFunc, then > the returned object is not the correct type, > but a TypeVariable. > Example: > class MyAbstractEvalFunc extends EvalFunc ... > class MyEvalFunc extends MyAbstractEvalFunc ... > when MyEvalFunc() is called, inside EvalFunc constructor the return type is > set to a TypeVariable rather than > String.class. > The workaround we've implemented is for the MyAbstractEvalFunc to > determine *its* type parameters using code > similar to that in the EvalFunc constructor, and then reset protected data > member returnType manually in the > MyAbstractEvalFunc constructor. (though this has the same drawback of not > working if someone then extends > MyAbstractEvalFunc) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1229) allow pig to write output into a JDBC db
[ https://issues.apache.org/jira/browse/PIG-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892999#action_12892999 ] Hadoop QA commented on PIG-1229: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450586/jira-1229-final.patch against trunk revision 979781. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/360/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/360/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/360/console This message is automatically generated. > allow pig to write output into a JDBC db > > > Key: PIG-1229 > URL: https://issues.apache.org/jira/browse/PIG-1229 > Project: Pig > Issue Type: New Feature > Components: impl >Reporter: Ian Holsman >Assignee: Ankur >Priority: Minor > Fix For: 0.8.0 > > Attachments: jira-1229-final.patch, jira-1229-v2.patch, > jira-1229-v3.patch, pig-1229.2.patch, pig-1229.patch > > > UDF to store data into a DB -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1249) Safe-guards against misconfigured Pig scripts without PARALLEL keyword
[ https://issues.apache.org/jira/browse/PIG-1249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892873#action_12892873 ] Hadoop QA commented on PIG-1249: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450579/PIG-1249_5.patch against trunk revision 979503. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/359/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/359/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/359/console This message is automatically generated. > Safe-guards against misconfigured Pig scripts without PARALLEL keyword > -- > > Key: PIG-1249 > URL: https://issues.apache.org/jira/browse/PIG-1249 > Project: Pig > Issue Type: Improvement >Affects Versions: 0.8.0 >Reporter: Arun C Murthy >Assignee: Jeff Zhang >Priority: Critical > Fix For: 0.8.0 > > Attachments: PIG-1249-4.patch, PIG-1249.patch, PIG-1249_5.patch, > PIG_1249_2.patch, PIG_1249_3.patch > > > It would be *very* useful for Pig to have safe-guards against naive scripts > which process a *lot* of data without the use of PARALLEL keyword. > We've seen a fair number of instances where naive users process huge > data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1517) Pig needs to support keywords in the package name
[ https://issues.apache.org/jira/browse/PIG-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892744#action_12892744 ] Hadoop QA commented on PIG-1517: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450481/pigusergroup656.patch against trunk revision 979503. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/358/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/358/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/358/console This message is automatically generated. > Pig needs to support keywords in the package name > - > > Key: PIG-1517 > URL: https://issues.apache.org/jira/browse/PIG-1517 > Project: Pig > Issue Type: Bug > Components: grunt >Reporter: Aniket Mokashi >Assignee: Aniket Mokashi >Priority: Minor > Fix For: 0.8.0 > > Attachments: pigusergroup656.patch > > > Pig needs to support keywords in the package name. Pig supports most of the > keywords as this was fixed in https://issues.apache.org/jira/browse/PIG-656. > There are a few missing tokens like "eq","gt","lt","gte","lte","neq" that > need to be supported. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1512) PlanPrinter does not print LOJoin operator in the new logical optimization framework
[ https://issues.apache.org/jira/browse/PIG-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892740#action_12892740 ] Hadoop QA commented on PIG-1512: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450145/printJoin.patch against trunk revision 979503. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 407 release audit warnings (more than the trunk's current 405 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/380/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/380/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/380/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/380/console This message is automatically generated. > PlanPrinter does not print LOJoin operator in the new logical optimization > framework > > > Key: PIG-1512 > URL: https://issues.apache.org/jira/browse/PIG-1512 > Project: Pig > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Swati Jain >Assignee: Swati Jain > Fix For: 0.8.0 > > Attachments: printJoin.patch > > > PlanPrinter does not print LOJoin relational operator. As such, the LOJoin > operator would not get printed when we do an explain. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-348) -j command line option doesn't work
[ https://issues.apache.org/jira/browse/PIG-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892663#action_12892663 ] Hadoop QA commented on PIG-348: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450362/PIG-348.path against trunk revision 979503. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/357/console This message is automatically generated. > -j command line option doesn't work > --- > > Key: PIG-348 > URL: https://issues.apache.org/jira/browse/PIG-348 > Project: Pig > Issue Type: Improvement > Components: documentation >Reporter: Amir Youssefi >Assignee: Richard Ding > Fix For: 0.8.0 > > Attachments: PIG-348.path > > > According to: > $ pig --help > ... > -j, -jar jarfile load jarfile > ... > yet > $pig -j my.jar > doesn't work in place of: > register my.jar > in Pig script. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too complex and hard to work with
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892658#action_12892658 ] Hadoop QA commented on PIG-1178: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450250/PIG-1178-4.patch against trunk revision 979362. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 48 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 446 release audit warnings (more than the trunk's current 398 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/355/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/355/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/355/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/355/console This message is automatically generated. > LogicalPlan and Optimizer are too complex and hard to work with > --- > > Key: PIG-1178 > URL: https://issues.apache.org/jira/browse/PIG-1178 > Project: Pig > Issue Type: Improvement >Reporter: Alan Gates >Assignee: Daniel Dai > Fix For: 0.8.0 > > Attachments: expressions-2.patch, expressions.patch, lp.patch, > lp.patch, PIG-1178-4.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, > pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, > pig_1178_3.patch > > > The current implementation of the logical plan and the logical optimizer in > Pig has proven to not be easily extensible. Developer feedback has indicated > that adding new rules to the optimizer is quite burdensome. In addition, the > logical plan has been an area of numerous bugs, many of which have been > difficult to fix. Developers also feel that the logical plan is difficult to > understand and maintain. The root cause for these issues is that a number of > design decisions that were made as part of the 0.2 rewrite of the front end > have now proven to be sub-optimal. The heart of this proposal is to revisit a > number of those proposals and rebuild the logical plan with a simpler design > that will make it much easier to maintain the logical plan as well as extend > the logical optimizer. > See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full > details. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1511) Pig removes packages from its own jar when building the JAR to ship to Hadoop
[ https://issues.apache.org/jira/browse/PIG-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892568#action_12892568 ] Hadoop QA commented on PIG-1511: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450112/pig-1511.diff against trunk revision 979362. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/354/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/354/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/354/console This message is automatically generated. > Pig removes packages from its own jar when building the JAR to ship to Hadoop > - > > Key: PIG-1511 > URL: https://issues.apache.org/jira/browse/PIG-1511 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Eric Tschetter > Attachments: pig-1511.diff > > > Pig generates a new jar file to ship over to Hadoop. Pig has a couple of > packages whitelisted that it includes from its own jar. Pig throws away > everything else. > I package all of my dependencies into a single jar file. Pig is included in > this jar file. I do it this way because my code needs to run reliably and > reproducibly in production. Pig throws away all of my dependencies. > I don't know what the performance gain is of shaving ~5MB off of a jar that > is pushed to a job tracker once and then used to run over 100s of GB of data. > The overhead is minimal on my cluster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1505) support jars and scripts in dfs
[ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892564#action_12892564 ] Hadoop QA commented on PIG-1505: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450123/pig-jars-and-scripts-from-dfs-3.patch against trunk revision 979362. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/379/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/379/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/379/console This message is automatically generated. > support jars and scripts in dfs > --- > > Key: PIG-1505 > URL: https://issues.apache.org/jira/browse/PIG-1505 > Project: Pig > Issue Type: Improvement >Reporter: Andrew Hitchcock >Assignee: Andrew Hitchcock > Attachments: pig-jars-and-scripts-from-dfs-3.patch, > pig-jars-and-scripts-from-dfs-trunk-1.patch, > pig-jars-and-scripts-from-dfs-trunk-2.patch, > pig-jars-and-scripts-from-dfs-trunk.patch > > > Pig can't operate on files stored in Amazon S3. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1500) guava.jar should be removed from the lib folder
[ https://issues.apache.org/jira/browse/PIG-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892409#action_12892409 ] Hadoop QA commented on PIG-1500: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450378/guava.jar.r06.patch against trunk revision 979362. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/376/console This message is automatically generated. > guava.jar should be removed from the lib folder > --- > > Key: PIG-1500 > URL: https://issues.apache.org/jira/browse/PIG-1500 > Project: Pig > Issue Type: Bug > Components: build >Reporter: Giridharan Kesavan >Assignee: niraj rai > Fix For: 0.8.0 > > Attachments: guava.jar.r06.patch, removeGuavaJar.patch > > > guava jar is available in the maven repository but still its is checked into > the pig trunk's lib folder. > I ve checked the availability of guava jar in the maven repository. > http://mvnrepository.com/artifact/com.google.guava/guava -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1508) Make 'docs' target (forrest) work with Java 1.6
[ https://issues.apache.org/jira/browse/PIG-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890596#action_12890596 ] Hadoop QA commented on PIG-1508: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449977/PIG-1508.patch.txt against trunk revision 965559. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/349/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/349/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/349/console This message is automatically generated. > Make 'docs' target (forrest) work with Java 1.6 > --- > > Key: PIG-1508 > URL: https://issues.apache.org/jira/browse/PIG-1508 > Project: Pig > Issue Type: Bug > Components: documentation >Affects Versions: 0.7.0 >Reporter: Carl Steinbach > Attachments: PIG-1508.patch.txt > > > FOR-984 covers the very inconvenient fact that Forrest 0.8 does not work with > Java 1.6 > The same ticket also suggests a workaround: disabling sitemap and stylesheet > validation > by setting the forrest.validate.sitemap and forrest.validate.stylesheets > properties to false. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1507) Full outer join fails while doing a filter on joined data
[
https://issues.apache.org/jira/browse/PIG-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890521#action_12890521
]
Hadoop QA commented on PIG-1507:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12449962/PIG-1507-1.patch
against trunk revision 965559.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/348/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/348/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/348/console
This message is automatically generated.
> Full outer join fails while doing a filter on joined data
> -
>
> Key: PIG-1507
> URL: https://issues.apache.org/jira/browse/PIG-1507
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.8.0
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: PIG-1507-1.patch
>
>
> The following script produce wrong result:
> test1.dat:
> 1
> 2
> 3
> test2.dat:
> 1
> 2
> pig script:
> {code}
> a = LOAD 'test1.dat' USING PigStorage() AS (d1:int);
> b = LOAD 'test2.dat' USING PigStorage() AS (d2:int);
> c = JOIN a BY d1 FULL OUTER, b BY d2;
> d = FILTER c BY d2 IS NULL;
> STORE d INTO 'test.out' USING PigStorage();
> {code}
> expected:
> 3
> We get:
> 1
> 2
> 3
> This is because we erroneously push the filter before full outer join.
> Similar issue is addressed in
> [PIG-1289|https://issues.apache.org/jira/browse/PIG-1289], but we only fix
> left/right outer join.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1434) Allow casting relations to scalars
[ https://issues.apache.org/jira/browse/PIG-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890159#action_12890159 ] Hadoop QA commented on PIG-1434: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449903/ScalarImpl1.patch against trunk revision 965559. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/347/testReport/ Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/347/console This message is automatically generated. > Allow casting relations to scalars > -- > > Key: PIG-1434 > URL: https://issues.apache.org/jira/browse/PIG-1434 > Project: Pig > Issue Type: Improvement >Reporter: Olga Natkovich >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: scalarImpl.patch, ScalarImpl1.patch > > > This jira is to implement a simplified version of the functionality described > in https://issues.apache.org/jira/browse/PIG-801. > The proposal is to allow casting relations to scalar types in foreach. > Example: > A = load 'data' as (x, y, z); > B = group A all; > C = foreach B generate COUNT(A); > . > X = > Y = foreach X generate $1/(long) C; > Couple of additional comments: > (1) You can only cast relations including a single value or an error will be > reported > (2) Name resolution is needed since relation X might have field named C in > which case that field takes precedence. > (3) Y will look for C closest to it. > Implementation thoughts: > The idea is to store C into a file and then convert it into scalar via a UDF. > I believe we already have a UDF that Ben Reed contributed for this purpose. > Most of the work would be to update the logical plan to > (1) Store C > (2) convert the cast to the UDF -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1379) Jars registered from command line should override the ones present in the script
[ https://issues.apache.org/jira/browse/PIG-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890157#action_12890157 ] Hadoop QA commented on PIG-1379: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449873/PIG-1379.patch against trunk revision 965559. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/346/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/346/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/346/console This message is automatically generated. > Jars registered from command line should override the ones present in the > script > - > > Key: PIG-1379 > URL: https://issues.apache.org/jira/browse/PIG-1379 > Project: Pig > Issue Type: Improvement >Reporter: Ankur >Assignee: Richard Ding > Fix For: 0.8.0 > > Attachments: PIG-1379.patch > > > Jars that are registered from the command line when executing the pig script > should override the ones that are specified via 'register' in the pig script > itself. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1505) support jars and scripts in dfs
[ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890118#action_12890118 ] Hadoop QA commented on PIG-1505: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449741/pig-jars-and-scripts-from-dfs-trunk-1.patch against trunk revision 965559. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/372/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/372/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/372/console This message is automatically generated. > support jars and scripts in dfs > --- > > Key: PIG-1505 > URL: https://issues.apache.org/jira/browse/PIG-1505 > Project: Pig > Issue Type: Improvement >Reporter: Andrew Hitchcock > Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, > pig-jars-and-scripts-from-dfs-trunk.patch > > > Pig can't operate on files stored in Amazon S3. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1435) make sure dependent jobs fail when a jon in multiquery fails
[ https://issues.apache.org/jira/browse/PIG-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889461#action_12889461 ] Hadoop QA commented on PIG-1435: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449609/depJobsFailure.patch against trunk revision 964182. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 405 release audit warnings (more than the trunk's current 404 warnings). +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/371/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/371/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/371/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/371/console This message is automatically generated. > make sure dependent jobs fail when a jon in multiquery fails > > > Key: PIG-1435 > URL: https://issues.apache.org/jira/browse/PIG-1435 > Project: Pig > Issue Type: Bug >Reporter: Olga Natkovich >Assignee: niraj rai > Fix For: 0.8.0 > > Attachments: depJobs.patch, depJobsFailure.patch > > > Currently if one of the MQ jobs fails, Pig tries to run all remainin jobs. As > the result, if data was partially generated by the failed job, you might get > incorrect results from dependent jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1505) support jars and scripts in dfs
[ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889421#action_12889421 ] Hadoop QA commented on PIG-1505: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449736/pig-jars-and-scripts-from-dfs-trunk.patch against trunk revision 964182. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/345/console This message is automatically generated. > support jars and scripts in dfs > --- > > Key: PIG-1505 > URL: https://issues.apache.org/jira/browse/PIG-1505 > Project: Pig > Issue Type: Improvement >Reporter: Andrew Hitchcock > Attachments: pig-jars-and-scripts-from-dfs-trunk.patch > > > Pig can't operate on files stored in Amazon S3. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1492) DefaultTuple and DefaultMemory understimate their memory footprint
[ https://issues.apache.org/jira/browse/PIG-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888952#action_12888952 ] Hadoop QA commented on PIG-1492: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449531/PIG-1492.1.patch against trunk revision 964182. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/370/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/370/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/370/console This message is automatically generated. > DefaultTuple and DefaultMemory understimate their memory footprint > -- > > Key: PIG-1492 > URL: https://issues.apache.org/jira/browse/PIG-1492 > Project: Pig > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Fix For: 0.8.0 > > Attachments: PIG-1492.1.patch > > > There are several places where we highly underestimate the memory footprint . > For example, for map datatypes, we don't account for the per entry cost for > the map container data structures. The estimated size of a tuple having map > with 100 integer key-value entries , as per current version of code is 3260 > bytes, while what is observed is around 6775 bytes . To verify the memory > footprint, i checked free memory before and after creating multiple instances > of the object , using code on the lines of > http://www.javaspecialists.eu/archive/Issue029.html . > In PIG-1443 similar change was done to fix this for CHARARRAY . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1435) make sure dependent jobs fail when a jon in multiquery fails
[ https://issues.apache.org/jira/browse/PIG-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888692#action_12888692 ] Hadoop QA commented on PIG-1435: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449486/depJobs.patch against trunk revision 964182. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 405 release audit warnings (more than the trunk's current 404 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/369/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/369/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/369/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/369/console This message is automatically generated. > make sure dependent jobs fail when a jon in multiquery fails > > > Key: PIG-1435 > URL: https://issues.apache.org/jira/browse/PIG-1435 > Project: Pig > Issue Type: Bug >Reporter: Olga Natkovich >Assignee: niraj rai > Fix For: 0.8.0 > > Attachments: depJobs.patch > > > Currently if one of the MQ jobs fails, Pig tries to run all remainin jobs. As > the result, if data was partially generated by the failed job, you might get > incorrect results from dependent jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1495) Add -q command line option to set queue name for Pig jobs from command line
[ https://issues.apache.org/jira/browse/PIG-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888279#action_12888279 ] Hadoop QA commented on PIG-1495: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449293/set_queue.patch against trunk revision 963830. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/368/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/368/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/368/console This message is automatically generated. > Add -q command line option to set queue name for Pig jobs from command line > --- > > Key: PIG-1495 > URL: https://issues.apache.org/jira/browse/PIG-1495 > Project: Pig > Issue Type: New Feature > Components: impl >Affects Versions: 0.7.0 >Reporter: Russell Jurney > Fix For: 0.8.0 > > Attachments: set_queue.patch > > > rjurney$ pig -q default > This sets the mapred.job.queue.name property in the execution engine from the > pig properties for MAPRED type jobs. > Patch attached. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-928) UDFs in scripting languages
[ https://issues.apache.org/jira/browse/PIG-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888068#action_12888068 ] Hadoop QA commented on PIG-928: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449134/RegisterPythonUDFFinale5.patch against trunk revision 963504. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 145 javac compiler warnings (more than the trunk's current 144 warnings). -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/344/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/344/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/344/console This message is automatically generated. > UDFs in scripting languages > --- > > Key: PIG-928 > URL: https://issues.apache.org/jira/browse/PIG-928 > Project: Pig > Issue Type: New Feature >Reporter: Alan Gates >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: calltrace.png, package.zip, PIG-928.patch, > pig-greek.tgz, pig.scripting.patch.arnab, pyg.tgz, RegisterPythonUDF3.patch, > RegisterPythonUDF4.patch, RegisterPythonUDF_Final.patch, > RegisterPythonUDFFinale.patch, RegisterPythonUDFFinale3.patch, > RegisterPythonUDFFinale4.patch, RegisterPythonUDFFinale5.patch, > RegisterScriptUDFDefineParse.patch, scripting.tgz, scripting.tgz, test.zip > > > It should be possible to write UDFs in scripting languages such as python, > ruby, etc. This frees users from needing to compile Java, generate a jar, > etc. It also opens Pig to programmers who prefer scripting languages over > Java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1493) Column Pruner throw exception "inconsistent pruning"
[
https://issues.apache.org/jira/browse/PIG-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888063#action_12888063
]
Hadoop QA commented on PIG-1493:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12449203/PIG-1493-1.patch
against trunk revision 963504.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/367/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/367/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/367/console
This message is automatically generated.
> Column Pruner throw exception "inconsistent pruning"
>
>
> Key: PIG-1493
> URL: https://issues.apache.org/jira/browse/PIG-1493
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.7.0
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.7.0, 0.8.0
>
> Attachments: PIG-1493-1.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' as (a0:chararray, a1:chararray, a2);
> b = foreach a generate CONCAT(a0,a1) as b0, a0, a2;
> c = foreach b generate a0, a2;
> dump c;
> {code}
> Error message:
> ERROR 2185: Column $0 of (Name: b: ForEach 1-50 Operator Key: 1-50)
> inconsistent pruning
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
> open iterator for alias c
> at org.apache.pig.PigServer.openIterator(PigServer.java:698)
> at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:595)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:291)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:162)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:138)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> at org.apache.pig.Main.run(Main.java:451)
> at org.apache.pig.Main.main(Main.java:103)
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002:
> Unable to store alias c
> at org.apache.pig.PigServer.storeEx(PigServer.java:804)
> at org.apache.pig.PigServer.store(PigServer.java:760)
> at org.apache.pig.PigServer.openIterator(PigServer.java:680)
> ... 7 more
> Caused by: org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2212:
> Unable to prune plan
> at
> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns.prune(PruneColumns.java:826)
> at
> org.apache.pig.impl.logicalLayer.optimizer.LogicalOptimizer.optimize(LogicalOptimizer.java:240)
> at org.apache.pig.PigServer.compileLp(PigServer.java:1180)
> at org.apache.pig.PigServer.storeEx(PigServer.java:799)
> ... 9 more
> Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 2188: Cannot
> prune columns for (Name: b: ForEach 1-50 Operator Key: 1-50)
> at
> org.apache.pig.impl.logicalLayer.ColumnPruner.prune(ColumnPruner.java:177)
> at
> org.apache.pig.impl.logicalLayer.ColumnPruner.visit(ColumnPruner.java:202)
> at
> org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:132)
> at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:47)
> at
> org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:69)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at
> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns.prune(PruneColumns.java:821)
> ... 12 more
> Caused by: org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2185:
> Column $0 of (Name: b: ForEach 1-50 Operator Key: 1-50) inconsistent pruning
> at
> org.apache.pig.impl.logicalLayer.ColumnPruner.prune(ColumnPruner.java:148)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1490) Make Pig storers work with remote HDFS in secure mode
[ https://issues.apache.org/jira/browse/PIG-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887015#action_12887015 ] Hadoop QA commented on PIG-1490: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449139/PIG-1490.patch against trunk revision 962722. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/366/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/366/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/366/console This message is automatically generated. > Make Pig storers work with remote HDFS in secure mode > - > > Key: PIG-1490 > URL: https://issues.apache.org/jira/browse/PIG-1490 > Project: Pig > Issue Type: Bug >Reporter: Richard Ding >Assignee: Richard Ding > Fix For: 0.7.0, 0.8.0 > > Attachments: PIG-1490.patch > > > PIG-1403 fixed the problem for Pig loaders. We need to do the same for Pig > storers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-928) UDFs in scripting languages
[ https://issues.apache.org/jira/browse/PIG-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886888#action_12886888 ] Hadoop QA commented on PIG-928: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449105/RegisterPythonUDFFinale4.patch against trunk revision 962628. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/365/console This message is automatically generated. > UDFs in scripting languages > --- > > Key: PIG-928 > URL: https://issues.apache.org/jira/browse/PIG-928 > Project: Pig > Issue Type: New Feature >Reporter: Alan Gates >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: calltrace.png, package.zip, PIG-928.patch, > pig-greek.tgz, pig.scripting.patch.arnab, pyg.tgz, RegisterPythonUDF2.patch, > RegisterPythonUDF3.patch, RegisterPythonUDF4.patch, > RegisterPythonUDF_Final.patch, RegisterPythonUDFFinale.patch, > RegisterPythonUDFFinale3.patch, RegisterPythonUDFFinale4.patch, > RegisterScriptUDFDefineParse.patch, scripting.tgz, scripting.tgz, test.zip > > > It should be possible to write UDFs in scripting languages such as python, > ruby, etc. This frees users from needing to compile Java, generate a jar, > etc. It also opens Pig to programmers who prefer scripting languages over > Java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1472) Optimize serialization/deserialization between Map and Reduce and between MR jobs
[ https://issues.apache.org/jira/browse/PIG-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886647#action_12886647 ] Hadoop QA commented on PIG-1472: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449033/PIG-1472.3.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 69 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 395 release audit warnings (more than the trunk's current 394 warnings). +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/343/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/343/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/343/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/343/console This message is automatically generated. > Optimize serialization/deserialization between Map and Reduce and between MR > jobs > - > > Key: PIG-1472 > URL: https://issues.apache.org/jira/browse/PIG-1472 > Project: Pig > Issue Type: Improvement >Affects Versions: 0.8.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Fix For: 0.8.0 > > Attachments: PIG-1472.2.patch, PIG-1472.3.patch, PIG-1472.patch > > > In certain types of pig queries most of the execution time is spent in > serializing/deserializing (sedes) records between Map and Reduce and between > MR jobs. > For example, if PigMix queries are modified to specify types for all the > fields in the load statement schema, some of the queries (L2,L3,L9, L10 in > pigmix v1) that have records with bags and maps being transmitted across map > or reduce boundaries run a lot longer (runtime increase of few times has been > seen. > There are a few optimizations that have shown to improve the performance of > sedes in my tests - > 1. Use smaller number of bytes to store length of the column . For example if > a bytearray is smaller than 255 bytes , a byte can be used to store the > length instead of the integer that is currently used. > 2. Instead of custom code to do sedes on Strings, use DataOutput.writeUTF and > DataInput.readUTF. This reduces the cost of serialization by more than 1/2. > Zebra and BinStorage are known to use DefaultTuple sedes functionality. The > serialization format that these loaders use cannot change, so after the > optimization their format is going to be different from the format used > between M/R boundaries. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-928) UDFs in scripting languages
[ https://issues.apache.org/jira/browse/PIG-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886610#action_12886610 ] Hadoop QA commented on PIG-928: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449018/RegisterPythonUDF_Final.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 146 javac compiler warnings (more than the trunk's current 145 warnings). -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/364/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/364/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/364/console This message is automatically generated. > UDFs in scripting languages > --- > > Key: PIG-928 > URL: https://issues.apache.org/jira/browse/PIG-928 > Project: Pig > Issue Type: New Feature >Reporter: Alan Gates >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: calltrace.png, package.zip, PIG-928.patch, > pig-greek.tgz, pig.scripting.patch.arnab, pyg.tgz, RegisterPythonUDF2.patch, > RegisterPythonUDF3.patch, RegisterPythonUDF4.patch, > RegisterPythonUDF_Final.patch, RegisterPythonUDFFinale.patch, > RegisterPythonUDFFinale3.patch, RegisterScriptUDFDefineParse.patch, > scripting.tgz, scripting.tgz, test.zip > > > It should be possible to write UDFs in scripting languages such as python, > ruby, etc. This frees users from needing to compile Java, generate a jar, > etc. It also opens Pig to programmers who prefer scripting languages over > Java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1484) BinStorage should support comma seperated path
[ https://issues.apache.org/jira/browse/PIG-1484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886591#action_12886591 ] Hadoop QA commented on PIG-1484: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449001/PIG-1484-3.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/342/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/342/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/342/console This message is automatically generated. > BinStorage should support comma seperated path > -- > > Key: PIG-1484 > URL: https://issues.apache.org/jira/browse/PIG-1484 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.7.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.7.0, 0.8.0 > > Attachments: PIG-1484-1.patch, PIG-1484-2.patch, PIG-1484-3.patch > > > BinStorage does not take comma seperated path. The following script fail: > a = load '1.bin,2.bin' using BinStorage(); > dump a; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1484) BinStorage should support comma seperated path
[ https://issues.apache.org/jira/browse/PIG-1484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886538#action_12886538 ] Hadoop QA commented on PIG-1484: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448988/PIG-1484-2.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/363/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/363/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/363/console This message is automatically generated. > BinStorage should support comma seperated path > -- > > Key: PIG-1484 > URL: https://issues.apache.org/jira/browse/PIG-1484 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.7.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.7.0, 0.8.0 > > Attachments: PIG-1484-1.patch, PIG-1484-2.patch, PIG-1484-3.patch > > > BinStorage does not take comma seperated path. The following script fail: > a = load '1.bin,2.bin' using BinStorage(); > dump a; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1472) Optimize serialization/deserialization between Map and Reduce and between MR jobs
[ https://issues.apache.org/jira/browse/PIG-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886281#action_12886281 ] Hadoop QA commented on PIG-1472: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448937/PIG-1472.2.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 69 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. -1 javac. The applied patch generated 148 javac compiler warnings (more than the trunk's current 145 warnings). -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. -1 release audit. The applied patch generated 400 release audit warnings (more than the trunk's current 399 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/362/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/362/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/362/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/362/console This message is automatically generated. > Optimize serialization/deserialization between Map and Reduce and between MR > jobs > - > > Key: PIG-1472 > URL: https://issues.apache.org/jira/browse/PIG-1472 > Project: Pig > Issue Type: Improvement >Affects Versions: 0.8.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Fix For: 0.8.0 > > Attachments: PIG-1472.2.patch, PIG-1472.patch > > > In certain types of pig queries most of the execution time is spent in > serializing/deserializing (sedes) records between Map and Reduce and between > MR jobs. > For example, if PigMix queries are modified to specify types for all the > fields in the load statement schema, some of the queries (L2,L3,L9, L10 in > pigmix v1) that have records with bags and maps being transmitted across map > or reduce boundaries run a lot longer (runtime increase of few times has been > seen. > There are a few optimizations that have shown to improve the performance of > sedes in my tests - > 1. Use smaller number of bytes to store length of the column . For example if > a bytearray is smaller than 255 bytes , a byte can be used to store the > length instead of the integer that is currently used. > 2. Instead of custom code to do sedes on Strings, use DataOutput.writeUTF and > DataInput.readUTF. This reduces the cost of serialization by more than 1/2. > Zebra and BinStorage are known to use DefaultTuple sedes functionality. The > serialization format that these loaders use cannot change, so after the > optimization their format is going to be different from the format used > between M/R boundaries. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1486) update ant eclipse-files target to include new jar and remove contrib dirs from build path
[ https://issues.apache.org/jira/browse/PIG-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886274#action_12886274 ] Hadoop QA commented on PIG-1486: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448935/PIG-1486.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/341/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/341/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/341/console This message is automatically generated. > update ant eclipse-files target to include new jar and remove contrib dirs > from build path > -- > > Key: PIG-1486 > URL: https://issues.apache.org/jira/browse/PIG-1486 > Project: Pig > Issue Type: Bug > Components: tools >Affects Versions: 0.8.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Minor > Fix For: 0.8.0 > > Attachments: PIG-1486.patch > > > .eclipse.templates/.classpath needs to be updated to address following - > 1. There is a new jar that is used by the code - guava-r03.jar > 2. The jar "ANT_HOME/lib/ant.jar" gives an 'unbounded jar' error in eclipse. > 3. Removing the contrib projects from class path as discussed in PIG-1390, > until all libs necessary for the contribs are included in classpath. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1484) BinStorage should support comma seperated path
[ https://issues.apache.org/jira/browse/PIG-1484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886175#action_12886175 ] Hadoop QA commented on PIG-1484: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448904/PIG-1484-1.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/361/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/361/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/361/console This message is automatically generated. > BinStorage should support comma seperated path > -- > > Key: PIG-1484 > URL: https://issues.apache.org/jira/browse/PIG-1484 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.7.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.8.0 > > Attachments: PIG-1484-1.patch > > > BinStorage does not take comma seperated path. The following script fail: > a = load '1.bin,2.bin' using BinStorage(); > dump a; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-928) UDFs in scripting languages
[ https://issues.apache.org/jira/browse/PIG-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885822#action_12885822 ] Hadoop QA commented on PIG-928: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448831/RegisterPythonUDFFinale3.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. -1 javac. The applied patch generated 146 javac compiler warnings (more than the trunk's current 145 warnings). -1 findbugs. The patch appears to introduce 4 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/340/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/340/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/340/console This message is automatically generated. > UDFs in scripting languages > --- > > Key: PIG-928 > URL: https://issues.apache.org/jira/browse/PIG-928 > Project: Pig > Issue Type: New Feature >Reporter: Alan Gates >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: calltrace.png, package.zip, PIG-928.patch, > pig-greek.tgz, pig.scripting.patch.arnab, pyg.tgz, RegisterPythonUDF2.patch, > RegisterPythonUDF3.patch, RegisterPythonUDF4.patch, > RegisterPythonUDFFinale.patch, RegisterPythonUDFFinale3.patch, > RegisterScriptUDFDefineParse.patch, scripting.tgz, scripting.tgz, test.zip > > > It should be possible to write UDFs in scripting languages such as python, > ruby, etc. This frees users from needing to compile Java, generate a jar, > etc. It also opens Pig to programmers who prefer scripting languages over > Java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1389) Implement Pig counter to track number of rows for each input files
[ https://issues.apache.org/jira/browse/PIG-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885804#action_12885804 ] Hadoop QA commented on PIG-1389: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448821/PIG-1389_2.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/360/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/360/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/360/console This message is automatically generated. > Implement Pig counter to track number of rows for each input files > --- > > Key: PIG-1389 > URL: https://issues.apache.org/jira/browse/PIG-1389 > Project: Pig > Issue Type: Improvement >Affects Versions: 0.7.0 >Reporter: Richard Ding >Assignee: Richard Ding > Fix For: 0.8.0 > > Attachments: PIG-1389.patch, PIG-1389.patch, PIG-1389_1.patch, > PIG-1389_2.patch > > > A MR job generated by Pig not only can have multiple outputs (in the case of > multiquery) but also can have multiple inputs (in the case of join or > cogroup). In both cases, the existing Hadoop counters (e.g. > MAP_INPUT_RECORDS, REDUCE_OUTPUT_RECORDS) can not be used to count the number > of records in the given input or output. PIG-1299 addressed the case of > multiple outputs. We need to add new counters for jobs with multiple inputs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1478) Add progress notification listener to PigRunner API
[
https://issues.apache.org/jira/browse/PIG-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885625#action_12885625
]
Hadoop QA commented on PIG-1478:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448532/PIG-1478.patch
against trunk revision 960062.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/339/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/339/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/339/console
This message is automatically generated.
> Add progress notification listener to PigRunner API
> ---
>
> Key: PIG-1478
> URL: https://issues.apache.org/jira/browse/PIG-1478
> Project: Pig
> Issue Type: Improvement
>Reporter: Richard Ding
>Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1478.patch
>
>
> PIG-1333 added PigRunner API to allow Pig users and tools to get a
> status/stats object back after executing a Pig script. The new API, however,
> is synchronous (blocking). It's known that a Pig script can spawn tens (even
> hundreds) MR jobs and take hours to complete. Therefore it'll be nice to give
> progress feedback to the callers during the execution.
> The proposal is to add an optional parameter to the API:
> {code}
> public abstract class PigRunner {
> public static PigStats run(String[] args, PigProgressNotificationListener
> listener) {...}
> }
> {code}
> The new listener is defined as following:
> {code}
> package org.apache.pig.tools.pigstats;
> public interface PigProgressNotificationListener extends
> java.util.EventListener {
> // just before the launch of MR jobs for the script
> public void LaunchStartedNotification(int numJobsToLaunch);
> // number of jobs submitted in a batch
> public void jobsSubmittedNotification(int numJobsSubmitted);
> // a job is started
> public void jobStartedNotification(String assignedJobId);
> // a job is completed successfully
> public void jobFinishedNotification(JobStats jobStats);
> // a job is failed
> public void jobFailedNotification(JobStats jobStats);
> // a user output is completed successfully
> public void outputCompletedNotification(OutputStats outputStats);
> // updates the progress as percentage
> public void progressUpdatedNotification(int progress);
> // the script execution is done
> public void launchCompletedNotification(int numJobsSucceeded);
> }
> {code}
> Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1404) PigUnit - Pig script testing simplified.
[
https://issues.apache.org/jira/browse/PIG-1404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884901#action_12884901
]
Hadoop QA commented on PIG-1404:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448463/PIG-1404-3-doc.patch
against trunk revision 960062.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 401 release audit warnings
(more than the trunk's current 399 warnings).
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/359/testReport/
Release audit warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/359/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/359/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/359/console
This message is automatically generated.
> PigUnit - Pig script testing simplified.
> -
>
> Key: PIG-1404
> URL: https://issues.apache.org/jira/browse/PIG-1404
> Project: Pig
> Issue Type: New Feature
>Reporter: Romain Rigaux
>Assignee: Romain Rigaux
> Fix For: 0.8.0
>
> Attachments: commons-lang-2.4.jar, PIG-1404-2.patch,
> PIG-1404-3-doc.patch, PIG-1404-3.patch, PIG-1404.patch
>
>
> The goal is to provide a simple xUnit framework that enables our Pig scripts
> to be easily:
> - unit tested
> - regression tested
> - quickly prototyped
> No cluster set up is required.
> For example:
> TestCase
> {code}
> @Test
> public void testTop3Queries() {
> String[] args = {
> "n=3",
> };
> test = new PigTest("top_queries.pig", args);
> String[] input = {
> "yahoo\t10",
> "twitter\t7",
> "facebook\t10",
> "yahoo\t15",
> "facebook\t5",
>
> };
> String[] output = {
> "(yahoo,25L)",
> "(facebook,15L)",
> "(twitter,7L)",
> };
> test.assertOutput("data", input, "queries_limit", output);
> }
> {code}
> top_queries.pig
> {code}
> data =
> LOAD '$input'
> AS (query:CHARARRAY, count:INT);
>
> ...
>
> queries_sum =
> FOREACH queries_group
> GENERATE
> group AS query,
> SUM(queries.count) AS count;
>
> ...
>
> queries_limit = LIMIT queries_ordered $n;
> STORE queries_limit INTO '$output';
> {code}
> They are 3 modes:
> * LOCAL (if "pigunit.exectype.local" properties is present)
> * MAPREDUCE (use the cluster specified in the classpath, same as
> HADOOP_CONF_DIR)
> ** automatic mini cluster (is the default and the HADOOP_CONF_DIR to have in
> the class path will be: ~/pigtest/conf)
> ** pointing to an existing cluster (if "pigunit.exectype.cluster" properties
> is present)
> For now, it would be nice to see how this idea could be integrated in
> Piggybank and if PigParser/PigServer could improve their interfaces in order
> to make PigUnit simple.
> Other components based on PigUnit could be built later:
> - standalone MiniCluster
> - notion of workspaces for each test
> - standalone utility that reads test configuration and generates a test
> report...
> It is a first prototype, open to suggestions and can definitely take
> advantage of feedbacks.
> How to test, in pig_trunk:
> {code}
> Apply patch
> $pig_trunk ant compile-test
> $pig_trunk ant
> $pig_trunk/contrib/piggybank/java ant test -Dtest.timeout=99
> {code}
> (it takes 15 min in MAPREDUCE minicluster, tests will need to be split in the
> future between 'unit' and 'integration')
> Many examples are in:
> {code}
> contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/pigunit/TestPigTest.java
> {code}
> When used as a standalone, do not forget commons-lang-2.4.jar and the
> HADOOP_CONF_DIR to your cluster in your CLASSPATH.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1478) Add progress notification listener to PigRunner API
[
https://issues.apache.org/jira/browse/PIG-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884732#action_12884732
]
Hadoop QA commented on PIG-1478:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448532/PIG-1478.patch
against trunk revision 959865.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/337/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/337/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/337/console
This message is automatically generated.
> Add progress notification listener to PigRunner API
> ---
>
> Key: PIG-1478
> URL: https://issues.apache.org/jira/browse/PIG-1478
> Project: Pig
> Issue Type: Improvement
>Reporter: Richard Ding
>Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1478.patch
>
>
> PIG-1333 added PigRunner API to allow Pig users and tools to get a
> status/stats object back after executing a Pig script. The new API, however,
> is synchronous (blocking). It's known that a Pig script can spawn tens (even
> hundreds) MR jobs and take hours to complete. Therefore it'll be nice to give
> progress feedback to the callers during the execution.
> The proposal is to add an optional parameter to the API:
> {code}
> public abstract class PigRunner {
> public static PigStats run(String[] args, PigProgressNotificationListener
> listener) {...}
> }
> {code}
> The new listener is defined as following:
> {code}
> package org.apache.pig.tools.pigstats;
> public interface PigProgressNotificationListener extends
> java.util.EventListener {
> // just before the launch of MR jobs for the script
> public void LaunchStartedNotification(int numJobsToLaunch);
> // number of jobs submitted in a batch
> public void jobsSubmittedNotification(int numJobsSubmitted);
> // a job is started
> public void jobStartedNotification(String assignedJobId);
> // a job is completed successfully
> public void jobFinishedNotification(JobStats jobStats);
> // a job is failed
> public void jobFailedNotification(JobStats jobStats);
> // a user output is completed successfully
> public void outputCompletedNotification(OutputStats outputStats);
> // updates the progress as percentage
> public void progressUpdatedNotification(int progress);
> // the script execution is done
> public void launchCompletedNotification(int numJobsSucceeded);
> }
> {code}
> Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1478) Add progress notification listener to PigRunner API
[
https://issues.apache.org/jira/browse/PIG-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884677#action_12884677
]
Hadoop QA commented on PIG-1478:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448532/PIG-1478.patch
against trunk revision 959865.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/358/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/358/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/358/console
This message is automatically generated.
> Add progress notification listener to PigRunner API
> ---
>
> Key: PIG-1478
> URL: https://issues.apache.org/jira/browse/PIG-1478
> Project: Pig
> Issue Type: Improvement
>Reporter: Richard Ding
>Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1478.patch
>
>
> PIG-1333 added PigRunner API to allow Pig users and tools to get a
> status/stats object back after executing a Pig script. The new API, however,
> is synchronous (blocking). It's known that a Pig script can spawn tens (even
> hundreds) MR jobs and take hours to complete. Therefore it'll be nice to give
> progress feedback to the callers during the execution.
> The proposal is to add an optional parameter to the API:
> {code}
> public abstract class PigRunner {
> public static PigStats run(String[] args, PigProgressNotificationListener
> listener) {...}
> }
> {code}
> The new listener is defined as following:
> {code}
> package org.apache.pig.tools.pigstats;
> public interface PigProgressNotificationListener extends
> java.util.EventListener {
> // just before the launch of MR jobs for the script
> public void LaunchStartedNotification(int numJobsToLaunch);
> // number of jobs submitted in a batch
> public void jobsSubmittedNotification(int numJobsSubmitted);
> // a job is started
> public void jobStartedNotification(String assignedJobId);
> // a job is completed successfully
> public void jobFinishedNotification(JobStats jobStats);
> // a job is failed
> public void jobFailedNotification(JobStats jobStats);
> // a user output is completed successfully
> public void outputCompletedNotification(OutputStats outputStats);
> // updates the progress as percentage
> public void progressUpdatedNotification(int progress);
> // the script execution is done
> public void launchCompletedNotification(int numJobsSucceeded);
> }
> {code}
> Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1478) Add progress notification listener to PigRunner API
[
https://issues.apache.org/jira/browse/PIG-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884554#action_12884554
]
Hadoop QA commented on PIG-1478:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448532/PIG-1478.patch
against trunk revision 958666.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/336/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/336/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/336/console
This message is automatically generated.
> Add progress notification listener to PigRunner API
> ---
>
> Key: PIG-1478
> URL: https://issues.apache.org/jira/browse/PIG-1478
> Project: Pig
> Issue Type: Improvement
>Reporter: Richard Ding
>Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1478.patch
>
>
> PIG-1333 added PigRunner API to allow Pig users and tools to get a
> status/stats object back after executing a Pig script. The new API, however,
> is synchronous (blocking). It's known that a Pig script can spawn tens (even
> hundreds) MR jobs and take hours to complete. Therefore it'll be nice to give
> progress feedback to the callers during the execution.
> The proposal is to add an optional parameter to the API:
> {code}
> public abstract class PigRunner {
> public static PigStats run(String[] args, PigProgressNotificationListener
> listener) {...}
> }
> {code}
> The new listener is defined as following:
> {code}
> package org.apache.pig.tools.pigstats;
> public interface PigProgressNotificationListener extends
> java.util.EventListener {
> // just before the launch of MR jobs for the script
> public void LaunchStartedNotification(int numJobsToLaunch);
> // number of jobs submitted in a batch
> public void jobsSubmittedNotification(int numJobsSubmitted);
> // a job is started
> public void jobStartedNotification(String assignedJobId);
> // a job is completed successfully
> public void jobFinishedNotification(JobStats jobStats);
> // a job is failed
> public void jobFailedNotification(JobStats jobStats);
> // a user output is completed successfully
> public void outputCompletedNotification(OutputStats outputStats);
> // updates the progress as percentage
> public void progressUpdatedNotification(int progress);
> // the script execution is done
> public void launchCompletedNotification(int numJobsSucceeded);
> }
> {code}
> Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1449) RegExLoader hangs on lines that don't match the regular expression
[
https://issues.apache.org/jira/browse/PIG-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884539#action_12884539
]
Hadoop QA commented on PIG-1449:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448516/PIG-1449-RegExLoaderInfiniteLoopFix.patch
against trunk revision 958666.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/357/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/357/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/357/console
This message is automatically generated.
> RegExLoader hangs on lines that don't match the regular expression
> --
>
> Key: PIG-1449
> URL: https://issues.apache.org/jira/browse/PIG-1449
> Project: Pig
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Justin Sanders
>Priority: Minor
> Attachments: PIG-1449-RegExLoaderInfiniteLoopFix.patch,
> RegExLoader.patch
>
>
> In the 0.7.0 changes to RegExLoader there was a bug introduced where the code
> will stay in the while loop if the line isn't matched. Before 0.7.0 these
> lines would be skipped if they didn't match the regular expression. The
> result is the mapper will not respond and will time out with "Task attempt_X
> failed to report status for 600 seconds. Killing!".
> Here are the steps to recreate the bug:
> Create a text file in HDFS with the following lines:
> test1
> testA
> test2
> Run the following pig script:
> REGISTER /usr/local/pig/contrib/piggybank/java/piggybank.jar;
> test = LOAD '/path/to/test.txt' using
> org.apache.pig.piggybank.storage.MyRegExLoader('(test\\d)') AS (line);
> dump test;
> Expected result:
> (test1)
> (test3)
> Actual result:
> Job fails to complete after 600 second timeout waiting on the mapper to
> complete. The mapper hangs at 33% since it can process the first line but
> gets stuck into the while loop on the second line.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1367) [zebra] Map-side Cogroup Test case is needed on 0.7 if the feature is supported in 0.7
[ https://issues.apache.org/jira/browse/PIG-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884094#action_12884094 ] Hadoop QA commented on PIG-1367: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448416/PIG-1367.patch against trunk revision 958666. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/356/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/356/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/356/console This message is automatically generated. > [zebra] Map-side Cogroup Test case is needed on 0.7 if the feature is > supported in 0.7 > -- > > Key: PIG-1367 > URL: https://issues.apache.org/jira/browse/PIG-1367 > Project: Pig > Issue Type: New Feature >Affects Versions: 0.7.0 >Reporter: Yan Zhou > Fix For: 0.8.0 > > Attachments: PIG-1367.patch > > > PIG-1315 has the Zebra support for this feature and the map-side group-by. It > also has the test case for map-side COGROUP; while the test case for map-side > GROUP-BY is in PIG-1357. > However PIG-1315 is committed to the trunk as a whole; but only committed to > the 0.7 branch without the map-side group-by test case because PIG has yet to > decide if the feature will be in the 0.7 release. > This JIRA is created for tracking purpose should the decision to support > map-side COGROUP in 0.7 by PIG is made. If not, this should be made invalid > eventually. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1295) Binary comparator for secondary sort
[ https://issues.apache.org/jira/browse/PIG-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12883486#action_12883486 ] Hadoop QA commented on PIG-1295: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448251/PIG-1295_0.6.patch against trunk revision 958666. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 150 javac compiler warnings (more than the trunk's current 145 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 402 release audit warnings (more than the trunk's current 399 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/355/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/355/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/355/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/355/console This message is automatically generated. > Binary comparator for secondary sort > > > Key: PIG-1295 > URL: https://issues.apache.org/jira/browse/PIG-1295 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.7.0 >Reporter: Daniel Dai >Assignee: Gianmarco De Francisci Morales > Fix For: 0.8.0 > > Attachments: PIG-1295_0.1.patch, PIG-1295_0.2.patch, > PIG-1295_0.3.patch, PIG-1295_0.4.patch, PIG-1295_0.5.patch, PIG-1295_0.6.patch > > > When hadoop framework doing the sorting, it will try to use binary version of > comparator if available. The benefit of binary comparator is we do not need > to instantiate the object before we compare. We see a ~30% speedup after we > switch to binary comparator. Currently, Pig use binary comparator in > following case: > 1. When semantics of order doesn't matter. For example, in distinct, we need > to do a sort in order to filter out duplicate values; however, we do not care > how comparator sort keys. Groupby also share this character. In this case, we > rely on hadoop's default binary comparator > 2. Semantics of order matter, but the key is of simple type. In this case, we > have implementation for simple types, such as integer, long, float, > chararray, databytearray, string > However, if the key is a tuple and the sort semantics matters, we do not have > a binary comparator implementation. This especially matters when we switch to > use secondary sort. In secondary sort, we convert the inner sort of nested > foreach into the secondary key and rely on hadoop to sorting on both main key > and secondary key. The sorting key will become a two items tuple. Since the > secondary key the sorting key of the nested foreach, so the sorting semantics > matters. It turns out we do not have binary comparator once we use secondary > sort, and we see a significant slow down. > Binary comparator for tuple should be doable once we understand the binary > structure of the serialized tuple. We can focus on most common use cases > first, which is "group by" followed by a nested sort. In this case, we will > use secondary sort. Semantics of the first key does not matter but semantics > of secondary key matters. We need to identify the boundary of main key and > secondary key in the binary tuple buffer without instantiate tuple itself. > Then if the first key equals, we use a binary comparator to compare secondary > key. Secondary key can also be a complex data type, but for the first step, > we focus on simple secondary key, which is the most common use case. > We mark this issue to be a candidate project for "Google summer of code 2010" > program. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1389) Implement Pig counter to track number of rows for each input files
[ https://issues.apache.org/jira/browse/PIG-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12883424#action_12883424 ] Hadoop QA commented on PIG-1389: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448259/PIG-1389_1.patch against trunk revision 958666. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/335/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/335/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/335/console This message is automatically generated. > Implement Pig counter to track number of rows for each input files > --- > > Key: PIG-1389 > URL: https://issues.apache.org/jira/browse/PIG-1389 > Project: Pig > Issue Type: Improvement >Affects Versions: 0.7.0 >Reporter: Richard Ding >Assignee: Richard Ding > Fix For: 0.8.0 > > Attachments: PIG-1389.patch, PIG-1389.patch, PIG-1389_1.patch > > > A MR job generated by Pig not only can have multiple outputs (in the case of > multiquery) but also can have multiple inputs (in the case of join or > cogroup). In both cases, the existing Hadoop counters (e.g. > MAP_INPUT_RECORDS, REDUCE_OUTPUT_RECORDS) can not be used to count the number > of records in the given input or output. PIG-1299 addressed the case of > multiple outputs. We need to add new counters for jobs with multiple inputs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1468) DataByteArray.compareTo() does not compare in lexicographic order
[ https://issues.apache.org/jira/browse/PIG-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882985#action_12882985 ] Hadoop QA commented on PIG-1468: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448155/PIG-1468.patch against trunk revision 958053. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/354/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/354/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/354/console This message is automatically generated. > DataByteArray.compareTo() does not compare in lexicographic order > - > > Key: PIG-1468 > URL: https://issues.apache.org/jira/browse/PIG-1468 > Project: Pig > Issue Type: Bug >Reporter: Gianmarco De Francisci Morales >Assignee: Gianmarco De Francisci Morales > Attachments: PIG-1468.patch > > > The compareTo() method of org.apache.pig.data.DataByteArray does not compare > items in lexicographic order. > Actually, it takes into account the signum of the bytes that compose the > DataByteArray. > So, for example, 0xff compares to less than 0x00 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1469) DefaultDataBag assumes ArrayList as default List type
[ https://issues.apache.org/jira/browse/PIG-1469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882983#action_12882983 ] Hadoop QA commented on PIG-1469: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448156/PIG-1469.patch against trunk revision 958053. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/334/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/334/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/334/console This message is automatically generated. > DefaultDataBag assumes ArrayList as default List type > - > > Key: PIG-1469 > URL: https://issues.apache.org/jira/browse/PIG-1469 > Project: Pig > Issue Type: Bug > Components: data >Affects Versions: 0.8.0 >Reporter: Gianmarco De Francisci Morales >Assignee: Gianmarco De Francisci Morales > Fix For: 0.8.0 > > Attachments: PIG-1469.patch > > > In org.apache.pig.data.DefaultDataBag, the field mContents is assumed to be > of type ArrayList but the user can actually pass a different List to the > constructor. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1467) order by fail when set "fs.file.impl.disable.cache" to true
[ https://issues.apache.org/jira/browse/PIG-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882813#action_12882813 ] Hadoop QA commented on PIG-1467: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448105/PIG-1467-2.patch against trunk revision 958053. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 145 javac compiler warnings (more than the trunk's current 140 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/353/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/353/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/353/console This message is automatically generated. > order by fail when set "fs.file.impl.disable.cache" to true > --- > > Key: PIG-1467 > URL: https://issues.apache.org/jira/browse/PIG-1467 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.7.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.7.0, 0.8.0 > > Attachments: PIG-1467-1.patch, PIG-1467-2.patch > > > Order by fail with the message: > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.WeightedRangePartitioner.setConf(WeightedRangePartitioner.java:135) > at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:551) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:630) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314) > at org.apache.hadoop.mapred.Child$4.run(Child.java:217) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1062) > at org.apache.hadoop.mapred.Child.main(Child.java:211) > This happens with the following hadoop settings: > fs.file.impl.disable.cache=true > fs.hdfs.impl.disable.cache=true -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1467) order by fail when set "fs.file.impl.disable.cache" to true
[ https://issues.apache.org/jira/browse/PIG-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882754#action_12882754 ] Hadoop QA commented on PIG-1467: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448103/PIG-1467-1.patch against trunk revision 958053. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/352/console This message is automatically generated. > order by fail when set "fs.file.impl.disable.cache" to true > --- > > Key: PIG-1467 > URL: https://issues.apache.org/jira/browse/PIG-1467 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.7.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.7.0, 0.8.0 > > Attachments: PIG-1467-1.patch > > > Order by fail with the message: > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.WeightedRangePartitioner.setConf(WeightedRangePartitioner.java:135) > at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:551) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:630) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314) > at org.apache.hadoop.mapred.Child$4.run(Child.java:217) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1062) > at org.apache.hadoop.mapred.Child.main(Child.java:211) > This happens with the following hadoop settings: > fs.file.impl.disable.cache=true > fs.hdfs.impl.disable.cache=true -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1434) Allow casting relations to scalars
[ https://issues.apache.org/jira/browse/PIG-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882732#action_12882732 ] Hadoop QA commented on PIG-1434: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448098/scalarImpl.patch against trunk revision 958053. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/351/console This message is automatically generated. > Allow casting relations to scalars > -- > > Key: PIG-1434 > URL: https://issues.apache.org/jira/browse/PIG-1434 > Project: Pig > Issue Type: Improvement >Reporter: Olga Natkovich >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: scalarImpl.patch > > > This jira is to implement a simplified version of the functionality described > in https://issues.apache.org/jira/browse/PIG-801. > The proposal is to allow casting relations to scalar types in foreach. > Example: > A = load 'data' as (x, y, z); > B = group A all; > C = foreach B generate COUNT(A); > . > X = > Y = foreach X generate $1/(long) C; > Couple of additional comments: > (1) You can only cast relations including a single value or an error will be > reported > (2) Name resolution is needed since relation X might have field named C in > which case that field takes precedence. > (3) Y will look for C closest to it. > Implementation thoughts: > The idea is to store C into a file and then convert it into scalar via a UDF. > I believe we already have a UDF that Ben Reed contributed for this purpose. > Most of the work would be to update the logical plan to > (1) Store C > (2) convert the cast to the UDF -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1464) Should clean the Graph when register another Pig Script
[
https://issues.apache.org/jira/browse/PIG-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882583#action_12882583
]
Hadoop QA commented on PIG-1464:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448030/PIG_1463.patch
against trunk revision 957753.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/350/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/350/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/350/console
This message is automatically generated.
> Should clean the Graph when register another Pig Script
> ---
>
> Key: PIG-1464
> URL: https://issues.apache.org/jira/browse/PIG-1464
> Project: Pig
> Issue Type: Bug
> Components: grunt
>Affects Versions: 0.8.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
> Fix For: 0.8.0
>
> Attachments: PIG_1463.patch
>
>
> In the current implementation, the variable names in pig script are all
> global variable. This make one pig script know the variable in other scripts.
> In my opinion, this is not right. Every relation name in pig script should be
> local variable, otherwise it will bring in unexpected result. This issue
> relates to PIG-1423
> E.g there are two pig script as follows:
> Test_1.pig
> {code}
> a = load 'data/b.txt' ;
> {code}
> Test_2.pig
> {code}
> b = foreach a generate $0; // "a" is recognized by Grunt although it is in
> Test_1.pig
> {code}
> And the following execute normally, do not throw any exception
> {code}
> PigServer pig=new PigServer(ExecType.Local);
> pig.registerScript("Test_1.pig");
> pig.registerScript("Test_2.pig");
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1389) Implement Pig counter to track number of rows for each input files
[ https://issues.apache.org/jira/browse/PIG-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882077#action_12882077 ] Hadoop QA commented on PIG-1389: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447912/PIG-1389.patch against trunk revision 957399. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/349/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/349/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/349/console This message is automatically generated. > Implement Pig counter to track number of rows for each input files > --- > > Key: PIG-1389 > URL: https://issues.apache.org/jira/browse/PIG-1389 > Project: Pig > Issue Type: Improvement >Affects Versions: 0.7.0 >Reporter: Richard Ding >Assignee: Richard Ding > Fix For: 0.8.0 > > Attachments: PIG-1389.patch > > > A MR job generated by Pig not only can have multiple outputs (in the case of > multiquery) but also can have multiple inputs (in the case of join or > cogroup). In both cases, the existing Hadoop counters (e.g. > MAP_INPUT_RECORDS, REDUCE_OUTPUT_RECORDS) can not be used to count the number > of records in the given input or output. PIG-1299 addressed the case of > multiple outputs. We need to add new counters for jobs with multiple inputs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1454) Consider clean up backend code
[ https://issues.apache.org/jira/browse/PIG-1454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882031#action_12882031 ] Hadoop QA commented on PIG-1454: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447897/PIG-1454.patch against trunk revision 957277. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 394 release audit warnings (more than the trunk's current 389 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/333/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/333/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/333/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/333/console This message is automatically generated. > Consider clean up backend code > -- > > Key: PIG-1454 > URL: https://issues.apache.org/jira/browse/PIG-1454 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.7.0 >Reporter: Richard Ding >Assignee: Richard Ding > Fix For: 0.8.0 > > Attachments: PIG-1454.patch > > > Prior to 0.7, Pig had its own local execution mode, in addition to hadoop map > reduce execution mode. To support these two different execution modes, Pig > implemented an abstraction layer with a set of interfaces and abstract > classes. Pig 0.7 replaced the local mode with hadoop local mode and made > this abstraction layer redundant. > Our goal is to remove those extra code. But we need also keep code backward > compatible since some interfaces are exposed by top-level API. > So we propose the first steps: > * Deprecate methods on FileLocalizer that have DataStorage as parameter. > * Remove ExecPhysicalOperator, ExecPhysicalPlan, ExecScopedLogicalOperator, > ExecutionEngine and util/ExecTools from > org.apache.pig.backend.executionengine package. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1453) [zebra] Intermittent failure for TestOrderPreserveUnionHDFS
[ https://issues.apache.org/jira/browse/PIG-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881996#action_12881996 ] Hadoop QA commented on PIG-1453: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447494/PIG-1453.patch against trunk revision 957277. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 36 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/348/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/348/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/348/console This message is automatically generated. > [zebra] Intermittent failure for TestOrderPreserveUnionHDFS > --- > > Key: PIG-1453 > URL: https://issues.apache.org/jira/browse/PIG-1453 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.8.0 >Reporter: Daniel Dai >Assignee: Yan Zhou > Fix For: 0.8.0 > > Attachments: PIG-1453.patch, PIG-1453.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1333) API interface to Pig
[
https://issues.apache.org/jira/browse/PIG-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881586#action_12881586
]
Hadoop QA commented on PIG-1333:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12447767/PIG-1333_3.patch
against trunk revision 957046.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 11 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 140 javac compiler warnings (more
than the trunk's current 138 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 391 release audit warnings
(more than the trunk's current 387 warnings).
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/347/testReport/
Release audit warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/347/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/347/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/347/console
This message is automatically generated.
> API interface to Pig
>
>
> Key: PIG-1333
> URL: https://issues.apache.org/jira/browse/PIG-1333
> Project: Pig
> Issue Type: Improvement
>Reporter: Olga Natkovich
>Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1333.patch, PIG-1333_1.patch, PIG-1333_2.patch,
> PIG-1333_3.patch
>
>
> It would be nice to make Pig more friendly for applications like workflow
> that would be executing pig scripts on user behalf.
> Currently, they would have to use pig command line to execute the code;
> however, this has limitation on the kind of output that would be delivered.
> For instance, it is hard to produce error information that is easy to use
> programatically or collect statistics.
> The proposal is to create a class that mimics the behavior of the Main but
> gives users a status object back. The the main code of pig would look
> somethig like:
> public static void main(String args[])
> {
> PigStatus ps = PigMain.exec(args);
> exit (PigStatus.rc);
> }
> We need to define the following:
> - Content of PigStatus. It should at least include
>* return code
>* error string
>* exception
>* statistics
> - A way to propagate the status class through pig code
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1405) Need to move many standard functions from piggybank into Pig
[ https://issues.apache.org/jira/browse/PIG-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880965#action_12880965 ] Hadoop QA commented on PIG-1405: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447615/StandardUDFtoPigFinale.patch against trunk revision 956662. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/345/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/345/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/345/console This message is automatically generated. > Need to move many standard functions from piggybank into Pig > > > Key: PIG-1405 > URL: https://issues.apache.org/jira/browse/PIG-1405 > Project: Pig > Issue Type: Improvement >Reporter: Alan Gates >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: StandardUDFtoPig.patch, StandardUDFtoPig3.patch, > StandardUDFtoPig4.patch, StandardUDFtoPigFinale.patch > > > There are currently a number of functions in Piggybank that represent > features commonly supported by languages and database engines. We need to > decide which of these Pig should support as built in functions and put them > in org.apache.pig.builtin. This will also mean adding unit tests and > javadocs for some UDFs. The existing classes will be left in Piggybank for > some time for backward compatibility. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1034) Pig does not support ORDER ... BY group alias
[
https://issues.apache.org/jira/browse/PIG-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880808#action_12880808
]
Hadoop QA commented on PIG-1034:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12447586/PIG_1034.patch
against trunk revision 956440.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/344/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/344/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/344/console
This message is automatically generated.
> Pig does not support ORDER ... BY group alias
> -
>
> Key: PIG-1034
> URL: https://issues.apache.org/jira/browse/PIG-1034
> Project: Pig
> Issue Type: Bug
>Affects Versions: 0.8.0
>Reporter: David Ciemiewicz
>Assignee: Jeff Zhang
> Fix For: 0.8.0
>
> Attachments: PIG_1034.patch
>
>
> GROUP ... ALL and GROUP ... BY produce an alias "group".
> Pig produces a syntax error if you attempt to ORDER ... BY group.
> This does seem like a perfectly reasonable thing to do.
> The workaround is to create an alias for group using an AS clause. But I
> think this workaround should be unnecessary.
> Here's sample code which elicits the syntax error:
> {code}
> A = load 'one.txt' using PigStorage as (one: int);
> B = group A all;
> C = foreach B generate
> group,
> COUNT(A) as count;
> D = order C by group parallel 1; -- group is one of the aliases in C, why
> does this throw a syntax error?
> dump D;
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1453) [zebra] Intermittent failure for TestOrderPreserveUnionHDFS
[ https://issues.apache.org/jira/browse/PIG-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880425#action_12880425 ] Hadoop QA commented on PIG-1453: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447494/PIG-1453.patch against trunk revision 955763. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 36 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/331/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/331/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/331/console This message is automatically generated. > [zebra] Intermittent failure for TestOrderPreserveUnionHDFS > --- > > Key: PIG-1453 > URL: https://issues.apache.org/jira/browse/PIG-1453 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.8.0 >Reporter: Daniel Dai >Assignee: Yan Zhou > Fix For: 0.8.0 > > Attachments: PIG-1453.patch, PIG-1453.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1405) Need to move many standard functions from piggybank into Pig
[ https://issues.apache.org/jira/browse/PIG-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880421#action_12880421 ] Hadoop QA commented on PIG-1405: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447492/StandardUDFtoPig4.patch against trunk revision 955763. -1 @author. The patch appears to contain 2 @author tags which the Pig community has agreed to not allow in code contributions. +1 tests included. The patch appears to include 5 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/343/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/343/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/343/console This message is automatically generated. > Need to move many standard functions from piggybank into Pig > > > Key: PIG-1405 > URL: https://issues.apache.org/jira/browse/PIG-1405 > Project: Pig > Issue Type: Improvement >Reporter: Alan Gates >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: StandardUDFtoPig.patch, StandardUDFtoPig3.patch, > StandardUDFtoPig4.patch > > > There are currently a number of functions in Piggybank that represent > features commonly supported by languages and database engines. We need to > decide which of these Pig should support as built in functions and put them > in org.apache.pig.builtin. This will also mean adding unit tests and > javadocs for some UDFs. The existing classes will be left in Piggybank for > some time for backward compatibility. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1405) Need to move many standard functions from piggybank into Pig
[ https://issues.apache.org/jira/browse/PIG-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880049#action_12880049 ] Hadoop QA commented on PIG-1405: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447381/StandardUDFtoPig3.patch against trunk revision 955701. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. -1 javac. The applied patch generated 146 javac compiler warnings (more than the trunk's current 138 warnings). -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/330/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/330/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/330/console This message is automatically generated. > Need to move many standard functions from piggybank into Pig > > > Key: PIG-1405 > URL: https://issues.apache.org/jira/browse/PIG-1405 > Project: Pig > Issue Type: Improvement >Reporter: Alan Gates >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: StandardUDFtoPig.patch, StandardUDFtoPig3.patch > > > There are currently a number of functions in Piggybank that represent > features commonly supported by languages and database engines. We need to > decide which of these Pig should support as built in functions and put them > in org.apache.pig.builtin. This will also mean adding unit tests and > javadocs for some UDFs. The existing classes will be left in Piggybank for > some time for backward compatibility. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1453) [zebra] Intermittent failure for TestOrderPreserveUnionHDFS
[ https://issues.apache.org/jira/browse/PIG-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880025#action_12880025 ] Hadoop QA commented on PIG-1453: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447373/PIG-1453.patch against trunk revision 955701. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 36 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/341/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/341/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/341/console This message is automatically generated. > [zebra] Intermittent failure for TestOrderPreserveUnionHDFS > --- > > Key: PIG-1453 > URL: https://issues.apache.org/jira/browse/PIG-1453 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.8.0 >Reporter: Daniel Dai >Assignee: Yan Zhou > Fix For: 0.8.0 > > Attachments: PIG-1453.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1221) Filter equality does not work for tuples
[ https://issues.apache.org/jira/browse/PIG-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879765#action_12879765 ] Hadoop QA commented on PIG-1221: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447317/PIG_1221.patch against trunk revision 955028. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/340/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/340/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/340/console This message is automatically generated. > Filter equality does not work for tuples > > > Key: PIG-1221 > URL: https://issues.apache.org/jira/browse/PIG-1221 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.8.0 > Environment: Windows and Linux. Java 1.6 hadoop 0.20.1 >Reporter: Neil Blue >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: PIG_1221.patch > > > From the documentation I understand that it should be possible to filter a > relation based on the equality of tuples. > http://wiki.apache.org/pig/PigTypesFunctionalSpec , > http://hadoop.apache.org/pig/docs/r0.5.0/piglatin_reference.html#deref: > However with this data file > -- indext.txt: > (1,one) (1,ONE) > (2,two) (22, twentytwo) > (3,three) (3,three) > I run this pig script: > A = LOAD 'indext.txt' AS (t1:(a:int, b:chararray), t2:(a:int, b:chararray)); > B = FILTER A BY t1==t2; DUMP B; > Expecting the output: > ((3,three),(3,three)) > However there is an error: > 2010-02-03 09:05:20,523 [main] ERROR org.apache.pig.tools.grunt.Grunt > - ERROR 2067: EqualToExpr does not know how to handle type: tuple > > Pig Stack Trace > > --- > > ERROR 2067: EqualToExpr does not know how to handle type: tuple > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: > > Unable to > > open iterator for alias B > >at org.apache.pig.PigServer.openIterator(PigServer.java:475) > >at > > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java: > > 532) > >at > > org > > .apache > > .pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser. > > java:190) > >at > > org > > .apache > > .pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:166 > > ) > >at > > org > > .apache > > .pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:142 > > ) > >at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89) > >at org.apache.pig.Main.main(Main.java:397) > > Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR > > 1002: > > Unable to store alias B > >at org.apache.pig.PigServer.store(PigServer.java:530) > >at org.apache.pig.PigServer.openIterator(PigServer.java:458) > >... 6 more > > Caused by: org.apache.pig.backend.executionengine.ExecException: > > ERROR 2067: > > EqualToExpr does not know how to handle type: tuple > >at > > org > > .apache > > .pig.backend.hadoop.executionengine.physicalLayer.expressionOperat > > ors.EqualToExpr.getNext(EqualToExpr.java:108) > >at > > org > > .apache > > .pig.backend.hadoop.executionengine.physicalLayer.relationalOperat > > ors.POFilter.getNext(POFilter.java:148) > >at > > org > > .apache > > .pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator > > .processInput(PhysicalOperator.java:231) > >at > > org > > .apache > > .pig.backend.local.executionengine.physicalLayer.counters.POCounte > > r.getNext(POCounter.java:71) > >at > > org > > .apache > > .pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator > > .processInput(PhysicalOperator.java:231) > >at > > org > > .apache > > .pig.backend.hadoop.executionengine.physicalLayer.relationalOperat > > ors.POStore.getNext(POStore.java:117) > >at > > org > > .apache > > .pig.backend.local.executionengine.LocalPigLauncher.runPipeline(Lo > > calPigLauncher.java:146) > >at > > org > > .apache >
[jira] Commented: (PIG-1452) to remove hadoop20.jar from lib and use hadoop from the apache maven repo.
[ https://issues.apache.org/jira/browse/PIG-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879414#action_12879414 ] Hadoop QA commented on PIG-1452: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447216/PIG-1452.PATCH against trunk revision 955028. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/339/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/339/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/339/console This message is automatically generated. > to remove hadoop20.jar from lib and use hadoop from the apache maven repo. > -- > > Key: PIG-1452 > URL: https://issues.apache.org/jira/browse/PIG-1452 > Project: Pig > Issue Type: Improvement > Components: build >Affects Versions: 0.8.0 >Reporter: Giridharan Kesavan >Assignee: Giridharan Kesavan > Attachments: PIG-1452.PATCH > > > pig use ivy for dependency management. But still it uses hadoop20.jar from > the lib folder. > Now that we have the hadoop-0.20.2 artifacts available in the maven repo, pig > should leverage ivy for resolving/retrieving hadoop artifacts. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1451) [zebra] change the build.test property in build to test.build.dir to be in consistent with PIG
[ https://issues.apache.org/jira/browse/PIG-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879209#action_12879209 ] Hadoop QA commented on PIG-1451: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447159/PIG-1451.patch against trunk revision 954772. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 14 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/338/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/338/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/338/console This message is automatically generated. > [zebra] change the build.test property in build to test.build.dir to be in > consistent with PIG > -- > > Key: PIG-1451 > URL: https://issues.apache.org/jira/browse/PIG-1451 > Project: Pig > Issue Type: Improvement >Affects Versions: 0.6.0, 0.7.0, 0.8.0 >Reporter: Yan Zhou >Assignee: Yan Zhou >Priority: Minor > Fix For: 0.6.0, 0.7.0, 0.8.0 > > Attachments: PIG-1451.patch > > > Because build process handles PIG and Zebra builds in the same settings, the > property should be the same so the build process have consistent controls. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1333) API interface to Pig
[
https://issues.apache.org/jira/browse/PIG-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878869#action_12878869
]
Hadoop QA commented on PIG-1333:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12447048/PIG-1333_1.patch
against trunk revision 953798.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 11 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 387 release audit warnings
(more than the trunk's current 383 warnings).
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/329/testReport/
Release audit warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/329/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/329/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/329/console
This message is automatically generated.
> API interface to Pig
>
>
> Key: PIG-1333
> URL: https://issues.apache.org/jira/browse/PIG-1333
> Project: Pig
> Issue Type: Improvement
>Reporter: Olga Natkovich
>Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1333.patch, PIG-1333_1.patch
>
>
> It would be nice to make Pig more friendly for applications like workflow
> that would be executing pig scripts on user behalf.
> Currently, they would have to use pig command line to execute the code;
> however, this has limitation on the kind of output that would be delivered.
> For instance, it is hard to produce error information that is easy to use
> programatically or collect statistics.
> The proposal is to create a class that mimics the behavior of the Main but
> gives users a status object back. The the main code of pig would look
> somethig like:
> public static void main(String args[])
> {
> PigStatus ps = PigMain.exec(args);
> exit (PigStatus.rc);
> }
> We need to define the following:
> - Content of PigStatus. It should at least include
>* return code
>* error string
>* exception
>* statistics
> - A way to propagate the status class through pig code
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1449) RegExLoader hangs on lines that don't match the regular expression
[
https://issues.apache.org/jira/browse/PIG-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878812#action_12878812
]
Hadoop QA commented on PIG-1449:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12447045/RegExLoader.patch
against trunk revision 953798.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no tests are needed for this patch.
-1 patch. The patch command could not apply the patch.
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/328/console
This message is automatically generated.
> RegExLoader hangs on lines that don't match the regular expression
> --
>
> Key: PIG-1449
> URL: https://issues.apache.org/jira/browse/PIG-1449
> Project: Pig
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Justin Sanders
>Priority: Minor
> Attachments: RegExLoader.patch
>
>
> In the 0.7.0 changes to RegExLoader there was a bug introduced where the code
> will stay in the while loop if the line isn't matched. Before 0.7.0 these
> lines would be skipped if they didn't match the regular expression. The
> result is the mapper will not respond and will time out with "Task attempt_X
> failed to report status for 600 seconds. Killing!".
> Here are the steps to recreate the bug:
> Create a text file in HDFS with the following lines:
> test1
> testA
> test2
> Run the following pig script:
> REGISTER /usr/local/pig/contrib/piggybank/java/piggybank.jar;
> test = LOAD '/path/to/test.txt' using
> org.apache.pig.piggybank.storage.MyRegExLoader('(test\\d)') AS (line);
> dump test;
> Expected result:
> (test1)
> (test3)
> Actual result:
> Job fails to complete after 600 second timeout waiting on the mapper to
> complete. The mapper hangs at 33% since it can process the first line but
> gets stuck into the while loop on the second line.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-972) Make describe work with nested foreach
[ https://issues.apache.org/jira/browse/PIG-972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878810#action_12878810 ] Hadoop QA commented on PIG-972: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447041/NestedDescribeFinale1.patch against trunk revision 953798. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 384 release audit warnings (more than the trunk's current 383 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/327/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/327/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/327/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/327/console This message is automatically generated. > Make describe work with nested foreach > -- > > Key: PIG-972 > URL: https://issues.apache.org/jira/browse/PIG-972 > Project: Pig > Issue Type: Improvement >Reporter: Olga Natkovich >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: NestedDescribeFinale.patch, NestedDescribeFinale1.patch, > NestedDescribeProp1.patch, NestedDescribeProp2Initial.patch > > > Currently Parser can't deal with that. This is because describe is part of > Grunt parser while the rest of nested foreach is handled by the QueryParser -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1302) Include zebra's "pigtest" ant target as a part of pig's ant test target
[ https://issues.apache.org/jira/browse/PIG-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878702#action_12878702 ] Hadoop QA commented on PIG-1302: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446596/PIG-1302.patch against trunk revision 953798. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/326/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/326/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/326/console This message is automatically generated. > Include zebra's "pigtest" ant target as a part of pig's ant test target > --- > > Key: PIG-1302 > URL: https://issues.apache.org/jira/browse/PIG-1302 > Project: Pig > Issue Type: Improvement >Affects Versions: 0.7.0 >Reporter: Pradeep Kamath >Assignee: Giridharan Kesavan > Attachments: PIG-1302.patch > > > There are changes made in Pig interfaces which break zebra loaders/storers. > It would be good to run the pig tests in the zebra unit tests as part of > running pig's core-test for each patch submission. So essentially in the > "test" ant target in pig, we would need to invoke zebra's "pigtest" target. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-972) Make describe work with nested foreach
[ https://issues.apache.org/jira/browse/PIG-972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878545#action_12878545 ] Hadoop QA commented on PIG-972: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446735/NestedDescribeFinale.patch against trunk revision 953798. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. -1 release audit. The applied patch generated 384 release audit warnings (more than the trunk's current 383 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/324/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/324/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/324/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/324/console This message is automatically generated. > Make describe work with nested foreach > -- > > Key: PIG-972 > URL: https://issues.apache.org/jira/browse/PIG-972 > Project: Pig > Issue Type: Improvement >Reporter: Olga Natkovich >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: NestedDescribeFinale.patch, NestedDescribeProp1.patch, > NestedDescribeProp2Initial.patch > > > Currently Parser can't deal with that. This is because describe is part of > Grunt parser while the rest of nested foreach is handled by the QueryParser -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1428) Add getPigStatusReporter() to PigHadoopLogger
[ https://issues.apache.org/jira/browse/PIG-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877324#action_12877324 ] Hadoop QA commented on PIG-1428: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446095/PIG-1428.patch against trunk revision 949057. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 379 release audit warnings (more than the trunk's current 378 warnings). -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/318/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/318/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/318/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/318/console This message is automatically generated. > Add getPigStatusReporter() to PigHadoopLogger > - > > Key: PIG-1428 > URL: https://issues.apache.org/jira/browse/PIG-1428 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Ashutosh Chauhan >Assignee: Dmitriy V. Ryaboy > Fix For: 0.8.0 > > Attachments: PIG-1428.patch, PIG-1428.patch > > > Without this getter method, its not possible to get counters, report progress > etc. from UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1445) Pig error: ERROR 2013: Moving LOLimit in front of LOStream is not implemented
[
https://issues.apache.org/jira/browse/PIG-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877318#action_12877318
]
Hadoop QA commented on PIG-1445:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12446718/PIG-1445-1.patch
against trunk revision 953109.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 383 release audit warnings
(more than the trunk's current 382 warnings).
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/322/testReport/
Release audit warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/322/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/322/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/322/console
This message is automatically generated.
> Pig error: ERROR 2013: Moving LOLimit in front of LOStream is not implemented
> --
>
> Key: PIG-1445
> URL: https://issues.apache.org/jira/browse/PIG-1445
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.7.0
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: PIG-1445-1.patch
>
>
> The following script fail due to "ERROR 2013: Moving LOLimit in front of
> LOStream is not implemented".
> {code}
> A = LOAD 'data';
> B = STREAM A THROUGH `stream.pl`;
> C = LIMIT B 10;
> explain C;
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1443) DefaultTuple underestimate the memory footprint for string
[
https://issues.apache.org/jira/browse/PIG-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877256#action_12877256
]
Hadoop QA commented on PIG-1443:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12446712/PIG-1443-1.patch
against trunk revision 952098.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 139 javac compiler warnings (more
than the trunk's current 138 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/321/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/321/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/321/console
This message is automatically generated.
> DefaultTuple underestimate the memory footprint for string
> --
>
> Key: PIG-1443
> URL: https://issues.apache.org/jira/browse/PIG-1443
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.7.0
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: PIG-1443-1.patch
>
>
> Currently, in DefaultTuple, we estimate the memory footprint for string as if
> it is char array. The formula we use is: length * 2 + 12. It turns out we
> underestimate the memory usage for string. Here is a list of real memory
> footprint for string we get from memory dump:
> | length of string | memory in bytes |
> | 7 | 56 |
> | 3 | 48 |
> | 1 | 40 |
> I did a search and find the following formula can accurately estimate the
> memory footprint for string:
> {code}
> 8 * (int) (((length * 2) + 45) / 8)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1438) [Performance] MultiQueryOptimizer should also merge DISTINCT jobs
[ https://issues.apache.org/jira/browse/PIG-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876980#action_12876980 ] Hadoop QA commented on PIG-1438: +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446652/PIG-1438_1.patch against trunk revision 952098. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/334/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/334/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/334/console This message is automatically generated. > [Performance] MultiQueryOptimizer should also merge DISTINCT jobs > - > > Key: PIG-1438 > URL: https://issues.apache.org/jira/browse/PIG-1438 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.7.0 >Reporter: Richard Ding >Assignee: Richard Ding > Fix For: 0.8.0 > > Attachments: PIG-1438.patch, PIG-1438_1.patch > > > Current implementation doesn't merge jobs derived from DISTINCT statements. > The reason is that DISTINCT jobs are implemented using a special combiner > (DistinctCombiner). But we should be able to merge jobs that have the same > type of combiner (e.g. merge multiple DISTINCT jobs into one). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1438) [Performance] MultiQueryOptimizer should also merge DISTINCT jobs
[ https://issues.apache.org/jira/browse/PIG-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876840#action_12876840 ] Hadoop QA commented on PIG-1438: +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446604/PIG-1438.patch against trunk revision 952098. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/333/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/333/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/333/console This message is automatically generated. > [Performance] MultiQueryOptimizer should also merge DISTINCT jobs > - > > Key: PIG-1438 > URL: https://issues.apache.org/jira/browse/PIG-1438 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.7.0 >Reporter: Richard Ding >Assignee: Richard Ding > Fix For: 0.8.0 > > Attachments: PIG-1438.patch > > > Current implementation doesn't merge jobs derived from DISTINCT statements. > The reason is that DISTINCT jobs are implemented using a special combiner > (DistinctCombiner). But we should be able to merge jobs that have the same > type of combiner (e.g. merge multiple DISTINCT jobs into one). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1428) Add getPigStatusReporter() to PigHadoopLogger
[ https://issues.apache.org/jira/browse/PIG-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876708#action_12876708 ] Hadoop QA commented on PIG-1428: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446095/PIG-1428.patch against trunk revision 952098. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 383 release audit warnings (more than the trunk's current 382 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/332/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/332/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/332/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/332/console This message is automatically generated. > Add getPigStatusReporter() to PigHadoopLogger > - > > Key: PIG-1428 > URL: https://issues.apache.org/jira/browse/PIG-1428 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Ashutosh Chauhan >Assignee: Dmitriy V. Ryaboy > Fix For: 0.8.0 > > Attachments: PIG-1428.patch, PIG-1428.patch > > > Without this getter method, its not possible to get counters, report progress > etc. from UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1433) pig should create success file if mapreduce.fileoutputcommitter.marksuccessfuljobs is true
[ https://issues.apache.org/jira/browse/PIG-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875639#action_12875639 ] Hadoop QA commented on PIG-1433: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446222/PIG-1433.patch against trunk revision 951229. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/330/console This message is automatically generated. > pig should create success file if > mapreduce.fileoutputcommitter.marksuccessfuljobs is true > -- > > Key: PIG-1433 > URL: https://issues.apache.org/jira/browse/PIG-1433 > Project: Pig > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Pradeep Kamath >Assignee: Pradeep Kamath > Fix For: 0.8.0 > > Attachments: PIG-1433.patch > > > pig should create success file if > mapreduce.fileoutputcommitter.marksuccessfuljobs is true -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-282) Custom Partitioner
[ https://issues.apache.org/jira/browse/PIG-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875554#action_12875554 ] Hadoop QA commented on PIG-282: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446172/CustomPartitionerFinale.patch against trunk revision 951229. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 380 release audit warnings (more than the trunk's current 379 warnings). -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/320/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/320/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/320/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/320/console This message is automatically generated. > Custom Partitioner > -- > > Key: PIG-282 > URL: https://issues.apache.org/jira/browse/PIG-282 > Project: Pig > Issue Type: New Feature >Affects Versions: 0.7.0 >Reporter: Amir Youssefi >Assignee: Aniket Mokashi >Priority: Minor > Fix For: 0.8.0 > > Attachments: CustomPartitioner.patch, CustomPartitionerFinale.patch, > CustomPartitionerTest.patch > > > By adding custom partitioner we can give control over which output partition > a key (/value) goes to. We can add keywords to language e.g. > PARTITION BY UDF(...) > or a similar syntax. UDF returns a number between 0 and n-1 where n is number > of output partitions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1249) Safe-guards against misconfigured Pig scripts without PARALLEL keyword
[ https://issues.apache.org/jira/browse/PIG-1249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875551#action_12875551 ] Hadoop QA commented on PIG-1249: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446173/PIG-1249-4.patch against trunk revision 951229. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/329/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/329/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/329/console This message is automatically generated. > Safe-guards against misconfigured Pig scripts without PARALLEL keyword > -- > > Key: PIG-1249 > URL: https://issues.apache.org/jira/browse/PIG-1249 > Project: Pig > Issue Type: Improvement >Affects Versions: 0.8.0 >Reporter: Arun C Murthy >Assignee: Jeff Zhang >Priority: Critical > Fix For: 0.8.0 > > Attachments: PIG-1249-4.patch, PIG-1249.patch, PIG_1249_2.patch, > PIG_1249_3.patch > > > It would be *very* useful for Pig to have safe-guards against naive scripts > which process a *lot* of data without the use of PARALLEL keyword. > We've seen a fair number of instances where naive users process huge > data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1432) [zebra] There are some debuging info output to STDOUT in PIG's TableStorer call path
[ https://issues.apache.org/jira/browse/PIG-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874394#action_12874394 ] Hadoop QA commented on PIG-1432: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446078/PIG-1432.patch against trunk revision 949057. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/19/console This message is automatically generated. > [zebra] There are some debuging info output to STDOUT in PIG's TableStorer > call path > > > Key: PIG-1432 > URL: https://issues.apache.org/jira/browse/PIG-1432 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Yan Zhou >Assignee: Yan Zhou >Priority: Trivial > Fix For: 0.7.0 > > Attachments: PIG-1432.patch > > > Users redirecting STDOUT to disk file got "disk full" errors. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-282) Custom Partitioner
[ https://issues.apache.org/jira/browse/PIG-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874393#action_12874393 ] Hadoop QA commented on PIG-282: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446067/CustomPartitionerTest.patch against trunk revision 949057. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 386 release audit warnings (more than the trunk's current 385 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/18/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/18/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/18/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/18/console This message is automatically generated. > Custom Partitioner > -- > > Key: PIG-282 > URL: https://issues.apache.org/jira/browse/PIG-282 > Project: Pig > Issue Type: New Feature >Affects Versions: 0.7.0 >Reporter: Amir Youssefi >Assignee: Aniket Mokashi >Priority: Minor > Fix For: 0.8.0 > > Attachments: CustomPartitioner.patch, CustomPartitionerTest.patch > > > By adding custom partitioner we can give control over which output partition > a key (/value) goes to. We can add keywords to language e.g. > PARTITION BY UDF(...) > or a similar syntax. UDF returns a number between 0 and n-1 where n is number > of output partitions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1428) Add getPigStatusReporter() to PigHadoopLogger
[ https://issues.apache.org/jira/browse/PIG-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873901#action_12873901 ] Hadoop QA commented on PIG-1428: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445985/PIG-1428.patch against trunk revision 949057. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. -1 release audit. The applied patch generated 386 release audit warnings (more than the trunk's current 385 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/17/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/17/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/17/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/17/console This message is automatically generated. > Add getPigStatusReporter() to PigHadoopLogger > - > > Key: PIG-1428 > URL: https://issues.apache.org/jira/browse/PIG-1428 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Ashutosh Chauhan >Assignee: Dmitriy V. Ryaboy > Fix For: 0.8.0 > > Attachments: PIG-1428.patch > > > Without this getter method, its not possible to get counters, report progress > etc. from UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1333) API interface to Pig
[
https://issues.apache.org/jira/browse/PIG-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873145#action_12873145
]
Hadoop QA commented on PIG-1333:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12445786/PIG-1333.patch
against trunk revision 949057.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 99 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 147 javac compiler warnings (more
than the trunk's current 139 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 395 release audit warnings
(more than the trunk's current 385 warnings).
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/16/testReport/
Release audit warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/16/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/16/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/16/console
This message is automatically generated.
> API interface to Pig
>
>
> Key: PIG-1333
> URL: https://issues.apache.org/jira/browse/PIG-1333
> Project: Pig
> Issue Type: Improvement
>Reporter: Olga Natkovich
>Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1333.patch
>
>
> It would be nice to make Pig more friendly for applications like workflow
> that would be executing pig scripts on user behalf.
> Currently, they would have to use pig command line to execute the code;
> however, this has limitation on the kind of output that would be delivered.
> For instance, it is hard to produce error information that is easy to use
> programatically or collect statistics.
> The proposal is to create a class that mimics the behavior of the Main but
> gives users a status object back. The the main code of pig would look
> somethig like:
> public static void main(String args[])
> {
> PigStatus ps = PigMain.exec(args);
> exit (PigStatus.rc);
> }
> We need to define the following:
> - Content of PigStatus. It should at least include
>* return code
>* error string
>* exception
>* statistics
> - A way to propagate the status class through pig code
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-283) Allow to set arbitrary jobconf key-value pairs inside pig program
[ https://issues.apache.org/jira/browse/PIG-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872969#action_12872969 ] Hadoop QA commented on PIG-283: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445710/pig-282.patch against trunk revision 949057. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/15/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/15/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/15/console This message is automatically generated. > Allow to set arbitrary jobconf key-value pairs inside pig program > - > > Key: PIG-283 > URL: https://issues.apache.org/jira/browse/PIG-283 > Project: Pig > Issue Type: New Feature > Components: grunt >Affects Versions: 0.7.0 >Reporter: Christian Kunz >Assignee: Ashutosh Chauhan > Fix For: 0.8.0 > > Attachments: pig-282.patch > > > It would be useful to be able to set arbitrary JobConf key-value pairs inside > a pig program (e.g. in front of a COGROUP statement). > I wonder whether the simplest way to add this feature is by expanding the > 'set' command functionality. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1333) API interface to Pig
[
https://issues.apache.org/jira/browse/PIG-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872894#action_12872894
]
Hadoop QA commented on PIG-1333:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12445727/PIG-1333.patch
against trunk revision 949057.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 99 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/14/console
This message is automatically generated.
> API interface to Pig
>
>
> Key: PIG-1333
> URL: https://issues.apache.org/jira/browse/PIG-1333
> Project: Pig
> Issue Type: Improvement
>Reporter: Olga Natkovich
>Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1333.patch
>
>
> It would be nice to make Pig more friendly for applications like workflow
> that would be executing pig scripts on user behalf.
> Currently, they would have to use pig command line to execute the code;
> however, this has limitation on the kind of output that would be delivered.
> For instance, it is hard to produce error information that is easy to use
> programatically or collect statistics.
> The proposal is to create a class that mimics the behavior of the Main but
> gives users a status object back. The the main code of pig would look
> somethig like:
> public static void main(String args[])
> {
> PigStatus ps = PigMain.exec(args);
> exit (PigStatus.rc);
> }
> We need to define the following:
> - Content of PigStatus. It should at least include
>* return code
>* error string
>* exception
>* statistics
> - A way to propagate the status class through pig code
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
