[jira] Commented: (PIG-1461) support union operation that merges based on column names
[
https://issues.apache.org/jira/browse/PIG-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12895212#action_12895212
]
Hadoop QA commented on PIG-1461:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12451175/PIG-1461.1.patch
against trunk revision 981984.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 407 release audit warnings
(more than the trunk's current 405 warnings).
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/372/testReport/
Release audit warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/372/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/372/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/372/console
This message is automatically generated.
support union operation that merges based on column names
-
Key: PIG-1461
URL: https://issues.apache.org/jira/browse/PIG-1461
Project: Pig
Issue Type: New Feature
Components: impl
Affects Versions: 0.8.0
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Fix For: 0.8.0
Attachments: PIG-1461.1.patch, PIG-1461.patch
When the data has schema, it often makes sense to union on column names in
schema rather than the position of the columns.
The behavior of existing union operator should remain backward compatible .
This feature can be supported using either a new operator or extending union
to support 'using' clause . I am thinking of having a new operator called
either unionschema or merge . Does anybody have any other suggestions for the
syntax ?
example -
L1 = load 'x' as (a,b);
L2 = load 'y' as (b,c);
U = unionschema L1, L2;
describe U;
U: {a:bytearray, b:byetarray, c:bytearray}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1527) No need to deserialize UDFContext on the client side
[ https://issues.apache.org/jira/browse/PIG-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12895310#action_12895310 ] Hadoop QA commented on PIG-1527: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12451181/PIG-1527.patch against trunk revision 981984. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 406 release audit warnings (more than the trunk's current 405 warnings). +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/373/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/373/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/373/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/373/console This message is automatically generated. No need to deserialize UDFContext on the client side Key: PIG-1527 URL: https://issues.apache.org/jira/browse/PIG-1527 Project: Pig Issue Type: Bug Affects Versions: 0.7.0 Reporter: Richard Ding Assignee: Richard Ding Fix For: 0.8.0 Attachments: PIG-1527.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1199) help includes obsolete options
[ https://issues.apache.org/jira/browse/PIG-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12895460#action_12895460 ] Hadoop QA commented on PIG-1199: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12451182/PIG-1199.patch against trunk revision 981984. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 406 release audit warnings (more than the trunk's current 405 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/374/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/374/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/374/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/374/console This message is automatically generated. help includes obsolete options -- Key: PIG-1199 URL: https://issues.apache.org/jira/browse/PIG-1199 Project: Pig Issue Type: Bug Affects Versions: 0.6.0 Reporter: Olga Natkovich Assignee: Olga Natkovich Fix For: 0.8.0 Attachments: PIG-1199.patch This is confusing to users -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too complex and hard to work with
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12895463#action_12895463 ] Hadoop QA commented on PIG-1178: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12451203/PIG-1178-5.patch against trunk revision 982423. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 91 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/375/console This message is automatically generated. LogicalPlan and Optimizer are too complex and hard to work with --- Key: PIG-1178 URL: https://issues.apache.org/jira/browse/PIG-1178 Project: Pig Issue Type: Improvement Reporter: Alan Gates Assignee: Daniel Dai Fix For: 0.8.0 Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, PIG-1178-5.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer. See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1434) Allow casting relations to scalars
[ https://issues.apache.org/jira/browse/PIG-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12894837#action_12894837 ] Hadoop QA commented on PIG-1434: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12451096/ScalarImplFinale1.patch against trunk revision 980930. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. -1 release audit. The applied patch generated 409 release audit warnings (more than the trunk's current 403 warnings). +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/368/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/368/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/368/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/368/console This message is automatically generated. Allow casting relations to scalars -- Key: PIG-1434 URL: https://issues.apache.org/jira/browse/PIG-1434 Project: Pig Issue Type: Improvement Reporter: Olga Natkovich Assignee: Aniket Mokashi Fix For: 0.8.0 Attachments: scalarImpl.patch, ScalarImpl1.patch, ScalarImpl5.patch, ScalarImplFinale.patch, ScalarImplFinale1.patch This jira is to implement a simplified version of the functionality described in https://issues.apache.org/jira/browse/PIG-801. The proposal is to allow casting relations to scalar types in foreach. Example: A = load 'data' as (x, y, z); B = group A all; C = foreach B generate COUNT(A); . X = Y = foreach X generate $1/(long) C; Couple of additional comments: (1) You can only cast relations including a single value or an error will be reported (2) Name resolution is needed since relation X might have field named C in which case that field takes precedence. (3) Y will look for C closest to it. Implementation thoughts: The idea is to store C into a file and then convert it into scalar via a UDF. I believe we already have a UDF that Ben Reed contributed for this purpose. Most of the work would be to update the logical plan to (1) Store C (2) convert the cast to the UDF -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1526) HiveColumnarLoader Partitioning Support
[ https://issues.apache.org/jira/browse/PIG-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12894933#action_12894933 ] Hadoop QA commented on PIG-1526: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12451115/PIG-1526-2.patch against trunk revision 980930. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/369/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/369/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/369/console This message is automatically generated. HiveColumnarLoader Partitioning Support --- Key: PIG-1526 URL: https://issues.apache.org/jira/browse/PIG-1526 Project: Pig Issue Type: Improvement Affects Versions: 0.8.0 Reporter: Gerrit Jansen van Vuuren Assignee: Gerrit Jansen van Vuuren Priority: Minor Fix For: 0.8.0 Attachments: PIG-1526-2.patch, PIG-1526.patch I've made allot improvements on the HiveColumnarLoader: - Added support for LoadMetadata and data path Partitioning - Improved and simplefied column loading Data Path Partitioning: Hive stores partitions as folders like to /mytable/partition1=[value]/partition2=[value]. That is the table mytable contains 2 partitions [partition1, partition2]. The HiveColumnarLoader will scan the inputpath /mytable and add to the PigSchema the columns partition2 and partition2. These columns can then be used in filtering. For example: We've got year,month,day,hour partitions in our data uploads. So a table might look like mytable/year=2010/month=02/day=01. Loading with the HiveColumnarLoader allows our pig scripts do filter by date using the standard pig Filter operator. I've added 2 classes for this: - PathPartitioner - PathPartitionHelper These classes are not hive dependent and could be used by any other loader that wants to support partitioning and helps with implementing the LoadMetadata interface. For this reason I though it best to put it into the package org.apache.pig.piggybank.storage.partition. What would be nice is in the future have the PigStorage also use these 2 classes to provide automatic path partitioning support. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1461) support union operation that merges based on column names
[
https://issues.apache.org/jira/browse/PIG-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12895067#action_12895067
]
Hadoop QA commented on PIG-1461:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12451133/PIG-1461.patch
against trunk revision 980930.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 405 release audit warnings
(more than the trunk's current 403 warnings).
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/370/testReport/
Release audit warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/370/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/370/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/370/console
This message is automatically generated.
support union operation that merges based on column names
-
Key: PIG-1461
URL: https://issues.apache.org/jira/browse/PIG-1461
Project: Pig
Issue Type: New Feature
Components: impl
Affects Versions: 0.8.0
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Fix For: 0.8.0
Attachments: PIG-1461.patch
When the data has schema, it often makes sense to union on column names in
schema rather than the position of the columns.
The behavior of existing union operator should remain backward compatible .
This feature can be supported using either a new operator or extending union
to support 'using' clause . I am thinking of having a new operator called
either unionschema or merge . Does anybody have any other suggestions for the
syntax ?
example -
L1 = load 'x' as (a,b);
L2 = load 'y' as (b,c);
U = unionschema L1, L2;
describe U;
U: {a:bytearray, b:byetarray, c:bytearray}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1533) Compression codec should be a per-store property
[
https://issues.apache.org/jira/browse/PIG-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12895139#action_12895139
]
Hadoop QA commented on PIG-1533:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12451140/PIG-1533.patch
against trunk revision 981984.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/371/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/371/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/371/console
This message is automatically generated.
Compression codec should be a per-store property
Key: PIG-1533
URL: https://issues.apache.org/jira/browse/PIG-1533
Project: Pig
Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Richard Ding
Assignee: Richard Ding
Fix For: 0.8.0
Attachments: PIG-1533.patch
The following script with multi-query optimization
{code}
a = load 'input';
store a into 'outout.bz2';
store a into 'outout2'
{code}
generates two .bz files, while only one of them should be compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1521) explain plan does not show correct Physical operator in MR plan when POSortedDistinct, POPackageLite are used
[ https://issues.apache.org/jira/browse/PIG-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12893661#action_12893661 ] Hadoop QA commented on PIG-1521: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450784/PIG-1521.patch against trunk revision 980276. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 409 release audit warnings (more than the trunk's current 406 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/385/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/385/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/385/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/385/console This message is automatically generated. explain plan does not show correct Physical operator in MR plan when POSortedDistinct, POPackageLite are used - Key: PIG-1521 URL: https://issues.apache.org/jira/browse/PIG-1521 Project: Pig Issue Type: Bug Reporter: Thejas M Nair Assignee: Thejas M Nair Priority: Minor Fix For: 0.8.0 Attachments: PIG-1521.patch MR plan in explain shows PODistinct and Package (POPackage), when the operators POSortedDistinct and PackageLite (POPackageLite) are actually being used. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1452) to remove hadoop20.jar from lib and use hadoop from the apache maven repo.
[ https://issues.apache.org/jira/browse/PIG-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12893793#action_12893793 ] Hadoop QA commented on PIG-1452: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450812/PIG-1452V2.PATCH against trunk revision 980276. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/365/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/365/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/365/console This message is automatically generated. to remove hadoop20.jar from lib and use hadoop from the apache maven repo. -- Key: PIG-1452 URL: https://issues.apache.org/jira/browse/PIG-1452 Project: Pig Issue Type: Improvement Components: build Affects Versions: 0.8.0 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Fix For: 0.8.0 Attachments: PIG-1452.PATCH, PIG-1452V2.PATCH pig use ivy for dependency management. But still it uses hadoop20.jar from the lib folder. Now that we have the hadoop-0.20.2 artifacts available in the maven repo, pig should leverage ivy for resolving/retrieving hadoop artifacts. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1434) Allow casting relations to scalars
[ https://issues.apache.org/jira/browse/PIG-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12893939#action_12893939 ] Hadoop QA commented on PIG-1434: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450872/ScalarImplFinale.patch against trunk revision 980276. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. -1 javac. The applied patch generated 146 javac compiler warnings (more than the trunk's current 145 warnings). -1 findbugs. The patch appears to introduce 5 new Findbugs warnings. -1 release audit. The applied patch generated 406 release audit warnings (more than the trunk's current 400 warnings). +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/366/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/366/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/366/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/366/console This message is automatically generated. Allow casting relations to scalars -- Key: PIG-1434 URL: https://issues.apache.org/jira/browse/PIG-1434 Project: Pig Issue Type: Improvement Reporter: Olga Natkovich Assignee: Aniket Mokashi Fix For: 0.8.0 Attachments: scalarImpl.patch, ScalarImpl1.patch, ScalarImpl5.patch, ScalarImplFinale.patch This jira is to implement a simplified version of the functionality described in https://issues.apache.org/jira/browse/PIG-801. The proposal is to allow casting relations to scalar types in foreach. Example: A = load 'data' as (x, y, z); B = group A all; C = foreach B generate COUNT(A); . X = Y = foreach X generate $1/(long) C; Couple of additional comments: (1) You can only cast relations including a single value or an error will be reported (2) Name resolution is needed since relation X might have field named C in which case that field takes precedence. (3) Y will look for C closest to it. Implementation thoughts: The idea is to store C into a file and then convert it into scalar via a UDF. I believe we already have a UDF that Ben Reed contributed for this purpose. Most of the work would be to update the logical plan to (1) Store C (2) convert the cast to the UDF -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1520) Remove Owl from Pig contrib
[ https://issues.apache.org/jira/browse/PIG-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12893149#action_12893149 ] Hadoop QA commented on PIG-1520: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450615/PIG-1520.patch against trunk revision 979918. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 345 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/382/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/382/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/382/console This message is automatically generated. Remove Owl from Pig contrib --- Key: PIG-1520 URL: https://issues.apache.org/jira/browse/PIG-1520 Project: Pig Issue Type: Task Components: impl Affects Versions: 0.8.0 Reporter: Alan Gates Assignee: Alan Gates Fix For: 0.8.0 Attachments: PIG-1520.patch Yahoo has transitioned work on Owl to Howl (which will not be a Pig contrib project). Since no one else is working on Owl and there will be no one to support it we should remove it from our contrib before releasing 0.8. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1513) Pig doesn't handle empty input directory
[
https://issues.apache.org/jira/browse/PIG-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12893433#action_12893433
]
Hadoop QA commented on PIG-1513:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12450727/PIG-1513.patch
against trunk revision 979918.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/383/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/383/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/383/console
This message is automatically generated.
Pig doesn't handle empty input directory
Key: PIG-1513
URL: https://issues.apache.org/jira/browse/PIG-1513
Project: Pig
Issue Type: Bug
Reporter: Richard Ding
Assignee: Richard Ding
Fix For: 0.8.0
Attachments: PIG-1513.patch
The following script
{code}
A = load 'input';
B = load 'emptydir';
C = join B by $0, A by $0 using 'skewed';
store C into 'output';
{code}
fails with ERROR: java.lang.RuntimeException: Empty samples file';
In this case, the sample job has 0 maps. Pig doesn't expect this and fails .
For merge join the script
The merge join script
{code}
A = load 'input';
B = load 'emptydir';
C = join A by $0, B by $0 using 'merge';
store C into 'output';
{code}
the sample job again has 0 maps and the script fails with ERROR 2176:
Error processing right input during merge join.
But if we change the join order:
{code}
A = load 'input';
B = load 'emptydir';
C = join B by $0, A by $0 using 'merge';
store C into 'output';
{code}
The second job (merge) now has 0 maps and 0 reduces. And it generates an
empty 'output' directory.
Order by on empty directory works fine and generates empty part files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too complex and hard to work with
[ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12892658#action_12892658 ] Hadoop QA commented on PIG-1178: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450250/PIG-1178-4.patch against trunk revision 979362. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 48 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 446 release audit warnings (more than the trunk's current 398 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/355/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/355/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/355/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/355/console This message is automatically generated. LogicalPlan and Optimizer are too complex and hard to work with --- Key: PIG-1178 URL: https://issues.apache.org/jira/browse/PIG-1178 Project: Pig Issue Type: Improvement Reporter: Alan Gates Assignee: Daniel Dai Fix For: 0.8.0 Attachments: expressions-2.patch, expressions.patch, lp.patch, lp.patch, PIG-1178-4.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch, pig_1178_2.patch, pig_1178_3.2.patch, pig_1178_3.3.patch, pig_1178_3.4.patch, pig_1178_3.patch The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer. See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-348) -j command line option doesn't work
[ https://issues.apache.org/jira/browse/PIG-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12892663#action_12892663 ] Hadoop QA commented on PIG-348: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450362/PIG-348.path against trunk revision 979503. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/357/console This message is automatically generated. -j command line option doesn't work --- Key: PIG-348 URL: https://issues.apache.org/jira/browse/PIG-348 Project: Pig Issue Type: Improvement Components: documentation Reporter: Amir Youssefi Assignee: Richard Ding Fix For: 0.8.0 Attachments: PIG-348.path According to: $ pig --help ... -j, -jar jarfile load jarfile ... yet $pig -j my.jar doesn't work in place of: register my.jar in Pig script. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1512) PlanPrinter does not print LOJoin operator in the new logical optimization framework
[ https://issues.apache.org/jira/browse/PIG-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12892740#action_12892740 ] Hadoop QA commented on PIG-1512: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450145/printJoin.patch against trunk revision 979503. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 407 release audit warnings (more than the trunk's current 405 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/380/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/380/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/380/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/380/console This message is automatically generated. PlanPrinter does not print LOJoin operator in the new logical optimization framework Key: PIG-1512 URL: https://issues.apache.org/jira/browse/PIG-1512 Project: Pig Issue Type: Bug Affects Versions: 0.8.0 Reporter: Swati Jain Assignee: Swati Jain Fix For: 0.8.0 Attachments: printJoin.patch PlanPrinter does not print LOJoin relational operator. As such, the LOJoin operator would not get printed when we do an explain. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1249) Safe-guards against misconfigured Pig scripts without PARALLEL keyword
[ https://issues.apache.org/jira/browse/PIG-1249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12892873#action_12892873 ] Hadoop QA commented on PIG-1249: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450579/PIG-1249_5.patch against trunk revision 979503. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/359/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/359/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/359/console This message is automatically generated. Safe-guards against misconfigured Pig scripts without PARALLEL keyword -- Key: PIG-1249 URL: https://issues.apache.org/jira/browse/PIG-1249 Project: Pig Issue Type: Improvement Affects Versions: 0.8.0 Reporter: Arun C Murthy Assignee: Jeff Zhang Priority: Critical Fix For: 0.8.0 Attachments: PIG-1249-4.patch, PIG-1249.patch, PIG-1249_5.patch, PIG_1249_2.patch, PIG_1249_3.patch It would be *very* useful for Pig to have safe-guards against naive scripts which process a *lot* of data without the use of PARALLEL keyword. We've seen a fair number of instances where naive users process huge data-sets (10TB) with badly mis-configured #reduces e.g. 1 reduce. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1229) allow pig to write output into a JDBC db
[ https://issues.apache.org/jira/browse/PIG-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12892999#action_12892999 ] Hadoop QA commented on PIG-1229: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450586/jira-1229-final.patch against trunk revision 979781. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/360/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/360/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/360/console This message is automatically generated. allow pig to write output into a JDBC db Key: PIG-1229 URL: https://issues.apache.org/jira/browse/PIG-1229 Project: Pig Issue Type: New Feature Components: impl Reporter: Ian Holsman Assignee: Ankur Priority: Minor Fix For: 0.8.0 Attachments: jira-1229-final.patch, jira-1229-v2.patch, jira-1229-v3.patch, pig-1229.2.patch, pig-1229.patch UDF to store data into a DB -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1500) guava.jar should be removed from the lib folder
[ https://issues.apache.org/jira/browse/PIG-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12892409#action_12892409 ] Hadoop QA commented on PIG-1500: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450378/guava.jar.r06.patch against trunk revision 979362. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/376/console This message is automatically generated. guava.jar should be removed from the lib folder --- Key: PIG-1500 URL: https://issues.apache.org/jira/browse/PIG-1500 Project: Pig Issue Type: Bug Components: build Reporter: Giridharan Kesavan Assignee: niraj rai Fix For: 0.8.0 Attachments: guava.jar.r06.patch, removeGuavaJar.patch guava jar is available in the maven repository but still its is checked into the pig trunk's lib folder. I ve checked the availability of guava jar in the maven repository. http://mvnrepository.com/artifact/com.google.guava/guava -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1508) Make 'docs' target (forrest) work with Java 1.6
[ https://issues.apache.org/jira/browse/PIG-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12890596#action_12890596 ] Hadoop QA commented on PIG-1508: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449977/PIG-1508.patch.txt against trunk revision 965559. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/349/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/349/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/349/console This message is automatically generated. Make 'docs' target (forrest) work with Java 1.6 --- Key: PIG-1508 URL: https://issues.apache.org/jira/browse/PIG-1508 Project: Pig Issue Type: Bug Components: documentation Affects Versions: 0.7.0 Reporter: Carl Steinbach Attachments: PIG-1508.patch.txt FOR-984 covers the very inconvenient fact that Forrest 0.8 does not work with Java 1.6 The same ticket also suggests a workaround: disabling sitemap and stylesheet validation by setting the forrest.validate.sitemap and forrest.validate.stylesheets properties to false. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1507) Full outer join fails while doing a filter on joined data
[
https://issues.apache.org/jira/browse/PIG-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12890521#action_12890521
]
Hadoop QA commented on PIG-1507:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12449962/PIG-1507-1.patch
against trunk revision 965559.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/348/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/348/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/348/console
This message is automatically generated.
Full outer join fails while doing a filter on joined data
-
Key: PIG-1507
URL: https://issues.apache.org/jira/browse/PIG-1507
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.8.0
Reporter: Daniel Dai
Assignee: Daniel Dai
Fix For: 0.8.0
Attachments: PIG-1507-1.patch
The following script produce wrong result:
test1.dat:
1
2
3
test2.dat:
1
2
pig script:
{code}
a = LOAD 'test1.dat' USING PigStorage() AS (d1:int);
b = LOAD 'test2.dat' USING PigStorage() AS (d2:int);
c = JOIN a BY d1 FULL OUTER, b BY d2;
d = FILTER c BY d2 IS NULL;
STORE d INTO 'test.out' USING PigStorage();
{code}
expected:
3
We get:
1
2
3
This is because we erroneously push the filter before full outer join.
Similar issue is addressed in
[PIG-1289|https://issues.apache.org/jira/browse/PIG-1289], but we only fix
left/right outer join.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1505) support jars and scripts in dfs
[ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12889421#action_12889421 ] Hadoop QA commented on PIG-1505: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449736/pig-jars-and-scripts-from-dfs-trunk.patch against trunk revision 964182. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/345/console This message is automatically generated. support jars and scripts in dfs --- Key: PIG-1505 URL: https://issues.apache.org/jira/browse/PIG-1505 Project: Pig Issue Type: Improvement Reporter: Andrew Hitchcock Attachments: pig-jars-and-scripts-from-dfs-trunk.patch Pig can't operate on files stored in Amazon S3. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1492) DefaultTuple and DefaultMemory understimate their memory footprint
[ https://issues.apache.org/jira/browse/PIG-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12888952#action_12888952 ] Hadoop QA commented on PIG-1492: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449531/PIG-1492.1.patch against trunk revision 964182. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/370/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/370/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/370/console This message is automatically generated. DefaultTuple and DefaultMemory understimate their memory footprint -- Key: PIG-1492 URL: https://issues.apache.org/jira/browse/PIG-1492 Project: Pig Issue Type: Bug Affects Versions: 0.8.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.8.0 Attachments: PIG-1492.1.patch There are several places where we highly underestimate the memory footprint . For example, for map datatypes, we don't account for the per entry cost for the map container data structures. The estimated size of a tuple having map with 100 integer key-value entries , as per current version of code is 3260 bytes, while what is observed is around 6775 bytes . To verify the memory footprint, i checked free memory before and after creating multiple instances of the object , using code on the lines of http://www.javaspecialists.eu/archive/Issue029.html . In PIG-1443 similar change was done to fix this for CHARARRAY . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1495) Add -q command line option to set queue name for Pig jobs from command line
[ https://issues.apache.org/jira/browse/PIG-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12888279#action_12888279 ] Hadoop QA commented on PIG-1495: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449293/set_queue.patch against trunk revision 963830. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/368/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/368/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/368/console This message is automatically generated. Add -q command line option to set queue name for Pig jobs from command line --- Key: PIG-1495 URL: https://issues.apache.org/jira/browse/PIG-1495 Project: Pig Issue Type: New Feature Components: impl Affects Versions: 0.7.0 Reporter: Russell Jurney Fix For: 0.8.0 Attachments: set_queue.patch rjurney$ pig -q default This sets the mapred.job.queue.name property in the execution engine from the pig properties for MAPRED type jobs. Patch attached. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1435) make sure dependent jobs fail when a jon in multiquery fails
[ https://issues.apache.org/jira/browse/PIG-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12888692#action_12888692 ] Hadoop QA commented on PIG-1435: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449486/depJobs.patch against trunk revision 964182. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 405 release audit warnings (more than the trunk's current 404 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/369/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/369/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/369/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/369/console This message is automatically generated. make sure dependent jobs fail when a jon in multiquery fails Key: PIG-1435 URL: https://issues.apache.org/jira/browse/PIG-1435 Project: Pig Issue Type: Bug Reporter: Olga Natkovich Assignee: niraj rai Fix For: 0.8.0 Attachments: depJobs.patch Currently if one of the MQ jobs fails, Pig tries to run all remainin jobs. As the result, if data was partially generated by the failed job, you might get incorrect results from dependent jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1493) Column Pruner throw exception inconsistent pruning
[
https://issues.apache.org/jira/browse/PIG-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12888063#action_12888063
]
Hadoop QA commented on PIG-1493:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12449203/PIG-1493-1.patch
against trunk revision 963504.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/367/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/367/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/367/console
This message is automatically generated.
Column Pruner throw exception inconsistent pruning
Key: PIG-1493
URL: https://issues.apache.org/jira/browse/PIG-1493
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.7.0
Reporter: Daniel Dai
Assignee: Daniel Dai
Fix For: 0.7.0, 0.8.0
Attachments: PIG-1493-1.patch
The following script fail:
{code}
a = load '1.txt' as (a0:chararray, a1:chararray, a2);
b = foreach a generate CONCAT(a0,a1) as b0, a0, a2;
c = foreach b generate a0, a2;
dump c;
{code}
Error message:
ERROR 2185: Column $0 of (Name: b: ForEach 1-50 Operator Key: 1-50)
inconsistent pruning
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
open iterator for alias c
at org.apache.pig.PigServer.openIterator(PigServer.java:698)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:595)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:291)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:162)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:138)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:451)
at org.apache.pig.Main.main(Main.java:103)
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002:
Unable to store alias c
at org.apache.pig.PigServer.storeEx(PigServer.java:804)
at org.apache.pig.PigServer.store(PigServer.java:760)
at org.apache.pig.PigServer.openIterator(PigServer.java:680)
... 7 more
Caused by: org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2212:
Unable to prune plan
at
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns.prune(PruneColumns.java:826)
at
org.apache.pig.impl.logicalLayer.optimizer.LogicalOptimizer.optimize(LogicalOptimizer.java:240)
at org.apache.pig.PigServer.compileLp(PigServer.java:1180)
at org.apache.pig.PigServer.storeEx(PigServer.java:799)
... 9 more
Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 2188: Cannot
prune columns for (Name: b: ForEach 1-50 Operator Key: 1-50)
at
org.apache.pig.impl.logicalLayer.ColumnPruner.prune(ColumnPruner.java:177)
at
org.apache.pig.impl.logicalLayer.ColumnPruner.visit(ColumnPruner.java:202)
at
org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:132)
at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:47)
at
org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:69)
at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
at
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns.prune(PruneColumns.java:821)
... 12 more
Caused by: org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2185:
Column $0 of (Name: b: ForEach 1-50 Operator Key: 1-50) inconsistent pruning
at
org.apache.pig.impl.logicalLayer.ColumnPruner.prune(ColumnPruner.java:148)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-928) UDFs in scripting languages
[ https://issues.apache.org/jira/browse/PIG-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12888068#action_12888068 ] Hadoop QA commented on PIG-928: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449134/RegisterPythonUDFFinale5.patch against trunk revision 963504. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 145 javac compiler warnings (more than the trunk's current 144 warnings). -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/344/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/344/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/344/console This message is automatically generated. UDFs in scripting languages --- Key: PIG-928 URL: https://issues.apache.org/jira/browse/PIG-928 Project: Pig Issue Type: New Feature Reporter: Alan Gates Assignee: Aniket Mokashi Fix For: 0.8.0 Attachments: calltrace.png, package.zip, PIG-928.patch, pig-greek.tgz, pig.scripting.patch.arnab, pyg.tgz, RegisterPythonUDF3.patch, RegisterPythonUDF4.patch, RegisterPythonUDF_Final.patch, RegisterPythonUDFFinale.patch, RegisterPythonUDFFinale3.patch, RegisterPythonUDFFinale4.patch, RegisterPythonUDFFinale5.patch, RegisterScriptUDFDefineParse.patch, scripting.tgz, scripting.tgz, test.zip It should be possible to write UDFs in scripting languages such as python, ruby, etc. This frees users from needing to compile Java, generate a jar, etc. It also opens Pig to programmers who prefer scripting languages over Java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1472) Optimize serialization/deserialization between Map and Reduce and between MR jobs
[ https://issues.apache.org/jira/browse/PIG-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886647#action_12886647 ] Hadoop QA commented on PIG-1472: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449033/PIG-1472.3.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 69 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 395 release audit warnings (more than the trunk's current 394 warnings). +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/343/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/343/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/343/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/343/console This message is automatically generated. Optimize serialization/deserialization between Map and Reduce and between MR jobs - Key: PIG-1472 URL: https://issues.apache.org/jira/browse/PIG-1472 Project: Pig Issue Type: Improvement Affects Versions: 0.8.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.8.0 Attachments: PIG-1472.2.patch, PIG-1472.3.patch, PIG-1472.patch In certain types of pig queries most of the execution time is spent in serializing/deserializing (sedes) records between Map and Reduce and between MR jobs. For example, if PigMix queries are modified to specify types for all the fields in the load statement schema, some of the queries (L2,L3,L9, L10 in pigmix v1) that have records with bags and maps being transmitted across map or reduce boundaries run a lot longer (runtime increase of few times has been seen. There are a few optimizations that have shown to improve the performance of sedes in my tests - 1. Use smaller number of bytes to store length of the column . For example if a bytearray is smaller than 255 bytes , a byte can be used to store the length instead of the integer that is currently used. 2. Instead of custom code to do sedes on Strings, use DataOutput.writeUTF and DataInput.readUTF. This reduces the cost of serialization by more than 1/2. Zebra and BinStorage are known to use DefaultTuple sedes functionality. The serialization format that these loaders use cannot change, so after the optimization their format is going to be different from the format used between M/R boundaries. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-928) UDFs in scripting languages
[ https://issues.apache.org/jira/browse/PIG-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886888#action_12886888 ] Hadoop QA commented on PIG-928: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449105/RegisterPythonUDFFinale4.patch against trunk revision 962628. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/365/console This message is automatically generated. UDFs in scripting languages --- Key: PIG-928 URL: https://issues.apache.org/jira/browse/PIG-928 Project: Pig Issue Type: New Feature Reporter: Alan Gates Assignee: Aniket Mokashi Fix For: 0.8.0 Attachments: calltrace.png, package.zip, PIG-928.patch, pig-greek.tgz, pig.scripting.patch.arnab, pyg.tgz, RegisterPythonUDF2.patch, RegisterPythonUDF3.patch, RegisterPythonUDF4.patch, RegisterPythonUDF_Final.patch, RegisterPythonUDFFinale.patch, RegisterPythonUDFFinale3.patch, RegisterPythonUDFFinale4.patch, RegisterScriptUDFDefineParse.patch, scripting.tgz, scripting.tgz, test.zip It should be possible to write UDFs in scripting languages such as python, ruby, etc. This frees users from needing to compile Java, generate a jar, etc. It also opens Pig to programmers who prefer scripting languages over Java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1486) update ant eclipse-files target to include new jar and remove contrib dirs from build path
[ https://issues.apache.org/jira/browse/PIG-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886274#action_12886274 ] Hadoop QA commented on PIG-1486: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448935/PIG-1486.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/341/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/341/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/341/console This message is automatically generated. update ant eclipse-files target to include new jar and remove contrib dirs from build path -- Key: PIG-1486 URL: https://issues.apache.org/jira/browse/PIG-1486 Project: Pig Issue Type: Bug Components: tools Affects Versions: 0.8.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Priority: Minor Fix For: 0.8.0 Attachments: PIG-1486.patch .eclipse.templates/.classpath needs to be updated to address following - 1. There is a new jar that is used by the code - guava-r03.jar 2. The jar ANT_HOME/lib/ant.jar gives an 'unbounded jar' error in eclipse. 3. Removing the contrib projects from class path as discussed in PIG-1390, until all libs necessary for the contribs are included in classpath. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1472) Optimize serialization/deserialization between Map and Reduce and between MR jobs
[ https://issues.apache.org/jira/browse/PIG-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886281#action_12886281 ] Hadoop QA commented on PIG-1472: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448937/PIG-1472.2.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 69 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. -1 javac. The applied patch generated 148 javac compiler warnings (more than the trunk's current 145 warnings). -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. -1 release audit. The applied patch generated 400 release audit warnings (more than the trunk's current 399 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/362/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/362/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/362/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/362/console This message is automatically generated. Optimize serialization/deserialization between Map and Reduce and between MR jobs - Key: PIG-1472 URL: https://issues.apache.org/jira/browse/PIG-1472 Project: Pig Issue Type: Improvement Affects Versions: 0.8.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.8.0 Attachments: PIG-1472.2.patch, PIG-1472.patch In certain types of pig queries most of the execution time is spent in serializing/deserializing (sedes) records between Map and Reduce and between MR jobs. For example, if PigMix queries are modified to specify types for all the fields in the load statement schema, some of the queries (L2,L3,L9, L10 in pigmix v1) that have records with bags and maps being transmitted across map or reduce boundaries run a lot longer (runtime increase of few times has been seen. There are a few optimizations that have shown to improve the performance of sedes in my tests - 1. Use smaller number of bytes to store length of the column . For example if a bytearray is smaller than 255 bytes , a byte can be used to store the length instead of the integer that is currently used. 2. Instead of custom code to do sedes on Strings, use DataOutput.writeUTF and DataInput.readUTF. This reduces the cost of serialization by more than 1/2. Zebra and BinStorage are known to use DefaultTuple sedes functionality. The serialization format that these loaders use cannot change, so after the optimization their format is going to be different from the format used between M/R boundaries. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1484) BinStorage should support comma seperated path
[ https://issues.apache.org/jira/browse/PIG-1484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886538#action_12886538 ] Hadoop QA commented on PIG-1484: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448988/PIG-1484-2.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/363/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/363/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/363/console This message is automatically generated. BinStorage should support comma seperated path -- Key: PIG-1484 URL: https://issues.apache.org/jira/browse/PIG-1484 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.7.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.7.0, 0.8.0 Attachments: PIG-1484-1.patch, PIG-1484-2.patch, PIG-1484-3.patch BinStorage does not take comma seperated path. The following script fail: a = load '1.bin,2.bin' using BinStorage(); dump a; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1484) BinStorage should support comma seperated path
[ https://issues.apache.org/jira/browse/PIG-1484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886591#action_12886591 ] Hadoop QA commented on PIG-1484: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449001/PIG-1484-3.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/342/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/342/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/342/console This message is automatically generated. BinStorage should support comma seperated path -- Key: PIG-1484 URL: https://issues.apache.org/jira/browse/PIG-1484 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.7.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.7.0, 0.8.0 Attachments: PIG-1484-1.patch, PIG-1484-2.patch, PIG-1484-3.patch BinStorage does not take comma seperated path. The following script fail: a = load '1.bin,2.bin' using BinStorage(); dump a; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-928) UDFs in scripting languages
[ https://issues.apache.org/jira/browse/PIG-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886610#action_12886610 ] Hadoop QA commented on PIG-928: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449018/RegisterPythonUDF_Final.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 146 javac compiler warnings (more than the trunk's current 145 warnings). -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/364/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/364/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/364/console This message is automatically generated. UDFs in scripting languages --- Key: PIG-928 URL: https://issues.apache.org/jira/browse/PIG-928 Project: Pig Issue Type: New Feature Reporter: Alan Gates Assignee: Aniket Mokashi Fix For: 0.8.0 Attachments: calltrace.png, package.zip, PIG-928.patch, pig-greek.tgz, pig.scripting.patch.arnab, pyg.tgz, RegisterPythonUDF2.patch, RegisterPythonUDF3.patch, RegisterPythonUDF4.patch, RegisterPythonUDF_Final.patch, RegisterPythonUDFFinale.patch, RegisterPythonUDFFinale3.patch, RegisterScriptUDFDefineParse.patch, scripting.tgz, scripting.tgz, test.zip It should be possible to write UDFs in scripting languages such as python, ruby, etc. This frees users from needing to compile Java, generate a jar, etc. It also opens Pig to programmers who prefer scripting languages over Java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1484) BinStorage should support comma seperated path
[ https://issues.apache.org/jira/browse/PIG-1484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12886175#action_12886175 ] Hadoop QA commented on PIG-1484: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448904/PIG-1484-1.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/361/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/361/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/361/console This message is automatically generated. BinStorage should support comma seperated path -- Key: PIG-1484 URL: https://issues.apache.org/jira/browse/PIG-1484 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.7.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.8.0 Attachments: PIG-1484-1.patch BinStorage does not take comma seperated path. The following script fail: a = load '1.bin,2.bin' using BinStorage(); dump a; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1389) Implement Pig counter to track number of rows for each input files
[ https://issues.apache.org/jira/browse/PIG-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12885804#action_12885804 ] Hadoop QA commented on PIG-1389: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448821/PIG-1389_2.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/360/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/360/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/360/console This message is automatically generated. Implement Pig counter to track number of rows for each input files --- Key: PIG-1389 URL: https://issues.apache.org/jira/browse/PIG-1389 Project: Pig Issue Type: Improvement Affects Versions: 0.7.0 Reporter: Richard Ding Assignee: Richard Ding Fix For: 0.8.0 Attachments: PIG-1389.patch, PIG-1389.patch, PIG-1389_1.patch, PIG-1389_2.patch A MR job generated by Pig not only can have multiple outputs (in the case of multiquery) but also can have multiple inputs (in the case of join or cogroup). In both cases, the existing Hadoop counters (e.g. MAP_INPUT_RECORDS, REDUCE_OUTPUT_RECORDS) can not be used to count the number of records in the given input or output. PIG-1299 addressed the case of multiple outputs. We need to add new counters for jobs with multiple inputs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-928) UDFs in scripting languages
[ https://issues.apache.org/jira/browse/PIG-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12885822#action_12885822 ] Hadoop QA commented on PIG-928: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448831/RegisterPythonUDFFinale3.patch against trunk revision 960062. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. -1 javac. The applied patch generated 146 javac compiler warnings (more than the trunk's current 145 warnings). -1 findbugs. The patch appears to introduce 4 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/340/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/340/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/340/console This message is automatically generated. UDFs in scripting languages --- Key: PIG-928 URL: https://issues.apache.org/jira/browse/PIG-928 Project: Pig Issue Type: New Feature Reporter: Alan Gates Assignee: Aniket Mokashi Fix For: 0.8.0 Attachments: calltrace.png, package.zip, PIG-928.patch, pig-greek.tgz, pig.scripting.patch.arnab, pyg.tgz, RegisterPythonUDF2.patch, RegisterPythonUDF3.patch, RegisterPythonUDF4.patch, RegisterPythonUDFFinale.patch, RegisterPythonUDFFinale3.patch, RegisterScriptUDFDefineParse.patch, scripting.tgz, scripting.tgz, test.zip It should be possible to write UDFs in scripting languages such as python, ruby, etc. This frees users from needing to compile Java, generate a jar, etc. It also opens Pig to programmers who prefer scripting languages over Java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1404) PigUnit - Pig script testing simplified.
[
https://issues.apache.org/jira/browse/PIG-1404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12884901#action_12884901
]
Hadoop QA commented on PIG-1404:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448463/PIG-1404-3-doc.patch
against trunk revision 960062.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 401 release audit warnings
(more than the trunk's current 399 warnings).
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/359/testReport/
Release audit warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/359/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/359/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/359/console
This message is automatically generated.
PigUnit - Pig script testing simplified.
-
Key: PIG-1404
URL: https://issues.apache.org/jira/browse/PIG-1404
Project: Pig
Issue Type: New Feature
Reporter: Romain Rigaux
Assignee: Romain Rigaux
Fix For: 0.8.0
Attachments: commons-lang-2.4.jar, PIG-1404-2.patch,
PIG-1404-3-doc.patch, PIG-1404-3.patch, PIG-1404.patch
The goal is to provide a simple xUnit framework that enables our Pig scripts
to be easily:
- unit tested
- regression tested
- quickly prototyped
No cluster set up is required.
For example:
TestCase
{code}
@Test
public void testTop3Queries() {
String[] args = {
n=3,
};
test = new PigTest(top_queries.pig, args);
String[] input = {
yahoo\t10,
twitter\t7,
facebook\t10,
yahoo\t15,
facebook\t5,
};
String[] output = {
(yahoo,25L),
(facebook,15L),
(twitter,7L),
};
test.assertOutput(data, input, queries_limit, output);
}
{code}
top_queries.pig
{code}
data =
LOAD '$input'
AS (query:CHARARRAY, count:INT);
...
queries_sum =
FOREACH queries_group
GENERATE
group AS query,
SUM(queries.count) AS count;
...
queries_limit = LIMIT queries_ordered $n;
STORE queries_limit INTO '$output';
{code}
They are 3 modes:
* LOCAL (if pigunit.exectype.local properties is present)
* MAPREDUCE (use the cluster specified in the classpath, same as
HADOOP_CONF_DIR)
** automatic mini cluster (is the default and the HADOOP_CONF_DIR to have in
the class path will be: ~/pigtest/conf)
** pointing to an existing cluster (if pigunit.exectype.cluster properties
is present)
For now, it would be nice to see how this idea could be integrated in
Piggybank and if PigParser/PigServer could improve their interfaces in order
to make PigUnit simple.
Other components based on PigUnit could be built later:
- standalone MiniCluster
- notion of workspaces for each test
- standalone utility that reads test configuration and generates a test
report...
It is a first prototype, open to suggestions and can definitely take
advantage of feedbacks.
How to test, in pig_trunk:
{code}
Apply patch
$pig_trunk ant compile-test
$pig_trunk ant
$pig_trunk/contrib/piggybank/java ant test -Dtest.timeout=99
{code}
(it takes 15 min in MAPREDUCE minicluster, tests will need to be split in the
future between 'unit' and 'integration')
Many examples are in:
{code}
contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/pigunit/TestPigTest.java
{code}
When used as a standalone, do not forget commons-lang-2.4.jar and the
HADOOP_CONF_DIR to your cluster in your CLASSPATH.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1478) Add progress notification listener to PigRunner API
[
https://issues.apache.org/jira/browse/PIG-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12884554#action_12884554
]
Hadoop QA commented on PIG-1478:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448532/PIG-1478.patch
against trunk revision 958666.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/336/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/336/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/336/console
This message is automatically generated.
Add progress notification listener to PigRunner API
---
Key: PIG-1478
URL: https://issues.apache.org/jira/browse/PIG-1478
Project: Pig
Issue Type: Improvement
Reporter: Richard Ding
Assignee: Richard Ding
Fix For: 0.8.0
Attachments: PIG-1478.patch
PIG-1333 added PigRunner API to allow Pig users and tools to get a
status/stats object back after executing a Pig script. The new API, however,
is synchronous (blocking). It's known that a Pig script can spawn tens (even
hundreds) MR jobs and take hours to complete. Therefore it'll be nice to give
progress feedback to the callers during the execution.
The proposal is to add an optional parameter to the API:
{code}
public abstract class PigRunner {
public static PigStats run(String[] args, PigProgressNotificationListener
listener) {...}
}
{code}
The new listener is defined as following:
{code}
package org.apache.pig.tools.pigstats;
public interface PigProgressNotificationListener extends
java.util.EventListener {
// just before the launch of MR jobs for the script
public void LaunchStartedNotification(int numJobsToLaunch);
// number of jobs submitted in a batch
public void jobsSubmittedNotification(int numJobsSubmitted);
// a job is started
public void jobStartedNotification(String assignedJobId);
// a job is completed successfully
public void jobFinishedNotification(JobStats jobStats);
// a job is failed
public void jobFailedNotification(JobStats jobStats);
// a user output is completed successfully
public void outputCompletedNotification(OutputStats outputStats);
// updates the progress as percentage
public void progressUpdatedNotification(int progress);
// the script execution is done
public void launchCompletedNotification(int numJobsSucceeded);
}
{code}
Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1478) Add progress notification listener to PigRunner API
[
https://issues.apache.org/jira/browse/PIG-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12884677#action_12884677
]
Hadoop QA commented on PIG-1478:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448532/PIG-1478.patch
against trunk revision 959865.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/358/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/358/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/358/console
This message is automatically generated.
Add progress notification listener to PigRunner API
---
Key: PIG-1478
URL: https://issues.apache.org/jira/browse/PIG-1478
Project: Pig
Issue Type: Improvement
Reporter: Richard Ding
Assignee: Richard Ding
Fix For: 0.8.0
Attachments: PIG-1478.patch
PIG-1333 added PigRunner API to allow Pig users and tools to get a
status/stats object back after executing a Pig script. The new API, however,
is synchronous (blocking). It's known that a Pig script can spawn tens (even
hundreds) MR jobs and take hours to complete. Therefore it'll be nice to give
progress feedback to the callers during the execution.
The proposal is to add an optional parameter to the API:
{code}
public abstract class PigRunner {
public static PigStats run(String[] args, PigProgressNotificationListener
listener) {...}
}
{code}
The new listener is defined as following:
{code}
package org.apache.pig.tools.pigstats;
public interface PigProgressNotificationListener extends
java.util.EventListener {
// just before the launch of MR jobs for the script
public void LaunchStartedNotification(int numJobsToLaunch);
// number of jobs submitted in a batch
public void jobsSubmittedNotification(int numJobsSubmitted);
// a job is started
public void jobStartedNotification(String assignedJobId);
// a job is completed successfully
public void jobFinishedNotification(JobStats jobStats);
// a job is failed
public void jobFailedNotification(JobStats jobStats);
// a user output is completed successfully
public void outputCompletedNotification(OutputStats outputStats);
// updates the progress as percentage
public void progressUpdatedNotification(int progress);
// the script execution is done
public void launchCompletedNotification(int numJobsSucceeded);
}
{code}
Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1478) Add progress notification listener to PigRunner API
[
https://issues.apache.org/jira/browse/PIG-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12884732#action_12884732
]
Hadoop QA commented on PIG-1478:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448532/PIG-1478.patch
against trunk revision 959865.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/337/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/337/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/337/console
This message is automatically generated.
Add progress notification listener to PigRunner API
---
Key: PIG-1478
URL: https://issues.apache.org/jira/browse/PIG-1478
Project: Pig
Issue Type: Improvement
Reporter: Richard Ding
Assignee: Richard Ding
Fix For: 0.8.0
Attachments: PIG-1478.patch
PIG-1333 added PigRunner API to allow Pig users and tools to get a
status/stats object back after executing a Pig script. The new API, however,
is synchronous (blocking). It's known that a Pig script can spawn tens (even
hundreds) MR jobs and take hours to complete. Therefore it'll be nice to give
progress feedback to the callers during the execution.
The proposal is to add an optional parameter to the API:
{code}
public abstract class PigRunner {
public static PigStats run(String[] args, PigProgressNotificationListener
listener) {...}
}
{code}
The new listener is defined as following:
{code}
package org.apache.pig.tools.pigstats;
public interface PigProgressNotificationListener extends
java.util.EventListener {
// just before the launch of MR jobs for the script
public void LaunchStartedNotification(int numJobsToLaunch);
// number of jobs submitted in a batch
public void jobsSubmittedNotification(int numJobsSubmitted);
// a job is started
public void jobStartedNotification(String assignedJobId);
// a job is completed successfully
public void jobFinishedNotification(JobStats jobStats);
// a job is failed
public void jobFailedNotification(JobStats jobStats);
// a user output is completed successfully
public void outputCompletedNotification(OutputStats outputStats);
// updates the progress as percentage
public void progressUpdatedNotification(int progress);
// the script execution is done
public void launchCompletedNotification(int numJobsSucceeded);
}
{code}
Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1449) RegExLoader hangs on lines that don't match the regular expression
[
https://issues.apache.org/jira/browse/PIG-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12884539#action_12884539
]
Hadoop QA commented on PIG-1449:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448516/PIG-1449-RegExLoaderInfiniteLoopFix.patch
against trunk revision 958666.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/357/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/357/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/357/console
This message is automatically generated.
RegExLoader hangs on lines that don't match the regular expression
--
Key: PIG-1449
URL: https://issues.apache.org/jira/browse/PIG-1449
Project: Pig
Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Justin Sanders
Priority: Minor
Attachments: PIG-1449-RegExLoaderInfiniteLoopFix.patch,
RegExLoader.patch
In the 0.7.0 changes to RegExLoader there was a bug introduced where the code
will stay in the while loop if the line isn't matched. Before 0.7.0 these
lines would be skipped if they didn't match the regular expression. The
result is the mapper will not respond and will time out with Task attempt_X
failed to report status for 600 seconds. Killing!.
Here are the steps to recreate the bug:
Create a text file in HDFS with the following lines:
test1
testA
test2
Run the following pig script:
REGISTER /usr/local/pig/contrib/piggybank/java/piggybank.jar;
test = LOAD '/path/to/test.txt' using
org.apache.pig.piggybank.storage.MyRegExLoader('(test\\d)') AS (line);
dump test;
Expected result:
(test1)
(test3)
Actual result:
Job fails to complete after 600 second timeout waiting on the mapper to
complete. The mapper hangs at 33% since it can process the first line but
gets stuck into the while loop on the second line.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1367) [zebra] Map-side Cogroup Test case is needed on 0.7 if the feature is supported in 0.7
[ https://issues.apache.org/jira/browse/PIG-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12884094#action_12884094 ] Hadoop QA commented on PIG-1367: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448416/PIG-1367.patch against trunk revision 958666. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/356/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/356/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/356/console This message is automatically generated. [zebra] Map-side Cogroup Test case is needed on 0.7 if the feature is supported in 0.7 -- Key: PIG-1367 URL: https://issues.apache.org/jira/browse/PIG-1367 Project: Pig Issue Type: New Feature Affects Versions: 0.7.0 Reporter: Yan Zhou Fix For: 0.8.0 Attachments: PIG-1367.patch PIG-1315 has the Zebra support for this feature and the map-side group-by. It also has the test case for map-side COGROUP; while the test case for map-side GROUP-BY is in PIG-1357. However PIG-1315 is committed to the trunk as a whole; but only committed to the 0.7 branch without the map-side group-by test case because PIG has yet to decide if the feature will be in the 0.7 release. This JIRA is created for tracking purpose should the decision to support map-side COGROUP in 0.7 by PIG is made. If not, this should be made invalid eventually. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1389) Implement Pig counter to track number of rows for each input files
[ https://issues.apache.org/jira/browse/PIG-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12883424#action_12883424 ] Hadoop QA commented on PIG-1389: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448259/PIG-1389_1.patch against trunk revision 958666. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/335/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/335/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/335/console This message is automatically generated. Implement Pig counter to track number of rows for each input files --- Key: PIG-1389 URL: https://issues.apache.org/jira/browse/PIG-1389 Project: Pig Issue Type: Improvement Affects Versions: 0.7.0 Reporter: Richard Ding Assignee: Richard Ding Fix For: 0.8.0 Attachments: PIG-1389.patch, PIG-1389.patch, PIG-1389_1.patch A MR job generated by Pig not only can have multiple outputs (in the case of multiquery) but also can have multiple inputs (in the case of join or cogroup). In both cases, the existing Hadoop counters (e.g. MAP_INPUT_RECORDS, REDUCE_OUTPUT_RECORDS) can not be used to count the number of records in the given input or output. PIG-1299 addressed the case of multiple outputs. We need to add new counters for jobs with multiple inputs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1295) Binary comparator for secondary sort
[ https://issues.apache.org/jira/browse/PIG-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12883486#action_12883486 ] Hadoop QA commented on PIG-1295: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448251/PIG-1295_0.6.patch against trunk revision 958666. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 150 javac compiler warnings (more than the trunk's current 145 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 402 release audit warnings (more than the trunk's current 399 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/355/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/355/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/355/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/355/console This message is automatically generated. Binary comparator for secondary sort Key: PIG-1295 URL: https://issues.apache.org/jira/browse/PIG-1295 Project: Pig Issue Type: Improvement Components: impl Affects Versions: 0.7.0 Reporter: Daniel Dai Assignee: Gianmarco De Francisci Morales Fix For: 0.8.0 Attachments: PIG-1295_0.1.patch, PIG-1295_0.2.patch, PIG-1295_0.3.patch, PIG-1295_0.4.patch, PIG-1295_0.5.patch, PIG-1295_0.6.patch When hadoop framework doing the sorting, it will try to use binary version of comparator if available. The benefit of binary comparator is we do not need to instantiate the object before we compare. We see a ~30% speedup after we switch to binary comparator. Currently, Pig use binary comparator in following case: 1. When semantics of order doesn't matter. For example, in distinct, we need to do a sort in order to filter out duplicate values; however, we do not care how comparator sort keys. Groupby also share this character. In this case, we rely on hadoop's default binary comparator 2. Semantics of order matter, but the key is of simple type. In this case, we have implementation for simple types, such as integer, long, float, chararray, databytearray, string However, if the key is a tuple and the sort semantics matters, we do not have a binary comparator implementation. This especially matters when we switch to use secondary sort. In secondary sort, we convert the inner sort of nested foreach into the secondary key and rely on hadoop to sorting on both main key and secondary key. The sorting key will become a two items tuple. Since the secondary key the sorting key of the nested foreach, so the sorting semantics matters. It turns out we do not have binary comparator once we use secondary sort, and we see a significant slow down. Binary comparator for tuple should be doable once we understand the binary structure of the serialized tuple. We can focus on most common use cases first, which is group by followed by a nested sort. In this case, we will use secondary sort. Semantics of the first key does not matter but semantics of secondary key matters. We need to identify the boundary of main key and secondary key in the binary tuple buffer without instantiate tuple itself. Then if the first key equals, we use a binary comparator to compare secondary key. Secondary key can also be a complex data type, but for the first step, we focus on simple secondary key, which is the most common use case. We mark this issue to be a candidate project for Google summer of code 2010 program. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1469) DefaultDataBag assumes ArrayList as default List type
[ https://issues.apache.org/jira/browse/PIG-1469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882983#action_12882983 ] Hadoop QA commented on PIG-1469: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448156/PIG-1469.patch against trunk revision 958053. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/334/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/334/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/334/console This message is automatically generated. DefaultDataBag assumes ArrayList as default List type - Key: PIG-1469 URL: https://issues.apache.org/jira/browse/PIG-1469 Project: Pig Issue Type: Bug Components: data Affects Versions: 0.8.0 Reporter: Gianmarco De Francisci Morales Assignee: Gianmarco De Francisci Morales Fix For: 0.8.0 Attachments: PIG-1469.patch In org.apache.pig.data.DefaultDataBag, the field mContents is assumed to be of type ArrayList but the user can actually pass a different List to the constructor. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1468) DataByteArray.compareTo() does not compare in lexicographic order
[ https://issues.apache.org/jira/browse/PIG-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882985#action_12882985 ] Hadoop QA commented on PIG-1468: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448155/PIG-1468.patch against trunk revision 958053. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/354/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/354/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/354/console This message is automatically generated. DataByteArray.compareTo() does not compare in lexicographic order - Key: PIG-1468 URL: https://issues.apache.org/jira/browse/PIG-1468 Project: Pig Issue Type: Bug Reporter: Gianmarco De Francisci Morales Assignee: Gianmarco De Francisci Morales Attachments: PIG-1468.patch The compareTo() method of org.apache.pig.data.DataByteArray does not compare items in lexicographic order. Actually, it takes into account the signum of the bytes that compose the DataByteArray. So, for example, 0xff compares to less than 0x00 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1464) Should clean the Graph when register another Pig Script
[
https://issues.apache.org/jira/browse/PIG-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882583#action_12882583
]
Hadoop QA commented on PIG-1464:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448030/PIG_1463.patch
against trunk revision 957753.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/350/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/350/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/350/console
This message is automatically generated.
Should clean the Graph when register another Pig Script
---
Key: PIG-1464
URL: https://issues.apache.org/jira/browse/PIG-1464
Project: Pig
Issue Type: Bug
Components: grunt
Affects Versions: 0.8.0
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Fix For: 0.8.0
Attachments: PIG_1463.patch
In the current implementation, the variable names in pig script are all
global variable. This make one pig script know the variable in other scripts.
In my opinion, this is not right. Every relation name in pig script should be
local variable, otherwise it will bring in unexpected result. This issue
relates to PIG-1423
E.g there are two pig script as follows:
Test_1.pig
{code}
a = load 'data/b.txt' ;
{code}
Test_2.pig
{code}
b = foreach a generate $0; // a is recognized by Grunt although it is in
Test_1.pig
{code}
And the following execute normally, do not throw any exception
{code}
PigServer pig=new PigServer(ExecType.Local);
pig.registerScript(Test_1.pig);
pig.registerScript(Test_2.pig);
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1467) order by fail when set fs.file.impl.disable.cache to true
[ https://issues.apache.org/jira/browse/PIG-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882813#action_12882813 ] Hadoop QA commented on PIG-1467: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448105/PIG-1467-2.patch against trunk revision 958053. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 145 javac compiler warnings (more than the trunk's current 140 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/353/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/353/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/353/console This message is automatically generated. order by fail when set fs.file.impl.disable.cache to true --- Key: PIG-1467 URL: https://issues.apache.org/jira/browse/PIG-1467 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.7.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.7.0, 0.8.0 Attachments: PIG-1467-1.patch, PIG-1467-2.patch Order by fail with the message: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.WeightedRangePartitioner.setConf(WeightedRangePartitioner.java:135) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.init(MapTask.java:551) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:630) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314) at org.apache.hadoop.mapred.Child$4.run(Child.java:217) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1062) at org.apache.hadoop.mapred.Child.main(Child.java:211) This happens with the following hadoop settings: fs.file.impl.disable.cache=true fs.hdfs.impl.disable.cache=true -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1389) Implement Pig counter to track number of rows for each input files
[ https://issues.apache.org/jira/browse/PIG-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882077#action_12882077 ] Hadoop QA commented on PIG-1389: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447912/PIG-1389.patch against trunk revision 957399. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/349/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/349/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/349/console This message is automatically generated. Implement Pig counter to track number of rows for each input files --- Key: PIG-1389 URL: https://issues.apache.org/jira/browse/PIG-1389 Project: Pig Issue Type: Improvement Affects Versions: 0.7.0 Reporter: Richard Ding Assignee: Richard Ding Fix For: 0.8.0 Attachments: PIG-1389.patch A MR job generated by Pig not only can have multiple outputs (in the case of multiquery) but also can have multiple inputs (in the case of join or cogroup). In both cases, the existing Hadoop counters (e.g. MAP_INPUT_RECORDS, REDUCE_OUTPUT_RECORDS) can not be used to count the number of records in the given input or output. PIG-1299 addressed the case of multiple outputs. We need to add new counters for jobs with multiple inputs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1333) API interface to Pig
[
https://issues.apache.org/jira/browse/PIG-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12881586#action_12881586
]
Hadoop QA commented on PIG-1333:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12447767/PIG-1333_3.patch
against trunk revision 957046.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 11 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 140 javac compiler warnings (more
than the trunk's current 138 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 391 release audit warnings
(more than the trunk's current 387 warnings).
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/347/testReport/
Release audit warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/347/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/347/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/347/console
This message is automatically generated.
API interface to Pig
Key: PIG-1333
URL: https://issues.apache.org/jira/browse/PIG-1333
Project: Pig
Issue Type: Improvement
Reporter: Olga Natkovich
Assignee: Richard Ding
Fix For: 0.8.0
Attachments: PIG-1333.patch, PIG-1333_1.patch, PIG-1333_2.patch,
PIG-1333_3.patch
It would be nice to make Pig more friendly for applications like workflow
that would be executing pig scripts on user behalf.
Currently, they would have to use pig command line to execute the code;
however, this has limitation on the kind of output that would be delivered.
For instance, it is hard to produce error information that is easy to use
programatically or collect statistics.
The proposal is to create a class that mimics the behavior of the Main but
gives users a status object back. The the main code of pig would look
somethig like:
public static void main(String args[])
{
PigStatus ps = PigMain.exec(args);
exit (PigStatus.rc);
}
We need to define the following:
- Content of PigStatus. It should at least include
* return code
* error string
* exception
* statistics
- A way to propagate the status class through pig code
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1453) [zebra] Intermittent failure for TestOrderPreserveUnionHDFS
[ https://issues.apache.org/jira/browse/PIG-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12881996#action_12881996 ] Hadoop QA commented on PIG-1453: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447494/PIG-1453.patch against trunk revision 957277. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 36 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/348/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/348/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/348/console This message is automatically generated. [zebra] Intermittent failure for TestOrderPreserveUnionHDFS --- Key: PIG-1453 URL: https://issues.apache.org/jira/browse/PIG-1453 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.8.0 Reporter: Daniel Dai Assignee: Yan Zhou Fix For: 0.8.0 Attachments: PIG-1453.patch, PIG-1453.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1454) Consider clean up backend code
[ https://issues.apache.org/jira/browse/PIG-1454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882031#action_12882031 ] Hadoop QA commented on PIG-1454: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447897/PIG-1454.patch against trunk revision 957277. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 394 release audit warnings (more than the trunk's current 389 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/333/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/333/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/333/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/333/console This message is automatically generated. Consider clean up backend code -- Key: PIG-1454 URL: https://issues.apache.org/jira/browse/PIG-1454 Project: Pig Issue Type: Improvement Components: impl Affects Versions: 0.7.0 Reporter: Richard Ding Assignee: Richard Ding Fix For: 0.8.0 Attachments: PIG-1454.patch Prior to 0.7, Pig had its own local execution mode, in addition to hadoop map reduce execution mode. To support these two different execution modes, Pig implemented an abstraction layer with a set of interfaces and abstract classes. Pig 0.7 replaced the local mode with hadoop local mode and made this abstraction layer redundant. Our goal is to remove those extra code. But we need also keep code backward compatible since some interfaces are exposed by top-level API. So we propose the first steps: * Deprecate methods on FileLocalizer that have DataStorage as parameter. * Remove ExecPhysicalOperator, ExecPhysicalPlan, ExecScopedLogicalOperator, ExecutionEngine and util/ExecTools from org.apache.pig.backend.executionengine package. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1034) Pig does not support ORDER ... BY group alias
[
https://issues.apache.org/jira/browse/PIG-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12880808#action_12880808
]
Hadoop QA commented on PIG-1034:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12447586/PIG_1034.patch
against trunk revision 956440.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/344/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/344/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/344/console
This message is automatically generated.
Pig does not support ORDER ... BY group alias
-
Key: PIG-1034
URL: https://issues.apache.org/jira/browse/PIG-1034
Project: Pig
Issue Type: Bug
Affects Versions: 0.8.0
Reporter: David Ciemiewicz
Assignee: Jeff Zhang
Fix For: 0.8.0
Attachments: PIG_1034.patch
GROUP ... ALL and GROUP ... BY produce an alias group.
Pig produces a syntax error if you attempt to ORDER ... BY group.
This does seem like a perfectly reasonable thing to do.
The workaround is to create an alias for group using an AS clause. But I
think this workaround should be unnecessary.
Here's sample code which elicits the syntax error:
{code}
A = load 'one.txt' using PigStorage as (one: int);
B = group A all;
C = foreach B generate
group,
COUNT(A) as count;
D = order C by group parallel 1; -- group is one of the aliases in C, why
does this throw a syntax error?
dump D;
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1405) Need to move many standard functions from piggybank into Pig
[ https://issues.apache.org/jira/browse/PIG-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12880965#action_12880965 ] Hadoop QA commented on PIG-1405: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447615/StandardUDFtoPigFinale.patch against trunk revision 956662. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/345/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/345/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/345/console This message is automatically generated. Need to move many standard functions from piggybank into Pig Key: PIG-1405 URL: https://issues.apache.org/jira/browse/PIG-1405 Project: Pig Issue Type: Improvement Reporter: Alan Gates Assignee: Aniket Mokashi Fix For: 0.8.0 Attachments: StandardUDFtoPig.patch, StandardUDFtoPig3.patch, StandardUDFtoPig4.patch, StandardUDFtoPigFinale.patch There are currently a number of functions in Piggybank that represent features commonly supported by languages and database engines. We need to decide which of these Pig should support as built in functions and put them in org.apache.pig.builtin. This will also mean adding unit tests and javadocs for some UDFs. The existing classes will be left in Piggybank for some time for backward compatibility. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1405) Need to move many standard functions from piggybank into Pig
[ https://issues.apache.org/jira/browse/PIG-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12880421#action_12880421 ] Hadoop QA commented on PIG-1405: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447492/StandardUDFtoPig4.patch against trunk revision 955763. -1 @author. The patch appears to contain 2 @author tags which the Pig community has agreed to not allow in code contributions. +1 tests included. The patch appears to include 5 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/343/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/343/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/343/console This message is automatically generated. Need to move many standard functions from piggybank into Pig Key: PIG-1405 URL: https://issues.apache.org/jira/browse/PIG-1405 Project: Pig Issue Type: Improvement Reporter: Alan Gates Assignee: Aniket Mokashi Fix For: 0.8.0 Attachments: StandardUDFtoPig.patch, StandardUDFtoPig3.patch, StandardUDFtoPig4.patch There are currently a number of functions in Piggybank that represent features commonly supported by languages and database engines. We need to decide which of these Pig should support as built in functions and put them in org.apache.pig.builtin. This will also mean adding unit tests and javadocs for some UDFs. The existing classes will be left in Piggybank for some time for backward compatibility. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1221) Filter equality does not work for tuples
[ https://issues.apache.org/jira/browse/PIG-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12879765#action_12879765 ] Hadoop QA commented on PIG-1221: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447317/PIG_1221.patch against trunk revision 955028. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/340/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/340/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/340/console This message is automatically generated. Filter equality does not work for tuples Key: PIG-1221 URL: https://issues.apache.org/jira/browse/PIG-1221 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.8.0 Environment: Windows and Linux. Java 1.6 hadoop 0.20.1 Reporter: Neil Blue Assignee: Jeff Zhang Fix For: 0.8.0 Attachments: PIG_1221.patch From the documentation I understand that it should be possible to filter a relation based on the equality of tuples. http://wiki.apache.org/pig/PigTypesFunctionalSpec , http://hadoop.apache.org/pig/docs/r0.5.0/piglatin_reference.html#deref: However with this data file -- indext.txt: (1,one) (1,ONE) (2,two) (22, twentytwo) (3,three) (3,three) I run this pig script: A = LOAD 'indext.txt' AS (t1:(a:int, b:chararray), t2:(a:int, b:chararray)); B = FILTER A BY t1==t2; DUMP B; Expecting the output: ((3,three),(3,three)) However there is an error: 2010-02-03 09:05:20,523 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2067: EqualToExpr does not know how to handle type: tuple Pig Stack Trace --- ERROR 2067: EqualToExpr does not know how to handle type: tuple org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias B at org.apache.pig.PigServer.openIterator(PigServer.java:475) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java: 532) at org .apache .pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser. java:190) at org .apache .pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:166 ) at org .apache .pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:142 ) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89) at org.apache.pig.Main.main(Main.java:397) Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias B at org.apache.pig.PigServer.store(PigServer.java:530) at org.apache.pig.PigServer.openIterator(PigServer.java:458) ... 6 more Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2067: EqualToExpr does not know how to handle type: tuple at org .apache .pig.backend.hadoop.executionengine.physicalLayer.expressionOperat ors.EqualToExpr.getNext(EqualToExpr.java:108) at org .apache .pig.backend.hadoop.executionengine.physicalLayer.relationalOperat ors.POFilter.getNext(POFilter.java:148) at org .apache .pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator .processInput(PhysicalOperator.java:231) at org .apache .pig.backend.local.executionengine.physicalLayer.counters.POCounte r.getNext(POCounter.java:71) at org .apache .pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator .processInput(PhysicalOperator.java:231) at org .apache .pig.backend.hadoop.executionengine.physicalLayer.relationalOperat ors.POStore.getNext(POStore.java:117) at org .apache .pig.backend.local.executionengine.LocalPigLauncher.runPipeline(Lo calPigLauncher.java:146) at org .apache .pig.backend.local.executionengine.LocalPigLauncher.launchPig(Loca lPigLauncher.java:109) at org .apache
[jira] Commented: (PIG-1453) [zebra] Intermittent failure for TestOrderPreserveUnionHDFS
[ https://issues.apache.org/jira/browse/PIG-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12880025#action_12880025 ] Hadoop QA commented on PIG-1453: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447373/PIG-1453.patch against trunk revision 955701. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 36 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/341/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/341/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/341/console This message is automatically generated. [zebra] Intermittent failure for TestOrderPreserveUnionHDFS --- Key: PIG-1453 URL: https://issues.apache.org/jira/browse/PIG-1453 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.8.0 Reporter: Daniel Dai Assignee: Yan Zhou Fix For: 0.8.0 Attachments: PIG-1453.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1405) Need to move many standard functions from piggybank into Pig
[ https://issues.apache.org/jira/browse/PIG-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12880049#action_12880049 ] Hadoop QA commented on PIG-1405: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447381/StandardUDFtoPig3.patch against trunk revision 955701. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. -1 javac. The applied patch generated 146 javac compiler warnings (more than the trunk's current 138 warnings). -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/330/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/330/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/330/console This message is automatically generated. Need to move many standard functions from piggybank into Pig Key: PIG-1405 URL: https://issues.apache.org/jira/browse/PIG-1405 Project: Pig Issue Type: Improvement Reporter: Alan Gates Assignee: Aniket Mokashi Fix For: 0.8.0 Attachments: StandardUDFtoPig.patch, StandardUDFtoPig3.patch There are currently a number of functions in Piggybank that represent features commonly supported by languages and database engines. We need to decide which of these Pig should support as built in functions and put them in org.apache.pig.builtin. This will also mean adding unit tests and javadocs for some UDFs. The existing classes will be left in Piggybank for some time for backward compatibility. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1452) to remove hadoop20.jar from lib and use hadoop from the apache maven repo.
[ https://issues.apache.org/jira/browse/PIG-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12879414#action_12879414 ] Hadoop QA commented on PIG-1452: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447216/PIG-1452.PATCH against trunk revision 955028. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/339/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/339/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/339/console This message is automatically generated. to remove hadoop20.jar from lib and use hadoop from the apache maven repo. -- Key: PIG-1452 URL: https://issues.apache.org/jira/browse/PIG-1452 Project: Pig Issue Type: Improvement Components: build Affects Versions: 0.8.0 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Attachments: PIG-1452.PATCH pig use ivy for dependency management. But still it uses hadoop20.jar from the lib folder. Now that we have the hadoop-0.20.2 artifacts available in the maven repo, pig should leverage ivy for resolving/retrieving hadoop artifacts. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1333) API interface to Pig
[
https://issues.apache.org/jira/browse/PIG-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12878869#action_12878869
]
Hadoop QA commented on PIG-1333:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12447048/PIG-1333_1.patch
against trunk revision 953798.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 11 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 387 release audit warnings
(more than the trunk's current 383 warnings).
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/329/testReport/
Release audit warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/329/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/329/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/329/console
This message is automatically generated.
API interface to Pig
Key: PIG-1333
URL: https://issues.apache.org/jira/browse/PIG-1333
Project: Pig
Issue Type: Improvement
Reporter: Olga Natkovich
Assignee: Richard Ding
Fix For: 0.8.0
Attachments: PIG-1333.patch, PIG-1333_1.patch
It would be nice to make Pig more friendly for applications like workflow
that would be executing pig scripts on user behalf.
Currently, they would have to use pig command line to execute the code;
however, this has limitation on the kind of output that would be delivered.
For instance, it is hard to produce error information that is easy to use
programatically or collect statistics.
The proposal is to create a class that mimics the behavior of the Main but
gives users a status object back. The the main code of pig would look
somethig like:
public static void main(String args[])
{
PigStatus ps = PigMain.exec(args);
exit (PigStatus.rc);
}
We need to define the following:
- Content of PigStatus. It should at least include
* return code
* error string
* exception
* statistics
- A way to propagate the status class through pig code
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1451) [zebra] change the build.test property in build to test.build.dir to be in consistent with PIG
[ https://issues.apache.org/jira/browse/PIG-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12879209#action_12879209 ] Hadoop QA commented on PIG-1451: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447159/PIG-1451.patch against trunk revision 954772. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 14 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/338/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/338/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/338/console This message is automatically generated. [zebra] change the build.test property in build to test.build.dir to be in consistent with PIG -- Key: PIG-1451 URL: https://issues.apache.org/jira/browse/PIG-1451 Project: Pig Issue Type: Improvement Affects Versions: 0.6.0, 0.7.0, 0.8.0 Reporter: Yan Zhou Assignee: Yan Zhou Priority: Minor Fix For: 0.6.0, 0.7.0, 0.8.0 Attachments: PIG-1451.patch Because build process handles PIG and Zebra builds in the same settings, the property should be the same so the build process have consistent controls. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-972) Make describe work with nested foreach
[ https://issues.apache.org/jira/browse/PIG-972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12878545#action_12878545 ] Hadoop QA commented on PIG-972: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446735/NestedDescribeFinale.patch against trunk revision 953798. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. -1 release audit. The applied patch generated 384 release audit warnings (more than the trunk's current 383 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/324/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/324/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/324/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/324/console This message is automatically generated. Make describe work with nested foreach -- Key: PIG-972 URL: https://issues.apache.org/jira/browse/PIG-972 Project: Pig Issue Type: Improvement Reporter: Olga Natkovich Assignee: Aniket Mokashi Fix For: 0.8.0 Attachments: NestedDescribeFinale.patch, NestedDescribeProp1.patch, NestedDescribeProp2Initial.patch Currently Parser can't deal with that. This is because describe is part of Grunt parser while the rest of nested foreach is handled by the QueryParser -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1302) Include zebra's pigtest ant target as a part of pig's ant test target
[ https://issues.apache.org/jira/browse/PIG-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12878702#action_12878702 ] Hadoop QA commented on PIG-1302: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446596/PIG-1302.patch against trunk revision 953798. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/326/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/326/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/326/console This message is automatically generated. Include zebra's pigtest ant target as a part of pig's ant test target --- Key: PIG-1302 URL: https://issues.apache.org/jira/browse/PIG-1302 Project: Pig Issue Type: Improvement Affects Versions: 0.7.0 Reporter: Pradeep Kamath Assignee: Giridharan Kesavan Attachments: PIG-1302.patch There are changes made in Pig interfaces which break zebra loaders/storers. It would be good to run the pig tests in the zebra unit tests as part of running pig's core-test for each patch submission. So essentially in the test ant target in pig, we would need to invoke zebra's pigtest target. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-972) Make describe work with nested foreach
[ https://issues.apache.org/jira/browse/PIG-972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12878810#action_12878810 ] Hadoop QA commented on PIG-972: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447041/NestedDescribeFinale1.patch against trunk revision 953798. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 384 release audit warnings (more than the trunk's current 383 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/327/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/327/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/327/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/327/console This message is automatically generated. Make describe work with nested foreach -- Key: PIG-972 URL: https://issues.apache.org/jira/browse/PIG-972 Project: Pig Issue Type: Improvement Reporter: Olga Natkovich Assignee: Aniket Mokashi Fix For: 0.8.0 Attachments: NestedDescribeFinale.patch, NestedDescribeFinale1.patch, NestedDescribeProp1.patch, NestedDescribeProp2Initial.patch Currently Parser can't deal with that. This is because describe is part of Grunt parser while the rest of nested foreach is handled by the QueryParser -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1449) RegExLoader hangs on lines that don't match the regular expression
[
https://issues.apache.org/jira/browse/PIG-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12878812#action_12878812
]
Hadoop QA commented on PIG-1449:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12447045/RegExLoader.patch
against trunk revision 953798.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no tests are needed for this patch.
-1 patch. The patch command could not apply the patch.
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/328/console
This message is automatically generated.
RegExLoader hangs on lines that don't match the regular expression
--
Key: PIG-1449
URL: https://issues.apache.org/jira/browse/PIG-1449
Project: Pig
Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Justin Sanders
Priority: Minor
Attachments: RegExLoader.patch
In the 0.7.0 changes to RegExLoader there was a bug introduced where the code
will stay in the while loop if the line isn't matched. Before 0.7.0 these
lines would be skipped if they didn't match the regular expression. The
result is the mapper will not respond and will time out with Task attempt_X
failed to report status for 600 seconds. Killing!.
Here are the steps to recreate the bug:
Create a text file in HDFS with the following lines:
test1
testA
test2
Run the following pig script:
REGISTER /usr/local/pig/contrib/piggybank/java/piggybank.jar;
test = LOAD '/path/to/test.txt' using
org.apache.pig.piggybank.storage.MyRegExLoader('(test\\d)') AS (line);
dump test;
Expected result:
(test1)
(test3)
Actual result:
Job fails to complete after 600 second timeout waiting on the mapper to
complete. The mapper hangs at 33% since it can process the first line but
gets stuck into the while loop on the second line.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1438) [Performance] MultiQueryOptimizer should also merge DISTINCT jobs
[ https://issues.apache.org/jira/browse/PIG-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12876980#action_12876980 ] Hadoop QA commented on PIG-1438: +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446652/PIG-1438_1.patch against trunk revision 952098. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/334/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/334/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/334/console This message is automatically generated. [Performance] MultiQueryOptimizer should also merge DISTINCT jobs - Key: PIG-1438 URL: https://issues.apache.org/jira/browse/PIG-1438 Project: Pig Issue Type: Improvement Components: impl Affects Versions: 0.7.0 Reporter: Richard Ding Assignee: Richard Ding Fix For: 0.8.0 Attachments: PIG-1438.patch, PIG-1438_1.patch Current implementation doesn't merge jobs derived from DISTINCT statements. The reason is that DISTINCT jobs are implemented using a special combiner (DistinctCombiner). But we should be able to merge jobs that have the same type of combiner (e.g. merge multiple DISTINCT jobs into one). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1443) DefaultTuple underestimate the memory footprint for string
[
https://issues.apache.org/jira/browse/PIG-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12877256#action_12877256
]
Hadoop QA commented on PIG-1443:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12446712/PIG-1443-1.patch
against trunk revision 952098.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 139 javac compiler warnings (more
than the trunk's current 138 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/321/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/321/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/321/console
This message is automatically generated.
DefaultTuple underestimate the memory footprint for string
--
Key: PIG-1443
URL: https://issues.apache.org/jira/browse/PIG-1443
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.7.0
Reporter: Daniel Dai
Assignee: Daniel Dai
Fix For: 0.8.0
Attachments: PIG-1443-1.patch
Currently, in DefaultTuple, we estimate the memory footprint for string as if
it is char array. The formula we use is: length * 2 + 12. It turns out we
underestimate the memory usage for string. Here is a list of real memory
footprint for string we get from memory dump:
| length of string | memory in bytes |
| 7 | 56 |
| 3 | 48 |
| 1 | 40 |
I did a search and find the following formula can accurately estimate the
memory footprint for string:
{code}
8 * (int) (((length * 2) + 45) / 8)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1445) Pig error: ERROR 2013: Moving LOLimit in front of LOStream is not implemented
[
https://issues.apache.org/jira/browse/PIG-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12877318#action_12877318
]
Hadoop QA commented on PIG-1445:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12446718/PIG-1445-1.patch
against trunk revision 953109.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 383 release audit warnings
(more than the trunk's current 382 warnings).
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/322/testReport/
Release audit warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/322/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/322/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/322/console
This message is automatically generated.
Pig error: ERROR 2013: Moving LOLimit in front of LOStream is not implemented
--
Key: PIG-1445
URL: https://issues.apache.org/jira/browse/PIG-1445
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.7.0
Reporter: Daniel Dai
Assignee: Daniel Dai
Fix For: 0.8.0
Attachments: PIG-1445-1.patch
The following script fail due to ERROR 2013: Moving LOLimit in front of
LOStream is not implemented.
{code}
A = LOAD 'data';
B = STREAM A THROUGH `stream.pl`;
C = LIMIT B 10;
explain C;
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1428) Add getPigStatusReporter() to PigHadoopLogger
[ https://issues.apache.org/jira/browse/PIG-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12876708#action_12876708 ] Hadoop QA commented on PIG-1428: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446095/PIG-1428.patch against trunk revision 952098. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 383 release audit warnings (more than the trunk's current 382 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/332/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/332/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/332/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/332/console This message is automatically generated. Add getPigStatusReporter() to PigHadoopLogger - Key: PIG-1428 URL: https://issues.apache.org/jira/browse/PIG-1428 Project: Pig Issue Type: Bug Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Assignee: Dmitriy V. Ryaboy Fix For: 0.8.0 Attachments: PIG-1428.patch, PIG-1428.patch Without this getter method, its not possible to get counters, report progress etc. from UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1438) [Performance] MultiQueryOptimizer should also merge DISTINCT jobs
[ https://issues.apache.org/jira/browse/PIG-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12876840#action_12876840 ] Hadoop QA commented on PIG-1438: +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446604/PIG-1438.patch against trunk revision 952098. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/333/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/333/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/333/console This message is automatically generated. [Performance] MultiQueryOptimizer should also merge DISTINCT jobs - Key: PIG-1438 URL: https://issues.apache.org/jira/browse/PIG-1438 Project: Pig Issue Type: Improvement Components: impl Affects Versions: 0.7.0 Reporter: Richard Ding Assignee: Richard Ding Fix For: 0.8.0 Attachments: PIG-1438.patch Current implementation doesn't merge jobs derived from DISTINCT statements. The reason is that DISTINCT jobs are implemented using a special combiner (DistinctCombiner). But we should be able to merge jobs that have the same type of combiner (e.g. merge multiple DISTINCT jobs into one). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1249) Safe-guards against misconfigured Pig scripts without PARALLEL keyword
[ https://issues.apache.org/jira/browse/PIG-1249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12875551#action_12875551 ] Hadoop QA commented on PIG-1249: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446173/PIG-1249-4.patch against trunk revision 951229. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/329/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/329/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/329/console This message is automatically generated. Safe-guards against misconfigured Pig scripts without PARALLEL keyword -- Key: PIG-1249 URL: https://issues.apache.org/jira/browse/PIG-1249 Project: Pig Issue Type: Improvement Affects Versions: 0.8.0 Reporter: Arun C Murthy Assignee: Jeff Zhang Priority: Critical Fix For: 0.8.0 Attachments: PIG-1249-4.patch, PIG-1249.patch, PIG_1249_2.patch, PIG_1249_3.patch It would be *very* useful for Pig to have safe-guards against naive scripts which process a *lot* of data without the use of PARALLEL keyword. We've seen a fair number of instances where naive users process huge data-sets (10TB) with badly mis-configured #reduces e.g. 1 reduce. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-282) Custom Partitioner
[ https://issues.apache.org/jira/browse/PIG-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12875554#action_12875554 ] Hadoop QA commented on PIG-282: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446172/CustomPartitionerFinale.patch against trunk revision 951229. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 380 release audit warnings (more than the trunk's current 379 warnings). -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/320/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/320/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/320/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/320/console This message is automatically generated. Custom Partitioner -- Key: PIG-282 URL: https://issues.apache.org/jira/browse/PIG-282 Project: Pig Issue Type: New Feature Affects Versions: 0.7.0 Reporter: Amir Youssefi Assignee: Aniket Mokashi Priority: Minor Fix For: 0.8.0 Attachments: CustomPartitioner.patch, CustomPartitionerFinale.patch, CustomPartitionerTest.patch By adding custom partitioner we can give control over which output partition a key (/value) goes to. We can add keywords to language e.g. PARTITION BY UDF(...) or a similar syntax. UDF returns a number between 0 and n-1 where n is number of output partitions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1433) pig should create success file if mapreduce.fileoutputcommitter.marksuccessfuljobs is true
[ https://issues.apache.org/jira/browse/PIG-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12875639#action_12875639 ] Hadoop QA commented on PIG-1433: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446222/PIG-1433.patch against trunk revision 951229. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/330/console This message is automatically generated. pig should create success file if mapreduce.fileoutputcommitter.marksuccessfuljobs is true -- Key: PIG-1433 URL: https://issues.apache.org/jira/browse/PIG-1433 Project: Pig Issue Type: Bug Affects Versions: 0.8.0 Reporter: Pradeep Kamath Assignee: Pradeep Kamath Fix For: 0.8.0 Attachments: PIG-1433.patch pig should create success file if mapreduce.fileoutputcommitter.marksuccessfuljobs is true -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1428) Add getPigStatusReporter() to PigHadoopLogger
[ https://issues.apache.org/jira/browse/PIG-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12873901#action_12873901 ] Hadoop QA commented on PIG-1428: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445985/PIG-1428.patch against trunk revision 949057. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. -1 release audit. The applied patch generated 386 release audit warnings (more than the trunk's current 385 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/17/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/17/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/17/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/17/console This message is automatically generated. Add getPigStatusReporter() to PigHadoopLogger - Key: PIG-1428 URL: https://issues.apache.org/jira/browse/PIG-1428 Project: Pig Issue Type: Bug Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Assignee: Dmitriy V. Ryaboy Fix For: 0.8.0 Attachments: PIG-1428.patch Without this getter method, its not possible to get counters, report progress etc. from UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-282) Custom Partitioner
[ https://issues.apache.org/jira/browse/PIG-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12874393#action_12874393 ] Hadoop QA commented on PIG-282: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446067/CustomPartitionerTest.patch against trunk revision 949057. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 386 release audit warnings (more than the trunk's current 385 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/18/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/18/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/18/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/18/console This message is automatically generated. Custom Partitioner -- Key: PIG-282 URL: https://issues.apache.org/jira/browse/PIG-282 Project: Pig Issue Type: New Feature Affects Versions: 0.7.0 Reporter: Amir Youssefi Assignee: Aniket Mokashi Priority: Minor Fix For: 0.8.0 Attachments: CustomPartitioner.patch, CustomPartitionerTest.patch By adding custom partitioner we can give control over which output partition a key (/value) goes to. We can add keywords to language e.g. PARTITION BY UDF(...) or a similar syntax. UDF returns a number between 0 and n-1 where n is number of output partitions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1432) [zebra] There are some debuging info output to STDOUT in PIG's TableStorer call path
[ https://issues.apache.org/jira/browse/PIG-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12874394#action_12874394 ] Hadoop QA commented on PIG-1432: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446078/PIG-1432.patch against trunk revision 949057. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/19/console This message is automatically generated. [zebra] There are some debuging info output to STDOUT in PIG's TableStorer call path Key: PIG-1432 URL: https://issues.apache.org/jira/browse/PIG-1432 Project: Pig Issue Type: Bug Affects Versions: 0.7.0 Reporter: Yan Zhou Assignee: Yan Zhou Priority: Trivial Fix For: 0.7.0 Attachments: PIG-1432.patch Users redirecting STDOUT to disk file got disk full errors. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-282) Custom Partitioner
[ https://issues.apache.org/jira/browse/PIG-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12872893#action_12872893 ] Hadoop QA commented on PIG-282: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445704/CustomPartitioner.patch against trunk revision 949057. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 385 release audit warnings (more than the trunk's current 384 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/13/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/13/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/13/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/13/console This message is automatically generated. Custom Partitioner -- Key: PIG-282 URL: https://issues.apache.org/jira/browse/PIG-282 Project: Pig Issue Type: New Feature Affects Versions: 0.7.0 Reporter: Amir Youssefi Assignee: Aniket Mokashi Priority: Minor Fix For: 0.8.0 Attachments: CustomPartitioner.patch By adding custom partitioner we can give control over which output partition a key (/value) goes to. We can add keywords to language e.g. PARTITION BY UDF(...) or a similar syntax. UDF returns a number between 0 and n-1 where n is number of output partitions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1333) API interface to Pig
[
https://issues.apache.org/jira/browse/PIG-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12872894#action_12872894
]
Hadoop QA commented on PIG-1333:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12445727/PIG-1333.patch
against trunk revision 949057.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 99 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/14/console
This message is automatically generated.
API interface to Pig
Key: PIG-1333
URL: https://issues.apache.org/jira/browse/PIG-1333
Project: Pig
Issue Type: Improvement
Reporter: Olga Natkovich
Assignee: Richard Ding
Fix For: 0.8.0
Attachments: PIG-1333.patch
It would be nice to make Pig more friendly for applications like workflow
that would be executing pig scripts on user behalf.
Currently, they would have to use pig command line to execute the code;
however, this has limitation on the kind of output that would be delivered.
For instance, it is hard to produce error information that is easy to use
programatically or collect statistics.
The proposal is to create a class that mimics the behavior of the Main but
gives users a status object back. The the main code of pig would look
somethig like:
public static void main(String args[])
{
PigStatus ps = PigMain.exec(args);
exit (PigStatus.rc);
}
We need to define the following:
- Content of PigStatus. It should at least include
* return code
* error string
* exception
* statistics
- A way to propagate the status class through pig code
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-283) Allow to set arbitrary jobconf key-value pairs inside pig program
[ https://issues.apache.org/jira/browse/PIG-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12872969#action_12872969 ] Hadoop QA commented on PIG-283: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445710/pig-282.patch against trunk revision 949057. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/15/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/15/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/15/console This message is automatically generated. Allow to set arbitrary jobconf key-value pairs inside pig program - Key: PIG-283 URL: https://issues.apache.org/jira/browse/PIG-283 Project: Pig Issue Type: New Feature Components: grunt Affects Versions: 0.7.0 Reporter: Christian Kunz Assignee: Ashutosh Chauhan Fix For: 0.8.0 Attachments: pig-282.patch It would be useful to be able to set arbitrary JobConf key-value pairs inside a pig program (e.g. in front of a COGROUP statement). I wonder whether the simplest way to add this feature is by expanding the 'set' command functionality. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1333) API interface to Pig
[
https://issues.apache.org/jira/browse/PIG-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12873145#action_12873145
]
Hadoop QA commented on PIG-1333:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12445786/PIG-1333.patch
against trunk revision 949057.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 99 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 147 javac compiler warnings (more
than the trunk's current 139 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 395 release audit warnings
(more than the trunk's current 385 warnings).
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/16/testReport/
Release audit warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/16/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/16/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/16/console
This message is automatically generated.
API interface to Pig
Key: PIG-1333
URL: https://issues.apache.org/jira/browse/PIG-1333
Project: Pig
Issue Type: Improvement
Reporter: Olga Natkovich
Assignee: Richard Ding
Fix For: 0.8.0
Attachments: PIG-1333.patch
It would be nice to make Pig more friendly for applications like workflow
that would be executing pig scripts on user behalf.
Currently, they would have to use pig command line to execute the code;
however, this has limitation on the kind of output that would be delivered.
For instance, it is hard to produce error information that is easy to use
programatically or collect statistics.
The proposal is to create a class that mimics the behavior of the Main but
gives users a status object back. The the main code of pig would look
somethig like:
public static void main(String args[])
{
PigStatus ps = PigMain.exec(args);
exit (PigStatus.rc);
}
We need to define the following:
- Content of PigStatus. It should at least include
* return code
* error string
* exception
* statistics
- A way to propagate the status class through pig code
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1373) We need to add jdiff output to docs on the website
[ https://issues.apache.org/jira/browse/PIG-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12872192#action_12872192 ] Hadoop QA commented on PIG-1373: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444728/PIG-1373-2.patch against trunk revision 948526. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/4/console This message is automatically generated. We need to add jdiff output to docs on the website -- Key: PIG-1373 URL: https://issues.apache.org/jira/browse/PIG-1373 Project: Pig Issue Type: Bug Reporter: Alan Gates Assignee: Daniel Dai Priority: Minor Fix For: 0.8.0 Attachments: PIG-1373-1.patch, PIG-1373-2.patch Our build process constructs a jdiff between APIs for different versions. But we don't post the results of that to the website when we deploy the docs. We should, in order to help users understand changes across versions of pig. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1420) Make CONCAT act on all fields of a tuple, instead of just the first two fields of a tuple
[ https://issues.apache.org/jira/browse/PIG-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12872194#action_12872194 ] Hadoop QA commented on PIG-1420: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444886/addconcat2.patch against trunk revision 948526. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/5/console This message is automatically generated. Make CONCAT act on all fields of a tuple, instead of just the first two fields of a tuple - Key: PIG-1420 URL: https://issues.apache.org/jira/browse/PIG-1420 Project: Pig Issue Type: Improvement Components: impl Affects Versions: 0.8.0 Reporter: Russell Jurney Assignee: Russell Jurney Fix For: 0.8.0 Attachments: addconcat2.patch Original Estimate: 24h Remaining Estimate: 24h org.apache.pig.builtin.CONCAT (which acts on DataByteArray's internally) and org.apache.pig.builtin.StringConcat (which acts on Strings internally), both act on the first two fields of a tuple. This results in ugly nested CONCAT calls like: CONCAT(CONCAT(A, ' '), B) The more desirable form is: CONCAT(A, ' ', B) This change will be backwards compatible, provided that no one was relying on the fact that CONCAT ignores fields after the first two in a tuple. This seems a reasonable assumption to make, or at least a small break in compatibility for a sizable improvement. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1249) Safe-guards against misconfigured Pig scripts without PARALLEL keyword
[ https://issues.apache.org/jira/browse/PIG-1249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12872196#action_12872196 ] Hadoop QA commented on PIG-1249: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445559/PIG_1249_3.patch against trunk revision 948526. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/6/console This message is automatically generated. Safe-guards against misconfigured Pig scripts without PARALLEL keyword -- Key: PIG-1249 URL: https://issues.apache.org/jira/browse/PIG-1249 Project: Pig Issue Type: Improvement Affects Versions: 0.8.0 Reporter: Arun C Murthy Assignee: Jeff Zhang Priority: Critical Fix For: 0.8.0 Attachments: PIG-1249.patch, PIG_1249_2.patch, PIG_1249_3.patch It would be *very* useful for Pig to have safe-guards against naive scripts which process a *lot* of data without the use of PARALLEL keyword. We've seen a fair number of instances where naive users process huge data-sets (10TB) with badly mis-configured #reduces e.g. 1 reduce. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1404) PigUnit - Pig script testing simplified.
[
https://issues.apache.org/jira/browse/PIG-1404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12872293#action_12872293
]
Hadoop QA commented on PIG-1404:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12444822/PIG-1404-2.patch
against trunk revision 948526.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/8/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/8/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/8/console
This message is automatically generated.
PigUnit - Pig script testing simplified.
-
Key: PIG-1404
URL: https://issues.apache.org/jira/browse/PIG-1404
Project: Pig
Issue Type: New Feature
Reporter: Romain Rigaux
Assignee: Romain Rigaux
Fix For: 0.8.0
Attachments: commons-lang-2.4.jar, PIG-1404-2.patch, PIG-1404.patch
The goal is to provide a simple xUnit framework that enables our Pig scripts
to be easily:
- unit tested
- regression tested
- quickly prototyped
No cluster set up is required.
For example:
TestCase
{code}
@Test
public void testTop3Queries() {
String[] args = {
n=3,
};
test = new PigTest(top_queries.pig, args);
String[] input = {
yahoo\t10,
twitter\t7,
facebook\t10,
yahoo\t15,
facebook\t5,
};
String[] output = {
(yahoo,25L),
(facebook,15L),
(twitter,7L),
};
test.assertOutput(data, input, queries_limit, output);
}
{code}
top_queries.pig
{code}
data =
LOAD '$input'
AS (query:CHARARRAY, count:INT);
...
queries_sum =
FOREACH queries_group
GENERATE
group AS query,
SUM(queries.count) AS count;
...
queries_limit = LIMIT queries_ordered $n;
STORE queries_limit INTO '$output';
{code}
They are 3 modes:
* LOCAL (if pigunit.exectype.local properties is present)
* MAPREDUCE (use the cluster specified in the classpath, same as
HADOOP_CONF_DIR)
** automatic mini cluster (is the default and the HADOOP_CONF_DIR to have in
the class path will be: ~/pigtest/conf)
** pointing to an existing cluster (if pigunit.exectype.cluster properties
is present)
For now, it would be nice to see how this idea could be integrated in
Piggybank and if PigParser/PigServer could improve their interfaces in order
to make PigUnit simple.
Other components based on PigUnit could be built later:
- standalone MiniCluster
- notion of workspaces for each test
- standalone utility that reads test configuration and generates a test
report...
It is a first prototype, open to suggestions and can definitely take
advantage of feedbacks.
How to test, in pig_trunk:
{code}
Apply patch
$pig_trunk ant compile-test
$pig_trunk ant
$pig_trunk/contrib/piggybank/java ant test -Dtest.timeout=99
{code}
(it takes 15 min in MAPREDUCE minicluster, tests will need to be split in the
future between 'unit' and 'integration')
Many examples are in:
{code}
contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/pigunit/TestPigTest.java
{code}
When used as a standalone, do not forget commons-lang-2.4.jar and the
HADOOP_CONF_DIR to your cluster in your CLASSPATH.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1423) Suggest to add clear command in grunt to clear the relation variable
[
https://issues.apache.org/jira/browse/PIG-1423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12872664#action_12872664
]
Hadoop QA commented on PIG-1423:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12445029/PIG-1423.patch
against trunk revision 948526.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/9/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/9/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/9/console
This message is automatically generated.
Suggest to add clear command in grunt to clear the relation variable
Key: PIG-1423
URL: https://issues.apache.org/jira/browse/PIG-1423
Project: Pig
Issue Type: New Feature
Components: grunt
Affects Versions: 0.8.0
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Fix For: 0.8.0
Attachments: PIG-1423.patch
The relation name in pig script can been considered as variable of
programming language. One risk of the current grunt is that user may use the
previous defined relation name by typo after a long period of work on grunt.
And it is difficult for users to track this problem. E.g. the following red
students is not the user intend to use, but here grunt won't throw any
error meesage.
students = load 'a.txt';
student = load 'b.txt';
result = foreach {color:red}students{color} generate $0;
The clear command is to clear the variable defined before, then if users use
the relation name defined before, grunt will throw error message. And this
command also will be useful to let user reuse the relation names especially
when he's doing lots of experiments for one specific task.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1426) Change the size of Tuple from Int to VInt when Serialize Tuple
[ https://issues.apache.org/jira/browse/PIG-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12872810#action_12872810 ] Hadoop QA commented on PIG-1426: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445288/PIG_1426.patch against trunk revision 948526. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/11/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/11/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/11/console This message is automatically generated. Change the size of Tuple from Int to VInt when Serialize Tuple -- Key: PIG-1426 URL: https://issues.apache.org/jira/browse/PIG-1426 Project: Pig Issue Type: Improvement Components: data Affects Versions: 0.8.0 Reporter: Jeff Zhang Assignee: Jeff Zhang Fix For: 0.8.0 Attachments: PIG_1426.patch Most of time, the size of tuple is not very large, one byte is enough for store the size of tuple. So I suggest to use VInt instead of Int for the size of tuple when doing Serialization. Because the key type of map output is Tuple, so this can reduce the amount of data transferred from mapper to reducer. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1381) Need a way for Pig to take an alternative property file
[ https://issues.apache.org/jira/browse/PIG-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866688#action_12866688 ] Hadoop QA commented on PIG-1381: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444330/PIG-1381-3.patch against trunk revision 943578. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/315/console This message is automatically generated. Need a way for Pig to take an alternative property file --- Key: PIG-1381 URL: https://issues.apache.org/jira/browse/PIG-1381 Project: Pig Issue Type: Improvement Components: impl Affects Versions: 0.7.0 Reporter: Daniel Dai Assignee: V.V.Chaitanya Krishna Fix For: 0.7.0, 0.8.0 Attachments: PIG-1381-1.patch, PIG-1381-2.patch, PIG-1381-3.patch Currently, Pig read the first ever pig.properties in CLASSPATH. Pig has a default pig.properties and if user have a different pig.properties, there will be a conflict since we can only read one. There are couple of ways to solve it: 1. Give a command line option for user to pass an additional property file 2. Change the name for default pig.properties to pig-default.properties, and user can give a pig.properties to override 3. Further, can we consider to use pig-default.xml/pig-site.xml, which seems to be more natural for hadoop community. If so, we shall provide backward compatibility to also read pig.properties, pig-cluster-hadoop-site.xml. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-566) Dump and store outputs do not match for PigStorage
[
https://issues.apache.org/jira/browse/PIG-566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866728#action_12866728
]
Hadoop QA commented on PIG-566:
---
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12444306/PIG-566.patch
against trunk revision 943522.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 18 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/327/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/327/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/327/console
This message is automatically generated.
Dump and store outputs do not match for PigStorage
--
Key: PIG-566
URL: https://issues.apache.org/jira/browse/PIG-566
Project: Pig
Issue Type: Bug
Affects Versions: 0.7.0, 0.8.0
Reporter: Santhosh Srinivasan
Assignee: Gianmarco De Francisci Morales
Priority: Minor
Fix For: 0.7.0, 0.8.0
Attachments: PIG-566.patch, PIG-566.patch, PIG-566.patch,
PIG-566.patch
The dump and store formats for PigStorage do not match for longs and floats.
{code}
grunt y = foreach x generate {(2985671202194220139L)};
grunt describe y;
y: {{(long)}}
grunt dump y;
({(2985671202194220139L)})
grunt store y into 'y';
grunt cat y
{(2985671202194220139)}
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1381) Need a way for Pig to take an alternative property file
[ https://issues.apache.org/jira/browse/PIG-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866942#action_12866942 ] Hadoop QA commented on PIG-1381: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444335/PIG-1381-4.patch against trunk revision 943578. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/328/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/328/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/328/console This message is automatically generated. Need a way for Pig to take an alternative property file --- Key: PIG-1381 URL: https://issues.apache.org/jira/browse/PIG-1381 Project: Pig Issue Type: Improvement Components: impl Affects Versions: 0.7.0 Reporter: Daniel Dai Assignee: V.V.Chaitanya Krishna Fix For: 0.7.0, 0.8.0 Attachments: PIG-1381-1.patch, PIG-1381-2.patch, PIG-1381-3.patch, PIG-1381-4.patch Currently, Pig read the first ever pig.properties in CLASSPATH. Pig has a default pig.properties and if user have a different pig.properties, there will be a conflict since we can only read one. There are couple of ways to solve it: 1. Give a command line option for user to pass an additional property file 2. Change the name for default pig.properties to pig-default.properties, and user can give a pig.properties to override 3. Further, can we consider to use pig-default.xml/pig-site.xml, which seems to be more natural for hadoop community. If so, we shall provide backward compatibility to also read pig.properties, pig-cluster-hadoop-site.xml. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1414) Problem with parameter substitution
[
https://issues.apache.org/jira/browse/PIG-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866952#action_12866952
]
Hadoop QA commented on PIG-1414:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12444353/PIG-1414.patch
against trunk revision 943578.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/316/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/316/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/316/console
This message is automatically generated.
Problem with parameter substitution
---
Key: PIG-1414
URL: https://issues.apache.org/jira/browse/PIG-1414
Project: Pig
Issue Type: Bug
Reporter: Richard Ding
Assignee: Richard Ding
Attachments: PIG-1414.patch
The following script:
{code}
L = load 'input';
store L into 'output' using MyClass$StorerAsInnerClass();
{code}
causes Pig to fail with this error message:
{code}
ERROR org.apache.pig.Main - ERROR 2999: Unexpected internal error. Undefined
parameter : StorerAsInnerClass
java.lang.RuntimeException: Undefined parameter : StorerAsInnerClass
at
org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232)
at
org.apache.pig.tools.parameters.PigFileParser.input(PigFileParser.java:60)
at
org.apache.pig.tools.parameters.PigFileParser.Parse(PigFileParser.java:42)
at
org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:105)
at
org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:98)
at org.apache.pig.Main.runParamPreprocessor(Main.java:576)
at org.apache.pig.Main.main(Main.java:418)
{code}
even though no parameter substitution is specified from the command line.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1406) Allow to run shell commands from grunt
[ https://issues.apache.org/jira/browse/PIG-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866328#action_12866328 ] Hadoop QA commented on PIG-1406: +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444201/Pig-1406_2.patch against trunk revision 943003. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/326/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/326/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/326/console This message is automatically generated. Allow to run shell commands from grunt -- Key: PIG-1406 URL: https://issues.apache.org/jira/browse/PIG-1406 Project: Pig Issue Type: Improvement Affects Versions: 0.8.0 Reporter: Olga Natkovich Assignee: Jeff Zhang Fix For: 0.8.0 Attachments: Pig-1406.patch, Pig-1406_2.patch We had several users asking to be able to run arbitrary shell commands from within grunt. This would work similarly to fs command. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1381) Need a way for Pig to take an alternative property file
[ https://issues.apache.org/jira/browse/PIG-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866427#action_12866427 ] Hadoop QA commented on PIG-1381: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444240/PIG-1381-1.patch against trunk revision 943003. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/314/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/314/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/314/console This message is automatically generated. Need a way for Pig to take an alternative property file --- Key: PIG-1381 URL: https://issues.apache.org/jira/browse/PIG-1381 Project: Pig Issue Type: Improvement Components: impl Affects Versions: 0.7.0 Reporter: Daniel Dai Assignee: V.V.Chaitanya Krishna Fix For: 0.7.0, 0.8.0 Attachments: PIG-1381-1.patch, PIG-1381-2.patch Currently, Pig read the first ever pig.properties in CLASSPATH. Pig has a default pig.properties and if user have a different pig.properties, there will be a conflict since we can only read one. There are couple of ways to solve it: 1. Give a command line option for user to pass an additional property file 2. Change the name for default pig.properties to pig-default.properties, and user can give a pig.properties to override 3. Further, can we consider to use pig-default.xml/pig-site.xml, which seems to be more natural for hadoop community. If so, we shall provide backward compatibility to also read pig.properties, pig-cluster-hadoop-site.xml. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1410) Make PigServer can handle files with parameters
[
https://issues.apache.org/jira/browse/PIG-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12865722#action_12865722
]
Hadoop QA commented on PIG-1410:
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12444081/Pig_1410.patch
against trunk revision 941976.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 11 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/321/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/321/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/321/console
This message is automatically generated.
Make PigServer can handle files with parameters
---
Key: PIG-1410
URL: https://issues.apache.org/jira/browse/PIG-1410
Project: Pig
Issue Type: Improvement
Affects Versions: 0.8.0
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Fix For: 0.8.0
Attachments: Pig_1410.patch
The current PigServer only support handle files without parameters, this task
is to extend PigServer's API.
Add there public methods:
{code}
public void registerScript(String fileName, MapString,String params) //
put the parameters in the Map data datastructure
public void registerScript(String fileName, ListString paramsFiles) // put
the parameters in the files
public void registerScript(String fileName, MapString,String
params,ListString paramsFiles) // put the parameter both in files and Map,
the Map should override the parameter file if they have the same parameters
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1280) Add a pig-script-id to the JobConf of all jobs run in a pig-script
[ https://issues.apache.org/jira/browse/PIG-1280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12865962#action_12865962 ] Hadoop QA commented on PIG-1280: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444018/PIG-1280.patch against trunk revision 941976. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/311/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/311/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/311/console This message is automatically generated. Add a pig-script-id to the JobConf of all jobs run in a pig-script -- Key: PIG-1280 URL: https://issues.apache.org/jira/browse/PIG-1280 Project: Pig Issue Type: Improvement Components: impl Reporter: Arun C Murthy Assignee: Richard Ding Fix For: 0.8.0 Attachments: PIG-1280.patch It would be very useful for tools like gridmix if pig could add a 'pig-script-id' to all Map-Reduce jobs spawned by a single pig-script. Potentially we could use this to re-construct the DAG of jobs in gridmix and so on. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-566) Dump and store outputs do not match for PigStorage
[
https://issues.apache.org/jira/browse/PIG-566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866057#action_12866057
]
Hadoop QA commented on PIG-566:
---
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12444157/PIG-566.patch
against trunk revision 941976.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 15 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/312/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/312/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/312/console
This message is automatically generated.
Dump and store outputs do not match for PigStorage
--
Key: PIG-566
URL: https://issues.apache.org/jira/browse/PIG-566
Project: Pig
Issue Type: Bug
Affects Versions: 0.7.0, 0.8.0
Reporter: Santhosh Srinivasan
Assignee: Gianmarco De Francisci Morales
Priority: Minor
Fix For: 0.7.0, 0.8.0
Attachments: PIG-566.patch, PIG-566.patch, PIG-566.patch
The dump and store formats for PigStorage do not match for longs and floats.
{code}
grunt y = foreach x generate {(2985671202194220139L)};
grunt describe y;
y: {{(long)}}
grunt dump y;
({(2985671202194220139L)})
grunt store y into 'y';
grunt cat y
{(2985671202194220139)}
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1408) Annotate explain plans with aliases
[
https://issues.apache.org/jira/browse/PIG-1408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866061#action_12866061
]
Hadoop QA commented on PIG-1408:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12443996/PIG-1408.patch
against trunk revision 941976.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 45 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/323/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/323/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/323/console
This message is automatically generated.
Annotate explain plans with aliases
---
Key: PIG-1408
URL: https://issues.apache.org/jira/browse/PIG-1408
Project: Pig
Issue Type: Improvement
Affects Versions: 0.7.0
Reporter: Richard Ding
Assignee: Richard Ding
Fix For: 0.8.0
Attachments: PIG-1408.patch
PIG-1156 added aliases in Pig scripts to the corresponding LogicalOperators
and PhysicalOperators. The aliases in the operators, however, are not
displayed in the output created by the explain command.
Since a Pig script can generate many MR jobs, it will be helpful, for
debugging purposes, to annotate the explain output plans with aliases, so
that users can correlate the jobs with the statements in their scripts.
Here is an example: given the following script
{code}
A = load 'input';
B = group A by $0;
C = foreach B generate group, flatten(A);
explain C
{code}
The output without alias annotation is
{code}
MapReduce node 1-28
Map Plan
Local Rearrange[tuple]{bytearray}(false) - 1-22
| |
| Project[bytearray][0] - 1-23
|
|---Load(file:///test/input:org.apache.pig.builtin.PigStorage) - 1-19
Reduce Plan
Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-27
|
|---New For Each(false,true)[bag] - 1-26
| |
| Project[bytearray][0] - 1-24
| |
| Project[bag][1] - 1-25
|
|---Package[tuple]{bytearray} - 1-21
Global sort: false
{code}
While the output with alias annotation will be
{code}
MapReduce node 1-28
Map Plan
B: Local Rearrange[tuple]{bytearray}(false) - 1-22
| |
| Project[bytearray][0] - 1-23
|
|---A: Load(file:///test/input:org.apache.pig.builtin.PigStorage) -
1-19
Reduce Plan
C: Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-27
|
|---C: New For Each(false,true)[bag] - 1-26
| |
| Project[bytearray][0] - 1-24
| |
| Project[bag][1] - 1-25
|
|---B: Package[tuple]{bytearray} - 1-21
Global sort: false
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1406) Allow to run shell commands from grunt
[ https://issues.apache.org/jira/browse/PIG-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12865467#action_12865467 ] Hadoop QA commented on PIG-1406: +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444035/Pig-1406.patch against trunk revision 941976. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/319/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/319/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/319/console This message is automatically generated. Allow to run shell commands from grunt -- Key: PIG-1406 URL: https://issues.apache.org/jira/browse/PIG-1406 Project: Pig Issue Type: Improvement Affects Versions: 0.8.0 Reporter: Olga Natkovich Assignee: Jeff Zhang Fix For: 0.8.0 Attachments: Pig-1406.patch We had several users asking to be able to run arbitrary shell commands from within grunt. This would work similarly to fs command. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-566) Dump and store outputs do not match for PigStorage
[
https://issues.apache.org/jira/browse/PIG-566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12865497#action_12865497
]
Hadoop QA commented on PIG-566:
---
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12444036/PIG-566.patch
against trunk revision 941976.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/320/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/320/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/320/console
This message is automatically generated.
Dump and store outputs do not match for PigStorage
--
Key: PIG-566
URL: https://issues.apache.org/jira/browse/PIG-566
Project: Pig
Issue Type: Bug
Affects Versions: 0.7.0, 0.8.0
Reporter: Santhosh Srinivasan
Assignee: Gianmarco De Francisci Morales
Priority: Minor
Fix For: 0.7.0, 0.8.0
Attachments: PIG-566.patch, PIG-566.patch
The dump and store formats for PigStorage do not match for longs and floats.
{code}
grunt y = foreach x generate {(2985671202194220139L)};
grunt describe y;
y: {{(long)}}
grunt dump y;
({(2985671202194220139L)})
grunt store y into 'y';
grunt cat y
{(2985671202194220139)}
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1407) Logging starts before being configured
[ https://issues.apache.org/jira/browse/PIG-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12865166#action_12865166 ] Hadoop QA commented on PIG-1407: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12443944/PIG-1407.patch against trunk revision 941976. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/316/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/316/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/316/console This message is automatically generated. Logging starts before being configured -- Key: PIG-1407 URL: https://issues.apache.org/jira/browse/PIG-1407 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.7.0, 0.8.0 Reporter: Gianmarco De Francisci Morales Assignee: Gianmarco De Francisci Morales Priority: Trivial Fix For: 0.7.0, 0.8.0 Attachments: PIG-1407.patch Original Estimate: 0.02h Time Spent: 0.02h Remaining Estimate: 0h Pig's Main starts logging before log4j has been configured. This way logging messages are mixed with pig's output. $ cat script.pig A = LOAD 'input' AS (num:long, fruit:chararray); DUMP A; $ cat input 1 orange 2 apple 3 coconut 4 mango 5 grape 6 pear $ pig -x local prova.pig dump $ cat dump 0[main] INFO org.apache.pig.Main - Logging error messages to: /home/gianmarcodfm/pig-sbox/pig_127306353.log (1,orange) (2,apple) (3,coconut) (4,mango) (5,grape) (6,pear) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
