[jira] Updated: (PIG-849) Local engine loses records in splits
[ https://issues.apache.org/jira/browse/PIG-849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated PIG-849: --- Status: Patch Available (was: Open) Local engine loses records in splits Key: PIG-849 URL: https://issues.apache.org/jira/browse/PIG-849 Project: Pig Issue Type: Bug Reporter: Gunther Hagleitner Attachments: local_engine.patch When there is a split in the physical plan records can be dropped in certain circumstances. The local split operator puts all records in a databag and turns over iterators to the POSplitOutput operators. The problem is that the local split also adds STATUS_NULL records to the bag. That will cause the databag's iterator to prematurely return false on the hasNext call (so a STATUS_NULL becomes a STATUS_EOP in the split output operators). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-753) Provide support for UDFs without parameters
[ https://issues.apache.org/jira/browse/PIG-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12719501#action_12719501 ] Hadoop QA commented on PIG-753: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12410538/Pig_753_Patch.txt against trunk revision 784333. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 162 release audit warnings (more than the trunk's current 160 warnings). -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/80/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/80/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/80/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/80/console This message is automatically generated. Provide support for UDFs without parameters --- Key: PIG-753 URL: https://issues.apache.org/jira/browse/PIG-753 Project: Pig Issue Type: Improvement Affects Versions: 0.3.0 Reporter: Jeff Zhang Fix For: 0.3.0 Attachments: Pig_753_Patch.txt Pig do not support UDF without parameters, it force me provide a parameter. like the following statement: B = FOREACH A GENERATE bagGenerator(); this will generate error. I have to provide a parameter like following B = FOREACH A GENERATE bagGenerator($0); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Build failed in Hudson: Pig-Patch-minerva.apache.org #80
See http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/80/changes Changes: [pradeepkth] MultiQuery optimization in some cases has an issue when there is a split in the map plan (pradeepkth) -- [...truncated 93411 lines...] [exec] [junit] 09/06/15 03:37:09 INFO dfs.DataNode: PacketResponder 2 for block blk_2882322542475422864_1011 terminating [exec] [junit] 09/06/15 03:37:09 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:42296 is added to blk_2882322542475422864_1011 size 6 [exec] [junit] 09/06/15 03:37:09 INFO executionengine.HExecutionEngine: Connecting to hadoop file system at: hdfs://localhost:48714 [exec] [junit] 09/06/15 03:37:09 INFO executionengine.HExecutionEngine: Connecting to map-reduce job tracker at: localhost:36097 [exec] [junit] 09/06/15 03:37:09 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size before optimization: 1 [exec] [junit] 09/06/15 03:37:09 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size after optimization: 1 [exec] [junit] 09/06/15 03:37:09 INFO dfs.DataNode: Deleting block blk_-6977836447127840030_1006 file dfs/data/data2/current/blk_-6977836447127840030 [exec] [junit] 09/06/15 03:37:09 INFO dfs.DataNode: Deleting block blk_-128469568138717644_1004 file dfs/data/data2/current/blk_-128469568138717644 [exec] [junit] 09/06/15 03:37:09 INFO dfs.DataNode: Deleting block blk_6613276072194154064_1005 file dfs/data/data1/current/blk_6613276072194154064 [exec] [junit] 09/06/15 03:37:10 INFO dfs.StateChange: BLOCK* ask 127.0.0.1:42296 to delete blk_-128469568138717644_1004 blk_6613276072194154064_1005 [exec] [junit] 09/06/15 03:37:10 INFO dfs.StateChange: BLOCK* ask 127.0.0.1:59576 to delete blk_-6977836447127840030_1006 [exec] [junit] 09/06/15 03:37:10 INFO mapReduceLayer.JobControlCompiler: Setting up single store job [exec] [junit] 09/06/15 03:37:10 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. [exec] [junit] 09/06/15 03:37:10 INFO dfs.StateChange: BLOCK* NameSystem.allocateBlock: /tmp/hadoop-hudson/mapred/system/job_200906150336_0002/job.jar. blk_-6848100406696102585_1012 [exec] [junit] 09/06/15 03:37:10 INFO dfs.DataNode: Receiving block blk_-6848100406696102585_1012 src: /127.0.0.1:44418 dest: /127.0.0.1:42296 [exec] [junit] 09/06/15 03:37:10 INFO dfs.DataNode: Receiving block blk_-6848100406696102585_1012 src: /127.0.0.1:39726 dest: /127.0.0.1:59576 [exec] [junit] 09/06/15 03:37:10 INFO dfs.DataNode: Receiving block blk_-6848100406696102585_1012 src: /127.0.0.1:49093 dest: /127.0.0.1:37409 [exec] [junit] 09/06/15 03:37:10 INFO dfs.DataNode: Received block blk_-6848100406696102585_1012 of size 1413553 from /127.0.0.1 [exec] [junit] 09/06/15 03:37:10 INFO dfs.DataNode: PacketResponder 0 for block blk_-6848100406696102585_1012 terminating [exec] [junit] 09/06/15 03:37:10 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:37409 is added to blk_-6848100406696102585_1012 size 1413553 [exec] [junit] 09/06/15 03:37:10 INFO dfs.DataNode: Received block blk_-6848100406696102585_1012 of size 1413553 from /127.0.0.1 [exec] [junit] 09/06/15 03:37:10 INFO dfs.DataNode: PacketResponder 1 for block blk_-6848100406696102585_1012 terminating [exec] [junit] 09/06/15 03:37:10 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:59576 is added to blk_-6848100406696102585_1012 size 1413553 [exec] [junit] 09/06/15 03:37:10 INFO dfs.DataNode: Received block blk_-6848100406696102585_1012 of size 1413553 from /127.0.0.1 [exec] [junit] 09/06/15 03:37:10 INFO dfs.DataNode: PacketResponder 2 for block blk_-6848100406696102585_1012 terminating [exec] [junit] 09/06/15 03:37:10 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:42296 is added to blk_-6848100406696102585_1012 size 1413553 [exec] [junit] 09/06/15 03:37:10 INFO fs.FSNamesystem: Increasing replication for file /tmp/hadoop-hudson/mapred/system/job_200906150336_0002/job.jar. New replication is 2 [exec] [junit] 09/06/15 03:37:10 INFO fs.FSNamesystem: Reducing replication for file /tmp/hadoop-hudson/mapred/system/job_200906150336_0002/job.jar. New replication is 2 [exec] [junit] 09/06/15 03:37:10 INFO dfs.StateChange: BLOCK* NameSystem.allocateBlock: /tmp/hadoop-hudson/mapred/system/job_200906150336_0002/job.split. blk_-3678369822503354886_1013 [exec] [junit] 09/06/15 03:37:10 INFO dfs.DataNode: Receiving block blk_-3678369822503354886_1013 src: /127.0.0.1:39728 dest: /127.0.0.1:59576 [exec] [junit] 09/06/15 03:37:10 INFO dfs.DataNode: Receiving block
Build failed in Hudson: Pig-Patch-minerva.apache.org #81
See http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/81/ -- [...truncated 93385 lines...] [exec] [junit] 09/06/15 09:29:36 INFO dfs.DataNode: Receiving block blk_2227271342495886703_1011 src: /127.0.0.1:38667 dest: /127.0.0.1:51194 [exec] [junit] 09/06/15 09:29:36 INFO dfs.DataNode: Receiving block blk_2227271342495886703_1011 src: /127.0.0.1:51057 dest: /127.0.0.1:50635 [exec] [junit] 09/06/15 09:29:36 INFO dfs.DataNode: Received block blk_2227271342495886703_1011 of size 6 from /127.0.0.1 [exec] [junit] 09/06/15 09:29:36 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:50635 is added to blk_2227271342495886703_1011 size 6 [exec] [junit] 09/06/15 09:29:36 INFO dfs.DataNode: PacketResponder 0 for block blk_2227271342495886703_1011 terminating [exec] [junit] 09/06/15 09:29:36 INFO dfs.DataNode: Received block blk_2227271342495886703_1011 of size 6 from /127.0.0.1 [exec] [junit] 09/06/15 09:29:36 INFO dfs.DataNode: PacketResponder 1 for block blk_2227271342495886703_1011 terminating [exec] [junit] 09/06/15 09:29:36 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:51194 is added to blk_2227271342495886703_1011 size 6 [exec] [junit] 09/06/15 09:29:36 INFO dfs.DataNode: Received block blk_2227271342495886703_1011 of size 6 from /127.0.0.1 [exec] [junit] 09/06/15 09:29:36 INFO dfs.DataNode: PacketResponder 2 for block blk_2227271342495886703_1011 terminating [exec] [junit] 09/06/15 09:29:36 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:41079 is added to blk_2227271342495886703_1011 size 6 [exec] [junit] 09/06/15 09:29:36 INFO executionengine.HExecutionEngine: Connecting to hadoop file system at: hdfs://localhost:58176 [exec] [junit] 09/06/15 09:29:36 INFO executionengine.HExecutionEngine: Connecting to map-reduce job tracker at: localhost:35554 [exec] [junit] 09/06/15 09:29:36 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size before optimization: 1 [exec] [junit] 09/06/15 09:29:36 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size after optimization: 1 [exec] [junit] 09/06/15 09:29:37 INFO dfs.StateChange: BLOCK* ask 127.0.0.1:51194 to delete blk_-1265025192484193669_1005 blk_-6491838709978587799_1006 blk_4503059020361815811_1004 [exec] [junit] 09/06/15 09:29:37 INFO dfs.StateChange: BLOCK* ask 127.0.0.1:50635 to delete blk_-6491838709978587799_1006 [exec] [junit] 09/06/15 09:29:37 INFO mapReduceLayer.JobControlCompiler: Setting up single store job [exec] [junit] 09/06/15 09:29:37 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. [exec] [junit] 09/06/15 09:29:37 INFO dfs.StateChange: BLOCK* NameSystem.allocateBlock: /tmp/hadoop-hudson/mapred/system/job_200906150928_0002/job.jar. blk_5849256845639308233_1012 [exec] [junit] 09/06/15 09:29:37 INFO dfs.DataNode: Receiving block blk_5849256845639308233_1012 src: /127.0.0.1:36663 dest: /127.0.0.1:41079 [exec] [junit] 09/06/15 09:29:37 INFO dfs.DataNode: Receiving block blk_5849256845639308233_1012 src: /127.0.0.1:51059 dest: /127.0.0.1:50635 [exec] [junit] 09/06/15 09:29:37 INFO dfs.DataNode: Receiving block blk_5849256845639308233_1012 src: /127.0.0.1:38671 dest: /127.0.0.1:51194 [exec] [junit] 09/06/15 09:29:37 INFO dfs.DataNode: Received block blk_5849256845639308233_1012 of size 1413551 from /127.0.0.1 [exec] [junit] 09/06/15 09:29:37 INFO dfs.DataNode: PacketResponder 0 for block blk_5849256845639308233_1012 terminating [exec] [junit] 09/06/15 09:29:37 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:51194 is added to blk_5849256845639308233_1012 size 1413551 [exec] [junit] 09/06/15 09:29:37 INFO dfs.DataNode: Received block blk_5849256845639308233_1012 of size 1413551 from /127.0.0.1 [exec] [junit] 09/06/15 09:29:37 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:50635 is added to blk_5849256845639308233_1012 size 1413551 [exec] [junit] 09/06/15 09:29:37 INFO dfs.DataNode: Received block blk_5849256845639308233_1012 of size 1413551 from /127.0.0.1 [exec] [junit] 09/06/15 09:29:37 INFO dfs.DataNode: PacketResponder 1 for block blk_5849256845639308233_1012 terminating [exec] [junit] 09/06/15 09:29:37 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:41079 is added to blk_5849256845639308233_1012 size 1413551 [exec] [junit] 09/06/15 09:29:37 INFO dfs.DataNode: PacketResponder 2 for block blk_5849256845639308233_1012 terminating [exec] [junit] 09/06/15 09:29:37 INFO
[jira] Commented: (PIG-849) Local engine loses records in splits
[ https://issues.apache.org/jira/browse/PIG-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12719611#action_12719611 ] Hadoop QA commented on PIG-849: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12410627/local_engine.patch against trunk revision 784333. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/81/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/81/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/81/console This message is automatically generated. Local engine loses records in splits Key: PIG-849 URL: https://issues.apache.org/jira/browse/PIG-849 Project: Pig Issue Type: Bug Reporter: Gunther Hagleitner Attachments: local_engine.patch When there is a split in the physical plan records can be dropped in certain circumstances. The local split operator puts all records in a databag and turns over iterators to the POSplitOutput operators. The problem is that the local split also adds STATUS_NULL records to the bag. That will cause the databag's iterator to prematurely return false on the hasNext call (so a STATUS_NULL becomes a STATUS_EOP in the split output operators). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-753) Provide support for UDFs without parameters
[ https://issues.apache.org/jira/browse/PIG-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12719616#action_12719616 ] Alan Gates commented on PIG-753: The test failures are in bzip tests, which I doubt are affected by this. I'll run them myself with the patch to check. But the release audit warnings are real. The two new test files need to have apache headers put on them. You can grab the header from any of the other java files. Provide support for UDFs without parameters --- Key: PIG-753 URL: https://issues.apache.org/jira/browse/PIG-753 Project: Pig Issue Type: Improvement Affects Versions: 0.3.0 Reporter: Jeff Zhang Fix For: 0.3.0 Attachments: Pig_753_Patch.txt Pig do not support UDF without parameters, it force me provide a parameter. like the following statement: B = FOREACH A GENERATE bagGenerator(); this will generate error. I have to provide a parameter like following B = FOREACH A GENERATE bagGenerator($0); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-753) Provide support for UDFs without parameters
[ https://issues.apache.org/jira/browse/PIG-753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated PIG-753: --- Status: Open (was: Patch Available) Provide support for UDFs without parameters --- Key: PIG-753 URL: https://issues.apache.org/jira/browse/PIG-753 Project: Pig Issue Type: Improvement Affects Versions: 0.3.0 Reporter: Jeff Zhang Fix For: 0.3.0 Attachments: Pig_753_Patch.txt Pig do not support UDF without parameters, it force me provide a parameter. like the following statement: B = FOREACH A GENERATE bagGenerator(); this will generate error. I have to provide a parameter like following B = FOREACH A GENERATE bagGenerator($0); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-728) All backend error messages must be logged to preserve the original error messages
[ https://issues.apache.org/jira/browse/PIG-728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12719617#action_12719617 ] Santhosh Srinivasan commented on PIG-728: - In addition, when the framework is not able to parse the error message, the message should be annotated as such. Extraneous details like Unable to recreate exception, Cannot create exception from empty string, etc should not be communicated to the user. These messages reflect internal workings of the error handling framework and do not add value to the user. All backend error messages must be logged to preserve the original error messages - Key: PIG-728 URL: https://issues.apache.org/jira/browse/PIG-728 Project: Pig Issue Type: Bug Affects Versions: 0.2.1 Reporter: Santhosh Srinivasan Assignee: Santhosh Srinivasan Priority: Minor Fix For: 0.2.1 The current error handling framework logs backend error messages only when Pig is not able to parse the error message. Instead, Pig should log the backend error message irrespective of Pig's ability to parse backend error messages. On a side note, the use of instantiateFuncFromSpec in Launcher.java is not consistent and should avoid the use of class_name + ( + string_constructor_args + ). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
PigPen Source
I want to play with PigPen, but although I can find the patches here: https://issues.apache.org/jira/browse/PIG-366 on the Jira, I cannot find the source in trunk/contrib/pigpen, or in any path in any branch. Where does the PigPen source reside? Does it exist only as a patch? Russell Jurney rjur...@cloudstenography.com
Re: PigPen Source
It has not yet been integrated into contrib because it requires the eclipse libraries to build, and those weren't integrated. The ivy stuff used by pig's build should be configured to pick up the appropriate eclipse jars so that this can be added to contrib. Alan. On Jun 15, 2009, at 12:09 PM, Russell Jurney wrote: I want to play with PigPen, but although I can find the patches here: https://issues.apache.org/jira/browse/PIG-366 on the Jira, I cannot find the source in trunk/contrib/pigpen, or in any path in any branch. Where does the PigPen source reside? Does it exist only as a patch? Russell Jurney rjur...@cloudstenography.com
[jira] Commented: (PIG-605) Better explain and console output
[ https://issues.apache.org/jira/browse/PIG-605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12719731#action_12719731 ] Santhosh Srinivasan commented on PIG-605: - In addition, it will be very useful for users if the plans have the line numbers of the pig script that resulted in the final plan. For example, the plan should state Line number 10, 12, 14 to help users work backwards from the plan to the original script. Better explain and console output - Key: PIG-605 URL: https://issues.apache.org/jira/browse/PIG-605 Project: Pig Issue Type: Improvement Components: grunt Reporter: Yiping Han It would be nice if when we explain the script, the corresponding mapred jobs can be explicitly mark out in a neat way. While we execute the script, the console output could print the name and url of the corresponding hadoop jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-797) Limit with ORDER BY producing wrong results
[ https://issues.apache.org/jira/browse/PIG-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-797: --- Attachment: PIG-797.patch For the limited sort case, the extra limit map-reduce operator introduced in [PIG-364|http://issues.apache.org/jira/browse/PIG-364] should use the same output key as the previous sort map-reduce operator. Limit with ORDER BY producing wrong results --- Key: PIG-797 URL: https://issues.apache.org/jira/browse/PIG-797 Project: Pig Issue Type: Bug Affects Versions: 0.2.0 Reporter: Olga Natkovich Attachments: PIG-797.patch Query: A = load 'studenttab10k' as (name, age, gpa); B = group A by name; C = foreach B generate group, SUM(A.gpa) as rev; D = order C by rev; E = limit D 10; dump E; Output: (alice king,31.7) (alice laertes,26.453) (alice thompson,25.867) (alice van buren,23.59) (bob allen,19.902) (bob ichabod,29.0) (bob king,28.454) (bob miller,10.28) (bob underhill,28.137) (bob van buren,25.992) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-797) Limit with ORDER BY producing wrong results
[ https://issues.apache.org/jira/browse/PIG-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-797: --- Fix Version/s: site Status: Patch Available (was: Open) Limit with ORDER BY producing wrong results --- Key: PIG-797 URL: https://issues.apache.org/jira/browse/PIG-797 Project: Pig Issue Type: Bug Affects Versions: 0.2.0 Reporter: Olga Natkovich Fix For: site Attachments: PIG-797.patch Query: A = load 'studenttab10k' as (name, age, gpa); B = group A by name; C = foreach B generate group, SUM(A.gpa) as rev; D = order C by rev; E = limit D 10; dump E; Output: (alice king,31.7) (alice laertes,26.453) (alice thompson,25.867) (alice van buren,23.59) (bob allen,19.902) (bob ichabod,29.0) (bob king,28.454) (bob miller,10.28) (bob underhill,28.137) (bob van buren,25.992) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-849) Local engine loses records in splits
[ https://issues.apache.org/jira/browse/PIG-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12719790#action_12719790 ] Olga Natkovich commented on PIG-849: I rerun the tests manually and they all passed. Looks like an issue with automated patch testing as we so another case of it this morning. Local engine loses records in splits Key: PIG-849 URL: https://issues.apache.org/jira/browse/PIG-849 Project: Pig Issue Type: Bug Reporter: Gunther Hagleitner Attachments: local_engine.patch When there is a split in the physical plan records can be dropped in certain circumstances. The local split operator puts all records in a databag and turns over iterators to the POSplitOutput operators. The problem is that the local split also adds STATUS_NULL records to the bag. That will cause the databag's iterator to prematurely return false on the hasNext call (so a STATUS_NULL becomes a STATUS_EOP in the split output operators). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-797) Limit with ORDER BY producing wrong results
[ https://issues.apache.org/jira/browse/PIG-797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12719837#action_12719837 ] Hadoop QA commented on PIG-797: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12410713/PIG-797.patch against trunk revision 784333. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/82/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/82/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/82/console This message is automatically generated. Limit with ORDER BY producing wrong results --- Key: PIG-797 URL: https://issues.apache.org/jira/browse/PIG-797 Project: Pig Issue Type: Bug Affects Versions: 0.2.0 Reporter: Olga Natkovich Fix For: site Attachments: PIG-797.patch Query: A = load 'studenttab10k' as (name, age, gpa); B = group A by name; C = foreach B generate group, SUM(A.gpa) as rev; D = order C by rev; E = limit D 10; dump E; Output: (alice king,31.7) (alice laertes,26.453) (alice thompson,25.867) (alice van buren,23.59) (bob allen,19.902) (bob ichabod,29.0) (bob king,28.454) (bob miller,10.28) (bob underhill,28.137) (bob van buren,25.992) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (PIG-851) Map type used as return type in UDFs not recognized at all times
Map type used as return type in UDFs not recognized at all times
Key: PIG-851
URL: https://issues.apache.org/jira/browse/PIG-851
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.3.0
Reporter: Santhosh Srinivasan
Fix For: 0.3.0
When an UDF returns a map and the outputSchema method is not overridden, Pig
does not figure out the data type. As a result, the type is set to unknown
resulting in run time failure. An example script and UDF follow
{code}
public class mapUDF extends EvalFuncMapObject, Object {
@Override
public MapObject, Object exec(Tuple input) throws IOException {
return new HashMapObject, Object();
}
//Note that the outputSchema method is commented out
/*
@Override
public Schema outputSchema(Schema input) {
try {
return new Schema(new Schema.FieldSchema(null, null, DataType.MAP));
} catch (FrontendException e) {
return null;
}
}
*/
{code}
{code}
grunt a = load 'student_tab.data';
grunt b = foreach a generate EXPLODE(1);
grunt describe b;
b: {Unknown}
grunt dump b;
2009-06-15 17:59:01,776 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed!
2009-06-15 17:59:01,781 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
2080: Foreach currently does not handle type Unknown
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-851) Map type used as return type in UDFs not recognized at all times
[
https://issues.apache.org/jira/browse/PIG-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12719886#action_12719886
]
Santhosh Srinivasan commented on PIG-851:
-
A workaround for this issue is to override the outputSchema method and return
the appropriate schema.
Map type used as return type in UDFs not recognized at all times
Key: PIG-851
URL: https://issues.apache.org/jira/browse/PIG-851
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.3.0
Reporter: Santhosh Srinivasan
Fix For: 0.3.0
When an UDF returns a map and the outputSchema method is not overridden, Pig
does not figure out the data type. As a result, the type is set to unknown
resulting in run time failure. An example script and UDF follow
{code}
public class mapUDF extends EvalFuncMapObject, Object {
@Override
public MapObject, Object exec(Tuple input) throws IOException {
return new HashMapObject, Object();
}
//Note that the outputSchema method is commented out
/*
@Override
public Schema outputSchema(Schema input) {
try {
return new Schema(new Schema.FieldSchema(null, null,
DataType.MAP));
} catch (FrontendException e) {
return null;
}
}
*/
{code}
{code}
grunt a = load 'student_tab.data';
grunt b = foreach a generate EXPLODE(1);
grunt describe b;
b: {Unknown}
grunt dump b;
2009-06-15 17:59:01,776 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed!
2009-06-15 17:59:01,781 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
2080: Foreach currently does not handle type Unknown
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (PIG-852) pig -version or pig -help returns exit code of 1
pig -version or pig -help returns exit code of 1
Key: PIG-852
URL: https://issues.apache.org/jira/browse/PIG-852
Project: Pig
Issue Type: Bug
Components: grunt
Affects Versions: 0.3.0
Environment: All
Reporter: Milind Bhandarkar
Assignee: Milind Bhandarkar
{code}
java -jar pig.jar -x local [-version|-help]
{code}
returns an exit code of 1 to the shell.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-852) pig -version or pig -help returns exit code of 1
[
https://issues.apache.org/jira/browse/PIG-852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Milind Bhandarkar updated PIG-852:
--
Attachment: rc.patch
pig -version or pig -help returns exit code of 1
Key: PIG-852
URL: https://issues.apache.org/jira/browse/PIG-852
Project: Pig
Issue Type: Bug
Components: grunt
Affects Versions: 0.3.0
Environment: All
Reporter: Milind Bhandarkar
Assignee: Milind Bhandarkar
Attachments: rc.patch
{code}
java -jar pig.jar -x local [-version|-help]
{code}
returns an exit code of 1 to the shell.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-852) pig -version or pig -help returns exit code of 1
[
https://issues.apache.org/jira/browse/PIG-852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Milind Bhandarkar updated PIG-852:
--
Status: Patch Available (was: Open)
Making patch available. Manual testing done.
pig -version or pig -help returns exit code of 1
Key: PIG-852
URL: https://issues.apache.org/jira/browse/PIG-852
Project: Pig
Issue Type: Bug
Components: grunt
Affects Versions: 0.3.0
Environment: All
Reporter: Milind Bhandarkar
Assignee: Milind Bhandarkar
Attachments: rc.patch
{code}
java -jar pig.jar -x local [-version|-help]
{code}
returns an exit code of 1 to the shell.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
