[jira] Commented: (PIG-913) Error in Pig script when grouping on chararray column
[
https://issues.apache.org/jira/browse/PIG-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12743264#action_12743264
]
Daniel Dai commented on PIG-913:
This release audit warning is caused by a new golden file. We cannot add
release audit notes to golden files.
Error in Pig script when grouping on chararray column
-
Key: PIG-913
URL: https://issues.apache.org/jira/browse/PIG-913
Project: Pig
Issue Type: Bug
Affects Versions: 0.4.0
Reporter: Viraj Bhat
Priority: Critical
Fix For: 0.4.0
Attachments: PIG-913-2.patch, PIG-913.patch
I have a very simple script which fails at parsetime due to the schema I
specified in the loader.
{code}
data = LOAD '/user/viraj/studenttab10k' AS (s:chararray);
dataSmall = limit data 100;
bb = GROUP dataSmall by $0;
dump bb;
{code}
=
2009-08-06 18:47:56,297 [main] INFO org.apache.pig.Main - Logging error
messages to: /homes/viraj/pig-svn/trunk/pig_1249609676296.log
09/08/06 18:47:56 INFO pig.Main: Logging error messages to:
/homes/viraj/pig-svn/trunk/pig_1249609676296.log
2009-08-06 18:47:56,459 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: hdfs://localhost:9000
09/08/06 18:47:56 INFO executionengine.HExecutionEngine: Connecting to hadoop
file system at: hdfs://localhost:9000
2009-08-06 18:47:56,694 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to map-reduce job tracker at: localhost:9001
09/08/06 18:47:56 INFO executionengine.HExecutionEngine: Connecting to
map-reduce job tracker at: localhost:9001
2009-08-06 18:47:57,008 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
1002: Unable to store alias bb
09/08/06 18:47:57 ERROR grunt.Grunt: ERROR 1002: Unable to store alias bb
Details at logfile: /homes/viraj/pig-svn/trunk/pig_1249609676296.log
=
=
Pig Stack Trace
---
ERROR 1002: Unable to store alias bb
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
open iterator for alias bb
at org.apache.pig.PigServer.openIterator(PigServer.java:481)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:531)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:190)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
at org.apache.pig.Main.main(Main.java:397)
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002:
Unable to store alias bb
at org.apache.pig.PigServer.store(PigServer.java:536)
at org.apache.pig.PigServer.openIterator(PigServer.java:464)
... 6 more
Caused by: java.lang.NullPointerException
at
org.apache.pig.impl.logicalLayer.LOCogroup.unsetSchema(LOCogroup.java:359)
at
org.apache.pig.impl.logicalLayer.optimizer.SchemaRemover.visit(SchemaRemover.java:64)
at
org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:335)
at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:46)
at
org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
at
org.apache.pig.impl.logicalLayer.optimizer.LogicalTransformer.rebuildSchemas(LogicalTransformer.java:67)
at
org.apache.pig.impl.logicalLayer.optimizer.LogicalOptimizer.optimize(LogicalOptimizer.java:187)
at org.apache.pig.PigServer.compileLp(PigServer.java:854)
at org.apache.pig.PigServer.compileLp(PigServer.java:791)
at org.apache.pig.PigServer.store(PigServer.java:509)
... 7 more
=
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (PIG-922) Logical optimizer: push up project
Logical optimizer: push up project -- Key: PIG-922 URL: https://issues.apache.org/jira/browse/PIG-922 Project: Pig Issue Type: New Feature Components: impl Affects Versions: 0.3.0 Reporter: Daniel Dai Fix For: 0.4.0 This is a continuation work of [PIG-697|https://issues.apache.org/jira/browse/PIG-697]. We need to add another rule to the logical optimizer: Push up project, ie, prune columns as early as possible. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-922) Logical optimizer: push up project
[ https://issues.apache.org/jira/browse/PIG-922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai reassigned PIG-922: -- Assignee: Daniel Dai Logical optimizer: push up project -- Key: PIG-922 URL: https://issues.apache.org/jira/browse/PIG-922 Project: Pig Issue Type: New Feature Components: impl Affects Versions: 0.3.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.4.0 This is a continuation work of [PIG-697|https://issues.apache.org/jira/browse/PIG-697]. We need to add another rule to the logical optimizer: Push up project, ie, prune columns as early as possible. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-845) PERFORMANCE: Merge Join
[ https://issues.apache.org/jira/browse/PIG-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pradeep Kamath updated PIG-845: --- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Patch committed to trunk. Thanks Ashutosh for this significant contribution! PERFORMANCE: Merge Join --- Key: PIG-845 URL: https://issues.apache.org/jira/browse/PIG-845 Project: Pig Issue Type: Improvement Reporter: Olga Natkovich Assignee: Ashutosh Chauhan Attachments: merge-join.patch Thsi join would work if the data for both tables is sorted on the join key. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-922) Logical optimizer: push up project
[ https://issues.apache.org/jira/browse/PIG-922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-922: --- Attachment: PIG-922-p1_0.patch Attach patch for phase 1. Logical optimizer: push up project -- Key: PIG-922 URL: https://issues.apache.org/jira/browse/PIG-922 Project: Pig Issue Type: New Feature Components: impl Affects Versions: 0.3.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.4.0 Attachments: PIG-922-p1_0.patch This is a continuation work of [PIG-697|https://issues.apache.org/jira/browse/PIG-697]. We need to add another rule to the logical optimizer: Push up project, ie, prune columns as early as possible. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-922) Logical optimizer: push up project
[ https://issues.apache.org/jira/browse/PIG-922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-922: --- Status: Patch Available (was: Open) Logical optimizer: push up project -- Key: PIG-922 URL: https://issues.apache.org/jira/browse/PIG-922 Project: Pig Issue Type: New Feature Components: impl Affects Versions: 0.3.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.4.0 Attachments: PIG-922-p1_0.patch This is a continuation work of [PIG-697|https://issues.apache.org/jira/browse/PIG-697]. We need to add another rule to the logical optimizer: Push up project, ie, prune columns as early as possible. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Build failed in Hudson: Pig-Patch-minerva.apache.org #164
See http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/164/changes Changes: [pradeepkth] PIG-845: PERFORMANCE: Merge Join (ashutoshc via pradeepkth) - deleting renamed file - MRStreamHandler.java [pradeepkth] PIG-845: PERFORMANCE: Merge Join (ashutoshc via pradeepkth) [daijy] PIG-913: Error in Pig script when grouping on chararray column -- [...truncated 111633 lines...] [exec] [junit] [exec] [junit] 09/08/14 21:35:32 INFO mapReduceLayer.JobControlCompiler: Setting up single store job [exec] [junit] 09/08/14 21:35:32 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. [exec] [junit] 09/08/14 21:35:33 INFO dfs.StateChange: BLOCK* NameSystem.allocateBlock: /tmp/hadoop-hudson/mapred/system/job_200908142134_0002/job.jar. blk_-6402472781047644060_1012 [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: Receiving block blk_-6402472781047644060_1012 src: /127.0.0.1:44681 dest: /127.0.0.1:40670 [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: Receiving block blk_-6402472781047644060_1012 src: /127.0.0.1:39714 dest: /127.0.0.1:53345 [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: Receiving block blk_-6402472781047644060_1012 src: /127.0.0.1:41010 dest: /127.0.0.1:41033 [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: Received block blk_-6402472781047644060_1012 of size 1497453 from /127.0.0.1 [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: PacketResponder 0 for block blk_-6402472781047644060_1012 terminating [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: Received block blk_-6402472781047644060_1012 of size 1497453 from /127.0.0.1 [exec] [junit] 09/08/14 21:35:33 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:41033 is added to blk_-6402472781047644060_1012 size 1497453 [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: PacketResponder 1 for block blk_-6402472781047644060_1012 terminating [exec] [junit] 09/08/14 21:35:33 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:53345 is added to blk_-6402472781047644060_1012 size 1497453 [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: Received block blk_-6402472781047644060_1012 of size 1497453 from /127.0.0.1 [exec] [junit] 09/08/14 21:35:33 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:40670 is added to blk_-6402472781047644060_1012 size 1497453 [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: PacketResponder 2 for block blk_-6402472781047644060_1012 terminating [exec] [junit] 09/08/14 21:35:33 INFO fs.FSNamesystem: Increasing replication for file /tmp/hadoop-hudson/mapred/system/job_200908142134_0002/job.jar. New replication is 2 [exec] [junit] 09/08/14 21:35:33 INFO fs.FSNamesystem: Reducing replication for file /tmp/hadoop-hudson/mapred/system/job_200908142134_0002/job.jar. New replication is 2 [exec] [junit] 09/08/14 21:35:33 INFO dfs.StateChange: BLOCK* NameSystem.allocateBlock: /tmp/hadoop-hudson/mapred/system/job_200908142134_0002/job.split. blk_5455499385688750307_1013 [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: Receiving block blk_5455499385688750307_1013 src: /127.0.0.1:39716 dest: /127.0.0.1:53345 [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: Receiving block blk_5455499385688750307_1013 src: /127.0.0.1:44685 dest: /127.0.0.1:40670 [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: Receiving block blk_5455499385688750307_1013 src: /127.0.0.1:39881 dest: /127.0.0.1:59910 [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: Received block blk_5455499385688750307_1013 of size 1837 from /127.0.0.1 [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: PacketResponder 0 for block blk_5455499385688750307_1013 terminating [exec] [junit] 09/08/14 21:35:33 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:59910 is added to blk_5455499385688750307_1013 size 1837 [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: Received block blk_5455499385688750307_1013 of size 1837 from /127.0.0.1 [exec] [junit] 09/08/14 21:35:33 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:40670 is added to blk_5455499385688750307_1013 size 1837 [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: PacketResponder 1 for block blk_5455499385688750307_1013 terminating [exec] [junit] 09/08/14 21:35:33 INFO dfs.DataNode: Received block blk_5455499385688750307_1013 of size 1837 from /127.0.0.1 [exec] [junit] 09/08/14 21:35:33 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:53345 is added to
[jira] Commented: (PIG-892) Make COUNT and AVG deal with nulls accordingly with SQL standar
[ https://issues.apache.org/jira/browse/PIG-892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12743467#action_12743467 ] Daniel Dai commented on PIG-892: +1 Make COUNT and AVG deal with nulls accordingly with SQL standar --- Key: PIG-892 URL: https://issues.apache.org/jira/browse/PIG-892 Project: Pig Issue Type: Improvement Affects Versions: 0.3.0 Reporter: Olga Natkovich Assignee: Olga Natkovich Fix For: 0.4.0 Attachments: PIG-892.patch, PIG-892_v2.patch, PIG-892_v3.patch both COUNT and AVG need to ignore nulls. Also add COUNT_STAR to match COUNT(*) in SQL -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-922) Logical optimizer: push up project
[ https://issues.apache.org/jira/browse/PIG-922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-922: --- Attachment: PIG-922-p1_1.patch Address comments by Hudson. Logical optimizer: push up project -- Key: PIG-922 URL: https://issues.apache.org/jira/browse/PIG-922 Project: Pig Issue Type: New Feature Components: impl Affects Versions: 0.3.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.4.0 Attachments: PIG-922-p1_0.patch, PIG-922-p1_1.patch This is a continuation work of [PIG-697|https://issues.apache.org/jira/browse/PIG-697]. We need to add another rule to the logical optimizer: Push up project, ie, prune columns as early as possible. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-922) Logical optimizer: push up project
[ https://issues.apache.org/jira/browse/PIG-922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-922: --- Status: Open (was: Patch Available) Logical optimizer: push up project -- Key: PIG-922 URL: https://issues.apache.org/jira/browse/PIG-922 Project: Pig Issue Type: New Feature Components: impl Affects Versions: 0.3.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.4.0 Attachments: PIG-922-p1_0.patch, PIG-922-p1_1.patch This is a continuation work of [PIG-697|https://issues.apache.org/jira/browse/PIG-697]. We need to add another rule to the logical optimizer: Push up project, ie, prune columns as early as possible. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-914) Change the PIG hbase interface to use bytes along with strings
[ https://issues.apache.org/jira/browse/PIG-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12743493#action_12743493 ] Daniel Dai commented on PIG-914: Hi, Alex, Are you able to assign the issue to yourself through Jira? Same to Pig-915, Pig-916. Change the PIG hbase interface to use bytes along with strings -- Key: PIG-914 URL: https://issues.apache.org/jira/browse/PIG-914 Project: Pig Issue Type: Improvement Reporter: Alex Newman Priority: Minor Currently start rows, tablenames, column names are all strings, and HBase supports bytes we might want to change the Pig interface to support bytes along with strings. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Hudson build is back to normal: Pig-Patch-minerva.apache.org #165
See http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/165/
[jira] Commented: (PIG-922) Logical optimizer: push up project
[ https://issues.apache.org/jira/browse/PIG-922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12743498#action_12743498 ] Hadoop QA commented on PIG-922: --- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416618/PIG-922-p1_1.patch against trunk revision 804310. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/165/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/165/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/165/console This message is automatically generated. Logical optimizer: push up project -- Key: PIG-922 URL: https://issues.apache.org/jira/browse/PIG-922 Project: Pig Issue Type: New Feature Components: impl Affects Versions: 0.3.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.4.0 Attachments: PIG-922-p1_0.patch, PIG-922-p1_1.patch This is a continuation work of [PIG-697|https://issues.apache.org/jira/browse/PIG-697]. We need to add another rule to the logical optimizer: Push up project, ie, prune columns as early as possible. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-824) SQL interface for Pig
[ https://issues.apache.org/jira/browse/PIG-824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated PIG-824: -- Attachment: SQL_IN_PIG.html PIG-824.1.patch PIG-824.binfiles.tar.gz PIG-824.binfiles.tar.gz - contains libs that it depends on PIG-824.1.patch - patch SQL_IN_PIG.html - (brief) document JFlex.jar has not been included because it covered by GPL. It will have to be downloaded to lib dir for building with the patch. In future Ivy will be setup to download it . SQL interface for Pig - Key: PIG-824 URL: https://issues.apache.org/jira/browse/PIG-824 Project: Pig Issue Type: New Feature Reporter: Olga Natkovich Attachments: PIG-824.1.patch, PIG-824.binfiles.tar.gz, SQL_IN_PIG.html In the last 18 month PigLatin has gained significant popularity within the open source community. Many users like its data flow model, its rich type system and its ability to work with any data available on HDFS or outside. We have also heard from many users that having Pig speak SQL would bring many more users. Having a single system that exports multiple interfaces is a big advantage as it guarantees consistent semantics, custom code reuse, and reduces the amount of maintenance. This is especially relevant for project where using both interfaces for different parts of the system is relevant. For instance, in a data warehousing system, you would have ETL component that brings data into the warehouse and a component that analyzes the data and produces reports. PigLatin is uniquely suited for ETL processing while SQL might be a better fit for report generation. To start, it would make sense to implement a subset of SQL92 standard and to be as much as possible standard compliant. This would include all the standard constructs: select, from, where, group-by + having, order by, limit, join (inner + outer). Several extensions such as support for pig's UDFs and possibly streaming, multiquery and support for pig's complex types would be helpful. This work is dependent on metadata support outlined in https://issues.apache.org/jira/browse/PIG-823 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
