[jira] Updated: (PIG-938) Pig Docs for 0.4.0
[ https://issues.apache.org/jira/browse/PIG-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Corinne Chandel updated PIG-938: Attachment: PIG-938.patch Pig Docs for 0.4.0 -- Key: PIG-938 URL: https://issues.apache.org/jira/browse/PIG-938 Project: Pig Issue Type: Task Components: documentation Affects Versions: 0.4.0 Reporter: Corinne Chandel Priority: Minor Attachments: PIG-938.patch Pig docs for 0.4.0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-938) Pig Docs for 0.4.0
[ https://issues.apache.org/jira/browse/PIG-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Corinne Chandel updated PIG-938: Status: Patch Available (was: Open) Apply patch to this branch: http://svn.apache.org/repos/asf/hadoop/pig/trunk Note: No new test code necessary; changes to documentation only. Pig Docs for 0.4.0 -- Key: PIG-938 URL: https://issues.apache.org/jira/browse/PIG-938 Project: Pig Issue Type: Task Components: documentation Affects Versions: 0.4.0 Reporter: Corinne Chandel Priority: Minor Attachments: PIG-938.patch Pig docs for 0.4.0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-933) broken link in pig-latin reference manual to hadoop file glob pattern documentation
[ https://issues.apache.org/jira/browse/PIG-933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12749187#action_12749187 ] Corinne Chandel commented on PIG-933: - Link fixed in 0.4.0 docs (jira PIG-938). broken link in pig-latin reference manual to hadoop file glob pattern documentation --- Key: PIG-933 URL: https://issues.apache.org/jira/browse/PIG-933 Project: Pig Issue Type: Bug Components: documentation Reporter: Thejas M Nair Priority: Minor http://wiki.apache.org/pig-data/attachments/FrontPage/attachments/plrm.htm#_LOAD has a link to http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/FileSystem.html#globPaths(org.apache.hadoop.fs.Path)the , it should be - http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FileSystem.html#globStatus(org.apache.hadoop.fs.Path) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-934) Merge join implementation currently does not seek to right point on the right side input based on the offset provided by the index
[
https://issues.apache.org/jira/browse/PIG-934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12749188#action_12749188
]
Ashutosh Chauhan commented on PIG-934:
--
Seeking to an offset would only work for a single file - hence maybe have a
separate function...
Since open() returns an input stream it is not hard to conceive of usecase when
one would want to seek into that stream even when filespec points to a
directory or a glob. We have to define the semantics here. What does seeking in
a directory/glob means? One reasonable answer is to view all the files in
directory/glob as one big logical file and offset as an offset in this logical
file and then seek into this file. Something along the lines of :
{code}
iterator = DataStreamIterator
bytesSeen = 0;
while(itertor.hasNext()){
open current file pointed by iterator
bytesSeen += current file length
if (bytesSeen offset)
bind to adjusted offset in current file and return
else
continue;
}
{code}
But since there is no requirement for such currently, we can catch the
situation when seeking is asked for directory/glob and throw an exception (as
is done in this patch). Later on, if we decide to support it instead of
throwing exception, we can implement whatever semantics we decide on. If we
create a new function with separate name it will be confusing to do these
changes later on. Moreover, if there is a different function, user of the api
needs to know about it and deal with it (e.g., need of special constructor in
POLoad). Presence/absence of offset parameter in argument list I think is a
sufficient indicator to tell which version of overloaded open() to call if
there is a need for seek.
Thoughts?
Merge join implementation currently does not seek to right point on the right
side input based on the offset provided by the index
--
Key: PIG-934
URL: https://issues.apache.org/jira/browse/PIG-934
Project: Pig
Issue Type: Bug
Affects Versions: 0.3.1
Reporter: Pradeep Kamath
Assignee: Ashutosh Chauhan
Attachments: pig-934.patch
We use POLoad to seek into right file which has the following code:
{noformat}
public void setUp() throws IOException{
String filename = lFile.getFileName();
loader =
(LoadFunc)PigContext.instantiateFuncFromSpec(lFile.getFuncSpec());
is = FileLocalizer.open(filename, pc);
loader.bindTo(filename , new BufferedPositionedInputStream(is),
this.offset, Long.MAX_VALUE);
}
{noformat}
Between opening the stream and bindTo we do not seek to the right offset.
bindTo itself does not perform any seek.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-934) Merge join implementation currently does not seek to right point on the right side input based on the offset provided by the index
[
https://issues.apache.org/jira/browse/PIG-934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12749196#action_12749196
]
Dmitriy V. Ryaboy commented on PIG-934:
---
Throwing an exception when a seek is past the file boundary seems acceptable to
me (and preferable to adding new functions and changing upstream code that
shouldn't care about this detail). Especially since if there is a way to get a
consistent ordering among files in a directory, it's trivial to later update
this code to seek past file boundaries and into the next file.
Merge join implementation currently does not seek to right point on the right
side input based on the offset provided by the index
--
Key: PIG-934
URL: https://issues.apache.org/jira/browse/PIG-934
Project: Pig
Issue Type: Bug
Affects Versions: 0.3.1
Reporter: Pradeep Kamath
Assignee: Ashutosh Chauhan
Attachments: pig-934.patch
We use POLoad to seek into right file which has the following code:
{noformat}
public void setUp() throws IOException{
String filename = lFile.getFileName();
loader =
(LoadFunc)PigContext.instantiateFuncFromSpec(lFile.getFuncSpec());
is = FileLocalizer.open(filename, pc);
loader.bindTo(filename , new BufferedPositionedInputStream(is),
this.offset, Long.MAX_VALUE);
}
{noformat}
Between opening the stream and bindTo we do not seek to the right offset.
bindTo itself does not perform any seek.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Hudson build is back to normal: Pig-Patch-minerva.apache.org #180
See http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/180/
[jira] Updated: (PIG-935) Skewed join throws an exception when used with map keys
[ https://issues.apache.org/jira/browse/PIG-935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriranjan Manjunath updated PIG-935: Attachment: skjoinmapbug.patch Fixed the issue with unit tests Skewed join throws an exception when used with map keys --- Key: PIG-935 URL: https://issues.apache.org/jira/browse/PIG-935 Project: Pig Issue Type: Bug Reporter: Sriranjan Manjunath Attachments: skjoinmapbug.patch, skjoinmapbug.patch Skewed join throws a runtime exception for the following query: A = load 'map.txt' as (e); B = load 'map.txt' as (f); C = join A by (chararray)e#'a', B by (chararray)f#'a' using skewed; explain C; Exception: Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.PO Project at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.getSortCols(MRCompiler.java:1492) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.getSamplingJob(MRCompiler.java:1894) ... 27 more -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-935) Skewed join throws an exception when used with map keys
[ https://issues.apache.org/jira/browse/PIG-935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriranjan Manjunath updated PIG-935: Status: Open (was: Patch Available) Skewed join throws an exception when used with map keys --- Key: PIG-935 URL: https://issues.apache.org/jira/browse/PIG-935 Project: Pig Issue Type: Bug Reporter: Sriranjan Manjunath Attachments: skjoinmapbug.patch, skjoinmapbug.patch Skewed join throws a runtime exception for the following query: A = load 'map.txt' as (e); B = load 'map.txt' as (f); C = join A by (chararray)e#'a', B by (chararray)f#'a' using skewed; explain C; Exception: Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.PO Project at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.getSortCols(MRCompiler.java:1492) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.getSamplingJob(MRCompiler.java:1894) ... 27 more -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-935) Skewed join throws an exception when used with map keys
[ https://issues.apache.org/jira/browse/PIG-935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriranjan Manjunath updated PIG-935: Attachment: (was: skjoinmapbug.patch) Skewed join throws an exception when used with map keys --- Key: PIG-935 URL: https://issues.apache.org/jira/browse/PIG-935 Project: Pig Issue Type: Bug Reporter: Sriranjan Manjunath Attachments: skjoinmapbug.patch Skewed join throws a runtime exception for the following query: A = load 'map.txt' as (e); B = load 'map.txt' as (f); C = join A by (chararray)e#'a', B by (chararray)f#'a' using skewed; explain C; Exception: Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.PO Project at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.getSortCols(MRCompiler.java:1492) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.getSamplingJob(MRCompiler.java:1894) ... 27 more -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-935) Skewed join throws an exception when used with map keys
[ https://issues.apache.org/jira/browse/PIG-935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriranjan Manjunath updated PIG-935: Attachment: (was: skjoinmapbug.patch) Skewed join throws an exception when used with map keys --- Key: PIG-935 URL: https://issues.apache.org/jira/browse/PIG-935 Project: Pig Issue Type: Bug Reporter: Sriranjan Manjunath Attachments: skjoinmapbug.patch Skewed join throws a runtime exception for the following query: A = load 'map.txt' as (e); B = load 'map.txt' as (f); C = join A by (chararray)e#'a', B by (chararray)f#'a' using skewed; explain C; Exception: Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.PO Project at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.getSortCols(MRCompiler.java:1492) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.getSamplingJob(MRCompiler.java:1894) ... 27 more -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-935) Skewed join throws an exception when used with map keys
[ https://issues.apache.org/jira/browse/PIG-935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriranjan Manjunath updated PIG-935: Attachment: skjoinmapbug.patch Skewed join throws an exception when used with map keys --- Key: PIG-935 URL: https://issues.apache.org/jira/browse/PIG-935 Project: Pig Issue Type: Bug Reporter: Sriranjan Manjunath Attachments: skjoinmapbug.patch Skewed join throws a runtime exception for the following query: A = load 'map.txt' as (e); B = load 'map.txt' as (f); C = join A by (chararray)e#'a', B by (chararray)f#'a' using skewed; explain C; Exception: Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.PO Project at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.getSortCols(MRCompiler.java:1492) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.getSamplingJob(MRCompiler.java:1894) ... 27 more -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-935) Skewed join throws an exception when used with map keys
[ https://issues.apache.org/jira/browse/PIG-935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriranjan Manjunath updated PIG-935: Status: Patch Available (was: Open) Skewed join throws an exception when used with map keys --- Key: PIG-935 URL: https://issues.apache.org/jira/browse/PIG-935 Project: Pig Issue Type: Bug Reporter: Sriranjan Manjunath Attachments: skjoinmapbug.patch Skewed join throws a runtime exception for the following query: A = load 'map.txt' as (e); B = load 'map.txt' as (f); C = join A by (chararray)e#'a', B by (chararray)f#'a' using skewed; explain C; Exception: Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.PO Project at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.getSortCols(MRCompiler.java:1492) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.getSamplingJob(MRCompiler.java:1894) ... 27 more -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Build failed in Hudson: Pig-Patch-minerva.apache.org #181
See http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/181/ -- [...truncated 113025 lines...] [exec] [junit] 09/08/30 04:00:26 INFO dfs.DataNode: Receiving block blk_-589553873048556787_1011 src: /127.0.0.1:55077 dest: /127.0.0.1:60322 [exec] [junit] 09/08/30 04:00:26 INFO dfs.DataNode: Receiving block blk_-589553873048556787_1011 src: /127.0.0.1:45439 dest: /127.0.0.1:35627 [exec] [junit] 09/08/30 04:00:26 INFO dfs.DataNode: Received block blk_-589553873048556787_1011 of size 6 from /127.0.0.1 [exec] [junit] 09/08/30 04:00:26 INFO dfs.DataNode: Received block blk_-589553873048556787_1011 of size 6 from /127.0.0.1 [exec] [junit] 09/08/30 04:00:26 INFO dfs.DataNode: PacketResponder 0 for block blk_-589553873048556787_1011 terminating [exec] [junit] 09/08/30 04:00:26 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:35627 is added to blk_-589553873048556787_1011 size 6 [exec] [junit] 09/08/30 04:00:26 INFO dfs.DataNode: PacketResponder 1 for block blk_-589553873048556787_1011 terminating [exec] [junit] 09/08/30 04:00:26 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:60322 is added to blk_-589553873048556787_1011 size 6 [exec] [junit] 09/08/30 04:00:26 INFO dfs.DataNode: Received block blk_-589553873048556787_1011 of size 6 from /127.0.0.1 [exec] [junit] 09/08/30 04:00:26 INFO dfs.DataNode: PacketResponder 2 for block blk_-589553873048556787_1011 terminating [exec] [junit] 09/08/30 04:00:26 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:40802 is added to blk_-589553873048556787_1011 size 6 [exec] [junit] 09/08/30 04:00:26 INFO executionengine.HExecutionEngine: Connecting to hadoop file system at: hdfs://localhost:35438 [exec] [junit] 09/08/30 04:00:26 INFO executionengine.HExecutionEngine: Connecting to map-reduce job tracker at: localhost:34463 [exec] [junit] 09/08/30 04:00:26 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size before optimization: 1 [exec] [junit] 09/08/30 04:00:26 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size after optimization: 1 [exec] [junit] 09/08/30 04:00:26 INFO dfs.StateChange: BLOCK* ask 127.0.0.1:33074 to delete blk_5017402978013794719_1006 blk_699524133206203_1005 blk_7671665588420890724_1004 [exec] [junit] 09/08/30 04:00:26 INFO dfs.StateChange: BLOCK* ask 127.0.0.1:35627 to delete blk_5017402978013794719_1006 blk_699524133206203_1005 [exec] [junit] 09/08/30 04:00:26 INFO dfs.DataNode: Deleting block blk_5017402978013794719_1006 file dfs/data/data7/current/blk_5017402978013794719 [exec] [junit] 09/08/30 04:00:26 INFO dfs.DataNode: Deleting block blk_699524133206203_1005 file dfs/data/data8/current/blk_699524133206203 [exec] [junit] 09/08/30 04:00:27 INFO mapReduceLayer.JobControlCompiler: Setting up single store job [exec] [junit] 09/08/30 04:00:27 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. [exec] [junit] 09/08/30 04:00:27 INFO dfs.StateChange: BLOCK* NameSystem.allocateBlock: /tmp/hadoop-hudson/mapred/system/job_200908300359_0002/job.jar. blk_-8555955310195289721_1012 [exec] [junit] 09/08/30 04:00:27 INFO dfs.DataNode: Receiving block blk_-8555955310195289721_1012 src: /127.0.0.1:45440 dest: /127.0.0.1:35627 [exec] [junit] 09/08/30 04:00:27 INFO dfs.DataNode: Receiving block blk_-8555955310195289721_1012 src: /127.0.0.1:53458 dest: /127.0.0.1:33074 [exec] [junit] 09/08/30 04:00:27 INFO dfs.DataNode: Receiving block blk_-8555955310195289721_1012 src: /127.0.0.1:60043 dest: /127.0.0.1:40802 [exec] [junit] 09/08/30 04:00:27 INFO dfs.DataNode: Received block blk_-8555955310195289721_1012 of size 1498962 from /127.0.0.1 [exec] [junit] 09/08/30 04:00:27 INFO dfs.DataNode: PacketResponder 0 for block blk_-8555955310195289721_1012 terminating [exec] [junit] 09/08/30 04:00:27 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:40802 is added to blk_-8555955310195289721_1012 size 1498962 [exec] [junit] 09/08/30 04:00:27 INFO dfs.DataNode: Received block blk_-8555955310195289721_1012 of size 1498962 from /127.0.0.1 [exec] [junit] 09/08/30 04:00:27 INFO dfs.DataNode: PacketResponder 1 for block blk_-8555955310195289721_1012 terminating [exec] [junit] 09/08/30 04:00:27 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:33074 is added to blk_-8555955310195289721_1012 size 1498962 [exec] [junit] 09/08/30 04:00:27 INFO dfs.DataNode: Received block blk_-8555955310195289721_1012 of size 1498962 from /127.0.0.1 [exec]
[jira] Commented: (PIG-935) Skewed join throws an exception when used with map keys
[ https://issues.apache.org/jira/browse/PIG-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12749227#action_12749227 ] Hadoop QA commented on PIG-935: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418081/skjoinmapbug.patch against trunk revision 806668. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/181/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/181/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/181/console This message is automatically generated. Skewed join throws an exception when used with map keys --- Key: PIG-935 URL: https://issues.apache.org/jira/browse/PIG-935 Project: Pig Issue Type: Bug Reporter: Sriranjan Manjunath Attachments: skjoinmapbug.patch Skewed join throws a runtime exception for the following query: A = load 'map.txt' as (e); B = load 'map.txt' as (f); C = join A by (chararray)e#'a', B by (chararray)f#'a' using skewed; explain C; Exception: Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.PO Project at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.getSortCols(MRCompiler.java:1492) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.getSamplingJob(MRCompiler.java:1894) ... 27 more -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
