Hudson build is back to normal: Pig-trunk #641

2009-12-08 Thread Apache Hudson Server 
See

[jira] Commented: (PIG-1133) UDFContext should be made available to LoadFunc.bindTo

2009-12-08 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787409#action_12787409
 ] 

Hadoop QA commented on PIG-1133:


-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12427280/PIG-1133-1.patch
  against trunk revision 888186.

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no tests are needed for this patch.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed core unit tests.

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/103/testReport/
Findbugs warnings: 
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/103/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: 
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/103/console

This message is automatically generated.

> UDFContext should be made available to LoadFunc.bindTo
> --
>
> Key: PIG-1133
> URL: https://issues.apache.org/jira/browse/PIG-1133
> Project: Pig
>  Issue Type: Bug
>  Components: impl
>Affects Versions: 0.4.0
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1133-1.patch
>
>
> Currently UDFContext is available at frontend, map and reduce. However, 
> LoadFunc.bindTo will be called on InputFormat.getSplits, which is on a 
> different context then the ones mentioned before. UDFContext should be 
> available in that context as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1132) Column Pruner issues in dealing with unprunable loader

2009-12-08 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787506#action_12787506
 ] 

Hadoop QA commented on PIG-1132:


-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12427303/PIG-1132-2.patch
  against trunk revision 888186.

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

-1 core tests.  The patch failed core unit tests.

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/104/testReport/
Findbugs warnings: 
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/104/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: 
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/104/console

This message is automatically generated.

> Column Pruner issues in dealing with unprunable loader
> --
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch, PIG-1132-2.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement 
> column for column: 1 in the new predecessorat 
> org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at 
> org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at 
> org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at 
> org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at 
> org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1110) Handle compressed file formats -- Gz, BZip with the new proposal

2009-12-08 Thread Richard Ding (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787584#action_12787584
 ] 

Richard Ding commented on PIG-1110:
---

The output from running ant "test-patch" target locally:

{code}
 [exec] +1 overall.  
 [exec] 
 [exec] +1 @author.  The patch does not contain any @author tags.
 [exec] 
 [exec] +1 tests included.  The patch appears to include 3 new or 
modified tests.
 [exec] 
 [exec] +1 javadoc.  The javadoc tool did not generate any warning 
messages.
 [exec] 
 [exec] +1 javac.  The applied patch does not increase the total number 
of javac compiler warnings.
 [exec] 
 [exec] +1 findbugs.  The patch does not introduce any new Findbugs 
warnings.
 [exec] 
 [exec] +1 release audit.  The applied patch does not increase the 
total number of release audit warnings.
 [exec] 
{code}

> Handle compressed file formats -- Gz, BZip with the new proposal
> 
>
> Key: PIG-1110
> URL: https://issues.apache.org/jira/browse/PIG-1110
> Project: Pig
>  Issue Type: Sub-task
>Reporter: Richard Ding
>Assignee: Richard Ding
> Attachments: PIG-1110.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1125) [zebra] Using typed APIs for Zebra's Map/Reduce interface

2009-12-08 Thread Gaurav Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787649#action_12787649
 ] 

Gaurav Jain commented on PIG-1125:
--


in requireSortedTable,

we are taking array of strings for sortcolumns which can be changed to 
List to be consistent with typeApi semantics



> [zebra] Using typed APIs for Zebra's Map/Reduce interface
> -
>
> Key: PIG-1125
> URL: https://issues.apache.org/jira/browse/PIG-1125
> Project: Pig
>  Issue Type: Improvement
>Affects Versions: 0.4.0
>Reporter: Chao Wang
>Assignee: Chao Wang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: PIG-1125.patch
>
>
> We plan to modify Zebra's M/R interface to use typed APIs, i.e., APIs taking 
> object arguments, instead of String arguments.
> Take TableInputFormat as an example:
> setSchema(jobConf conf, String schema) is changing to setSchema(jobConf conf, 
> ZebraSchemaInfo schemaInfo)
> setProjection(jobConf conf, String projection) is changing to 
> setProjection(jobConf conf, ZebraProjectionInfo projectionInfo)
> and so on.
> Benefits: 1) Typed APIs make it easier to detect usage mistakes earlier and 
> 2) Typed APIs are richer and hide things better.
> In the meanwhile, we plan to make the old APIs deprecated, instead of 
> removing them, for the sake of safety.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1133) UDFContext should be made available to LoadFunc.bindTo

2009-12-08 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787660#action_12787660
 ] 

Alan Gates commented on PIG-1133:
-

Changes look good.  It might be good to change the unit tests in TestUDFContext 
to test that the conf is available in the bindTo.

> UDFContext should be made available to LoadFunc.bindTo
> --
>
> Key: PIG-1133
> URL: https://issues.apache.org/jira/browse/PIG-1133
> Project: Pig
>  Issue Type: Bug
>  Components: impl
>Affects Versions: 0.4.0
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1133-1.patch
>
>
> Currently UDFContext is available at frontend, map and reduce. However, 
> LoadFunc.bindTo will be called on InputFormat.getSplits, which is on a 
> different context then the ones mentioned before. UDFContext should be 
> available in that context as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1133) UDFContext should be made available to LoadFunc.bindTo

2009-12-08 Thread Daniel Dai (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787668#action_12787668
 ] 

Daniel Dai commented on PIG-1133:
-

The interesting thing is UDFContext is always available to SliceWrapper in 
MiniCluster and local mode. Only in real cluster, we cannot get correct 
UDFContext setup. I guess MiniCluster and real cluster use different context 
when invoking SliceWrapper. That's why I do not put unit test because none unit 
test can simulate the real cluster situation.

> UDFContext should be made available to LoadFunc.bindTo
> --
>
> Key: PIG-1133
> URL: https://issues.apache.org/jira/browse/PIG-1133
> Project: Pig
>  Issue Type: Bug
>  Components: impl
>Affects Versions: 0.4.0
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1133-1.patch
>
>
> Currently UDFContext is available at frontend, map and reduce. However, 
> LoadFunc.bindTo will be called on InputFormat.getSplits, which is on a 
> different context then the ones mentioned before. UDFContext should be 
> available in that context as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1132) Column Pruner issues in dealing with unprunable loader

2009-12-08 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787701#action_12787701
 ] 

Alan Gates commented on PIG-1132:
-

Is this bug specific to a script with two foreachs in a row, or will it 
manifest with any script with more than one foreach?  So would:

A = load
B = foreach ..
C = filter
D = group
E = foreach

trigger this bug as well?

> Column Pruner issues in dealing with unprunable loader
> --
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch, PIG-1132-2.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement 
> column for column: 1 in the new predecessorat 
> org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at 
> org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at 
> org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at 
> org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at 
> org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1133) UDFContext should be made available to LoadFunc.bindTo

2009-12-08 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787707#action_12787707
 ] 

Alan Gates commented on PIG-1133:
-

Yeah, I missed your first comment before I asked that question.  Ok, no test 
seems fine then.

> UDFContext should be made available to LoadFunc.bindTo
> --
>
> Key: PIG-1133
> URL: https://issues.apache.org/jira/browse/PIG-1133
> Project: Pig
>  Issue Type: Bug
>  Components: impl
>Affects Versions: 0.4.0
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1133-1.patch
>
>
> Currently UDFContext is available at frontend, map and reduce. However, 
> LoadFunc.bindTo will be called on InputFormat.getSplits, which is on a 
> different context then the ones mentioned before. UDFContext should be 
> available in that context as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1132) Column Pruner issues in dealing with unprunable loader

2009-12-08 Thread Daniel Dai (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787716#action_12787716
 ] 

Daniel Dai commented on PIG-1132:
-

Yes, it could happen for any logical operators after Load. The key issue for 
this problem is OperatorPlan.insertBetween assume logical plan in a consistent 
state. What I want to do here is inserting a LOForEach after LOLoad if the 
loader do not support pruning. I will need to achieve this in two steps:

1. insert a LOForEach after LoLoad
2. prune columns the logical operator below

I have to leave the logical plan in an inconsistent state in the middle. If I 
insert LOForEach first, then the underlying logical operator has not been 
pruned yet; If I prune underlying logical operator first, then there is a 
schema gap between underlying logical operator and LOLoad. Both situations are 
not allowed for insertBetween. I have to use a trick to get around this problem.

> Column Pruner issues in dealing with unprunable loader
> --
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch, PIG-1132-2.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement 
> column for column: 1 in the new predecessorat 
> org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at 
> org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at 
> org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at 
> org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at 
> org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (PIG-1125) [zebra] Using typed APIs for Zebra's Map/Reduce interface

2009-12-08 Thread Chao Wang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Wang updated PIG-1125:
---

Attachment: PIG-1125.patch

> [zebra] Using typed APIs for Zebra's Map/Reduce interface
> -
>
> Key: PIG-1125
> URL: https://issues.apache.org/jira/browse/PIG-1125
> Project: Pig
>  Issue Type: Improvement
>Affects Versions: 0.4.0
>Reporter: Chao Wang
>Assignee: Chao Wang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: PIG-1125.patch, PIG-1125.patch
>
>
> We plan to modify Zebra's M/R interface to use typed APIs, i.e., APIs taking 
> object arguments, instead of String arguments.
> Take TableInputFormat as an example:
> setSchema(jobConf conf, String schema) is changing to setSchema(jobConf conf, 
> ZebraSchemaInfo schemaInfo)
> setProjection(jobConf conf, String projection) is changing to 
> setProjection(jobConf conf, ZebraProjectionInfo projectionInfo)
> and so on.
> Benefits: 1) Typed APIs make it easier to detect usage mistakes earlier and 
> 2) Typed APIs are richer and hide things better.
> In the meanwhile, we plan to make the old APIs deprecated, instead of 
> removing them, for the sake of safety.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1125) [zebra] Using typed APIs for Zebra's Map/Reduce interface

2009-12-08 Thread Gaurav Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787722#action_12787722
 ] 

Gaurav Jain commented on PIG-1125:
--



+1

> [zebra] Using typed APIs for Zebra's Map/Reduce interface
> -
>
> Key: PIG-1125
> URL: https://issues.apache.org/jira/browse/PIG-1125
> Project: Pig
>  Issue Type: Improvement
>Affects Versions: 0.4.0
>Reporter: Chao Wang
>Assignee: Chao Wang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: PIG-1125.patch, PIG-1125.patch
>
>
> We plan to modify Zebra's M/R interface to use typed APIs, i.e., APIs taking 
> object arguments, instead of String arguments.
> Take TableInputFormat as an example:
> setSchema(jobConf conf, String schema) is changing to setSchema(jobConf conf, 
> ZebraSchemaInfo schemaInfo)
> setProjection(jobConf conf, String projection) is changing to 
> setProjection(jobConf conf, ZebraProjectionInfo projectionInfo)
> and so on.
> Benefits: 1) Typed APIs make it easier to detect usage mistakes earlier and 
> 2) Typed APIs are richer and hide things better.
> In the meanwhile, we plan to make the old APIs deprecated, instead of 
> removing them, for the sake of safety.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1125) [zebra] Using typed APIs for Zebra's Map/Reduce interface

2009-12-08 Thread Yan Zhou (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787723#action_12787723
 ] 

Yan Zhou commented on PIG-1125:
---

Now it takes a ZebraSortInfo object.

> [zebra] Using typed APIs for Zebra's Map/Reduce interface
> -
>
> Key: PIG-1125
> URL: https://issues.apache.org/jira/browse/PIG-1125
> Project: Pig
>  Issue Type: Improvement
>Affects Versions: 0.4.0
>Reporter: Chao Wang
>Assignee: Chao Wang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: PIG-1125.patch, PIG-1125.patch
>
>
> We plan to modify Zebra's M/R interface to use typed APIs, i.e., APIs taking 
> object arguments, instead of String arguments.
> Take TableInputFormat as an example:
> setSchema(jobConf conf, String schema) is changing to setSchema(jobConf conf, 
> ZebraSchemaInfo schemaInfo)
> setProjection(jobConf conf, String projection) is changing to 
> setProjection(jobConf conf, ZebraProjectionInfo projectionInfo)
> and so on.
> Benefits: 1) Typed APIs make it easier to detect usage mistakes earlier and 
> 2) Typed APIs are richer and hide things better.
> In the meanwhile, we plan to make the old APIs deprecated, instead of 
> removing them, for the sake of safety.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (PIG-1125) [zebra] Using typed APIs for Zebra's Map/Reduce interface

2009-12-08 Thread Chao Wang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Wang updated PIG-1125:
---

Status: Patch Available  (was: Open)

> [zebra] Using typed APIs for Zebra's Map/Reduce interface
> -
>
> Key: PIG-1125
> URL: https://issues.apache.org/jira/browse/PIG-1125
> Project: Pig
>  Issue Type: Improvement
>Affects Versions: 0.4.0
>Reporter: Chao Wang
>Assignee: Chao Wang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: PIG-1125.patch, PIG-1125.patch
>
>
> We plan to modify Zebra's M/R interface to use typed APIs, i.e., APIs taking 
> object arguments, instead of String arguments.
> Take TableInputFormat as an example:
> setSchema(jobConf conf, String schema) is changing to setSchema(jobConf conf, 
> ZebraSchemaInfo schemaInfo)
> setProjection(jobConf conf, String projection) is changing to 
> setProjection(jobConf conf, ZebraProjectionInfo projectionInfo)
> and so on.
> Benefits: 1) Typed APIs make it easier to detect usage mistakes earlier and 
> 2) Typed APIs are richer and hide things better.
> In the meanwhile, we plan to make the old APIs deprecated, instead of 
> removing them, for the sake of safety.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1132) Column Pruner issues in dealing with unprunable loader

2009-12-08 Thread Alan Gates (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787770#action_12787770
 ] 

Alan Gates commented on PIG-1132:
-

Thanks, that clears up several questions I had.  

+1, patch looks good.

> Column Pruner issues in dealing with unprunable loader
> --
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch, PIG-1132-2.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement 
> column for column: 1 in the new predecessorat 
> org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at 
> org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at 
> org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at 
> org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at 
> org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (PIG-1133) UDFContext should be made available to LoadFunc.bindTo

2009-12-08 Thread Daniel Dai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-1133:


  Resolution: Fixed
Hadoop Flags: [Reviewed]
  Status: Resolved  (was: Patch Available)

Patch committed to both trunk and 0.6 branch.

> UDFContext should be made available to LoadFunc.bindTo
> --
>
> Key: PIG-1133
> URL: https://issues.apache.org/jira/browse/PIG-1133
> Project: Pig
>  Issue Type: Bug
>  Components: impl
>Affects Versions: 0.4.0
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1133-1.patch
>
>
> Currently UDFContext is available at frontend, map and reduce. However, 
> LoadFunc.bindTo will be called on InputFormat.getSplits, which is on a 
> different context then the ones mentioned before. UDFContext should be 
> available in that context as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (PIG-1132) Column Pruner issues in dealing with unprunable loader

2009-12-08 Thread Daniel Dai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-1132:


  Resolution: Fixed
Hadoop Flags: [Reviewed]
  Status: Resolved  (was: Patch Available)

Core test failures are due to port conflict. Test manually and it is 
successful. Patch committed to both trunk and 0.6 branch. Thanks Alan!

> Column Pruner issues in dealing with unprunable loader
> --
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch, PIG-1132-2.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement 
> column for column: 1 in the new predecessorat 
> org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at 
> org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at 
> org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at 
> org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at 
> org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1125) [zebra] Using typed APIs for Zebra's Map/Reduce interface

2009-12-08 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787837#action_12787837
 ] 

Hadoop QA commented on PIG-1125:


+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12427381/PIG-1125.patch
  against trunk revision 888186.

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 23 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed core unit tests.

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/105/testReport/
Findbugs warnings: 
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/105/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: 
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/105/console

This message is automatically generated.

> [zebra] Using typed APIs for Zebra's Map/Reduce interface
> -
>
> Key: PIG-1125
> URL: https://issues.apache.org/jira/browse/PIG-1125
> Project: Pig
>  Issue Type: Improvement
>Affects Versions: 0.4.0
>Reporter: Chao Wang
>Assignee: Chao Wang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: PIG-1125.patch, PIG-1125.patch
>
>
> We plan to modify Zebra's M/R interface to use typed APIs, i.e., APIs taking 
> object arguments, instead of String arguments.
> Take TableInputFormat as an example:
> setSchema(jobConf conf, String schema) is changing to setSchema(jobConf conf, 
> ZebraSchemaInfo schemaInfo)
> setProjection(jobConf conf, String projection) is changing to 
> setProjection(jobConf conf, ZebraProjectionInfo projectionInfo)
> and so on.
> Benefits: 1) Typed APIs make it easier to detect usage mistakes earlier and 
> 2) Typed APIs are richer and hide things better.
> In the meanwhile, we plan to make the old APIs deprecated, instead of 
> removing them, for the sake of safety.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (PIG-1134) Skewed Join sampling job overwhelms the name node

2009-12-08 Thread Sriranjan Manjunath (JIRA) 
Skewed Join sampling job overwhelms the name node
-

 Key: PIG-1134
 URL: https://issues.apache.org/jira/browse/PIG-1134
 Project: Pig
  Issue Type: Bug
Reporter: Sriranjan Manjunath
Assignee: Sriranjan Manjunath


The map tasks of the sampling job estimate the file size. For a large directory 
and a large number of maps the file system calls over whelm the name node. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (PIG-1134) Skewed Join sampling job overwhelms the name node

2009-12-08 Thread Sriranjan Manjunath (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sriranjan Manjunath updated PIG-1134:
-

Attachment: PIG-1134.patch

As a stop gap, I have replaced PoissonSampleLoader with RandomSampleLoader. 
This does not obtain the input size. Instead it obtains 100 samples per block. 

> Skewed Join sampling job overwhelms the name node
> -
>
> Key: PIG-1134
> URL: https://issues.apache.org/jira/browse/PIG-1134
> Project: Pig
>  Issue Type: Bug
>Reporter: Sriranjan Manjunath
>Assignee: Sriranjan Manjunath
> Attachments: PIG-1134.patch
>
>
> The map tasks of the sampling job estimate the file size. For a large 
> directory and a large number of maps the file system calls over whelm the 
> name node. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (PIG-1134) Skewed Join sampling job overwhelms the name node

2009-12-08 Thread Sriranjan Manjunath (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sriranjan Manjunath updated PIG-1134:
-

Status: Patch Available  (was: Open)

> Skewed Join sampling job overwhelms the name node
> -
>
> Key: PIG-1134
> URL: https://issues.apache.org/jira/browse/PIG-1134
> Project: Pig
>  Issue Type: Bug
>Reporter: Sriranjan Manjunath
>Assignee: Sriranjan Manjunath
> Attachments: PIG-1134.patch
>
>
> The map tasks of the sampling job estimate the file size. For a large 
> directory and a large number of maps the file system calls over whelm the 
> name node. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (PIG-1134) Skewed Join sampling job overwhelms the name node

2009-12-08 Thread Olga Natkovich (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Olga Natkovich updated PIG-1134:


Fix Version/s: 0.6.0

> Skewed Join sampling job overwhelms the name node
> -
>
> Key: PIG-1134
> URL: https://issues.apache.org/jira/browse/PIG-1134
> Project: Pig
>  Issue Type: Bug
>Reporter: Sriranjan Manjunath
>Assignee: Sriranjan Manjunath
> Fix For: 0.6.0
>
> Attachments: PIG-1134.patch
>
>
> The map tasks of the sampling job estimate the file size. For a large 
> directory and a large number of maps the file system calls over whelm the 
> name node. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (PIG-1135) skewed join partitioner returns negative partition index

2009-12-08 Thread Ying He (JIRA) 
skewed join partitioner returns negative partition index 
-

 Key: PIG-1135
 URL: https://issues.apache.org/jira/browse/PIG-1135
 Project: Pig
  Issue Type: Improvement
Reporter: Ying He
Assignee: Sriranjan Manjunath


Fragmented replicated join has a few limitations:
 - One of the tables needs to be loaded into memory
 - Join is limited to two tables

Skewed join partitions the table and joins the records in the reduce phase. It 
computes a histogram of the key space to account for skewing in the input 
records. Further, it adjusts the number of reducers depending on the key 
distribution.

We need to implement the skewed join in pig.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1134) Skewed Join sampling job overwhelms the name node

2009-12-08 Thread Olga Natkovich (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787868#action_12787868
 ] 

Olga Natkovich commented on PIG-1134:
-

+1 on the code change.

Please, open a separate JIRA to resolve the problem with PoissomSampleLoader

> Skewed Join sampling job overwhelms the name node
> -
>
> Key: PIG-1134
> URL: https://issues.apache.org/jira/browse/PIG-1134
> Project: Pig
>  Issue Type: Bug
>Reporter: Sriranjan Manjunath
>Assignee: Sriranjan Manjunath
> Fix For: 0.6.0
>
> Attachments: PIG-1134.patch
>
>
> The map tasks of the sampling job estimate the file size. For a large 
> directory and a large number of maps the file system calls over whelm the 
> name node. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (PIG-1135) skewed join partitioner returns negative partition index

2009-12-08 Thread Ying He (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ying He updated PIG-1135:
-

Description: skewed join returns negative reducer index  (was: Fragmented 
replicated join has a few limitations:
 - One of the tables needs to be loaded into memory
 - Join is limited to two tables

Skewed join partitions the table and joins the records in the reduce phase. It 
computes a histogram of the key space to account for skewing in the input 
records. Further, it adjusts the number of reducers depending on the key 
distribution.

We need to implement the skewed join in pig.)

> skewed join partitioner returns negative partition index 
> -
>
> Key: PIG-1135
> URL: https://issues.apache.org/jira/browse/PIG-1135
> Project: Pig
>  Issue Type: Improvement
>Reporter: Ying He
>Assignee: Sriranjan Manjunath
>
> skewed join returns negative reducer index

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (PIG-1135) skewed join partitioner returns negative partition index

2009-12-08 Thread Ying He (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ying He updated PIG-1135:
-

Status: Patch Available  (was: Open)

> skewed join partitioner returns negative partition index 
> -
>
> Key: PIG-1135
> URL: https://issues.apache.org/jira/browse/PIG-1135
> Project: Pig
>  Issue Type: Improvement
>Reporter: Ying He
>Assignee: Sriranjan Manjunath
> Attachments: PIG_1135.patch
>
>
> skewed join returns negative reducer index

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (PIG-1135) skewed join partitioner returns negative partition index

2009-12-08 Thread Ying He (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ying He updated PIG-1135:
-

Attachment: PIG_1135.patch

if reducer index is greater than 128, the index of streaming table becomes 
negative because byte is used as data type.
It is fixed by changing partition index from byte to integer. 

> skewed join partitioner returns negative partition index 
> -
>
> Key: PIG-1135
> URL: https://issues.apache.org/jira/browse/PIG-1135
> Project: Pig
>  Issue Type: Improvement
>Reporter: Ying He
>Assignee: Sriranjan Manjunath
> Attachments: PIG_1135.patch
>
>
> skewed join returns negative reducer index

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1135) skewed join partitioner returns negative partition index

2009-12-08 Thread Sriranjan Manjunath (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787912#action_12787912
 ] 

Sriranjan Manjunath commented on PIG-1135:
--

Ran skewed join end-end / unit tests against this patch and random sample 
loader, and they passed.

> skewed join partitioner returns negative partition index 
> -
>
> Key: PIG-1135
> URL: https://issues.apache.org/jira/browse/PIG-1135
> Project: Pig
>  Issue Type: Improvement
>Reporter: Ying He
>Assignee: Sriranjan Manjunath
> Attachments: PIG_1135.patch
>
>
> skewed join returns negative reducer index

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (PIG-1125) [zebra] Using typed APIs for Zebra's Map/Reduce interface

2009-12-08 Thread Yan Zhou (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yan Zhou updated PIG-1125:
--

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to Apache trunk and 0.6 branch.

> [zebra] Using typed APIs for Zebra's Map/Reduce interface
> -
>
> Key: PIG-1125
> URL: https://issues.apache.org/jira/browse/PIG-1125
> Project: Pig
>  Issue Type: Improvement
>Affects Versions: 0.4.0
>Reporter: Chao Wang
>Assignee: Chao Wang
> Fix For: 0.6.0, 0.7.0
>
> Attachments: PIG-1125.patch, PIG-1125.patch
>
>
> We plan to modify Zebra's M/R interface to use typed APIs, i.e., APIs taking 
> object arguments, instead of String arguments.
> Take TableInputFormat as an example:
> setSchema(jobConf conf, String schema) is changing to setSchema(jobConf conf, 
> ZebraSchemaInfo schemaInfo)
> setProjection(jobConf conf, String projection) is changing to 
> setProjection(jobConf conf, ZebraProjectionInfo projectionInfo)
> and so on.
> Benefits: 1) Typed APIs make it easier to detect usage mistakes earlier and 
> 2) Typed APIs are richer and hide things better.
> In the meanwhile, we plan to make the old APIs deprecated, instead of 
> removing them, for the sake of safety.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1134) Skewed Join sampling job overwhelms the name node

2009-12-08 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787930#action_12787930
 ] 

Hadoop QA commented on PIG-1134:


-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12427404/PIG-1134.patch
  against trunk revision 888601.

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no tests are needed for this patch.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

-1 core tests.  The patch failed core unit tests.

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/106/testReport/
Findbugs warnings: 
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/106/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: 
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/106/console

This message is automatically generated.

> Skewed Join sampling job overwhelms the name node
> -
>
> Key: PIG-1134
> URL: https://issues.apache.org/jira/browse/PIG-1134
> Project: Pig
>  Issue Type: Bug
>Reporter: Sriranjan Manjunath
>Assignee: Sriranjan Manjunath
> Fix For: 0.6.0
>
> Attachments: PIG-1134.patch
>
>
> The map tasks of the sampling job estimate the file size. For a large 
> directory and a large number of maps the file system calls over whelm the 
> name node. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (PIG-1134) Skewed Join sampling job overwhelms the name node

2009-12-08 Thread Olga Natkovich (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Olga Natkovich updated PIG-1134:


Status: Patch Available  (was: Open)

resubmitting because of the test failures

> Skewed Join sampling job overwhelms the name node
> -
>
> Key: PIG-1134
> URL: https://issues.apache.org/jira/browse/PIG-1134
> Project: Pig
>  Issue Type: Bug
>Reporter: Sriranjan Manjunath
>Assignee: Sriranjan Manjunath
> Fix For: 0.6.0
>
> Attachments: PIG-1134.patch
>
>
> The map tasks of the sampling job estimate the file size. For a large 
> directory and a large number of maps the file system calls over whelm the 
> name node. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (PIG-1134) Skewed Join sampling job overwhelms the name node

2009-12-08 Thread Olga Natkovich (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Olga Natkovich updated PIG-1134:


Status: Open  (was: Patch Available)

> Skewed Join sampling job overwhelms the name node
> -
>
> Key: PIG-1134
> URL: https://issues.apache.org/jira/browse/PIG-1134
> Project: Pig
>  Issue Type: Bug
>Reporter: Sriranjan Manjunath
>Assignee: Sriranjan Manjunath
> Fix For: 0.6.0
>
> Attachments: PIG-1134.patch
>
>
> The map tasks of the sampling job estimate the file size. For a large 
> directory and a large number of maps the file system calls over whelm the 
> name node. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1134) Skewed Join sampling job overwhelms the name node

2009-12-08 Thread Olga Natkovich (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787933#action_12787933
 ] 

Olga Natkovich commented on PIG-1134:
-

this patch does not need new tests - existing skewed join tests cover the 
functionality

> Skewed Join sampling job overwhelms the name node
> -
>
> Key: PIG-1134
> URL: https://issues.apache.org/jira/browse/PIG-1134
> Project: Pig
>  Issue Type: Bug
>Reporter: Sriranjan Manjunath
>Assignee: Sriranjan Manjunath
> Fix For: 0.6.0
>
> Attachments: PIG-1134.patch
>
>
> The map tasks of the sampling job estimate the file size. For a large 
> directory and a large number of maps the file system calls over whelm the 
> name node. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1044) [zebra] Zebra should implement ReversibleLoadStoreFunc interface

2009-12-08 Thread Dmitriy V. Ryaboy (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787941#action_12787941
 ] 

Dmitriy V. Ryaboy commented on PIG-1044:


ReversibleLoadStore is going away in the load/store redesign. Should this be 
marked as "won't fix"?

> [zebra] Zebra should implement ReversibleLoadStoreFunc interface
> 
>
> Key: PIG-1044
> URL: https://issues.apache.org/jira/browse/PIG-1044
> Project: Pig
>  Issue Type: Improvement
>Reporter: Olga Natkovich
>
> This will allow some extra optimizations in Pig and will insure that Pig + 
> Zebra have the best possible performance.
> Need to do:
> - combine current load and store classes
> - implement the interface (no additional functions needed)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.