Double logs in grunt and ^d don't work ?

2009-09-11 Thread Vincent BARAT
Hello, I'm new to pig, I use it on MacOS, and I wonder if there is a way to avoid the double log traces in the grunt console, and if there is a way to make the ^D key work (the DEL key). I think this is really inconvenient. Thanks for you answer.

[jira] Resolved: (PIG-929) Default value of memusage for skewed join is not correct

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich resolved PIG-929. Resolution: Fixed > Default value of memusage for skewed join is not correct > ---

[jira] Commented: (PIG-882) log level not propogated to loggers

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754219#action_12754219 ] Olga Natkovich commented on PIG-882: +1 > log level not propogated to loggers >

[jira] Commented: (PIG-891) Fixing dfs statement for Pig

2009-09-11 Thread Daniel Dai (JIRA)
[ https://issues.apache.org/jira/browse/PIG-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754233#action_12754233 ] Daniel Dai commented on PIG-891: Tested in local mode also, the patch even works well in local

[jira] Resolved: (PIG-950) Pig Loader does not handle unix hidden files ( files starting with dot)

2009-09-11 Thread Daniel Dai (JIRA)
[ https://issues.apache.org/jira/browse/PIG-950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai resolved PIG-950. Resolution: Invalid Fix Version/s: 0.4.0 It is a limitation of Hadoop map-reduce, so we cannot solve it

Re: Request for feedback: cost-based optimizer

2009-09-11 Thread Alan Gates
This is a good start at adding a cost based optimizer to Pig. I have a number of comments: 1) Your argument for putting it in the physical layer rather than the logical is that the logical layer does not know physical statistics. This need not be true. You suggest adding a getStatistics

proposed changes to Pig UDFs

2009-09-11 Thread Olga Natkovich
Hi, As you know, a lot of work this year went into performance optimization of Pig. One of the main sources of performance problems is high memory usage. In an effort to address this problem we propose switching internal implementation of strings from Java Strings to Hadoop Text because text ha

[jira] Updated: (PIG-895) Default parallel for Pig

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-895: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Default parallel for Pig >

[jira] Updated: (PIG-892) Make COUNT and AVG deal with nulls accordingly with SQL standar

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-892: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Make COUNT and AVG deal with nulls

[jira] Updated: (PIG-660) Integration with Hadoop 0.20

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-660: --- Affects Version/s: (was: 0.2.0) 0.5.0 > Integration with Hadoop 0.20 > --

Re: Request for feedback: cost-based optimizer

2009-09-11 Thread Dmitriy Ryaboy
Hi Alan, Thanks for the detailed review. After getting Daniel's feedback (and grokking the relationship between Pig's logical and physical operators, which is a little different than that described in the literature), we agree that the proper place to put the optimizer is at the logical layer, alt

[jira] Created: (PIG-955) Skewed join generates incorrect results

2009-09-11 Thread Ying He (JIRA)
Skewed join generates incorrect results - Key: PIG-955 URL: https://issues.apache.org/jira/browse/PIG-955 Project: Pig Issue Type: Improvement Reporter: Ying He Fragmented replicated jo

[jira] Updated: (PIG-660) Integration with Hadoop 0.20

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-660: --- Affects Version/s: (was: 0.5.0) 0.4.0 Fix Version/s: 0.5.0 > Integrat

[jira] Created: (PIG-956) Reduce patch testing time

2009-09-11 Thread Olga Natkovich (JIRA)
Reduce patch testing time - Key: PIG-956 URL: https://issues.apache.org/jira/browse/PIG-956 Project: Pig Issue Type: Improvement Affects Versions: 0.4.0 Reporter: Olga Natkovich Assignee: O

[jira] Commented: (PIG-956) Reduce patch testing time

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754298#action_12754298 ] Olga Natkovich commented on PIG-956: My plan is to do the following: (1) Take all the tes

[jira] Updated: (PIG-955) Skewed join generates incorrect results

2009-09-11 Thread Ying He (JIRA)
[ https://issues.apache.org/jira/browse/PIG-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying He updated PIG-955: Attachment: PIG-955.patch use tuple type to lookup skewed key map > Skewed join generates incorrect results >

[jira] Issue Comment Edited: (PIG-956) Reduce patch testing time

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754298#action_12754298 ] Olga Natkovich edited comment on PIG-956 at 9/11/09 12:41 PM: -- My

[jira] Commented: (PIG-955) Skewed join generates incorrect results

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754306#action_12754306 ] Olga Natkovich commented on PIG-955: Hi Ying, Thanks for the patch. From the description

[jira] Commented: (PIG-955) Skewed join generates incorrect results

2009-09-11 Thread Ying He (JIRA)
[ https://issues.apache.org/jira/browse/PIG-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754319#action_12754319 ] Ying He commented on PIG-955: - the sampling process generated a file which contains skewed keys an

[jira] Updated: (PIG-954) Skewed join fails when pig.skewedjoin.reduce.memusage is not configured

2009-09-11 Thread Ying He (JIRA)
[ https://issues.apache.org/jira/browse/PIG-954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying He updated PIG-954: Attachment: PIG-954.patch2 add JUnit test > Skewed join fails when pig.skewedjoin.reduce.memusage is not configured >

[jira] Updated: (PIG-882) log level not propogated to loggers

2009-09-11 Thread Daniel Dai (JIRA)
[ https://issues.apache.org/jira/browse/PIG-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-882: --- Resolution: Fixed Fix Version/s: 0.4.0 Status: Resolved (was: Patch Available) I don't have a

[jira] Updated: (PIG-955) Skewed join generates incorrect results

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-955: --- Status: Patch Available (was: Open) > Skewed join generates incorrect results > -

[jira] Commented: (PIG-954) Skewed join fails when pig.skewedjoin.reduce.memusage is not configured

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754338#action_12754338 ] Olga Natkovich commented on PIG-954: +1 on the code changes. Need to wait for test results

[jira] Commented: (PIG-955) Skewed join generates incorrect results

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754335#action_12754335 ] Olga Natkovich commented on PIG-955: +1. Changes look good. Just need to wait for test res

[jira] Updated: (PIG-955) Skewed join generates incorrect results

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-955: --- Status: Open (was: Patch Available) > Skewed join generates incorrect results > -

[jira] Commented: (PIG-955) Skewed join generates incorrect results

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754336#action_12754336 ] Olga Natkovich commented on PIG-955: Updated wrong JIRA > Skewed join generates incorrec

[jira] Updated: (PIG-954) Skewed join fails when pig.skewedjoin.reduce.memusage is not configured

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-954: --- Status: Patch Available (was: Open) > Skewed join fails when pig.skewedjoin.reduce.memusage is not conf

[jira] Commented: (PIG-955) Skewed join generates incorrect results

2009-09-11 Thread Santhosh Srinivasan (JIRA)
[ https://issues.apache.org/jira/browse/PIG-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754349#action_12754349 ] Santhosh Srinivasan commented on PIG-955: - Hi Ying, How are Fragment Replicate Join a

[jira] Commented: (PIG-949) Zebra Bug: splitting map into multiple column group using storage hint causes unexpected behaviour

2009-09-11 Thread Jing Huang (JIRA)
[ https://issues.apache.org/jira/browse/PIG-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754353#action_12754353 ] Jing Huang commented on PIG-949: Thanks Alok. I am able to reproduce the problem. I was only

[jira] Updated: (PIG-955) Skewed join generates incorrect results

2009-09-11 Thread Ying He (JIRA)
[ https://issues.apache.org/jira/browse/PIG-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying He updated PIG-955: Description: SkewedPartitioner doesn't partition the skewed keys in partition table (first table) correctly. This can

[jira] Updated: (PIG-955) Skewed join generates incorrect results

2009-09-11 Thread Ying He (JIRA)
[ https://issues.apache.org/jira/browse/PIG-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying He updated PIG-955: Description: SkewedPartitioner doesn't the skewed keys in partition table correctly. This can cause data loss. (was:

[jira] Commented: (PIG-955) Skewed join generates incorrect results

2009-09-11 Thread Ying He (JIRA)
[ https://issues.apache.org/jira/browse/PIG-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754370#action_12754370 ] Ying He commented on PIG-955: - This is not related to replicate join. The original description is

[jira] Updated: (PIG-954) Skewed join fails when pig.skewedjoin.reduce.memusage is not configured

2009-09-11 Thread Ying He (JIRA)
[ https://issues.apache.org/jira/browse/PIG-954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying He updated PIG-954: Description: query fails if pig.skewedjoin.reduce.memusage is not configured. (was: Fragmented replicated join has a

[jira] Updated: (PIG-929) Default value of memusage for skewed join is not correct

2009-09-11 Thread Ying He (JIRA)
[ https://issues.apache.org/jira/browse/PIG-929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying He updated PIG-929: Description: default value pig.skewedjoin.reduce.memusage , which is used in skewed join, should be set to 0.3 (was:

[jira] Resolved: (PIG-641) Fragment replicate join does not work in local mode

2009-09-11 Thread Pradeep Kamath (JIRA)
[ https://issues.apache.org/jira/browse/PIG-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pradeep Kamath resolved PIG-641. Resolution: Fixed Fix Version/s: 0.4.0 This issue is fixed in current trunk since LocalLogToPhy

[jira] Commented: (PIG-954) Skewed join fails when pig.skewedjoin.reduce.memusage is not configured

2009-09-11 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/PIG-954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754396#action_12754396 ] Hadoop QA commented on PIG-954: --- +1 overall. Here are the results of testing the latest attachm

[jira] Updated: (PIG-954) Skewed join fails when pig.skewedjoin.reduce.memusage is not configured

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-954: --- Resolution: Fixed Fix Version/s: 0.4.0 Status: Resolved (was: Patch Available) patch

[jira] Updated: (PIG-955) Skewed join generates incorrect results

2009-09-11 Thread Ying He (JIRA)
[ https://issues.apache.org/jira/browse/PIG-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying He updated PIG-955: Attachment: PIG-955.patch2 add Junit test > Skewed join generates incorrect results > -

[jira] Updated: (PIG-955) Skewed join generates incorrect results

2009-09-11 Thread Olga Natkovich (JIRA)
[ https://issues.apache.org/jira/browse/PIG-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-955: --- Status: Patch Available (was: Open) > Skewed join generates incorrect results > -

[jira] Created: (PIG-957) Tutorial is broken with 0.4 branch and trunk

2009-09-11 Thread Olga Natkovich (JIRA)
Tutorial is broken with 0.4 branch and trunk Key: PIG-957 URL: https://issues.apache.org/jira/browse/PIG-957 Project: Pig Issue Type: Bug Affects Versions: 0.3.0 Reporter: Olga Nat

[jira] Updated: (PIG-957) Tutorial is broken with 0.4 branch and trunk

2009-09-11 Thread Pradeep Kamath (JIRA)
[ https://issues.apache.org/jira/browse/PIG-957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pradeep Kamath updated PIG-957: --- Attachment: PIG-957.patch Attached patch to address the issue. LOJoin's getSchema() now keeps both the

[jira] Updated: (PIG-957) Tutorial is broken with 0.4 branch and trunk

2009-09-11 Thread Pradeep Kamath (JIRA)
[ https://issues.apache.org/jira/browse/PIG-957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pradeep Kamath updated PIG-957: --- Status: Patch Available (was: Open) > Tutorial is broken with 0.4 branch and trunk > --

[jira] Commented: (PIG-955) Skewed join generates incorrect results

2009-09-11 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/PIG-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754466#action_12754466 ] Hadoop QA commented on PIG-955: --- +1 overall. Here are the results of testing the latest attachm

[jira] Commented: (PIG-957) Tutorial is broken with 0.4 branch and trunk

2009-09-11 Thread Daniel Dai (JIRA)
[ https://issues.apache.org/jira/browse/PIG-957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754468#action_12754468 ] Daniel Dai commented on PIG-957: +1 > Tutorial is broken with 0.4 branch and trunk >

[jira] Commented: (PIG-957) Tutorial is broken with 0.4 branch and trunk

2009-09-11 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/PIG-957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754478#action_12754478 ] Hadoop QA commented on PIG-957: --- -1 overall. Here are the results of testing the latest attachm