[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-11-22 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15689206#comment-15689206 ] liyunzhang_intel commented on PIG-5029: --- [~kexianda]: {quote} I hava a question:

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-11-22 Thread Xianda Ke (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688696#comment-15688696 ] Xianda Ke commented on PIG-5029: Hi [~kellyzly], Salted key solution seem OK. JDK's Random

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-10-31 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15624556#comment-15624556 ] liyunzhang_intel commented on PIG-5029: --- [~knoguchi]: Before we discussed a lot abo

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-28 Thread Koji Noguchi (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530152#comment-15530152 ] Koji Noguchi commented on PIG-5029: --- {quote} bq. how is pig handling skew for MR/TEZ? Samp

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-28 Thread Rohini Palaniswamy (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530114#comment-15530114 ] Rohini Palaniswamy commented on PIG-5029: - bq. how is pig handling skew for MR/TEZ?

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-28 Thread Koji Noguchi (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530104#comment-15530104 ] Koji Noguchi commented on PIG-5029: --- Thanks a million [~tgraves]. > Optimize sort case wh

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530065#comment-15530065 ] Thomas Graves commented on PIG-5029: Is the question whether spark supports maps that ar

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-27 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15528532#comment-15528532 ] liyunzhang_intel commented on PIG-5029: --- [~knoguchi]: thanks for patience on this jira

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-27 Thread Koji Noguchi (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15526775#comment-15526775 ] Koji Noguchi commented on PIG-5029: --- [~kellyzly], I believe I explained how pure RANDOM ke

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-26 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524845#comment-15524845 ] liyunzhang_intel commented on PIG-5029: --- [~vanzin]: I guess what [~rohini] and [~knog

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524832#comment-15524832 ] Marcelo Vanzin commented on PIG-5029: - I'm not sure I understand the question. If you're

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-26 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524780#comment-15524780 ] liyunzhang_intel commented on PIG-5029: --- [~rohini] and [~knoguchi]: thanks for your pa

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-26 Thread Koji Noguchi (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524159#comment-15524159 ] Koji Noguchi commented on PIG-5029: --- [~rohini], can you take this ? Obviously I'm not exp

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-25 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15521883#comment-15521883 ] liyunzhang_intel commented on PIG-5029: --- [~vanzin]: Thanks for your comment, here i h

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-23 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15516915#comment-15516915 ] Marcelo Vanzin commented on PIG-5029: - Not really my area of expertise; but this reminds

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-22 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515605#comment-15515605 ] liyunzhang_intel commented on PIG-5029: --- [~kexianda], [~mohitsabharwal] , [~pallavi.ra

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-19 Thread Xianda Ke (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15505624#comment-15505624 ] Xianda Ke commented on PIG-5029: [~kellyzly], when task re-run, the partitioning is not the

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-19 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15505394#comment-15505394 ] liyunzhang_intel commented on PIG-5029: --- [~knoguchi]: {quote} If node goes down after

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-19 Thread Koji Noguchi (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15505381#comment-15505381 ] Koji Noguchi commented on PIG-5029: --- bq. Hadoop will try new task attempt only when last t

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-19 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15505358#comment-15505358 ] liyunzhang_intel commented on PIG-5029: --- [~knoguchi]: if this has nothing to do with s

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-19 Thread Koji Noguchi (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503610#comment-15503610 ] Koji Noguchi commented on PIG-5029: --- [~kellyzly], this has nothing to do with speculative

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-17 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15500402#comment-15500402 ] liyunzhang_intel commented on PIG-5029: --- [~knoguchi]: Thanks for your reply. Here is

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-14 Thread Rohini Palaniswamy (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15490809#comment-15490809 ] Rohini Palaniswamy commented on PIG-5029: - bq. Although spark will sample the data a

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-14 Thread Koji Noguchi (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15490595#comment-15490595 ] Koji Noguchi commented on PIG-5029: --- bq. Can you explain more about why this cause data lo

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-13 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15489116#comment-15489116 ] liyunzhang_intel commented on PIG-5029: --- [~rohini] and [~knoguchi]: Thanks for intere

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-13 Thread Rohini Palaniswamy (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488515#comment-15488515 ] Rohini Palaniswamy commented on PIG-5029: - [~knoguchi] just pointed out this jira to

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-13 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486676#comment-15486676 ] liyunzhang_intel commented on PIG-5029: --- The solution to solve the skewed data sort in