[
https://issues.apache.org/jira/browse/PIG-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598749#comment-14598749
]
kexianda commented on PIG-4607:
-------------------------------
1. Why TestRank2 passed?
in the verifyExpected function, there is a bug. When the actual result is
empty, it also pass.
I will fix this bug.
2. what's the purpose of sparkCounter & localCounter in CounterConverter
>From my understanding, localCounter is for local rank. sparkCounter is the
>offset for next partition when merge them togather.
For instance:
{noformat}
Rank {A, B, B, B, C, C, D, D, D};
{noformat}
CounterConverter will convert it to(say we have two partitions)
{noformat}
partition 0:
{
(0, 1, 1,{(A)}), //0 is partition index, 1 is sparkCounter, 1 is localCounter
(0, 4, 2,{(B),(B),(B)}) //4 will be used for offset in the next partition when
merged
}
partition 1:
{
(1, 1, 1,{(C),(C)}),
(1, 5, 3,{(D),(D),(D)})
}
{noformat}
when merged, partition 1 will converted to:
{noformat}
{
(1+4,{(C),(C)}), /*localCounter + offset for previous partition*/
(3+4,{(D),(D),(D)})
}
{noformat}
> Enable "TestRank1","TestRank3" unit tests in spark mode
> -------------------------------------------------------
>
> Key: PIG-4607
> URL: https://issues.apache.org/jira/browse/PIG-4607
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: kexianda
> Fix For: spark-branch
>
> Attachments: PIG-4607.patch
>
>
> In https://builds.apache.org/job/Pig-spark/216/#showFailuresLink, unit tests
> about TestRank1, TestRank3:
> org.apache.pig.test.TestRank1.testRank02RowNumber
> org.apache.pig.test.TestRank1.testRank01RowNumber
> org.apache.pig.test.TestRank3.testRankWithSplitInMap
> org.apache.pig.test.TestRank3.testRankWithSplitInReduce
> org.apache.pig.test.TestRank3.testRankCascade
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
