[
https://issues.apache.org/jira/browse/PIG-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598741#comment-14598741
]
kexianda commented on PIG-4607:
-------------------------------
Hi [~mohitsabharwal],
1. Why TestRank2 passed?
in the verifyExpected function, there is a bug. When the actual result is
empty, it also pass.
I will fix this bug.
2. what's the purpose of sparkCounter & localCounter in CounterConverter
>From my understanding, localCounter is for local rank. sparkCounter is the
>offset for next partition when merge them togather.
For instance:
Rank {A, B, B, B, C, C, D, D, D};
CounterConverter will convert it to(say we have two partitions):
partition 0:
{
(0, 1, 1, {(A)}), /*0 is partition index, 1 is sparkCounter, 1 is localCounter*/
(0, 4, 2, {(B),(B),(B)}) /* 4 will be used for offset in the next partition
when merged*/
}
partition 1:
{
(1, 1, 1, {(C),(C)}),
(1, 5, 3, {(D),(D),(D)})
}
> Enable "TestRank1","TestRank3" unit tests in spark mode
> -------------------------------------------------------
>
> Key: PIG-4607
> URL: https://issues.apache.org/jira/browse/PIG-4607
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: kexianda
> Fix For: spark-branch
>
> Attachments: PIG-4607.patch
>
>
> In https://builds.apache.org/job/Pig-spark/216/#showFailuresLink, unit tests
> about TestRank1, TestRank3:
> org.apache.pig.test.TestRank1.testRank02RowNumber
> org.apache.pig.test.TestRank1.testRank01RowNumber
> org.apache.pig.test.TestRank3.testRankWithSplitInMap
> org.apache.pig.test.TestRank3.testRankWithSplitInReduce
> org.apache.pig.test.TestRank3.testRankCascade
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
