[
https://issues.apache.org/jira/browse/PIG-4616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120965#comment-15120965
]
liyunzhang_intel commented on PIG-4616:
---------------------------------------
Summarize some reasons for the unit test failures:
1. The job number is different between mr and spark mode. For multiple stores
case like TestPigRunner#testDisablePigCounters2, there will be two spark jobs
rather than one.
{code}
A = load '" + INPUT_FILE + "' as (a0:int, a1:int, a2:int);
B = filter A by a0 > 3;
store A into 'output';
store B into 'tmp/output';
{code}
The sparkplan has 1 sparkOperator after multiquery optimization but it has two
POStore. In spark mode,
1 POStore generates 1 spark job(see
org.apache.pig.tools.pigstats.spark.SparkPigStats#addJobStats). Two POStore
generates 2 spark jobs.
2.TestPigRunner#testGetHadoopCounters fails due to we did not implement
SparkJobStats#getHadoopCounters.In PIG-4616_1.patch, we only collect
hdfs_bytes_read, hdfs_bytes_written these two metrics data to HDFS_BYTES_READ
and HDFS_BYTES_WRITTEN counter.
> Fix UT errors of TestPigRunner in Spark mode
> --------------------------------------------
>
> Key: PIG-4616
> URL: https://issues.apache.org/jira/browse/PIG-4616
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Xianda Ke
> Assignee: Xianda Ke
> Fix For: spark-branch
>
> Attachments: PIG-4616.patch, PIG-4616_1.patch
>
>
> Following unit test failed:
> org.apache.pig.test.TestPigRunner.returnCodeTest
> org.apache.pig.test.TestPigRunner.testEmptyFileCounter
> org.apache.pig.test.TestPigRunner.testDisablePigCounters2
> org.apache.pig.test.TestPigRunner.simpleTest
> org.apache.pig.test.TestPigRunner.simpleTest2
> org.apache.pig.test.TestPigRunner.MQDepJobFailedTest
> org.apache.pig.test.TestPigRunner.scriptsInDfsTest
> org.apache.pig.test.TestPigRunner.testGetHadoopCounters
> org.apache.pig.test.TestPigRunner.simpleMultiQueryTest
> org.apache.pig.test.TestPigRunner.testDuplicateCounterName
> org.apache.pig.test.TestPigRunner.testRegisterExternalJar
> org.apache.pig.test.TestPigRunner.simpleMultiQueryTest2
> org.apache.pig.test.TestPigRunner.testDuplicateCounterName2
> org.apache.pig.test.TestPigRunner.returnCodeTest2
> org.apache.pig.test.TestPigRunner.orderByTest
> org.apache.pig.test.TestPigRunner.testDisablePigCounters
> org.apache.pig.test.TestPigRunner.testLongCounterName
> org.apache.pig.test.TestPigRunner.testEmptyFileCounter2
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
