[
https://issues.apache.org/jira/browse/PIG-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang_intel updated PIG-4593:
----------------------------------
Attachment: PIG-4593.patch
[~mohitsabharwal],[~kexianda],[~praveenr019],[~xuefuz]:
In PIG-4593.patch, changes are:
1. fix org.apache.pig.test.TestMultiQueryLocal.testMultiQueryWithTwoStores,
org.apache.pig.test.TestMultiQueryLocal.testMultiQueryWithThreeStores,
org.apache.pig.test.TestMultiQueryLocal.testMultiQueryWithTwoLoads
The reason why these three unit test fail are same, let's explain why
org.apache.pig.test.TestMultiQueryLocal.testMultiQueryWithTwoStores fails:
when "ant -Dtestcase=TestMultiQueryLocal -Dexectype=spark -Dhadoopversion=23
test" is executed,
it throws exception in
[e.printStackTrace();|https://github.com/apache/pig/blob/spark/test/org/apache/pig/test/TestMultiQueryLocal.java#L110].
The exception is "java.lang.RuntimeException: Unknown test.exec.type: spark"
The exception is from
https://github.com/apache/pig/blob/spark/test/org/apache/pig/test/MiniGenericCluster.java#L164
where we don't deal with spark case in MiniGenericCluster#getLauncher. In
PIG-4593.patch, it adds the spark case in MiniGenericCluster#getLauncher.
In PIG-4593.patch, it adds SparkMiniCluster#getLauncher in
pig/shims/test/hadoop23/org/apache/pig/test/SparkMiniCluster.java not
pig/shims/test/hadoop20/org/apache/pig/test/SparkMiniCluster.java. The reason
why i only add SparkMiniCluster#getLauncher in hadoop23 directory is it seems
that we don't support hadoop-1 anymore and from current pig release version, we
only support hadoop2.
> Enable "TestMultiQueryLocal" in spark mode
> ------------------------------------------
>
> Key: PIG-4593
> URL: https://issues.apache.org/jira/browse/PIG-4593
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4593.patch
>
>
> in https://builds.apache.org/job/Pig-spark/211/#showFailuresLink, it shows
> that following unit tests fail:
> org.apache.pig.test.TestMultiQueryLocal.testMultiQueryWithTwoStores
> org.apache.pig.test.TestMultiQueryLocal.testMultiQueryWithThreeStores
> org.apache.pig.test.TestMultiQueryLocal.testMultiQueryWithTwoLoads
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
