[
https://issues.apache.org/jira/browse/PIG-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329849#comment-14329849
]
Mohit Sabharwal commented on PIG-4374:
--------------------------------------
[~praveenr019] I know there are some follow-up items on this on RB, but since
it's blocking some other tasks, do you feel we should commit as is? It is not
adding unit tests failures other than skewed join and merge join ones, which is
expected since these are TODO items for Spark. I'd be happy to take the
follow-up items I mentioned on RB in follow-on JIRAs.
Other tasks that depend on this include PIG-4221 and PIG-4422. Once this patch
is committed, I can start work on PIG-4221/2 as well as follow-up items on this
jira.
> Add SparkPlan in spark package
> ------------------------------
>
> Key: PIG-4374
> URL: https://issues.apache.org/jira/browse/PIG-4374
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Attachments: How to change a physical plan to a spark plan.docx,
> PIG-4374_1.patch, PIG-4374_2.patch.1, PIG-4374_3.patch.1, PIG-4374_4.patch,
> PIG-4374_4.patch_Jenkins_result.png, PIG-4374_6.patch,
> PIG-4374_6_patch_jekins.png, Pig-spark #76 [Jenkins .png,
> jenkins_PIG-4374_1_patch.png.png
>
>
> in current code, there are following class in mapreduce and tez package.
> mapreduce:
> MRCompiler
> MROperPlan MROpPlanVisitor
> MapReduceOper
> tez:
> TezCompiler
> TezOperPlan TezOpPlanVisitor
> TezOperator
> Following class needs to be added in spark package:
> SparkCompiler
> SparkOperPlan SparkOpPlanVisitor
> SparkOperator
> Current code need to be refactored because of above adding classes. Some unit
> tests like TestStoreInstances fail because of it.
> following are the error of unit test TestStoreInstances:
> Error Message
> num jobs expected:<1> but was:<0>
> Stacktrace
> junit.framework.AssertionFailedError: num jobs expected:<1> but was:<0>
> at
> org.apache.pig.test.TestStoreInstances.testBackendStoreCommunication(TestStoreInstances.java:122)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
