[
https://issues.apache.org/jira/browse/PIG-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15361900#comment-15361900
]
liyunzhang_intel commented on PIG-4281:
---------------------------------------
[~kexianda]: TestFinish$MyEvalFunction.finish() is not called by the backed
JVM which is launched by spark.
in following stack info it shows that it is called by main thread
{code}
main@1, prio=5, in group 'main', status: 'RUNNING'
at
org.apache.pig.test.TestFinish$MyEvalFunction.finish(TestFinish.java:70)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.finish(POUserFunc.java:504)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.UDFFinishVisitor.visitUserFunc(UDFFinishVisitor.java:36)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.visit(POUserFunc.java:538)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.visit(POUserFunc.java:62)
at
org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:87)
at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.plans.PhyPlanVisitor.visitPOForEach(PhyPlanVisitor.java:150)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.visit(POForEach.java:139)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.visit(POForEach.java:55)
at
org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:87)
{code}
so we need PigMapReduce.sJobConfInternal.set() in the main thread at
SparkLauncher.initialize().
> Fix TestFinish for Spark engine
> -------------------------------
>
> Key: PIG-4281
> URL: https://issues.apache.org/jira/browse/PIG-4281
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4281.patch, PIG-4281_2.patch, PIG-4281_3.patch,
> TEST-org.apache.pig.test.TestFinish.txt
>
>
> error log is attached
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
