[
https://issues.apache.org/jira/browse/PIG-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15361055#comment-15361055
]
Xianda Ke commented on PIG-4281:
--------------------------------
Hi [~kellyzly],
I agree with community's comments that a user UDF should not have access to
cluster object.
But I wonder that it won't work if we just set PigMapReduce.sJobConfInternal in
SparkLaucher( driver program). When PigMapReduce is instantiated in the
backend JVMs, I guess that sJobConfInternal is still null in spark mode.
Why PigMapReduce.sJobConfInternal works in MR mode? I guess that the reason
is that sJobConfInternal is reset in reducer/mapper
{code}
// PigGenericMapReduce.java Line 318
sJobConfInternal.set(context.getConfiguration());
{code}
To make PigMapReduce.sJobConfInternal work, it seems that we have to pass the
serialized conf bytes to the functions which will be passed to spark's RDD.
> Fix TestFinish for Spark engine
> -------------------------------
>
> Key: PIG-4281
> URL: https://issues.apache.org/jira/browse/PIG-4281
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4281.patch, PIG-4281_2.patch, PIG-4281_3.patch,
> TEST-org.apache.pig.test.TestFinish.txt
>
>
> error log is attached
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
