[
https://issues.apache.org/jira/browse/PIG-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang_intel updated PIG-4281:
----------------------------------
Attachment: PIG-4281_3.patch
[~kexianda],[~mohitsabharwal],[~pallavi.rao]
PIG-4281_3.patch is to fix the comments from community about the merge work.
Changes:
1. {code: title=org.apache.pig.test.TestFinish.MyEvalFunction#finish}
@Override
public void finish() {
try {
FileSystem fs = null;
if (execType.equalsIgnoreCase("SPARK")) {
//Kelly's comment: before we get the
configuration by cluster#getConfiguration, but committee suggest that "The
cluster object is not something a user UDF will have access to". In spark mode,
PigMapReduce.sJobConfInternal.get() is null because we don't set the value for
it.
fs =
FileSystem.get(cluster.getConfiguration());
} else {
fs = FileSystem.get(PigMapReduce.sJobConfInternal.get());
}
fs.create(new Path(expectedFileName));
} catch (IOException e) {
throw new RuntimeException("Unable to create file:" +
expectedFileName);
}
}
{code}
2. set value for PigMapReduce.sJobConfInternal in SparkLauncher#initialize
> Fix TestFinish for Spark engine
> -------------------------------
>
> Key: PIG-4281
> URL: https://issues.apache.org/jira/browse/PIG-4281
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4281.patch, PIG-4281_2.patch, PIG-4281_3.patch,
> TEST-org.apache.pig.test.TestFinish.txt
>
>
> error log is attached
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
