[
https://issues.apache.org/jira/browse/PIG-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
prateek vaishnav updated PIG-4807:
----------------------------------
Release Note:
Following test cases have been fixed -
1. org.apache.pig.test.TestEvalPipelineLocal.testSetLocationCalledInFE
2. org.apache.pig.test.TestEvalPipelineLocal.testExplainInDotGraph
3. org.apache.pig.test.TestEvalPipelineLocal.testSortWithUDF
1 was failing because of not saving UDF_CONTEXT configuration in jobConf. This
leads UDFContext.getUDFProperties() to return NULL.
public Properties getUDFProperties(Class c) {
UDFContextKey k = generateKey(c, null);
Properties p = udfConfs.get(k);
if (p == null) {
p = new Properties();
udfConfs.put(k, p);
}
return p;
}
Here, udfConfs remains empty even when it was set while processing the pig
query.
udf configuration in jobConf is getting lost while running the job.
In the code udf configuration is meant to be saved by serializing them in
jobConf.
Currently, serialization is done before loading configuration in jobConf.
It is done in 'newJobConf(PigContext pigContext)'
It needs to be done after loading configuration.
JobConf jobConf = SparkUtil.newJobConf(pigContext);
configureLoader(physicalPlan, op, jobConf);
UDFContext.getUDFContext().serialize(jobConf);
2 was failing because of pig-spark not supporting 'explain' in dot format. I
have added the DotSparkPrinter to fix the same.
3 was failing because instead of UDFSortComparator, SortConveter class was
using SortComparator.
JavaPairRDD<Tuple, Object> sorted = r.sortByKey(
sortOperator.new SortComparator(), true);
It should be using mComparator stored in POSort class. I have changed it to
following
JavaPairRDD<Tuple, Object> sorted = r.sortByKey(
sortOperator.getMComparator(), true);
Status: Patch Available (was: Open)
> Fix test cases of "TestEvalPipelineLocal" test suite.
> -----------------------------------------------------
>
> Key: PIG-4807
> URL: https://issues.apache.org/jira/browse/PIG-4807
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Affects Versions: spark-branch
> Reporter: prateek vaishnav
> Assignee: prateek vaishnav
> Attachments: diff_1
>
>
> This jira is created to address the failure of test cases
> org.apache.pig.test.TestEvalPipelineLocal.testSetLocationCalledInFE
> org.apache.pig.test.TestEvalPipelineLocal.testExplainInDotGraph
> org.apache.pig.test.TestEvalPipelineLocal.testSortWithUDF
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
