[
https://issues.apache.org/jira/browse/PIG-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452265#comment-13452265
]
Rohini Palaniswamy commented on PIG-2912:
-----------------------------------------
Dmitriy,
This patch is mainly to address a behaviour change in hadoop between 20 and
23 in the way instantiation of new JobContext is done and to deal with it in
HadoopShims so that JobContext objects in the backend do not get overwritten in
case of multiple stores. PIG-2578 main problem was with JobControlCompiler and
it changed frontend behaviour. I will create a separate jira to add test case
for PIG-2578.
Hadoop 20:
{code}
JobContext(JobConf conf, org.apache.hadoop.mapreduce.JobID jobId,
Progressable progress) {
super(conf, jobId); //Gets cloned
this.job = conf;
this.progress = progress;
}
{code}
Hadoop 23:
{code}
public JobContextImpl(Configuration conf, JobID jobId) {
if (conf instanceof JobConf) {
this.conf = (JobConf)conf; //Gets assigned
} else {
this.conf = new JobConf(conf);
}
{code}
> Pig should clone JobConf while creating JobContextImpl and
> TaskAttemptContextImpl in Hadoop23
> ---------------------------------------------------------------------------------------------
>
> Key: PIG-2912
> URL: https://issues.apache.org/jira/browse/PIG-2912
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.3, 0.10.1
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.9.3, 0.11, 0.10.1
>
> Attachments: PIG-2912-branch09.patch, PIG-2912-branch10.patch,
> PIG-2912-trunk.patch
>
>
> There is change in the semantics of
> JobContext::JobContext(Configuration, JobID). While in .20, the Config was
> cloned, in .23 the Config is adopted (if it's a JobConf). This causes the same
> Configuration instance to be written-to for different tables in the same job.
> It would affect multi store commands in pig on Hadoop 23/2.0. The
> cloning in HadoopShims was part of PIG-2578 but was reverted to other issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
