Evert Lammerts created PIG-2872:
-----------------------------------
Summary: StoreFuncInterface.setStoreLocation get's a copy of a
Configuration object
Key: PIG-2872
URL: https://issues.apache.org/jira/browse/PIG-2872
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.11
Environment: Pig trunk, Hadoop 0.20.205 with Kerberos, ElasticSearch
trunk, Wonderdog trunk
Reporter: Evert Lammerts
When an implementation of StoreFuncInterface.setStoreLocation is called from
JobControlCompiler.getJob, it is passed a copy of the Configuration that will
be used for the Job that will be submitted:
{code:title=JobControlCompiler.java}
sFunc.setStoreLocation(st.getSFile().getFileName(), new
org.apache.hadoop.mapreduce.Job(nwJob.getConfiguration()));
{code}
When a new org.apache.hadoop.mapreduce.Job is created it creates a copy of the
Configuration object, as far as I know. Thus anything added to the
Configuration object in the implementation of setStoreLocation will not be
included in the Configuration of nwJob in JobControlCompiler.getJob.
I notice this goes wrong in Wonderdog, which needs to include the Elasticsearch
configuration file in the DistributedCache. It is added to mapred.cache.files
through setStoreLocation, but this setting doesn't make it back into the Job
returned by JobControlCompiler.getJob, and is therefore never localized.
This might be intentional semantics within Pig, but I'm not familiar enough
with StoreFuncs to know whether it is.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
