None of the parameters in mapred-site.xml are respected. they're being
over-ridden by default configurations in the following file -
hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
public String get(String name) {
String val = getProps().getProperty(name);
if (val == null) {
val = CustomConf.getDefault(name);
}
return substituteVars(val);
}
Running via java APIs, not via grunt shell.
mapred-site.xml exists on the classpath.
If I have to make a change to add resource mapred-site.xml in apache pig's
org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
the following changes aren't enough -
* private static final String MAPRED_SITE = "mapred-site.xml";*
* jc.addResource(MAPRED_SITE);
*
recomputeProperties(jc, properties);
My question is, how should I add a new configuration resource file
myconfig-site.xml to pig and get pig to use it without making changes to
hadoop layer (Configuration.java or JobConf.java)?
Cheers,
Suhas.
On Tue, Aug 6, 2013 at 1:32 PM, Prashant Kommireddi <[email protected]>wrote:
> Can you tell us how exactly you are running the pig script? Is your
> mapred-site.xml on the classpath? Are you trying to run this via grunt or
> Java APIs?
>
>
> On Tue, Aug 6, 2013 at 1:16 PM, Suhas Satish <[email protected]>
> wrote:
>
> > I am running pig on a custom hadoop implementation but it doesnt respect
> > params in mapred-site.xml.
> >
> > Looking into the code, I find that the following 2 files are slightly
> > different from stock hadoop in that some patches are not present.
> >
> > hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
> > and
> > src/mapred/org/apache/hadoop/mapred/JobConf.java
> >
> > Given the constraint that I cannot modify these files, what change
> should I
> > make within pig to recognize mapred-site.xml parameters?
> >
> > I pulled in PIG-3135 and PIG-3145 which make changes to
> > org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
> >
> > But the params in mapred-site.xml are still not getting recognized. Upon
> > remote eclipse debugging with breakpoints in the file above, this is
> what
> > I found -
> >
> > HExecutionEngine.java - jc = new jobConf()
> > calls
> > 1st call upon JobConf() constructor -
> > Configuration.get(String)
> > Configuration.getProps() --> if properties ==null, properties = new
> > Properties(); loadResources(properties, resources...);
> >
> > JobConf static constructor -
> > Configuration.addDefaultResource(mapred-site.xml)
> >
> > HExecutionEngine: 2nd call jc.addResource("mapred-site.xml") -
> > Configuration.get
> > val = getProps().getProperty(name)
> > if (val==null) val =
> > <custom_hadoop_impl_configuration_object>.getDefault(name);
> >
> > HExecutionEngine: 3rd call recomputeProperties(jc, properties) -->
> > clearing properties which were added so it gets reloaded again.
> >
> >
> > What do I ned to do to make sure the getProps().getProperty call is not
> > null so that the mapred-site.xml values are not over-ridden by defaults
> in
> > custom_hadoop implementation ?
> >
> > Thanks,
> > Suhas.
> >
>
