Can you tell us how exactly you are running the pig script? Is your mapred-site.xml on the classpath? Are you trying to run this via grunt or Java APIs?
On Tue, Aug 6, 2013 at 1:16 PM, Suhas Satish <[email protected]> wrote: > I am running pig on a custom hadoop implementation but it doesnt respect > params in mapred-site.xml. > > Looking into the code, I find that the following 2 files are slightly > different from stock hadoop in that some patches are not present. > > hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java > and > src/mapred/org/apache/hadoop/mapred/JobConf.java > > Given the constraint that I cannot modify these files, what change should I > make within pig to recognize mapred-site.xml parameters? > > I pulled in PIG-3135 and PIG-3145 which make changes to > org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java > > But the params in mapred-site.xml are still not getting recognized. Upon > remote eclipse debugging with breakpoints in the file above, this is what > I found - > > HExecutionEngine.java - jc = new jobConf() > calls > 1st call upon JobConf() constructor - > Configuration.get(String) > Configuration.getProps() --> if properties ==null, properties = new > Properties(); loadResources(properties, resources...); > > JobConf static constructor - > Configuration.addDefaultResource(mapred-site.xml) > > HExecutionEngine: 2nd call jc.addResource("mapred-site.xml") - > Configuration.get > val = getProps().getProperty(name) > if (val==null) val = > <custom_hadoop_impl_configuration_object>.getDefault(name); > > HExecutionEngine: 3rd call recomputeProperties(jc, properties) --> > clearing properties which were added so it gets reloaded again. > > > What do I ned to do to make sure the getProps().getProperty call is not > null so that the mapred-site.xml values are not over-ridden by defaults in > custom_hadoop implementation ? > > Thanks, > Suhas. >
