Hi Bhooshan, There is a patch that addresses what you need, and is part of 0.12 (unreleased). Take a look and see if you can apply the patch to the version you are using. https://issues.apache.org/jira/browse/PIG-3135.
With this patch, the following property will allow you to override the default and pass in your own configuration. pig.use.overriden.hadoop.configs=true On Thu, Mar 28, 2013 at 6:10 PM, Bhooshan Mogal <[email protected]>wrote: > Hi Folks, > > I had implemented the Hadoop FileSystem abstract class for a storage system > at work. This implementation uses some config files that are similar in > structure to hadoop config files. They have a *-default.xml and a > *-site.xml for users to override default properties. In the class that > implemented the Hadoop FileSystem, I had added these configuration files as > default resources in a static block using > Configuration.addDefaultResource("my-default.xml") and > Configuration.addDefaultResource("my-site.xml". This was working fine and > we were able to run the Hadoop Filesystem CLI and map-reduce jobs just fine > for our storage system. However, when we tried using this storage system in > pig scripts, we saw errors indicating that our configuration parameters > were not available. Upon further debugging, we saw that the config files > were added to the Configuration object as resources, but were part of > defaultResources. However, in Main.java in the pig source, we saw that the > Configuration object was created as Configuration conf = new > Configuration(false);, thereby setting loadDefaults to false in the conf > object. As a result, properties from the default resources (including my > config files) were not loaded and hence, unavailable. > > We solved the problem by using Configuration.addResource instead of > Configuration.addDefaultResource, but still could not figure out why Pig > does not use default resources? > > Could someone on the list explain why this is the case? > > Thanks, > -- > Bhooshan >
