[
https://issues.apache.org/jira/browse/PIG-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13561272#comment-13561272
]
Prashant Kommireddi commented on PIG-3135:
------------------------------------------
To get around the current limitation of depending on *site.xml files being set
on classpath, we could add a property "pig.use.override.configs" to be able to
provide your own configs and have JobConf be built from that.
Currently:
{code}
jc = new JobConf();
{code}
Proposal:
1. If "pig.use.override.configs" is present, generate JobConf using properties
{code}
jc = new JobConf(ConfigurationUtil.toConfiguration(properties));
{code}
This change would be backward compatible, and those who wish to bypass
classpath limitation can do so by setting the override property.
Thoughts?
> HExecutionEngine should look for resources in user passed Properties
> --------------------------------------------------------------------
>
> Key: PIG-3135
> URL: https://issues.apache.org/jira/browse/PIG-3135
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10.0
> Reporter: Prashant Kommireddi
>
> Looking at this snippet:
> {code}
> private void init(Properties properties) throws ExecException {
> .
> .
> .
> // Check existence of hadoop-site.xml or core-site.xml
> Configuration testConf = new Configuration();
> ClassLoader cl = testConf.getClassLoader();
> URL hadoop_site = cl.getResource( HADOOP_SITE );
> URL core_site = cl.getResource( CORE_SITE );
>
> if( hadoop_site == null && core_site == null ) {
> throw new ExecException("Cannot find hadoop configurations in
> classpath (neither hadoop-site.xml nor core-site.xml was found in the
> classpath)." +
> " If you plan to use local mode, please put -x local
> option in command line",
> 4010);
> }
> {code}
> This assumes the resources (*-site.xml) are set on the classpath, but this
> will not always be the case when run with Pig's Java APIs. One could want to
> programatically set the resources and the code here should additionally check
> if they are available in there.
> Example: When a Configuration object is created and resources are added
> before passing it on to Pig.
> {code}
> Configuration conf = new Configuration(false);
> conf.addResource("foo/core-site.xml");
> conf.addResource("bar/hadoop-site.xml");
> PigServer pServer = new PigServer(ExecType.MAPREDUCE, conf);
> {code}
> The above conf is not used right now to obtain resources.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
