[
https://issues.apache.org/jira/browse/HADOOP-14387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000501#comment-16000501
]
Steve Loughran commented on HADOOP-14387:
-----------------------------------------
The problem I'm having is
I'm trying to create a configuration object with: {{new Configuration(true)}}
in a test suite downstream. That's all.
The problem arises as the new code fails fast if any resource is missing, and
this test deployment doesn't have a core-site as it's not a live Hadoop cluster.
{code}
static {
// Add default resources
addDefaultResource("core-default.xml");
addDefaultResource("core-site.xml");
...+ check for hadoop-site.xml & load iff present
}
{code}
And the default configuration option requires not just the classpath resource
{{/core-default.xml}} (which I want) but {{/core-site.xml}}, which isn't here.
I suspect if I'd got past that and asked for an HdfsConfiguration or
YarnConfiguration I'd hit the same on yarn-site and hdfs-site XML files
mapred-site, httpfs-site.xml, kms-site.xml, ...
Essentially: we must not treat the absence of these resource files as a reason
to fail the construction. They are optional.
I like the patch and reporting, but it isn't going address this as the problem
isn't that the XML file isn't loading: it's that we're treating the failure to
find the -site.xml files as a failure
I think we need to move to making the -site.xml files optional, presumably by
adding one option for a mandatory resource, and one for an optional one. And
for backwards compatibility, probably making addDefaultResource() be for
optional., addMandatoryDefaultResource for the mandatory one, so that
downstream apps don't suffer the same fate (hbase-site.xml, ....)
> new Configuration().get() fails if core-site.xml isn't on the classpath
> -----------------------------------------------------------------------
>
> Key: HADOOP-14387
> URL: https://issues.apache.org/jira/browse/HADOOP-14387
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 3.0.0-alpha3
> Environment: test run in downstream project with no core-site in
> test/resources
> Reporter: Steve Loughran
> Assignee: Jonathan Eagles
> Priority: Blocker
> Attachments: HADOOP-14387.1.patch
>
>
> If you try to create a config via {{new Configuration()}} and there isn't a
> {{core-site.xml}} on the CP, you get a stack trace. Previously it'd just skip
> the failure to load.
> This is a regression which breaks downstream apps that don't need a core-site
> to run, but do want to load core-default &c
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
