[
https://issues.apache.org/jira/browse/HADOOP-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14976388#comment-14976388
]
Daniel Templeton commented on HADOOP-12512:
-------------------------------------------
Let me make sure I understand the issue. You have a custom root logger that
uses an optional config file. When Log4j instantiates your logger, you add the
optional config file to the list of default resources on the Configuration.
Then, when the FsShell starts, it tries to load your config file, fails, and
throws an exception. Is that correct?
Why are you adding your file as a default resource and not a regular resource
(addResource()) ? That's not the source of the issue, but I'm curious.
Looking at the code, it looks to me like the first thing FsShell does is load
all resources with quiet=false, meaning at that time there shouldn't be
anything optional in the resource list. All subsequent resource loading is
done with quiet=true, allowing optional resources. It seems to me the issue is
that you're adding the resource too early. Is it possible to do the
initialization of your class lazily so that it doesn't go hunting for resource
files until it's past the FsShell startup?
> hadoop fs -ls / fails when we use Custom -Dhadoop.root.logger
> --------------------------------------------------------------
>
> Key: HADOOP-12512
> URL: https://issues.apache.org/jira/browse/HADOOP-12512
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.7.0
> Reporter: Prabhu Joseph
> Attachments: HADOOP-12512.001.patch
>
>
> hadoop fs -ls / fails with below error when we use Custom
> -Dhadoop.root.logger that creates Configuration object and adds
> defaultResource custom-conf.xml with quiet = false.
> custom-conf.xml is optional configuration.
> Exception in thread "main" java.lang.RuntimeException: custom-conf.xml not
> found
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2612)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2531)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2444)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1156)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1128)
> at
> org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1464)
> at
> org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:321)
> at
> org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:487)
> at
> org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
> at
> org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
>
>
> ISSUE:
> ######
> There is a Logic issue in Configuration class and defaultResources list.
> Configuration is shared by classes. Configuration has a shared list of
> default resources added by the classes.
> If class A wants x,y,z resources and says all are optional using quiet =
> false, Configuration loads if they are present else skips and adds them to
> list.
> Now shared list i.e defaultResources has x,y,z
> Now if class B wants x resource and says it as mandatory , loadResources
> scans the entire list and treats them mandatory. So during scan of y, it will
> fail.
> where A is Custom Class
> and B is FsShell
> FsShell checks for custom-conf.xml and treats it mandatory and fails.
> 1. The mandatory/optional has to be resource wise. [OR]
> 2. defaultResources should not be shared.
> Both of them looks complex. And simple fix is the below.
> 1. when loadResource skips initially if resource not found, it has to remove
> the entry from defaultResource list as well. There is no use in having a
> resource
> which is not at classpath in the list.
> CODE CHANGE: class org.apache.hadoop.conf.Configuration
> ############
> private Resource loadResource(Properties properties, Resource wrapper, boolean
> quiet) {}
> ......
> if (root == null) {
> if (doc == null) {
> if (quiet) {
> defaultResources.remove(resource); // FIX: During skip, remove
> Resource from shared list
> return null;
> }
> throw new RuntimeException(resource + " not found");
> }
> root = doc.getDocumentElement();
> }
> ........
> Tested after code fix, runs successfully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
