[
https://issues.apache.org/jira/browse/HADOOP-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14980394#comment-14980394
]
Prabhu Joseph commented on HADOOP-12512:
----------------------------------------
Yes that solution is risk. Maybe how about the below ones:
1. Either mandatory / optional field has to be resource wise. [OR]
2. defaultResources list should not be shared among classes
> hadoop fs -ls / fails when we use Custom -Dhadoop.root.logger
> --------------------------------------------------------------
>
> Key: HADOOP-12512
> URL: https://issues.apache.org/jira/browse/HADOOP-12512
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.7.0
> Reporter: Prabhu Joseph
> Attachments: HADOOP-12512.001.patch
>
>
> hadoop fs -ls / fails with below error when we use Custom
> -Dhadoop.root.logger that creates Configuration object and adds
> defaultResource custom-conf.xml with quiet = false.
> custom-conf.xml is optional configuration.
> Exception in thread "main" java.lang.RuntimeException: custom-conf.xml not
> found
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2612)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2531)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2444)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1156)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1128)
> at
> org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1464)
> at
> org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:321)
> at
> org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:487)
> at
> org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
> at
> org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
>
>
> ISSUE:
> ######
> There is a Logic issue in Configuration class and defaultResources list.
> Configuration is shared by classes. Configuration has a shared list of
> default resources added by the classes.
> If class A wants x,y,z resources and says all are optional using quiet =
> false, Configuration loads if they are present else skips and adds them to
> list.
> Now shared list i.e defaultResources has x,y,z
> Now if class B wants x resource and says it as mandatory , loadResources
> scans the entire list and treats them mandatory. So during scan of y, it will
> fail.
> where A is Custom Class
> and B is FsShell
> FsShell checks for custom-conf.xml and treats it mandatory and fails.
> 1. The mandatory/optional has to be resource wise. [OR]
> 2. defaultResources should not be shared.
> Both of them looks complex. And simple fix is the below.
> 1. when loadResource skips initially if resource not found, it has to remove
> the entry from defaultResource list as well. There is no use in having a
> resource
> which is not at classpath in the list.
> CODE CHANGE: class org.apache.hadoop.conf.Configuration
> ############
> private Resource loadResource(Properties properties, Resource wrapper, boolean
> quiet) {}
> ......
> if (root == null) {
> if (doc == null) {
> if (quiet) {
> defaultResources.remove(resource); // FIX: During skip, remove
> Resource from shared list
> return null;
> }
> throw new RuntimeException(resource + " not found");
> }
> root = doc.getDocumentElement();
> }
> ........
> Tested after code fix, runs successfully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
