[
https://issues.apache.org/jira/browse/HADOOP-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16723015#comment-16723015
]
Jason Lowe commented on HADOOP-16007:
-------------------------------------
The behavior will only be noticed if an included resource overrides a
previously set property from the same resource doing the include. If the
include was overriding a value from a previously parsed resource (like
core-default.xml) then the problem does not manifest.
The parser directly sets the included properties on the conf as a side-effect
of parsing but the non-included properties are returned as a parse result and
those results are iterated to set them. The sideband processing of includes
effectively reverses the order in which properties are processed if the
xinclude appears after the property setting in the original resource.
Here's the simple code I used to test it:
{code:title=testconf.java}
import org.apache.hadoop.conf.Configuration;
class testconf {
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
System.out.println("myconf = " + conf.get("myconf"));
}
}
{code}
Using this sample code with a core-site.xml and included file setup as
described in the JIRA description, The following shows what I get at two
adjacent commits on the trunk line:
{noformat}
$ git log -1
commit f51da9c4d1423c2ac92eb4f40e973264e7e968cc
Author: Andrew Wang <[email protected]>
Date: Mon Jul 2 18:31:21 2018 +0200
HADOOP-15554. Improve JIT performance for Configuration parsing.
Contributed by Todd Lipcon.
$ mvn clean && mvn install -Pdist -DskipTests -DskipShade -Dmaven.javadoc.skip
-am -pl :hadoop-common
[...]
$ java -cp
"hadoop/testconf:hadoop/apache/hadoop/hadoop-common-project/hadoop-common/target/hadoop-common-3.2.0-SNAPSHOT/share/hadoop/common/*:hadoop/apache/hadoop/hadoop-common-project/hadoop-common/target/hadoop-common-3.2.0-SNAPSHOT/share/hadoop/common/lib/*:."
testconf
myconf = val1
{noformat}
So the above shows the broken behavior. core-site.xml set myconf to val1 then
xincluded another file which set it to val2, yet the property acts as if the
xinclude occurred at the top of core-site.xml. Moving one commit earlier in
time shows the expected behavior:
{noformat}
$ git checkout HEAD~1
Previous HEAD position was f51da9c... HADOOP-15554. Improve JIT performance for
Configuration parsing. Contributed by Todd Lipcon.
HEAD is now at 5d748bd... HDFS-13702. Remove HTrace hooks from DFSClient to
reduce CPU usage. Contributed by Todd Lipcon.
$ mvn clean && mvn install -Pdist -DskipTests -DskipShade -Dmaven.javadoc.skip
-am -pl :hadoop-common
[...]
$ java -cp
"hadoop/testconf:hadoop/apache/hadoop/hadoop-common-project/hadoop-common/target/hadoop-common-3.2.0-SNAPSHOT/share/hadoop/common/*:hadoop/apache/hadoop/hadoop-common-project/hadoop-common/target/hadoop-common-3.2.0-SNAPSHOT/share/hadoop/common/lib/*:."
testconf
myconf = val2
{noformat}
> Order of property settings is incorrect when includes are processed
> -------------------------------------------------------------------
>
> Key: HADOOP-16007
> URL: https://issues.apache.org/jira/browse/HADOOP-16007
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 3.2.0, 3.1.1, 3.0.4
> Reporter: Jason Lowe
> Priority: Blocker
>
> If a configuration file contains a setting for a property then later includes
> another file that also sets that property to a different value then the
> property will be parsed incorrectly. For example, consider the following
> configuration file:
> {noformat}
> <configuration xmlns:xi="http://www.w3.org/2001/XInclude";>
> <property>
> <name>myprop</name>
> <value>val1</value>
> </property>
> <xi:include href="/some/other/file.xml"/>
> </configuration>
> {noformat}
> with the contents of /some/other/file.xml as:
> {noformat}
> <property>
> <name>myprop</name>
> <value>val2</value>
> </property>
> {noformat}
> Parsing this configuration should result in myprop=val2, but it actually
> results in myprop=val1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
