tl;dr This was borne out of an investigation of a hadoop issue at our company. It appears that prior to https://issues.apache.org/jira/browse/HADOOP-9450 the hadoop configuration directory (HADOOP_CONF_DIR) was always the first entry in the classpath, regardless of HADOOP_USER_CLASSPATH_FIRST or HADOOP_CLASSPATH. But after HADOOP-9450, if HADOOP_USER_CLASSPATH_FIRST is set and the user provides his/her version of *-site.xml through HADOOP_CLASSPATH, the user would end up trumping the hadoop configuration. And I believe this behavior is preserved after Allen's changes (HADOOP-9902).
Is this an intended behavior? What I'm not sure of is whether we expect the client to be able to override the site.xml files the hadoop configuration provides. If that is true, then it is working as desired. If not, we'd need to fix this behavior. Thoughts? Thanks, Sangjin
