Thanks for the clarification. Yes I agree that there are many ways for users to override *-site.xml regardless of what the hadoop scripts do. It was more of a question of the interpretation of intent, and I got the answer. Thanks!
Sangjin On Wed, Sep 10, 2014 at 4:34 PM, Chris Nauroth <[email protected]> wrote: > I participated on the HADOOP-9450 code review. There was some debate about > the significance of keeping HADOOP_CONF_DIR first. Ultimately, there was > agreement that "FIRST" really ought to mean first. Apparently, other > Hadoop ecosystem projects have made the same interpretation and implemented > their scripts accordingly. Just echoing what Allen said, there were always > plenty of ways for a user to override *-site.xml too. > > Of course, I'm sorry to hear the change caused an issue at your company, > Sangjin. > > Chris Nauroth > Hortonworks > http://hortonworks.com/ > > > On Wed, Sep 10, 2014 at 10:55 AM, Allen Wittenauer <[email protected]> > wrote: > > > > > On Sep 10, 2014, at 10:14 AM, Sangjin Lee <[email protected]> wrote: > > > > > if HADOOP_USER_CLASSPATH_FIRST is > > > set and the user provides his/her version of *-site.xml through > > > HADOOP_CLASSPATH, the user would end up trumping the hadoop > > configuration. > > > And I believe this behavior is preserved after Allen's changes > > > (HADOOP-9902). > > > > > > > I'd be surprised if 9902 changed this behavior, especially given > > that HADOOP_CONF_DIR is added fairly quickly to the CLASSPATH and > > HADOOP_USER_CLASSPATH is processed last or near last. > > > > > Is this an intended behavior? What I'm not sure of is whether we expect > > the > > > client to be able to override the site.xml files the hadoop > configuration > > > provides. If that is true, then it is working as desired. If not, we'd > > need > > > to fix this behavior. > > > > > > Thoughts? > > > > > > > Users have always been able to override the *-site.xml files via > > java properties, jobconf/etc, pointing HADOOP_CONF_DIR to somewhere > else, > > or providing --conf flags to shell commands. In other words, this isn't > > new behavior, but provides yet another option to override the provided > > settings. > > > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
