[
https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130744#comment-16130744
]
Allen Wittenauer commented on YARN-7011:
----------------------------------------
bq. I am not sure if everyone would agree that it is by design at least I am
not convinced yet.
As the person who designed and wrote the code in trunk, yes, it is working as
intended.
The problem is one of consistency. There are a ton of different ways that
HADOOP_CONF_DIR can get set: external vs. internal, --config vs. no option on
CLI, HADOOP_CONF_DIR manual vs. auto-discovered (2-3 ways), present in
hadoop-env.sh vs. not, present but correct vs. present and incorrect, etc ...
branch-2's behavior when going through this is matrix is wildly unpredictable.
(and let's ignore's YARN_CONF_DIR because it just makes things worse and gets
us too far into the weeds)
Here's a fun experiment to show my point. Unpack a new 2.8.1 tar ball. Set
HADOOP_CONF_DIR in hadoop-2.8.1/etc/hadoop/hadoop-env.sh to something incorrect
like /bin. Now run hadoop classpath.
You'll find it starts out as "/bin".
Is that an incorrect behavior?
On the one hand: yes, HADOOP_CONF_DIR is pointed to a wrong location. But on
the other: no, the user explicitly told the system that it wanted /bin as the
configuration directory. branch-2 decided that in this instance /bin was
correct and went forward.
Is that example different than when --config is used to point to a
misconfigured directory? Yes. branch-2's code decided that it was going to
ignore what was in the configuration directory as told to us by the user.
Should that behavior be different?
My argument is no: the system should not suddenly decide the user was
incorrect in one instance vs. another. It absolutely must act predictably. The
user is expressly passing us a configuration directory that tells us our
configuration is actually in a different directory--very much a
misconfiguration. The system accepts that as truth and moves on.
It's also worthwhile pointing out that 3.x adds a powerful wrinkle here:
.hadoop-env allows the user to specify a different HADOOP_CONF_DIR, overwriting
the installed one. Here, redirection is key.
If you don't like the current behavior, there are only two ways out. (Let's be
clear: I'll reject any patch that continues branch-2's inconsistent behavior.)
* Option 1: always ignore HADOOP_CONF_DIR when set in hadoop-env.sh
If the code is reading hadoop-env.sh, then HADOOP_CONF_DIR is superfluous. It's
only used at that point is to give a different set of directories to act as
replacements for the files present in this one.
* Option 2: throw a warning
After reading hadoop-env.sh, detect if the value of HADOOP_CONF_DIR changed
(after calculating the full path to deal with symbolic links, etc.). If it
did, throw a warning up and then either accept the new value (current behavior)
or ignore it (see option 1).
> yarn-daemon.sh is not respecting --config option
> ------------------------------------------------
>
> Key: YARN-7011
> URL: https://issues.apache.org/jira/browse/YARN-7011
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Reporter: Sumana Sathish
> Priority: Blocker
>
> Steps to reproduce:
> 1. Copy the conf to a temporary location /tmp/Conf
> 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM
> address
> 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf
> 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml
> is not taken in while restarting RM
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
