Dmytro Vitiuk created AMBARI-25551:
--------------------------------------
Summary: NodeManager doesn't work after adding a new host to a
patch upgraded cluster
Key: AMBARI-25551
URL: https://issues.apache.org/jira/browse/AMBARI-25551
Project: Ambari
Issue Type: Bug
Components: ambari-server
Affects Versions: 2.7.5
Reporter: Dmytro Vitiuk
Fix For: 2.7.6
By default "mapreduce.admin.user.env" property for mapreduce2 and
"tez.am.launch.env"/"tez.task.launch.env" for tez contains a path with a
specific hdp version, like the next one:
{code}
LD_LIBRARY_PATH=/usr/hdp/3.1.5.xx-x/hadoop/lib/native:/usr/hdp/3.1.5.xx-x/hadoop/lib/native/Linux-{{architecture}}-64
{code}
As result after patch upgrade (for Tez, MR2, YARN) tez/mapreduce jobs point to
paths with old versions. And in the case we add new hosts with NodeManager only
(without any client) we don't have these libraries by the old paths. So I am
going to change the default value to the next one:
{code}
/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-{{architecture}}-64
{code}
This path is always exists (because NodeManager installs hadoop-client package
as dependency).
Also there is an issue with yarn-shuffle.jar path selection. By default we
choose the spark client's version in the case a cluster has at least one spark
client at any host:
{code}
/usr/hdp/<sparkClientHDPVersion>/spark2/aux/spark-*-yarn-shuffle.jar
{code}
But this way stops working in the case we have hosts without spark client. I am
going to use yarn's yarn-shuffle.jar path for the same hosts:
{code}
/usr/hdp/<yarnClientHDPVersion>/spark2/aux/spark-*-yarn-shuffle.jar
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
