[
https://issues.apache.org/jira/browse/AMBARI-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Wagle updated AMBARI-3074:
------------------------------------
Attachment: AMBARI-3074.patch
Ignore failure during NM directory creation, similar in behavior to the DN.
> Ambari wont start NodeManager because one of multiple folders not created
> -------------------------------------------------------------------------
>
> Key: AMBARI-3074
> URL: https://issues.apache.org/jira/browse/AMBARI-3074
> Project: Ambari
> Issue Type: Bug
> Components: agent
> Affects Versions: 1.4.0
> Reporter: Siddharth Wagle
> Assignee: Siddharth Wagle
> Fix For: 1.4.1
>
> Attachments: AMBARI-3074.patch
>
>
> {{yarn-site}} having:
> {noformat}
> "yarn.nodemanager.local-dirs" :
> "/grid/0/hadoop/yarn,/grid/1/hadoop/yarn,/grid/2/hadoop/yarn,/grid/3/hadoop/yarn,/grid/4/hadoop/yarn,/grid/5/hadoop/yarn",
> "yarn.nodemanager.log-dirs" :
> "/grid/0/hadoop/yarn,/grid/1/hadoop/yarn,/grid/2/hadoop/yarn,/grid/3/hadoop/yarn,/grid/4/hadoop/yarn,/grid/5/hadoop/yarn",
> {noformat}
> Now {{/grid/3}} was mounted as read-only due to some disk errors. Though
> other folders got successfully created, Ambari will not start the NodeManager
> process.
> {noformat}
> notice:
> /Stage[1]/Hdp::Snappy::Package/Hdp::Snappy::Package::Ln[32]/Hdp::Exec[hdp::snappy::package::ln
> 32]/Exec[hdp::snappy::package::ln 32]/returns: executed successfully
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Exec[mkdir
> -p /grid/3/hadoop/yarn]/Exec[mkdir -p /grid/3/hadoop/yarn]/returns: mkdir:
> cannot create directory `/grid/3/hadoop': Read-only file system
> err:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Exec[mkdir
> -p /grid/3/hadoop/yarn]/Exec[mkdir -p /grid/3/hadoop/yarn]/returns: change
> from notrun to 0 failed: mkdir -p /grid/3/hadoop/yarn returned 1 instead of
> one of [0] at /var/lib/ambari-agent/puppet/modules/hdp/manifests/init.pp:479
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Exec[mkdir
> -p /grid/3/hadoop/yarn]/Anchor[hdp::exec::mkdir -p
> /grid/3/hadoop/yarn::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has
> failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Exec[mkdir
> -p /grid/3/hadoop/yarn]/Anchor[hdp::exec::mkdir -p
> /grid/3/hadoop/yarn::end]: Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Directory[/grid/3/hadoop/yarn]/File[/grid/3/hadoop/yarn]:
> Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Directory[/grid/3/hadoop/yarn]/File[/grid/3/hadoop/yarn]:
> Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[capacity-scheduler]/File[/etc/hadoop/conf/capacity-scheduler.xml]/content:
> content changed '{md5}e5d17c21c7a5e1db9f3af35cba71df0a' to
> '{md5}2ca1d267a46f1aecac726caabaa16774'
> notice:
> /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[capacity-scheduler]/File[/etc/hadoop/conf/capacity-scheduler.xml]/owner:
> owner changed 'hdfs' to 'yarn'
> notice:
> /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[core-site]/File[/etc/hadoop/conf/core-site.xml]/content:
> content changed '{md5}86d742a780d59a957ea0a283dec03784' to
> '{md5}8506e4402ba8140ea4f9fed97b6f94e2'
> notice:
> /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[yarn-site]/File[/etc/hadoop/conf/yarn-site.xml]/content:
> content changed '{md5}d84a967ce47a6b77734ed8f53d817c6e' to
> '{md5}42940cca6e8f64ae5de50524fb131274'
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Anchor[hdp-yarn::service::nodemanager::begin]:
> Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Anchor[hdp-yarn::service::nodemanager::begin]:
> Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir
> -p /var/log/hadoop-yarn]/Anchor[hdp::exec::mkdir -p
> /var/log/hadoop-yarn::begin]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn]
> has failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir
> -p /var/log/hadoop-yarn]/Anchor[hdp::exec::mkdir -p
> /var/log/hadoop-yarn::begin]: Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir
> -p /var/log/hadoop-yarn]/Exec[mkdir -p /var/log/hadoop-yarn]: Dependency
> Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir
> -p /var/log/hadoop-yarn]/Exec[mkdir -p /var/log/hadoop-yarn]: Skipping
> because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir
> -p /var/log/hadoop-yarn]/Anchor[hdp::exec::mkdir -p
> /var/log/hadoop-yarn::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has
> failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir
> -p /var/log/hadoop-yarn]/Anchor[hdp::exec::mkdir -p
> /var/log/hadoop-yarn::end]: Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Directory[/var/log/hadoop-yarn]/File[/var/log/hadoop-yarn]:
> Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Directory[/var/log/hadoop-yarn]/File[/var/log/hadoop-yarn]:
> Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir
> -p /var/run/hadoop-yarn/yarn]/Anchor[hdp::exec::mkdir -p
> /var/run/hadoop-yarn/yarn::begin]: Dependency Exec[mkdir -p
> /grid/3/hadoop/yarn] has failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir
> -p /var/run/hadoop-yarn/yarn]/Anchor[hdp::exec::mkdir -p
> /var/run/hadoop-yarn/yarn::begin]: Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir
> -p /var/run/hadoop-yarn/yarn]/Exec[mkdir -p /var/run/hadoop-yarn/yarn]:
> Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir
> -p /var/run/hadoop-yarn/yarn]/Exec[mkdir -p /var/run/hadoop-yarn/yarn]:
> Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir
> -p /var/run/hadoop-yarn/yarn]/Anchor[hdp::exec::mkdir -p
> /var/run/hadoop-yarn/yarn::end]: Dependency Exec[mkdir -p
> /grid/3/hadoop/yarn] has failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir
> -p /var/run/hadoop-yarn/yarn]/Anchor[hdp::exec::mkdir -p
> /var/run/hadoop-yarn/yarn::end]: Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Directory[/var/run/hadoop-yarn/yarn]/File[/var/run/hadoop-yarn/yarn]:
> Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Directory[/var/run/hadoop-yarn/yarn]/File[/var/run/hadoop-yarn/yarn]:
> Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su -
> yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
> /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
> nodemanager']/Anchor[hdp::exec::su - yarn -c 'export
> HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
> /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
> nodemanager'::begin]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has
> failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su -
> yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
> /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
> nodemanager']/Anchor[hdp::exec::su - yarn -c 'export
> HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
> /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
> nodemanager'::begin]: Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su -
> yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
> /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
> nodemanager']/Exec[su - yarn -c 'export
> HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
> /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
> nodemanager']: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures:
> true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su -
> yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
> /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
> nodemanager']/Exec[su - yarn -c 'export
> HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
> /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
> nodemanager']: Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su -
> yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
> /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
> nodemanager']/Anchor[hdp::exec::su - yarn -c 'export
> HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
> /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
> nodemanager'::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has
> failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su -
> yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
> /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
> nodemanager']/Anchor[hdp::exec::su - yarn -c 'export
> HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
> /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
> nodemanager'::end]: Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep
> 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 &&
> ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null
> 2>&1]/Anchor[hdp::exec::sleep 5; ls
> /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps
> `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null
> 2>&1::begin]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep
> 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 &&
> ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null
> 2>&1]/Anchor[hdp::exec::sleep 5; ls
> /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps
> `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null
> 2>&1::begin]: Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep
> 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 &&
> ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null
> 2>&1]/Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid
> >/dev/null 2>&1 && ps `cat
> /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]:
> Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep
> 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 &&
> ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null
> 2>&1]/Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid
> >/dev/null 2>&1 && ps `cat
> /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]:
> Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep
> 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 &&
> ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null
> 2>&1]/Anchor[hdp::exec::sleep 5; ls
> /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps
> `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null
> 2>&1::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep
> 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 &&
> ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null
> 2>&1]/Anchor[hdp::exec::sleep 5; ls
> /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps
> `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null
> 2>&1::end]: Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Anchor[hdp-yarn::service::nodemanager::end]:
> Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning:
> /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Anchor[hdp-yarn::service::nodemanager::end]:
> Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Anchor[hdp-yarn::nodemanager::end]:
> Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Anchor[hdp-yarn::nodemanager::end]:
> Skipping because of failed dependencies
> notice:
> /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[mapred-site]/File[/etc/hadoop/conf/mapred-site.xml]/content:
> content changed '{md5}093cb1899b3c3b9dc4a7c1c93729c18b' to
> '{md5}4c462999cc47e6f6ba0e6381d71d81ba'
> notice:
> /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[mapred-site]/File[/etc/hadoop/conf/mapred-site.xml]/owner:
> owner changed 'mapred' to 'yarn'
> notice: Finished catalog run in 2.39 seconds
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
