[
https://issues.apache.org/jira/browse/AMBARI-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balázs Bence Sári updated AMBARI-16360:
---------------------------------------
Status: Patch Available (was: In Progress)
> RM fails to start after adding services in Kerb'd cluster
> ---------------------------------------------------------
>
> Key: AMBARI-16360
> URL: https://issues.apache.org/jira/browse/AMBARI-16360
> Project: Ambari
> Issue Type: Bug
> Components: ambari-agent
> Affects Versions: 2.4.0
> Reporter: Balázs Bence Sári
> Assignee: Balázs Bence Sári
> Priority: Critical
> Fix For: 2.4.0
>
> Attachments: patch.1
>
>
> Build #220
> 1) single node cluster, HDP 2.4
> 2) hdfs, zk, ams
> 3) enable kerb, mit KDC
> 4) then go to add hive, yarn, tez, storm
> 5) add service wizard fails. RM fails to start.
> 6) wizard exits. I can't start RM at all regardless, same error.
> {code}
> stderr: /var/lib/ambari-agent/data/errors-121.txt
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py",
> line 280, in <module>
> Resourcemanager().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 241, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py",
> line 125, in start
> self.wait_for_dfs_directories_created(params.entity_groupfs_store_dir,
> params.entity_groupfs_active_dir)
> File
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py",
> line 245, in wait_for_dfs_directories_created
> self.wait_for_dfs_directory_created(dir_path, ignored_dfs_dirs)
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/decorator.py",
> line 55, in wrapper
> return function(*args, **kwargs)
> File
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py",
> line 267, in wait_for_dfs_directory_created
> list_status = util.run_command(dir_path, 'GETFILESTATUS', method='GET',
> ignore_status_codes=['404'], assertable_result=False)
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
> line 195, in run_command
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w
> '%{http_code}' -X GET --negotiate -u :
> 'http://c6401.ambari.apache.org:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=yarn''
> returned status_code=403.
> {
> "RemoteException": {
> "exception": "SecurityException",
> "javaClassName": "java.lang.SecurityException",
> "message": "Failed to obtain user group information: java.io.IOException:
> Usernames not matched: name=yarn != expected=rm"
> }
> }
> stdout: /var/lib/ambari-agent/data/output-121.txt
> 2016-04-06 17:52:33,805 - The hadoop conf dir
> /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for
> version 2.4.0.0-169
> 2016-04-06 17:52:33,805 - Checking if need to create versioned conf dir
> /etc/hadoop/2.4.0.0-169/0
> 2016-04-06 17:52:33,805 - call[('ambari-python-wrap', '/usr/bin/conf-select',
> 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.4.0.0-169',
> '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False,
> 'stderr': -1}
> 2016-04-06 17:52:33,915 - call returned (1, '/etc/hadoop/2.4.0.0-169/0 exist
> already', '')
> 2016-04-06 17:52:33,915 - checked_call[('ambari-python-wrap',
> '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop',
> '--stack-version', '2.4.0.0-169', '--conf-version', '0')] {'logoutput':
> False, 'sudo': True, 'quiet': False}
> 2016-04-06 17:52:33,987 - checked_call returned (0,
> '/usr/hdp/2.4.0.0-169/hadoop/conf -> /etc/hadoop/2.4.0.0-169/0')
> 2016-04-06 17:52:33,988 - Ensuring that hadoop has the correct symlink
> structure
> 2016-04-06 17:52:33,988 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2016-04-06 17:52:34,551 - The hadoop conf dir
> /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for
> version 2.4.0.0-169
> 2016-04-06 17:52:34,552 - Checking if need to create versioned conf dir
> /etc/hadoop/2.4.0.0-169/0
> 2016-04-06 17:52:34,552 - call[('ambari-python-wrap', '/usr/bin/conf-select',
> 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.4.0.0-169',
> '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False,
> 'stderr': -1}
> 2016-04-06 17:52:34,657 - call returned (1, '/etc/hadoop/2.4.0.0-169/0 exist
> already', '')
> 2016-04-06 17:52:34,658 - checked_call[('ambari-python-wrap',
> '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop',
> '--stack-version', '2.4.0.0-169', '--conf-version', '0')] {'logoutput':
> False, 'sudo': True, 'quiet': False}
> 2016-04-06 17:52:34,740 - checked_call returned (0,
> '/usr/hdp/2.4.0.0-169/hadoop/conf -> /etc/hadoop/2.4.0.0-169/0')
> 2016-04-06 17:52:34,741 - Ensuring that hadoop has the correct symlink
> structure
> 2016-04-06 17:52:34,742 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2016-04-06 17:52:34,742 - Group['hadoop'] {}
> 2016-04-06 17:52:34,748 - Group['users'] {}
> 2016-04-06 17:52:34,748 - User['storm'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2016-04-06 17:52:34,750 - User['zookeeper'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2016-04-06 17:52:34,751 - User['ams'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2016-04-06 17:52:34,751 - User['ambari-qa'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['users']}
> 2016-04-06 17:52:34,752 - User['tez'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['users']}
> 2016-04-06 17:52:34,752 - User['hdfs'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2016-04-06 17:52:34,753 - User['yarn'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2016-04-06 17:52:34,753 - User['mapred'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2016-04-06 17:52:34,756 - File['/var/lib/ambari-agent/tmp/changeUid.sh']
> {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
> 2016-04-06 17:52:34,767 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh
> ambari-qa
> /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa']
> {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
> 2016-04-06 17:52:34,816 - Skipping
> Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa
> /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa']
> due to not_if
> 2016-04-06 17:52:34,816 - Group['hdfs'] {}
> 2016-04-06 17:52:34,817 - User['hdfs'] {'fetch_nonlocal_groups': True,
> 'groups': ['hadoop', 'hdfs']}
> 2016-04-06 17:52:34,818 - FS Type:
> 2016-04-06 17:52:34,818 - Directory['/etc/hadoop'] {'mode': 0755}
> 2016-04-06 17:52:34,860 -
> File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content':
> InlineTemplate(...), 'owner': 'root', 'group': 'hadoop'}
> 2016-04-06 17:52:34,864 -
> Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner':
> 'hdfs', 'group': 'hadoop', 'mode': 0777}
> 2016-04-06 17:52:34,905 - Execute[('setenforce', '0')] {'not_if': '(! which
> getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo':
> True, 'only_if': 'test -f /selinux/enforce'}
> 2016-04-06 17:52:34,975 - Skipping Execute[('setenforce', '0')] due to not_if
> 2016-04-06 17:52:34,975 - Directory['/var/log/hadoop'] {'owner': 'root',
> 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
> 2016-04-06 17:52:34,986 - Directory['/var/run/hadoop'] {'owner': 'root',
> 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
> 2016-04-06 17:52:34,987 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs',
> 'create_parents': True, 'cd_access': 'a'}
> 2016-04-06 17:52:34,997 -
> File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties']
> {'content': Template('commons-logging.properties.j2'), 'owner': 'root'}
> 2016-04-06 17:52:34,999 -
> File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content':
> Template('health_check.j2'), 'owner': 'root'}
> 2016-04-06 17:52:35,000 -
> File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ...,
> 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
> 2016-04-06 17:52:35,041 -
> File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties']
> {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs',
> 'group': 'hadoop'}
> 2016-04-06 17:52:35,045 -
> File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content':
> StaticFile('task-log4j.properties'), 'mode': 0755}
> 2016-04-06 17:52:35,048 -
> File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner':
> 'hdfs', 'group': 'hadoop'}
> 2016-04-06 17:52:35,053 - File['/etc/hadoop/conf/topology_mappings.data']
> {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'),
> 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
> 2016-04-06 17:52:35,062 - File['/etc/hadoop/conf/topology_script.py']
> {'content': StaticFile('topology_script.py'), 'only_if': 'test -d
> /etc/hadoop/conf', 'mode': 0755}
> 2016-04-06 17:52:35,487 - The hadoop conf dir
> /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for
> version 2.4.0.0-169
> 2016-04-06 17:52:35,488 - Checking if need to create versioned conf dir
> /etc/hadoop/2.4.0.0-169/0
> 2016-04-06 17:52:35,489 - call[('ambari-python-wrap', '/usr/bin/conf-select',
> 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.4.0.0-169',
> '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False,
> 'stderr': -1}
> 2016-04-06 17:52:35,568 - call returned (1, '/etc/hadoop/2.4.0.0-169/0 exist
> already', '')
> 2016-04-06 17:52:35,569 - checked_call[('ambari-python-wrap',
> '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop',
> '--stack-version', '2.4.0.0-169', '--conf-version', '0')] {'logoutput':
> False, 'sudo': True, 'quiet': False}
> 2016-04-06 17:52:35,635 - checked_call returned (0,
> '/usr/hdp/2.4.0.0-169/hadoop/conf -> /etc/hadoop/2.4.0.0-169/0')
> 2016-04-06 17:52:35,636 - Ensuring that hadoop has the correct symlink
> structure
> 2016-04-06 17:52:35,636 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2016-04-06 17:52:35,637 - call['ambari-python-wrap /usr/bin/hdp-select status
> hadoop-yarn-resourcemanager'] {'timeout': 20}
> 2016-04-06 17:52:35,699 - call returned (0, 'hadoop-yarn-resourcemanager -
> 2.4.0.0-169')
> 2016-04-06 17:52:35,701 - The hadoop conf dir
> /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for
> version 2.4.0.0-169
> 2016-04-06 17:52:35,702 - Checking if need to create versioned conf dir
> /etc/hadoop/2.4.0.0-169/0
> 2016-04-06 17:52:35,702 - call[('ambari-python-wrap', '/usr/bin/conf-select',
> 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.4.0.0-169',
> '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False,
> 'stderr': -1}
> 2016-04-06 17:52:35,768 - call returned (1, '/etc/hadoop/2.4.0.0-169/0 exist
> already', '')
> 2016-04-06 17:52:35,769 - checked_call[('ambari-python-wrap',
> '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop',
> '--stack-version', '2.4.0.0-169', '--conf-version', '0')] {'logoutput':
> False, 'sudo': True, 'quiet': False}
> 2016-04-06 17:52:35,860 - checked_call returned (0,
> '/usr/hdp/2.4.0.0-169/hadoop/conf -> /etc/hadoop/2.4.0.0-169/0')
> 2016-04-06 17:52:35,861 - Ensuring that hadoop has the correct symlink
> structure
> 2016-04-06 17:52:35,861 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2016-04-06 17:52:35,883 -
> Directory['/var/log/hadoop-yarn/nodemanager/recovery-state'] {'owner':
> 'yarn', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access':
> 'a'}
> 2016-04-06 17:52:35,896 - Directory['/var/run/hadoop-yarn'] {'owner': 'yarn',
> 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
> 2016-04-06 17:52:35,899 - Directory['/var/run/hadoop-yarn/yarn'] {'owner':
> 'yarn', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
> 2016-04-06 17:52:35,900 - Directory['/var/log/hadoop-yarn/yarn'] {'owner':
> 'yarn', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}
> 2016-04-06 17:52:35,905 - Directory['/var/run/hadoop-mapreduce'] {'owner':
> 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
> 2016-04-06 17:52:35,905 - Directory['/var/run/hadoop-mapreduce/mapred']
> {'owner': 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access':
> 'a'}
> 2016-04-06 17:52:35,913 - Directory['/var/log/hadoop-mapreduce'] {'owner':
> 'mapred', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
> 2016-04-06 17:52:35,914 - Directory['/var/log/hadoop-mapreduce/mapred']
> {'owner': 'mapred', 'group': 'hadoop', 'create_parents': True, 'cd_access':
> 'a'}
> 2016-04-06 17:52:35,914 - Directory['/var/log/hadoop-yarn'] {'owner': 'yarn',
> 'ignore_failures': True, 'create_parents': True, 'cd_access': 'a'}
> 2016-04-06 17:52:35,915 - XmlConfig['core-site.xml'] {'group': 'hadoop',
> 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644,
> 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
> 2016-04-06 17:52:35,947 - Generating config:
> /usr/hdp/current/hadoop-client/conf/core-site.xml
> 2016-04-06 17:52:35,947 -
> File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs',
> 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding':
> 'UTF-8'}
> 2016-04-06 17:52:36,100 - XmlConfig['hdfs-site.xml'] {'group': 'hadoop',
> 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644,
> 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
> 2016-04-06 17:52:36,127 - Generating config:
> /usr/hdp/current/hadoop-client/conf/hdfs-site.xml
> 2016-04-06 17:52:36,127 -
> File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs',
> 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding':
> 'UTF-8'}
> 2016-04-06 17:52:36,336 - XmlConfig['mapred-site.xml'] {'group': 'hadoop',
> 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644,
> 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
> 2016-04-06 17:52:36,374 - Generating config:
> /usr/hdp/current/hadoop-client/conf/mapred-site.xml
> 2016-04-06 17:52:36,374 -
> File['/usr/hdp/current/hadoop-client/conf/mapred-site.xml'] {'owner': 'yarn',
> 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding':
> 'UTF-8'}
> 2016-04-06 17:52:36,561 - Changing owner for
> /usr/hdp/current/hadoop-client/conf/mapred-site.xml from 1005 to yarn
> 2016-04-06 17:52:36,561 - XmlConfig['yarn-site.xml'] {'group': 'hadoop',
> 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644,
> 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
> 2016-04-06 17:52:36,590 - Generating config:
> /usr/hdp/current/hadoop-client/conf/yarn-site.xml
> 2016-04-06 17:52:36,590 -
> File['/usr/hdp/current/hadoop-client/conf/yarn-site.xml'] {'owner': 'yarn',
> 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding':
> 'UTF-8'}
> 2016-04-06 17:52:36,998 - XmlConfig['capacity-scheduler.xml'] {'group':
> 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644,
> 'configuration_attributes': {}, 'owner': 'yarn', 'configurations': ...}
> 2016-04-06 17:52:37,026 - Generating config:
> /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml
> 2016-04-06 17:52:37,030 -
> File['/usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml'] {'owner':
> 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644,
> 'encoding': 'UTF-8'}
> 2016-04-06 17:52:37,088 - Changing owner for
> /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml from 505 to yarn
> 2016-04-06 17:52:37,088 -
> File['/var/log/hadoop-yarn/yarn/hadoop-mapreduce.jobsummary.log'] {'owner':
> 'yarn', 'group': 'hadoop'}
> 2016-04-06 17:52:37,090 - Writing
> File['/var/log/hadoop-yarn/yarn/hadoop-mapreduce.jobsummary.log'] because it
> doesn't exist
> 2016-04-06 17:52:37,093 - Changing owner for
> /var/log/hadoop-yarn/yarn/hadoop-mapreduce.jobsummary.log from 0 to yarn
> 2016-04-06 17:52:37,094 - Changing group for
> /var/log/hadoop-yarn/yarn/hadoop-mapreduce.jobsummary.log from 0 to hadoop
> 2016-04-06 17:52:37,094 - File['/etc/hadoop/conf/yarn.exclude'] {'owner':
> 'yarn', 'group': 'hadoop'}
> 2016-04-06 17:52:37,106 - File['/etc/security/limits.d/yarn.conf']
> {'content': Template('yarn.conf.j2'), 'mode': 0644}
> 2016-04-06 17:52:37,111 - File['/etc/security/limits.d/mapreduce.conf']
> {'content': Template('mapreduce.conf.j2'), 'mode': 0644}
> 2016-04-06 17:52:37,152 -
> File['/usr/hdp/current/hadoop-client/conf/yarn-env.sh'] {'content':
> InlineTemplate(...), 'owner': 'yarn', 'group': 'hadoop', 'mode': 0755}
> 2016-04-06 17:52:37,153 - Writing
> File['/usr/hdp/current/hadoop-client/conf/yarn-env.sh'] because contents
> don't match
> 2016-04-06 17:52:37,174 -
> File['/usr/hdp/current/hadoop-yarn-resourcemanager/bin/container-executor']
> {'group': 'hadoop', 'mode': 06050}
> 2016-04-06 17:52:37,184 -
> File['/usr/hdp/current/hadoop-client/conf/container-executor.cfg']
> {'content': Template('container-executor.cfg.j2'), 'group': 'hadoop', 'mode':
> 0644}
> 2016-04-06 17:52:37,193 - Directory['/cgroups_test/cpu'] {'group': 'hadoop',
> 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
> 2016-04-06 17:52:37,200 -
> File['/usr/hdp/current/hadoop-client/conf/mapred-env.sh'] {'content':
> InlineTemplate(...), 'owner': 'root', 'mode': 0755}
> 2016-04-06 17:52:37,200 -
> File['/usr/hdp/current/hadoop-client/sbin/task-controller'] {'owner': 'root',
> 'group': 'hadoop', 'mode': 06050}
> 2016-04-06 17:52:37,207 -
> File['/usr/hdp/current/hadoop-client/conf/taskcontroller.cfg'] {'content':
> Template('taskcontroller.cfg.j2'), 'owner': 'root', 'group': 'hadoop',
> 'mode': 0644}
> 2016-04-06 17:52:37,207 - XmlConfig['mapred-site.xml'] {'owner': 'mapred',
> 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf',
> 'configuration_attributes': {}, 'configurations': ...}
> 2016-04-06 17:52:37,264 - Generating config:
> /usr/hdp/current/hadoop-client/conf/mapred-site.xml
> 2016-04-06 17:52:37,264 -
> File['/usr/hdp/current/hadoop-client/conf/mapred-site.xml'] {'owner':
> 'mapred', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None,
> 'encoding': 'UTF-8'}
> 2016-04-06 17:52:37,547 - Changing owner for
> /usr/hdp/current/hadoop-client/conf/mapred-site.xml from 1004 to mapred
> 2016-04-06 17:52:37,547 - XmlConfig['capacity-scheduler.xml'] {'owner':
> 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf',
> 'configuration_attributes': {}, 'configurations': ...}
> 2016-04-06 17:52:37,653 - Generating config:
> /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml
> 2016-04-06 17:52:37,653 -
> File['/usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml'] {'owner':
> 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None,
> 'encoding': 'UTF-8'}
> 2016-04-06 17:52:37,771 - Changing owner for
> /usr/hdp/current/hadoop-client/conf/capacity-scheduler.xml from 1004 to hdfs
> 2016-04-06 17:52:37,772 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs',
> 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf',
> 'configuration_attributes': {}, 'configurations': ...}
> 2016-04-06 17:52:37,838 - Generating config:
> /usr/hdp/current/hadoop-client/conf/ssl-client.xml
> 2016-04-06 17:52:37,838 -
> File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs',
> 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding':
> 'UTF-8'}
> 2016-04-06 17:52:37,894 -
> Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root',
> 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
> 2016-04-06 17:52:37,910 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs',
> 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure',
> 'configuration_attributes': {}, 'configurations': ...}
> 2016-04-06 17:52:37,958 - Generating config:
> /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml
> 2016-04-06 17:52:37,958 -
> File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner':
> 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None,
> 'encoding': 'UTF-8'}
> 2016-04-06 17:52:37,995 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs',
> 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf',
> 'configuration_attributes': {}, 'configurations': ...}
> 2016-04-06 17:52:38,071 - Generating config:
> /usr/hdp/current/hadoop-client/conf/ssl-server.xml
> 2016-04-06 17:52:38,072 -
> File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs',
> 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding':
> 'UTF-8'}
> 2016-04-06 17:52:38,098 -
> File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml.example'] {'owner':
> 'mapred', 'group': 'hadoop'}
> 2016-04-06 17:52:38,099 -
> File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml.example'] {'owner':
> 'mapred', 'group': 'hadoop'}
> 2016-04-06 17:52:38,099 - Verifying DFS directories where ATS stores time
> line data for active and completed applications.
> 2016-04-06 17:52:38,104 - Execute['/usr/bin/kinit -kt
> /etc/security/keytabs/rm.service.keytab
> rm/[email protected];'] {'user': 'yarn'}
> 2016-04-06 17:52:38,404 - Verifying if DFS directory '/ats/done/' exists.
> 2016-04-06 17:52:38,406 - call['ambari-sudo.sh su yarn -l -s /bin/bash -c
> 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET --negotiate -u :
> '"'"'http://c6401.ambari.apache.org:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=yarn'"'"'
> 1>/tmp/tmpTOXEBI 2>/tmp/tmpwHIFI_''] {'logoutput': None, 'quiet': False}
> 2016-04-06 17:52:38,572 - call returned (0, '')
> 2016-04-06 17:52:38,573 - Will retry 7 time(s), caught exception: Execution
> of 'curl -sS -L -w '%{http_code}' -X GET --negotiate -u :
> 'http://c6401.ambari.apache.org:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=yarn''
> returned status_code=403.
> {
> "RemoteException": {
> "exception": "SecurityException",
> "javaClassName": "java.lang.SecurityException",
> "message": "Failed to obtain user group information: java.io.IOException:
> Usernames not matched: name=yarn != expected=rm"
> }
> }. Sleeping for 20 sec(s)
> 2016-04-06 17:52:58,592 - Verifying if DFS directory '/ats/done/' exists.
> 2016-04-06 17:52:58,594 - call['ambari-sudo.sh su yarn -l -s /bin/bash -c
> 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET --negotiate -u :
> '"'"'http://c6401.ambari.apache.org:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=yarn'"'"'
> 1>/tmp/tmpenjO1U 2>/tmp/tmp7i9TPL''] {'logoutput': None, 'quiet': False}
> 2016-04-06 17:52:58,696 - call returned (0, '')
> 2016-04-06 17:52:58,696 - Will retry 6 time(s), caught exception: Execution
> of 'curl -sS -L -w '%{http_code}' -X GET --negotiate -u :
> 'http://c6401.ambari.apache.org:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=yarn''
> returned status_code=403.
> {
> "RemoteException": {
> "exception": "SecurityException",
> "javaClassName": "java.lang.SecurityException",
> "message": "Failed to obtain user group information: java.io.IOException:
> Usernames not matched: name=yarn != expected=rm"
> }
> }. Sleeping for 20 sec(s)
> 2016-04-06 17:53:18,717 - Verifying if DFS directory '/ats/done/' exists.
> 2016-04-06 17:53:18,718 - call['ambari-sudo.sh su yarn -l -s /bin/bash -c
> 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET --negotiate -u :
> '"'"'http://c6401.ambari.apache.org:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=yarn'"'"'
> 1>/tmp/tmpAfX3Cp 2>/tmp/tmpzb3MbZ''] {'logoutput': None, 'quiet': False}
> 2016-04-06 17:53:18,901 - call returned (0, '')
> 2016-04-06 17:53:18,912 - Will retry 5 time(s), caught exception: Execution
> of 'curl -sS -L -w '%{http_code}' -X GET --negotiate -u :
> 'http://c6401.ambari.apache.org:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=yarn''
> returned status_code=403.
> {
> "RemoteException": {
> "exception": "SecurityException",
> "javaClassName": "java.lang.SecurityException",
> "message": "Failed to obtain user group information: java.io.IOException:
> Usernames not matched: name=yarn != expected=rm"
> }
> }. Sleeping for 20 sec(s)
> 2016-04-06 17:53:38,933 - Verifying if DFS directory '/ats/done/' exists.
> 2016-04-06 17:53:38,935 - call['ambari-sudo.sh su yarn -l -s /bin/bash -c
> 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET --negotiate -u :
> '"'"'http://c6401.ambari.apache.org:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=yarn'"'"'
> 1>/tmp/tmplrXWp3 2>/tmp/tmpO6cD5t''] {'logoutput': None, 'quiet': False}
> 2016-04-06 17:53:39,027 - call returned (0, '')
> 2016-04-06 17:53:39,028 - Will retry 4 time(s), caught exception: Execution
> of 'curl -sS -L -w '%{http_code}' -X GET --negotiate -u :
> 'http://c6401.ambari.apache.org:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=yarn''
> returned status_code=403.
> {
> "RemoteException": {
> "exception": "SecurityException",
> "javaClassName": "java.lang.SecurityException",
> "message": "Failed to obtain user group information: java.io.IOException:
> Usernames not matched: name=yarn != expected=rm"
> }
> }. Sleeping for 20 sec(s)
> 2016-04-06 17:53:59,047 - Verifying if DFS directory '/ats/done/' exists.
> 2016-04-06 17:53:59,048 - call['ambari-sudo.sh su yarn -l -s /bin/bash -c
> 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET --negotiate -u :
> '"'"'http://c6401.ambari.apache.org:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=yarn'"'"'
> 1>/tmp/tmpxNBcDc 2>/tmp/tmpSCX8Wk''] {'logoutput': None, 'quiet': False}
> 2016-04-06 17:53:59,095 - call returned (0, '')
> 2016-04-06 17:53:59,096 - Will retry 3 time(s), caught exception: Execution
> of 'curl -sS -L -w '%{http_code}' -X GET --negotiate -u :
> 'http://c6401.ambari.apache.org:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=yarn''
> returned status_code=403.
> {
> "RemoteException": {
> "exception": "SecurityException",
> "javaClassName": "java.lang.SecurityException",
> "message": "Failed to obtain user group information: java.io.IOException:
> Usernames not matched: name=yarn != expected=rm"
> }
> }. Sleeping for 20 sec(s)
> 2016-04-06 17:54:19,116 - Verifying if DFS directory '/ats/done/' exists.
> 2016-04-06 17:54:19,117 - call['ambari-sudo.sh su yarn -l -s /bin/bash -c
> 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET --negotiate -u :
> '"'"'http://c6401.ambari.apache.org:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=yarn'"'"'
> 1>/tmp/tmplZ3LST 2>/tmp/tmpjtCElc''] {'logoutput': None, 'quiet': False}
> 2016-04-06 17:54:19,177 - call returned (0, '')
> 2016-04-06 17:54:19,178 - Will retry 2 time(s), caught exception: Execution
> of 'curl -sS -L -w '%{http_code}' -X GET --negotiate -u :
> 'http://c6401.ambari.apache.org:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=yarn''
> returned status_code=403.
> {
> "RemoteException": {
> "exception": "SecurityException",
> "javaClassName": "java.lang.SecurityException",
> "message": "Failed to obtain user group information: java.io.IOException:
> Usernames not matched: name=yarn != expected=rm"
> }
> }. Sleeping for 20 sec(s)
> 2016-04-06 17:54:39,198 - Verifying if DFS directory '/ats/done/' exists.
> 2016-04-06 17:54:39,199 - call['ambari-sudo.sh su yarn -l -s /bin/bash -c
> 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET --negotiate -u :
> '"'"'http://c6401.ambari.apache.org:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=yarn'"'"'
> 1>/tmp/tmpsZoCji 2>/tmp/tmpUvPE7K''] {'logoutput': None, 'quiet': False}
> 2016-04-06 17:54:39,273 - call returned (0, '')
> 2016-04-06 17:54:39,274 - Will retry 1 time(s), caught exception: Execution
> of 'curl -sS -L -w '%{http_code}' -X GET --negotiate -u :
> 'http://c6401.ambari.apache.org:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=yarn''
> returned status_code=403.
> {
> "RemoteException": {
> "exception": "SecurityException",
> "javaClassName": "java.lang.SecurityException",
> "message": "Failed to obtain user group information: java.io.IOException:
> Usernames not matched: name=yarn != expected=rm"
> }
> }. Sleeping for 20 sec(s)
> 2016-04-06 17:54:59,289 - Verifying if DFS directory '/ats/done/' exists.
> 2016-04-06 17:54:59,291 - call['ambari-sudo.sh su yarn -l -s /bin/bash -c
> 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET --negotiate -u :
> '"'"'http://c6401.ambari.apache.org:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=yarn'"'"'
> 1>/tmp/tmpzuHWuQ 2>/tmp/tmpobAQNa''] {'logoutput': None, 'quiet': False}
> 2016-04-06 17:54:59,396 - call returned (0, '')
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
