Hi,
*I am getting following exception in Hiveserver2 log file:* 2015-05-04 13:45:23,542 WARN [main]: ipc.Client (Client.java:run(676)) - Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 2015-05-04 13:45:23,543 INFO [main]: retry.RetryInvocationHandler (RetryInvocationHandler.java:invoke(140)) - Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over sv2lxbdp2mst05.corp.host.com/10.192.149.187:8020 after 8 fail over attempts. Trying to fail over immediately. java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Thanks, Shaik (If it is not right place to post this query, please help me to know the correct group) On 4 May 2015 at 18:55, Shaik M <[email protected]> wrote: > Hi, > > I am using Ambari 1.7 and HDP 2.2.4. I have enabled security in this > cluster. > > After enabling the security Ambari unable start the Hive server 2. > > I have verified keytabs and it's working fine. Please find the below > ambari string process log below. not shown any output in standerr window. > > please let me know the how to resolve this issue. > > Thanks, > Shaik M > > stdout: /var/lib/ambari-agent/data/output-1832.txt > > 2015-05-04 10:47:25,616 - Execute['mkdir -p > /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry > 10 > http://sv2lxbdp2mst05.corp.equinix.com:8080/resources//UnlimitedJCEPolicyJDK7.zip > -o > /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] > {'environment': ..., 'not_if': 'test -e > /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip', > 'ignore_failures': True, 'path': ['/bin', '/usr/bin/']} > 2015-05-04 10:47:25,646 - Skipping Execute['mkdir -p > /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry > 10 > http://sv2lxbdp2mst05.corp.equinix.com:8080/resources//UnlimitedJCEPolicyJDK7.zip > -o > /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] > due to not_if > 2015-05-04 10:47:25,648 - Execute['rm -f local_policy.jar; rm -f > US_export_policy.jar; unzip -o -j -q > /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] > {'path': ['/bin/', '/usr/bin'], 'only_if': 'test -e > /usr/jdk64/jdk1.7.0_67/jre/lib/security && test -f > /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip', > 'cwd': '/usr/jdk64/jdk1.7.0_67/jre/lib/security'} > 2015-05-04 10:47:25,718 - Group['hadoop'] {'ignore_failures': False} > 2015-05-04 10:47:25,720 - Modifying group hadoop > 2015-05-04 10:47:25,784 - Group['nobody'] {'ignore_failures': False} > 2015-05-04 10:47:25,785 - Modifying group nobody > 2015-05-04 10:47:25,839 - Group['users'] {'ignore_failures': False} > 2015-05-04 10:47:25,840 - Modifying group users > 2015-05-04 10:47:25,886 - Group['nagios'] {'ignore_failures': False} > 2015-05-04 10:47:25,887 - Modifying group nagios > 2015-05-04 10:47:25,936 - User['nobody'] {'gid': 'hadoop', 'ignore_failures': > False, 'groups': [u'nobody']} > 2015-05-04 10:47:25,937 - Modifying user nobody > 2015-05-04 10:47:26,012 - User['oozie'] {'gid': 'hadoop', 'ignore_failures': > False, 'groups': [u'users']} > 2015-05-04 10:47:26,013 - Modifying user oozie > 2015-05-04 10:47:26,063 - User['hive'] {'gid': 'hadoop', 'ignore_failures': > False, 'groups': [u'hadoop']} > 2015-05-04 10:47:26,064 - Modifying user hive > 2015-05-04 10:47:26,098 - User['mapred'] {'gid': 'hadoop', 'ignore_failures': > False, 'groups': [u'hadoop']} > 2015-05-04 10:47:26,099 - Modifying user mapred > 2015-05-04 10:47:26,131 - User['nagios'] {'gid': 'nagios', 'ignore_failures': > False, 'groups': [u'hadoop']} > 2015-05-04 10:47:26,132 - Modifying user nagios > 2015-05-04 10:47:26,165 - User['ambari-qa'] {'gid': 'hadoop', > 'ignore_failures': False, 'groups': [u'users']} > 2015-05-04 10:47:26,166 - Modifying user ambari-qa > 2015-05-04 10:47:26,199 - User['zookeeper'] {'gid': 'hadoop', > 'ignore_failures': False, 'groups': [u'hadoop']} > 2015-05-04 10:47:26,199 - Modifying user zookeeper > 2015-05-04 10:47:26,232 - User['tez'] {'gid': 'hadoop', 'ignore_failures': > False, 'groups': [u'users']} > 2015-05-04 10:47:26,233 - Modifying user tez > 2015-05-04 10:47:26,265 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': > False, 'groups': [u'hadoop']} > 2015-05-04 10:47:26,266 - Modifying user hdfs > 2015-05-04 10:47:26,298 - User['sqoop'] {'gid': 'hadoop', 'ignore_failures': > False, 'groups': [u'hadoop']} > 2015-05-04 10:47:26,299 - Modifying user sqoop > 2015-05-04 10:47:26,332 - User['hcat'] {'gid': 'hadoop', 'ignore_failures': > False, 'groups': [u'hadoop']} > 2015-05-04 10:47:26,332 - Modifying user hcat > 2015-05-04 10:47:26,365 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': > False, 'groups': [u'hadoop']} > 2015-05-04 10:47:26,366 - Modifying user yarn > 2015-05-04 10:47:26,399 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] > {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} > 2015-05-04 10:47:26,401 - > Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa > /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa > 2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'} > 2015-05-04 10:47:26,431 - Skipping > Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa > /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa > 2>/dev/null'] due to not_if > 2015-05-04 10:47:26,432 - Directory['/etc/hadoop/conf.empty'] {'owner': > 'root', 'group': 'root', 'recursive': True} > 2015-05-04 10:47:26,433 - Link['/etc/hadoop/conf'] {'not_if': 'ls > /etc/hadoop/conf', 'to': '/etc/hadoop/conf.empty'} > 2015-05-04 10:47:26,462 - Skipping Link['/etc/hadoop/conf'] due to not_if > 2015-05-04 10:47:26,491 - File['/etc/hadoop/conf/hadoop-env.sh'] {'content': > InlineTemplate(...), 'owner': 'root'} > 2015-05-04 10:47:26,516 - Execute['/bin/echo 0 > /selinux/enforce'] > {'only_if': 'test -f /selinux/enforce'} > 2015-05-04 10:47:26,577 - Directory['/var/log/hadoop'] {'owner': 'root', > 'group': 'hadoop', 'mode': 0775, 'recursive': True} > 2015-05-04 10:47:26,578 - Directory['/var/run/hadoop'] {'owner': 'root', > 'group': 'root', 'recursive': True} > 2015-05-04 10:47:26,579 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', > 'recursive': True} > 2015-05-04 10:47:26,590 - File['/etc/hadoop/conf/commons-logging.properties'] > {'content': Template('commons-logging.properties.j2'), 'owner': 'root'} > 2015-05-04 10:47:26,595 - File['/etc/hadoop/conf/health_check'] {'content': > Template('health_check-v2.j2'), 'owner': 'root'} > 2015-05-04 10:47:26,596 - File['/etc/hadoop/conf/log4j.properties'] > {'content': '...', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644} > 2015-05-04 10:47:26,608 - File['/etc/hadoop/conf/hadoop-metrics2.properties'] > {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'} > 2015-05-04 10:47:26,609 - File['/etc/hadoop/conf/task-log4j.properties'] > {'content': StaticFile('task-log4j.properties'), 'mode': 0755} > 2015-05-04 10:47:26,956 - Execute['kill `cat /var/run/hive/hive-server.pid` > >/dev/null 2>&1 && rm -f /var/run/hive/hive-server.pid'] {'not_if': '! (ls > /var/run/hive/hive-server.pid >/dev/null 2>&1 && ps `cat > /var/run/hive/hive-server.pid` >/dev/null 2>&1)'} > 2015-05-04 10:47:27,068 - HdfsDirectory['/apps/hive/warehouse'] > {'security_enabled': True, 'keytab': > '/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': '/etc/hadoop/conf', > 'hdfs_user': 'hdfs', 'kinit_path_local': '/usr/bin/kinit', 'mode': 0777, > 'owner': 'hive', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': > ['create_delayed']} > 2015-05-04 10:47:27,069 - HdfsDirectory['/user/hive'] {'security_enabled': > True, 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': > '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': > '/usr/bin/kinit', 'mode': 0700, 'owner': 'hive', 'bin_dir': > '/usr/hdp/current/hadoop-client/bin', 'action': ['create_delayed']} > 2015-05-04 10:47:27,070 - HdfsDirectory['None'] {'security_enabled': True, > 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': > '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': > '/usr/bin/kinit', 'action': ['create'], 'bin_dir': > '/usr/hdp/current/hadoop-client/bin'} > 2015-05-04 10:47:27,073 - Execute['/usr/bin/kinit -kt > /etc/security/keytabs/hdfs.headless.keytab hdfs'] {'user': 'hdfs'} > 2015-05-04 10:47:27,850 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir > `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` /apps/hive/warehouse > /user/hive && hadoop --config /etc/hadoop/conf fs -chmod 777 > /apps/hive/warehouse && hadoop --config /etc/hadoop/conf fs -chmod 700 > /user/hive && hadoop --config /etc/hadoop/conf fs -chown hive > /apps/hive/warehouse /user/hive'] {'not_if': "su - hdfs -c 'export > PATH=$PATH:/usr/hdp/current/hadoop-client/bin ; hadoop --config > /etc/hadoop/conf fs -ls /apps/hive/warehouse /user/hive'", 'user': 'hdfs', > 'path': ['/usr/hdp/current/hadoop-client/bin']} > 2015-05-04 10:47:31,133 - Skipping Execute['hadoop --config /etc/hadoop/conf > fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` > /apps/hive/warehouse /user/hive && hadoop --config /etc/hadoop/conf fs -chmod > 777 /apps/hive/warehouse && hadoop --config /etc/hadoop/conf fs -chmod 700 > /user/hive && hadoop --config /etc/hadoop/conf fs -chown hive > /apps/hive/warehouse /user/hive'] due to not_if > 2015-05-04 10:47:31,134 - Directory['/etc/hive/conf.server'] {'owner': > 'hive', 'group': 'hadoop', 'recursive': True} > 2015-05-04 10:47:31,135 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', > 'conf_dir': '/etc/hive/conf.server', 'mode': 0644, > 'configuration_attributes': ..., 'owner': 'hive', 'configurations': ...} > 2015-05-04 10:47:31,168 - Generating config: > /etc/hive/conf.server/mapred-site.xml > 2015-05-04 10:47:31,169 - File['/etc/hive/conf.server/mapred-site.xml'] > {'owner': 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': > 0644, 'encoding': 'UTF-8'} > 2015-05-04 10:47:31,173 - Writing > File['/etc/hive/conf.server/mapred-site.xml'] because contents don't match > 2015-05-04 10:47:31,174 - > File['/etc/hive/conf.server/hive-default.xml.template'] {'owner': 'hive', > 'group': 'hadoop'} > 2015-05-04 10:47:31,175 - File['/etc/hive/conf.server/hive-env.sh.template'] > {'owner': 'hive', 'group': 'hadoop'} > 2015-05-04 10:47:31,177 - > File['/etc/hive/conf.server/hive-exec-log4j.properties'] {'content': '...', > 'owner': 'hive', 'group': 'hadoop', 'mode': 0644} > 2015-05-04 10:47:31,178 - File['/etc/hive/conf.server/hive-log4j.properties'] > {'content': '...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644} > 2015-05-04 10:47:31,179 - Directory['/etc/hive/conf'] {'owner': 'hive', > 'group': 'hadoop', 'recursive': True} > 2015-05-04 10:47:31,180 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', > 'conf_dir': '/etc/hive/conf', 'mode': 0644, 'configuration_attributes': ..., > 'owner': 'hive', 'configurations': ...} > 2015-05-04 10:47:31,202 - Generating config: /etc/hive/conf/mapred-site.xml > 2015-05-04 10:47:31,203 - File['/etc/hive/conf/mapred-site.xml'] {'owner': > 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, > 'encoding': 'UTF-8'} > 2015-05-04 10:47:31,207 - Writing File['/etc/hive/conf/mapred-site.xml'] > because contents don't match > 2015-05-04 10:47:31,208 - File['/etc/hive/conf/hive-default.xml.template'] > {'owner': 'hive', 'group': 'hadoop'} > 2015-05-04 10:47:31,209 - File['/etc/hive/conf/hive-env.sh.template'] > {'owner': 'hive', 'group': 'hadoop'} > 2015-05-04 10:47:31,211 - File['/etc/hive/conf/hive-exec-log4j.properties'] > {'content': '...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644} > 2015-05-04 10:47:31,212 - File['/etc/hive/conf/hive-log4j.properties'] > {'content': '...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644} > 2015-05-04 10:47:31,213 - XmlConfig['hive-site.xml'] {'group': 'hadoop', > 'conf_dir': '/etc/hive/conf.server', 'mode': 0644, > 'configuration_attributes': ..., 'owner': 'hive', 'configurations': ...} > 2015-05-04 10:47:31,236 - Generating config: > /etc/hive/conf.server/hive-site.xml > 2015-05-04 10:47:31,237 - File['/etc/hive/conf.server/hive-site.xml'] > {'owner': 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': > 0644, 'encoding': 'UTF-8'} > 2015-05-04 10:47:31,245 - Writing File['/etc/hive/conf.server/hive-site.xml'] > because contents don't match > 2015-05-04 10:47:31,251 - File['/etc/hive/conf.server/hive-env.sh'] > {'content': InlineTemplate(...), 'owner': 'hive', 'group': 'hadoop'} > 2015-05-04 10:47:31,253 - Execute['hive mkdir -p > /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; rm -f > /usr/hdp/current/hive-client/lib/mysql-connector-java.jar ; cp > /usr/share/java/mysql-connector-java.jar > /usr/hdp/current/hive-client/lib/mysql-connector-java.jar'] {'environment': > ..., 'path': ['/bin', '/usr/bin/'], 'creates': > '/usr/hdp/current/hive-client/lib/mysql-connector-java.jar', 'not_if': 'test > -f /usr/hdp/current/hive-client/lib/mysql-connector-java.jar'} > 2015-05-04 10:47:31,282 - Skipping Execute['hive mkdir -p > /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; rm -f > /usr/hdp/current/hive-client/lib/mysql-connector-java.jar ; cp > /usr/share/java/mysql-connector-java.jar > /usr/hdp/current/hive-client/lib/mysql-connector-java.jar'] due to not_if > 2015-05-04 10:47:31,285 - Execute['/bin/sh -c 'cd /usr/lib/ambari-agent/ && > curl -kf -x "" --retry 5 > http://sv2lxbdp2mst05.corp.equinix.com:8080/resources/DBConnectionVerification.jar > -o DBConnectionVerification.jar''] {'environment': ..., 'not_if': '[ -f > DBConnectionVerification.jar]'} > 2015-05-04 10:47:31,370 - > File['/var/lib/ambari-agent/data/tmp/start_hiveserver2_script'] {'content': > Template('startHiveserver2.sh.j2'), 'mode': 0755} > 2015-05-04 10:47:31,372 - Directory['/var/run/hive'] {'owner': 'hive', > 'group': 'hadoop', 'mode': 0755, 'recursive': True} > 2015-05-04 10:47:31,372 - Directory['/var/log/hive'] {'owner': 'hive', > 'group': 'hadoop', 'mode': 0755, 'recursive': True} > 2015-05-04 10:47:31,373 - Directory['/var/lib/hive'] {'owner': 'hive', > 'group': 'hadoop', 'mode': 0755, 'recursive': True} > 2015-05-04 10:47:31,453 - Could not verify HDP version by calling > '/usr/bin/hdp-select versions > /tmp/tmpTrvIN3'. Return Code: 0, Output: > 2.2.4.2-2 > . > 2015-05-04 10:47:31,531 - Could not verify HDP version by calling > '/usr/bin/hdp-select versions > /tmp/tmpHFa97f'. Return Code: 0, Output: > 2.2.4.2-2 > . > 2015-05-04 10:47:31,600 - Execute['env JAVA_HOME=/usr/jdk64/jdk1.7.0_67 > /var/lib/ambari-agent/data/tmp/start_hiveserver2_script > /var/log/hive/hive-server2.out /var/log/hive/hive-server2.log > /var/run/hive/hive-server.pid /etc/hive/conf.server /var/log/hive'] > {'environment': ..., 'not_if': 'ls /var/run/hive/hive-server.pid >/dev/null > 2>&1 && ps `cat /var/run/hive/hive-server.pid` >/dev/null 2>&1', 'user': > 'hive', 'path': > ['/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/usr/hdp/current/hive-client/bin:/usr/hdp/current/hadoop-client/bin']} > 2015-05-04 10:47:31,693 - Execute['/usr/jdk64/jdk1.7.0_67/bin/java -cp > /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java.jar > org.apache.ambari.server.DBConnectionVerification > 'jdbc:mysql://sv2lxbdp2mst04.corp.equinix.com/hive?createDatabaseIfNotExist=true' > hive [PROTECTED] com.mysql.jdbc.Driver'] {'path': > ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 5, 'try_sleep': 10} > >
