Hi, After verifying the Amabri agent log, I logged in as "hive" user and I ran below command.
/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs after that I have started "hiveserver2" service from Amabri and it started the service. Now, I can able to connect to beeline client. I have re-verified, with destroying the key and the hiveserver2 service went down after destroying the key. Look like Ambari having some issue to initiate the HiveServer2 keytabs. -Shaik On 4 May 2015 at 21:54, Shaik M <[email protected]> wrote: > Hi, > > > *I am getting following exception in Hiveserver2 log file:* > > 2015-05-04 13:45:23,542 WARN [main]: ipc.Client (Client.java:run(676)) - > Exception encountered while connecting to the server : > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to > find any Kerberos tgt)] > 2015-05-04 13:45:23,543 INFO [main]: retry.RetryInvocationHandler > (RetryInvocationHandler.java:invoke(140)) - Exception while invoking > getFileInfo of class ClientNamenodeProtocolTranslatorPB over > sv2lxbdp2mst05.corp.host.com/10.192.149.187:8020 after 8 fail over > attempts. Trying to fail over immediately. > java.io.IOException: Failed on local exception: java.io.IOException: > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to > find any Kerberos tgt)]; > > Thanks, > Shaik > > (If it is not right place to post this query, please help me to know the > correct group) > > On 4 May 2015 at 18:55, Shaik M <[email protected]> wrote: > >> Hi, >> >> I am using Ambari 1.7 and HDP 2.2.4. I have enabled security in this >> cluster. >> >> After enabling the security Ambari unable start the Hive server 2. >> >> I have verified keytabs and it's working fine. Please find the below >> ambari string process log below. not shown any output in standerr window. >> >> please let me know the how to resolve this issue. >> >> Thanks, >> Shaik M >> >> stdout: /var/lib/ambari-agent/data/output-1832.txt >> >> 2015-05-04 10:47:25,616 - Execute['mkdir -p >> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry >> 10 >> http://sv2lxbdp2mst05.corp.equinix.com:8080/resources//UnlimitedJCEPolicyJDK7.zip >> -o >> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] >> {'environment': ..., 'not_if': 'test -e >> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip', >> 'ignore_failures': True, 'path': ['/bin', '/usr/bin/']} >> 2015-05-04 10:47:25,646 - Skipping Execute['mkdir -p >> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry >> 10 >> http://sv2lxbdp2mst05.corp.equinix.com:8080/resources//UnlimitedJCEPolicyJDK7.zip >> -o >> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] >> due to not_if >> 2015-05-04 10:47:25,648 - Execute['rm -f local_policy.jar; rm -f >> US_export_policy.jar; unzip -o -j -q >> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] >> {'path': ['/bin/', '/usr/bin'], 'only_if': 'test -e >> /usr/jdk64/jdk1.7.0_67/jre/lib/security && test -f >> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip', >> 'cwd': '/usr/jdk64/jdk1.7.0_67/jre/lib/security'} >> 2015-05-04 10:47:25,718 - Group['hadoop'] {'ignore_failures': False} >> 2015-05-04 10:47:25,720 - Modifying group hadoop >> 2015-05-04 10:47:25,784 - Group['nobody'] {'ignore_failures': False} >> 2015-05-04 10:47:25,785 - Modifying group nobody >> 2015-05-04 10:47:25,839 - Group['users'] {'ignore_failures': False} >> 2015-05-04 10:47:25,840 - Modifying group users >> 2015-05-04 10:47:25,886 - Group['nagios'] {'ignore_failures': False} >> 2015-05-04 10:47:25,887 - Modifying group nagios >> 2015-05-04 10:47:25,936 - User['nobody'] {'gid': 'hadoop', >> 'ignore_failures': False, 'groups': [u'nobody']} >> 2015-05-04 10:47:25,937 - Modifying user nobody >> 2015-05-04 10:47:26,012 - User['oozie'] {'gid': 'hadoop', 'ignore_failures': >> False, 'groups': [u'users']} >> 2015-05-04 10:47:26,013 - Modifying user oozie >> 2015-05-04 10:47:26,063 - User['hive'] {'gid': 'hadoop', 'ignore_failures': >> False, 'groups': [u'hadoop']} >> 2015-05-04 10:47:26,064 - Modifying user hive >> 2015-05-04 10:47:26,098 - User['mapred'] {'gid': 'hadoop', >> 'ignore_failures': False, 'groups': [u'hadoop']} >> 2015-05-04 10:47:26,099 - Modifying user mapred >> 2015-05-04 10:47:26,131 - User['nagios'] {'gid': 'nagios', >> 'ignore_failures': False, 'groups': [u'hadoop']} >> 2015-05-04 10:47:26,132 - Modifying user nagios >> 2015-05-04 10:47:26,165 - User['ambari-qa'] {'gid': 'hadoop', >> 'ignore_failures': False, 'groups': [u'users']} >> 2015-05-04 10:47:26,166 - Modifying user ambari-qa >> 2015-05-04 10:47:26,199 - User['zookeeper'] {'gid': 'hadoop', >> 'ignore_failures': False, 'groups': [u'hadoop']} >> 2015-05-04 10:47:26,199 - Modifying user zookeeper >> 2015-05-04 10:47:26,232 - User['tez'] {'gid': 'hadoop', 'ignore_failures': >> False, 'groups': [u'users']} >> 2015-05-04 10:47:26,233 - Modifying user tez >> 2015-05-04 10:47:26,265 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': >> False, 'groups': [u'hadoop']} >> 2015-05-04 10:47:26,266 - Modifying user hdfs >> 2015-05-04 10:47:26,298 - User['sqoop'] {'gid': 'hadoop', 'ignore_failures': >> False, 'groups': [u'hadoop']} >> 2015-05-04 10:47:26,299 - Modifying user sqoop >> 2015-05-04 10:47:26,332 - User['hcat'] {'gid': 'hadoop', 'ignore_failures': >> False, 'groups': [u'hadoop']} >> 2015-05-04 10:47:26,332 - Modifying user hcat >> 2015-05-04 10:47:26,365 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': >> False, 'groups': [u'hadoop']} >> 2015-05-04 10:47:26,366 - Modifying user yarn >> 2015-05-04 10:47:26,399 - >> File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': >> StaticFile('changeToSecureUid.sh'), 'mode': 0555} >> 2015-05-04 10:47:26,401 - >> Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa >> /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa >> 2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'} >> 2015-05-04 10:47:26,431 - Skipping >> Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa >> /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa >> 2>/dev/null'] due to not_if >> 2015-05-04 10:47:26,432 - Directory['/etc/hadoop/conf.empty'] {'owner': >> 'root', 'group': 'root', 'recursive': True} >> 2015-05-04 10:47:26,433 - Link['/etc/hadoop/conf'] {'not_if': 'ls >> /etc/hadoop/conf', 'to': '/etc/hadoop/conf.empty'} >> 2015-05-04 10:47:26,462 - Skipping Link['/etc/hadoop/conf'] due to not_if >> 2015-05-04 10:47:26,491 - File['/etc/hadoop/conf/hadoop-env.sh'] {'content': >> InlineTemplate(...), 'owner': 'root'} >> 2015-05-04 10:47:26,516 - Execute['/bin/echo 0 > /selinux/enforce'] >> {'only_if': 'test -f /selinux/enforce'} >> 2015-05-04 10:47:26,577 - Directory['/var/log/hadoop'] {'owner': 'root', >> 'group': 'hadoop', 'mode': 0775, 'recursive': True} >> 2015-05-04 10:47:26,578 - Directory['/var/run/hadoop'] {'owner': 'root', >> 'group': 'root', 'recursive': True} >> 2015-05-04 10:47:26,579 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', >> 'recursive': True} >> 2015-05-04 10:47:26,590 - >> File['/etc/hadoop/conf/commons-logging.properties'] {'content': >> Template('commons-logging.properties.j2'), 'owner': 'root'} >> 2015-05-04 10:47:26,595 - File['/etc/hadoop/conf/health_check'] {'content': >> Template('health_check-v2.j2'), 'owner': 'root'} >> 2015-05-04 10:47:26,596 - File['/etc/hadoop/conf/log4j.properties'] >> {'content': '...', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644} >> 2015-05-04 10:47:26,608 - >> File['/etc/hadoop/conf/hadoop-metrics2.properties'] {'content': >> Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'} >> 2015-05-04 10:47:26,609 - File['/etc/hadoop/conf/task-log4j.properties'] >> {'content': StaticFile('task-log4j.properties'), 'mode': 0755} >> 2015-05-04 10:47:26,956 - Execute['kill `cat /var/run/hive/hive-server.pid` >> >/dev/null 2>&1 && rm -f /var/run/hive/hive-server.pid'] {'not_if': '! (ls >> /var/run/hive/hive-server.pid >/dev/null 2>&1 && ps `cat >> /var/run/hive/hive-server.pid` >/dev/null 2>&1)'} >> 2015-05-04 10:47:27,068 - HdfsDirectory['/apps/hive/warehouse'] >> {'security_enabled': True, 'keytab': >> '/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': >> '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': >> '/usr/bin/kinit', 'mode': 0777, 'owner': 'hive', 'bin_dir': >> '/usr/hdp/current/hadoop-client/bin', 'action': ['create_delayed']} >> 2015-05-04 10:47:27,069 - HdfsDirectory['/user/hive'] {'security_enabled': >> True, 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': >> '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': >> '/usr/bin/kinit', 'mode': 0700, 'owner': 'hive', 'bin_dir': >> '/usr/hdp/current/hadoop-client/bin', 'action': ['create_delayed']} >> 2015-05-04 10:47:27,070 - HdfsDirectory['None'] {'security_enabled': True, >> 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': >> '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': >> '/usr/bin/kinit', 'action': ['create'], 'bin_dir': >> '/usr/hdp/current/hadoop-client/bin'} >> 2015-05-04 10:47:27,073 - Execute['/usr/bin/kinit -kt >> /etc/security/keytabs/hdfs.headless.keytab hdfs'] {'user': 'hdfs'} >> 2015-05-04 10:47:27,850 - Execute['hadoop --config /etc/hadoop/conf fs >> -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` >> /apps/hive/warehouse /user/hive && hadoop --config /etc/hadoop/conf fs >> -chmod 777 /apps/hive/warehouse && hadoop --config /etc/hadoop/conf fs >> -chmod 700 /user/hive && hadoop --config /etc/hadoop/conf fs -chown hive >> /apps/hive/warehouse /user/hive'] {'not_if': "su - hdfs -c 'export >> PATH=$PATH:/usr/hdp/current/hadoop-client/bin ; hadoop --config >> /etc/hadoop/conf fs -ls /apps/hive/warehouse /user/hive'", 'user': 'hdfs', >> 'path': ['/usr/hdp/current/hadoop-client/bin']} >> 2015-05-04 10:47:31,133 - Skipping Execute['hadoop --config /etc/hadoop/conf >> fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` >> /apps/hive/warehouse /user/hive && hadoop --config /etc/hadoop/conf fs >> -chmod 777 /apps/hive/warehouse && hadoop --config /etc/hadoop/conf fs >> -chmod 700 /user/hive && hadoop --config /etc/hadoop/conf fs -chown hive >> /apps/hive/warehouse /user/hive'] due to not_if >> 2015-05-04 10:47:31,134 - Directory['/etc/hive/conf.server'] {'owner': >> 'hive', 'group': 'hadoop', 'recursive': True} >> 2015-05-04 10:47:31,135 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', >> 'conf_dir': '/etc/hive/conf.server', 'mode': 0644, >> 'configuration_attributes': ..., 'owner': 'hive', 'configurations': ...} >> 2015-05-04 10:47:31,168 - Generating config: >> /etc/hive/conf.server/mapred-site.xml >> 2015-05-04 10:47:31,169 - File['/etc/hive/conf.server/mapred-site.xml'] >> {'owner': 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': >> 0644, 'encoding': 'UTF-8'} >> 2015-05-04 10:47:31,173 - Writing >> File['/etc/hive/conf.server/mapred-site.xml'] because contents don't match >> 2015-05-04 10:47:31,174 - >> File['/etc/hive/conf.server/hive-default.xml.template'] {'owner': 'hive', >> 'group': 'hadoop'} >> 2015-05-04 10:47:31,175 - File['/etc/hive/conf.server/hive-env.sh.template'] >> {'owner': 'hive', 'group': 'hadoop'} >> 2015-05-04 10:47:31,177 - >> File['/etc/hive/conf.server/hive-exec-log4j.properties'] {'content': '...', >> 'owner': 'hive', 'group': 'hadoop', 'mode': 0644} >> 2015-05-04 10:47:31,178 - >> File['/etc/hive/conf.server/hive-log4j.properties'] {'content': '...', >> 'owner': 'hive', 'group': 'hadoop', 'mode': 0644} >> 2015-05-04 10:47:31,179 - Directory['/etc/hive/conf'] {'owner': 'hive', >> 'group': 'hadoop', 'recursive': True} >> 2015-05-04 10:47:31,180 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', >> 'conf_dir': '/etc/hive/conf', 'mode': 0644, 'configuration_attributes': ..., >> 'owner': 'hive', 'configurations': ...} >> 2015-05-04 10:47:31,202 - Generating config: /etc/hive/conf/mapred-site.xml >> 2015-05-04 10:47:31,203 - File['/etc/hive/conf/mapred-site.xml'] {'owner': >> 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, >> 'encoding': 'UTF-8'} >> 2015-05-04 10:47:31,207 - Writing File['/etc/hive/conf/mapred-site.xml'] >> because contents don't match >> 2015-05-04 10:47:31,208 - File['/etc/hive/conf/hive-default.xml.template'] >> {'owner': 'hive', 'group': 'hadoop'} >> 2015-05-04 10:47:31,209 - File['/etc/hive/conf/hive-env.sh.template'] >> {'owner': 'hive', 'group': 'hadoop'} >> 2015-05-04 10:47:31,211 - File['/etc/hive/conf/hive-exec-log4j.properties'] >> {'content': '...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644} >> 2015-05-04 10:47:31,212 - File['/etc/hive/conf/hive-log4j.properties'] >> {'content': '...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644} >> 2015-05-04 10:47:31,213 - XmlConfig['hive-site.xml'] {'group': 'hadoop', >> 'conf_dir': '/etc/hive/conf.server', 'mode': 0644, >> 'configuration_attributes': ..., 'owner': 'hive', 'configurations': ...} >> 2015-05-04 10:47:31,236 - Generating config: >> /etc/hive/conf.server/hive-site.xml >> 2015-05-04 10:47:31,237 - File['/etc/hive/conf.server/hive-site.xml'] >> {'owner': 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': >> 0644, 'encoding': 'UTF-8'} >> 2015-05-04 10:47:31,245 - Writing >> File['/etc/hive/conf.server/hive-site.xml'] because contents don't match >> 2015-05-04 10:47:31,251 - File['/etc/hive/conf.server/hive-env.sh'] >> {'content': InlineTemplate(...), 'owner': 'hive', 'group': 'hadoop'} >> 2015-05-04 10:47:31,253 - Execute['hive mkdir -p >> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; rm -f >> /usr/hdp/current/hive-client/lib/mysql-connector-java.jar ; cp >> /usr/share/java/mysql-connector-java.jar >> /usr/hdp/current/hive-client/lib/mysql-connector-java.jar'] {'environment': >> ..., 'path': ['/bin', '/usr/bin/'], 'creates': >> '/usr/hdp/current/hive-client/lib/mysql-connector-java.jar', 'not_if': 'test >> -f /usr/hdp/current/hive-client/lib/mysql-connector-java.jar'} >> 2015-05-04 10:47:31,282 - Skipping Execute['hive mkdir -p >> /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; rm -f >> /usr/hdp/current/hive-client/lib/mysql-connector-java.jar ; cp >> /usr/share/java/mysql-connector-java.jar >> /usr/hdp/current/hive-client/lib/mysql-connector-java.jar'] due to not_if >> 2015-05-04 10:47:31,285 - Execute['/bin/sh -c 'cd /usr/lib/ambari-agent/ && >> curl -kf -x "" --retry 5 >> http://sv2lxbdp2mst05.corp.equinix.com:8080/resources/DBConnectionVerification.jar >> -o DBConnectionVerification.jar''] {'environment': ..., 'not_if': '[ -f >> DBConnectionVerification.jar]'} >> 2015-05-04 10:47:31,370 - >> File['/var/lib/ambari-agent/data/tmp/start_hiveserver2_script'] {'content': >> Template('startHiveserver2.sh.j2'), 'mode': 0755} >> 2015-05-04 10:47:31,372 - Directory['/var/run/hive'] {'owner': 'hive', >> 'group': 'hadoop', 'mode': 0755, 'recursive': True} >> 2015-05-04 10:47:31,372 - Directory['/var/log/hive'] {'owner': 'hive', >> 'group': 'hadoop', 'mode': 0755, 'recursive': True} >> 2015-05-04 10:47:31,373 - Directory['/var/lib/hive'] {'owner': 'hive', >> 'group': 'hadoop', 'mode': 0755, 'recursive': True} >> 2015-05-04 10:47:31,453 - Could not verify HDP version by calling >> '/usr/bin/hdp-select versions > /tmp/tmpTrvIN3'. Return Code: 0, Output: >> 2.2.4.2-2 >> . >> 2015-05-04 10:47:31,531 - Could not verify HDP version by calling >> '/usr/bin/hdp-select versions > /tmp/tmpHFa97f'. Return Code: 0, Output: >> 2.2.4.2-2 >> . >> 2015-05-04 10:47:31,600 - Execute['env JAVA_HOME=/usr/jdk64/jdk1.7.0_67 >> /var/lib/ambari-agent/data/tmp/start_hiveserver2_script >> /var/log/hive/hive-server2.out /var/log/hive/hive-server2.log >> /var/run/hive/hive-server.pid /etc/hive/conf.server /var/log/hive'] >> {'environment': ..., 'not_if': 'ls /var/run/hive/hive-server.pid >/dev/null >> 2>&1 && ps `cat /var/run/hive/hive-server.pid` >/dev/null 2>&1', 'user': >> 'hive', 'path': >> ['/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/usr/hdp/current/hive-client/bin:/usr/hdp/current/hadoop-client/bin']} >> 2015-05-04 10:47:31,693 - Execute['/usr/jdk64/jdk1.7.0_67/bin/java -cp >> /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java.jar >> org.apache.ambari.server.DBConnectionVerification >> 'jdbc:mysql://sv2lxbdp2mst04.corp.equinix.com/hive?createDatabaseIfNotExist=true' >> hive [PROTECTED] com.mysql.jdbc.Driver'] {'path': >> ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 5, 'try_sleep': >> 10} >> >> >
