Hi Shaik… That is a good question. According to https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2, it doesn’t appear that a kinit is needed before starting up the server.
Rob From: Shaik M <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Tuesday, May 5, 2015 at 5:15 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Ambari Unable to start Hive server 2 after enabling security Ambari Team, can you please check, why it is required to do manual "kinit" for HiveServer2 startup? On 4 May 2015 at 22:24, Shaik M <[email protected]<mailto:[email protected]>> wrote: Hi, After verifying the Amabri agent log, I logged in as "hive" user and I ran below command. /usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs after that I have started "hiveserver2" service from Amabri and it started the service. Now, I can able to connect to beeline client. I have re-verified, with destroying the key and the hiveserver2 service went down after destroying the key. Look like Ambari having some issue to initiate the HiveServer2 keytabs. -Shaik On 4 May 2015 at 21:54, Shaik M <[email protected]<mailto:[email protected]>> wrote: Hi, I am getting following exception in Hiveserver2 log file: 2015-05-04 13:45:23,542 WARN [main]: ipc.Client (Client.java:run(676)) - Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 2015-05-04 13:45:23,543 INFO [main]: retry.RetryInvocationHandler (RetryInvocationHandler.java:invoke(140)) - Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over sv2lxbdp2mst05.corp.host.com/10.192.149.187:8020<http://sv2lxbdp2mst05.corp.host.com/10.192.149.187:8020> after 8 fail over attempts. Trying to fail over immediately. java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Thanks, Shaik (If it is not right place to post this query, please help me to know the correct group) On 4 May 2015 at 18:55, Shaik M <[email protected]<mailto:[email protected]>> wrote: Hi, I am using Ambari 1.7 and HDP 2.2.4. I have enabled security in this cluster. After enabling the security Ambari unable start the Hive server 2. I have verified keytabs and it's working fine. Please find the below ambari string process log below. not shown any output in standerr window. please let me know the how to resolve this issue. Thanks, Shaik M stdout: /var/lib/ambari-agent/data/output-1832.txt 2015-05-04 10:47:25,616 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry 10 http://sv2lxbdp2mst05.corp.equinix.com:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] {'environment': ..., 'not_if': 'test -e /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip', 'ignore_failures': True, 'path': ['/bin', '/usr/bin/']} 2015-05-04 10:47:25,646 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry 10 http://sv2lxbdp2mst05.corp.equinix.com:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] due to not_if 2015-05-04 10:47:25,648 - Execute['rm -f local_policy.jar; rm -f US_export_policy.jar; unzip -o -j -q /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] {'path': ['/bin/', '/usr/bin'], 'only_if': 'test -e /usr/jdk64/jdk1.7.0_67/jre/lib/security && test -f /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip', 'cwd': '/usr/jdk64/jdk1.7.0_67/jre/lib/security'} 2015-05-04 10:47:25,718 - Group['hadoop'] {'ignore_failures': False} 2015-05-04 10:47:25,720 - Modifying group hadoop 2015-05-04 10:47:25,784 - Group['nobody'] {'ignore_failures': False} 2015-05-04 10:47:25,785 - Modifying group nobody 2015-05-04 10:47:25,839 - Group['users'] {'ignore_failures': False} 2015-05-04 10:47:25,840 - Modifying group users 2015-05-04 10:47:25,886 - Group['nagios'] {'ignore_failures': False} 2015-05-04 10:47:25,887 - Modifying group nagios 2015-05-04 10:47:25,936 - User['nobody'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'nobody']} 2015-05-04 10:47:25,937 - Modifying user nobody 2015-05-04 10:47:26,012 - User['oozie'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']} 2015-05-04 10:47:26,013 - Modifying user oozie 2015-05-04 10:47:26,063 - User['hive'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-05-04 10:47:26,064 - Modifying user hive 2015-05-04 10:47:26,098 - User['mapred'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-05-04 10:47:26,099 - Modifying user mapred 2015-05-04 10:47:26,131 - User['nagios'] {'gid': 'nagios', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-05-04 10:47:26,132 - Modifying user nagios 2015-05-04 10:47:26,165 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']} 2015-05-04 10:47:26,166 - Modifying user ambari-qa 2015-05-04 10:47:26,199 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-05-04 10:47:26,199 - Modifying user zookeeper 2015-05-04 10:47:26,232 - User['tez'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']} 2015-05-04 10:47:26,233 - Modifying user tez 2015-05-04 10:47:26,265 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-05-04 10:47:26,266 - Modifying user hdfs 2015-05-04 10:47:26,298 - User['sqoop'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-05-04 10:47:26,299 - Modifying user sqoop 2015-05-04 10:47:26,332 - User['hcat'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-05-04 10:47:26,332 - Modifying user hcat 2015-05-04 10:47:26,365 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-05-04 10:47:26,366 - Modifying user yarn 2015-05-04 10:47:26,399 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2015-05-04 10:47:26,401 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'} 2015-05-04 10:47:26,431 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] due to not_if 2015-05-04 10:47:26,432 - Directory['/etc/hadoop/conf.empty'] {'owner': 'root', 'group': 'root', 'recursive': True} 2015-05-04 10:47:26,433 - Link['/etc/hadoop/conf'] {'not_if': 'ls /etc/hadoop/conf', 'to': '/etc/hadoop/conf.empty'} 2015-05-04 10:47:26,462 - Skipping Link['/etc/hadoop/conf'] due to not_if 2015-05-04 10:47:26,491 - File['/etc/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'root'} 2015-05-04 10:47:26,516 - Execute['/bin/echo 0 > /selinux/enforce'] {'only_if': 'test -f /selinux/enforce'} 2015-05-04 10:47:26,577 - Directory['/var/log/hadoop'] {'owner': 'root', 'group': 'hadoop', 'mode': 0775, 'recursive': True} 2015-05-04 10:47:26,578 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True} 2015-05-04 10:47:26,579 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True} 2015-05-04 10:47:26,590 - File['/etc/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'root'} 2015-05-04 10:47:26,595 - File['/etc/hadoop/conf/health_check'] {'content': Template('health_check-v2.j2'), 'owner': 'root'} 2015-05-04 10:47:26,596 - File['/etc/hadoop/conf/log4j.properties'] {'content': '...', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644} 2015-05-04 10:47:26,608 - File['/etc/hadoop/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'} 2015-05-04 10:47:26,609 - File['/etc/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755} 2015-05-04 10:47:26,956 - Execute['kill `cat /var/run/hive/hive-server.pid` >/dev/null 2>&1 && rm -f /var/run/hive/hive-server.pid'] {'not_if': '! (ls /var/run/hive/hive-server.pid >/dev/null 2>&1 && ps `cat /var/run/hive/hive-server.pid` >/dev/null 2>&1)'} 2015-05-04 10:47:27,068 - HdfsDirectory['/apps/hive/warehouse'] {'security_enabled': True, 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '/usr/bin/kinit', 'mode': 0777, 'owner': 'hive', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create_delayed']} 2015-05-04 10:47:27,069 - HdfsDirectory['/user/hive'] {'security_enabled': True, 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '/usr/bin/kinit', 'mode': 0700, 'owner': 'hive', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create_delayed']} 2015-05-04 10:47:27,070 - HdfsDirectory['None'] {'security_enabled': True, 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '/usr/bin/kinit', 'action': ['create'], 'bin_dir': '/usr/hdp/current/hadoop-client/bin'} 2015-05-04 10:47:27,073 - Execute['/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs'] {'user': 'hdfs'} 2015-05-04 10:47:27,850 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` /apps/hive/warehouse /user/hive && hadoop --config /etc/hadoop/conf fs -chmod 777 /apps/hive/warehouse && hadoop --config /etc/hadoop/conf fs -chmod 700 /user/hive && hadoop --config /etc/hadoop/conf fs -chown hive /apps/hive/warehouse /user/hive'] {'not_if': "su - hdfs -c 'export PATH=$PATH:/usr/hdp/current/hadoop-client/bin ; hadoop --config /etc/hadoop/conf fs -ls /apps/hive/warehouse /user/hive'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']} 2015-05-04 10:47:31,133 - Skipping Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` /apps/hive/warehouse /user/hive && hadoop --config /etc/hadoop/conf fs -chmod 777 /apps/hive/warehouse && hadoop --config /etc/hadoop/conf fs -chmod 700 /user/hive && hadoop --config /etc/hadoop/conf fs -chown hive /apps/hive/warehouse /user/hive'] due to not_if 2015-05-04 10:47:31,134 - Directory['/etc/hive/conf.server'] {'owner': 'hive', 'group': 'hadoop', 'recursive': True} 2015-05-04 10:47:31,135 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', 'conf_dir': '/etc/hive/conf.server', 'mode': 0644, 'configuration_attributes': ..., 'owner': 'hive', 'configurations': ...} 2015-05-04 10:47:31,168 - Generating config: /etc/hive/conf.server/mapred-site.xml 2015-05-04 10:47:31,169 - File['/etc/hive/conf.server/mapred-site.xml'] {'owner': 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'} 2015-05-04 10:47:31,173 - Writing File['/etc/hive/conf.server/mapred-site.xml'] because contents don't match 2015-05-04 10:47:31,174 - File['/etc/hive/conf.server/hive-default.xml.template'] {'owner': 'hive', 'group': 'hadoop'} 2015-05-04 10:47:31,175 - File['/etc/hive/conf.server/hive-env.sh.template'] {'owner': 'hive', 'group': 'hadoop'} 2015-05-04 10:47:31,177 - File['/etc/hive/conf.server/hive-exec-log4j.properties'] {'content': '...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644} 2015-05-04 10:47:31,178 - File['/etc/hive/conf.server/hive-log4j.properties'] {'content': '...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644} 2015-05-04 10:47:31,179 - Directory['/etc/hive/conf'] {'owner': 'hive', 'group': 'hadoop', 'recursive': True} 2015-05-04 10:47:31,180 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', 'conf_dir': '/etc/hive/conf', 'mode': 0644, 'configuration_attributes': ..., 'owner': 'hive', 'configurations': ...} 2015-05-04 10:47:31,202 - Generating config: /etc/hive/conf/mapred-site.xml 2015-05-04 10:47:31,203 - File['/etc/hive/conf/mapred-site.xml'] {'owner': 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'} 2015-05-04 10:47:31,207 - Writing File['/etc/hive/conf/mapred-site.xml'] because contents don't match 2015-05-04 10:47:31,208 - File['/etc/hive/conf/hive-default.xml.template'] {'owner': 'hive', 'group': 'hadoop'} 2015-05-04 10:47:31,209 - File['/etc/hive/conf/hive-env.sh.template'] {'owner': 'hive', 'group': 'hadoop'} 2015-05-04 10:47:31,211 - File['/etc/hive/conf/hive-exec-log4j.properties'] {'content': '...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644} 2015-05-04 10:47:31,212 - File['/etc/hive/conf/hive-log4j.properties'] {'content': '...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644} 2015-05-04 10:47:31,213 - XmlConfig['hive-site.xml'] {'group': 'hadoop', 'conf_dir': '/etc/hive/conf.server', 'mode': 0644, 'configuration_attributes': ..., 'owner': 'hive', 'configurations': ...} 2015-05-04 10:47:31,236 - Generating config: /etc/hive/conf.server/hive-site.xml 2015-05-04 10:47:31,237 - File['/etc/hive/conf.server/hive-site.xml'] {'owner': 'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'} 2015-05-04 10:47:31,245 - Writing File['/etc/hive/conf.server/hive-site.xml'] because contents don't match 2015-05-04 10:47:31,251 - File['/etc/hive/conf.server/hive-env.sh'] {'content': InlineTemplate(...), 'owner': 'hive', 'group': 'hadoop'} 2015-05-04 10:47:31,253 - Execute['hive mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; rm -f /usr/hdp/current/hive-client/lib/mysql-connector-java.jar ; cp /usr/share/java/mysql-connector-java.jar /usr/hdp/current/hive-client/lib/mysql-connector-java.jar'] {'environment': ..., 'path': ['/bin', '/usr/bin/'], 'creates': '/usr/hdp/current/hive-client/lib/mysql-connector-java.jar', 'not_if': 'test -f /usr/hdp/current/hive-client/lib/mysql-connector-java.jar'} 2015-05-04 10:47:31,282 - Skipping Execute['hive mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; rm -f /usr/hdp/current/hive-client/lib/mysql-connector-java.jar ; cp /usr/share/java/mysql-connector-java.jar /usr/hdp/current/hive-client/lib/mysql-connector-java.jar'] due to not_if 2015-05-04 10:47:31,285 - Execute['/bin/sh -c 'cd /usr/lib/ambari-agent/ && curl -kf -x "" --retry 5 http://sv2lxbdp2mst05.corp.equinix.com:8080/resources/DBConnectionVerification.jar -o DBConnectionVerification.jar''] {'environment': ..., 'not_if': '[ -f DBConnectionVerification.jar]'} 2015-05-04 10:47:31,370 - File['/var/lib/ambari-agent/data/tmp/start_hiveserver2_script'] {'content': Template('startHiveserver2.sh.j2'), 'mode': 0755} 2015-05-04 10:47:31,372 - Directory['/var/run/hive'] {'owner': 'hive', 'group': 'hadoop', 'mode': 0755, 'recursive': True} 2015-05-04 10:47:31,372 - Directory['/var/log/hive'] {'owner': 'hive', 'group': 'hadoop', 'mode': 0755, 'recursive': True} 2015-05-04 10:47:31,373 - Directory['/var/lib/hive'] {'owner': 'hive', 'group': 'hadoop', 'mode': 0755, 'recursive': True} 2015-05-04 10:47:31,453 - Could not verify HDP version by calling '/usr/bin/hdp-select versions > /tmp/tmpTrvIN3'. Return Code: 0, Output: 2.2.4.2-2 . 2015-05-04 10:47:31,531 - Could not verify HDP version by calling '/usr/bin/hdp-select versions > /tmp/tmpHFa97f'. Return Code: 0, Output: 2.2.4.2-2 . 2015-05-04 10:47:31,600 - Execute['env JAVA_HOME=/usr/jdk64/jdk1.7.0_67 /var/lib/ambari-agent/data/tmp/start_hiveserver2_script /var/log/hive/hive-server2.out /var/log/hive/hive-server2.log /var/run/hive/hive-server.pid /etc/hive/conf.server /var/log/hive'] {'environment': ..., 'not_if': 'ls /var/run/hive/hive-server.pid >/dev/null 2>&1 && ps `cat /var/run/hive/hive-server.pid` >/dev/null 2>&1', 'user': 'hive', 'path': ['/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/usr/hdp/current/hive-client/bin:/usr/hdp/current/hadoop-client/bin']} 2015-05-04 10:47:31,693 - Execute['/usr/jdk64/jdk1.7.0_67/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java.jar org.apache.ambari.server.DBConnectionVerification 'jdbc:mysql://sv2lxbdp2mst04.corp.equinix.com/hive?createDatabaseIfNotExist=true<http://sv2lxbdp2mst04.corp.equinix.com/hive?createDatabaseIfNotExist=true>' hive [PROTECTED] com.mysql.jdbc.Driver'] {'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 5, 'try_sleep': 10}
