Sandor Magyari created AMBARI-14409:
---------------------------------------
Summary: Blueprints: Kerberos deployments fail intermittently due
to invalid keytabs
Key: AMBARI-14409
URL: https://issues.apache.org/jira/browse/AMBARI-14409
Project: Ambari
Issue Type: Bug
Components: ambari-server
Affects Versions: 2.0.0
Reporter: Sandor Magyari
Assignee: Sandor Magyari
Priority: Critical
Fix For: 2.2.1
The basic problem appears to be a concurrency problem, in which keytabs are
sometimes generated incorrectly. This results in certain keytabs across the
cluster having different created/modified times in the keytabs, which can cause
Kerberos Preauthentication failures, such as the following:
{code}
resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt
/etc/security/keytabs/smokeuser.headless.keytab
[email protected];' returned 1. kinit: Password incorrect while
getting initial credentials
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py",
line 179, in <module>
ZookeeperServer().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 217, in execute
method(env)
File
"/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py",
line 55, in start
zookeeper_service(action='start', upgrade_type=upgrade_type)
File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py",
line 89, in thunk
return fn(*args, **kwargs)
File
"/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py",
line 54, in zookeeper_service
user=params.smokeuser
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
line 154, in __init__
self.env.run()
File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 158, in run
self.run_action(resource, action)
File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 121, in run_action
provider_action()
File
"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
line 238, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 291, in _call
raise Fail(err_msg)
{code}
The basic underlying problem appears to be invalid keytabs. Looking to the
keytabs on the failing node:
{code}
[root@rwntest-2 keytabs]# klist -kt smokeuser.headless.keytab
Keytab name: FILE:smokeuser.headless.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
1 12/04/15 20:25:04 [email protected]
1 12/04/15 20:25:04 [email protected]
1 12/04/15 20:25:04 [email protected]
1 12/04/15 20:25:04 [email protected]
1 12/04/15 20:25:04 [email protected]
{code}
whereas other nodes in the cluster report the following value for the same
keytab:
{code}
[root@rwntest-4 keytabs]# klist -kt smokeuser.headless.keytab
Keytab name: FILE:smokeuser.headless.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
4 12/04/15 20:25:32 [email protected]
4 12/04/15 20:25:32 [email protected]
4 12/04/15 20:25:32 [email protected]
4 12/04/15 20:25:32 [email protected]
4 12/04/15 20:25:32 [email protected]
{code}
It looks like these keytabs should have the same timestamps, and this is
causing the Kerberos failures at cluster start up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
