Hi there,
We have IPA (VERSION: 4.9.10, API_VERSION: 2.248) running on Alma Linux on 8.7
with total of 4 replicas. We're running in a cloud, so have an automated
process in place where new instances automatically enrol to IPA when launching
(they all use the same IPA user and fetch the password from a secrets manager).
For a while now we have been seeing instances fail to enrol to IPA on random
occasions, which is more pronounced when multiple instances are starting at the
same time.
Each instance runs ipa-client-install, like below, when it starts:
ipa-client-install --mkhomedir --ssh-trust-dns --domain=example.com -w${PASSW}
-phost-enrollment --unattended --force-join --no-dns-sshfp
This sometimes fails with the following:
Starting external process
args=['/usr/bin/certutil', '-d', 'sql:/tmp/tmpdqzuq_ts', '-A', '-n', 'CA
certificate 1', '-t', 'C,,', '-a', '-f', '/tmp/tmpdqzuq_ts/pwdfile.txt']
Process finished, return code=0
stdout=
stderr=
failed to find session_cookie in persistent storage for principal
'host/[email protected]'
trying https://ipa2.example.com/ipa/json
New HTTP connection (ipa2.example.com)
HTTP connection destroyed (ipa2.example.com)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/__init__.py",
line 120, in get_package
plugins = api._remote_plugins
AttributeError: 'API' object has no attribute '_remote_plugins'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 644, in
get_auth_info
response = self._sec_context.step()
File "<decorator-gen-15>", line 2, in step
File "/usr/lib64/python3.6/site-packages/gssapi/_utils.py", line 167, in
check_last_err
return func(self, *args, **kwargs)
File "<decorator-gen-5>", line 2, in step
File "/usr/lib64/python3.6/site-packages/gssapi/_utils.py", line 127, in
catch_and_return_token
return func(self, *args, **kwargs)
File "/usr/lib64/python3.6/site-packages/gssapi/sec_contexts.py", line 521,
in step
return self._initiator_step(token=token)
File "/usr/lib64/python3.6/site-packages/gssapi/sec_contexts.py", line 542,
in _initiator_step
token)
File "gssapi/raw/sec_contexts.pyx", line 244, in
gssapi.raw.sec_contexts.init_sec_context
gssapi.raw.misc.GSSError: Major (851968): Unspecified GSS failure. Minor code
may provide more information, Minor (2529639068): Cannot contact any KDC for
realm 'EXAMPLE.COM'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 697, in
single_request
self.get_auth_info()
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 646, in
get_auth_info
self._handle_exception(e, service=service)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 605, in
_handle_exception
raise errors.KerberosError(message=unicode(e))
ipalib.errors.KerberosError: Major (851968): Unspecified GSS failure. Minor
code may provide more information, Minor (2529639068): Cannot contact any KDC
for realm 'EXAMPLE.COM'
File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 180, in
execute
return_value = self.run()
File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py", line 344,
in run
return cfgr.run()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 360,
in run
return self.execute()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 386,
in execute
for rval in self._executor():
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431,
in __runner
exc_handler(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460,
in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450,
in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421,
in __runner
step()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418,
in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81,
in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59,
in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 655,
in _configure
next(executor)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431,
in __runner
exc_handler(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460,
in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 518,
in _handle_exception
self.__parent._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450,
in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 515,
in _handle_exception
super(ComponentBase, self)._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450,
in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421,
in __runner
step()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418,
in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81,
in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59,
in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.6/site-packages/ipapython/install/common.py", line 65,
in _install
for unused in self._installer(self.parent):
File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line
3961, in main
install(self)
File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line
2655, in install
_install(options)
File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line
2972, in _install
api.finalize()
File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 753, in
finalize
self.__do_if_not_done('load_plugins')
File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 432, in
__do_if_not_done
getattr(self, name)()
File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 632, in
load_plugins
for package in self.packages:
File "/usr/lib/python3.6/site-packages/ipalib/__init__.py", line 952, in
packages
ipaclient.remote_plugins.get_package(self),
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/__init__.py",
line 128, in get_package
plugins = schema.get_package(server_info, client)
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py",
line 546, in get_package
schema = Schema(client)
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py",
line 395, in __init__
fingerprint, ttl = self._fetch(client, ignore_cache=read_failed)
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py",
line 407, in _fetch
client.connect(verbose=False)
File "/usr/lib/python3.6/site-packages/ipalib/backend.py", line 69, in connect
conn = self.create_connection(*args, **kw)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1064, in
create_connection
command([],
{}
)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1276, in _call
return self.__request(name, args)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1243, in __request
verbose=self.__verbose >= 3,
File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 697, in
single_request
self.get_auth_info()
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 646, in
get_auth_info
self._handle_exception(e, service=service)
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 605, in
_handle_exception
raise errors.KerberosError(message=unicode(e))
The ipa-client-install command failed, exception: KerberosError: Major
(851968): Unspecified GSS failure. Minor code may provide more information,
Minor (2529639068): Cannot contact any KDC for realm 'EXAMPLE.COM'
Major (851968): Unspecified GSS failure. Minor code may provide more
information, Minor (2529639068): Cannot contact any KDC for realm 'EXAMPLE.COM'
The ipa-client-install command failed. See /var/log/ipaclient-install.log for
more information
This program will set up IPA client.
Version 4.9.10
On IPA server the following pops up in logs:
ERR - is_allowed_to_access_attr - [file ipa_pwd_extop.c, line 825]:
slapi_access_allowed does not allow WRITE to ipaProtectedOperation;write_keys!
ERR - ipapwd_getkeytab - [file ipa_pwd_extop.c, line 1714]: Not allowed to set
keytab on [host/[email protected]]!
This doesn't happen every time - even when multiple instances are launched from
the same image, some will fail and some will enrol successfully. It's worse
when instances are in different cloud region that IPA (even when they are very
close, network-wise, so latency shouldn't be an issue), but can still happen
within the same region. For some reason, this has also become worse when we
switched from forcing a specific IPA server (--server to ipa-client-install) to
DNS auto-discovery.
We commonly have situations where 5 instances try to launch at mostly the same
time and try to enrol using 2 replicas - and all 5 will fail, both IPAs showing
the same errors (as above).
We've run out of ideas of what to debug and how, so any clues would be
appreciated.
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[email protected]
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
