On la, 05 helmi 2022, Brian J. Murrell via FreeIPA-users wrote:
I am still trying to debug why the webUI on my new replica is not
authenticating me.
One difference I have noticed between my two replicas, one working and
one not is:
working replica:
# KRB5RCACHEDIR=/var/lib/gssproxy/rcache klist
Ticket cache: KCM:0
Default principal: [email protected]
Valid starting Expires Service principal
2022-02-04 08:58:59 2022-02-05 08:58:56 krbtgt/[email protected]
2022-02-04 08:59:03 2022-02-05 08:58:56
HTTP/[email protected]
Non-working replica:
# KRB5RCACHEDIR=/var/lib/gssproxy/rcache klist
Ticket cache: KEYRING:persistent:0:krb_ccache_AunlIbq
Default principal: host/[email protected]
Valid starting Expires Service principal
1969-12-31 19:00:00 1969-12-31 19:00:00 Encrypted/Credentials/v1@X-GSSPROXY:
What could cause the latter to not be getting any tickets like the
former is?
The above is unrelated completely to IPA server operation.
KRB5RCACHEDIR has nothing to do with credentials caches visible with
klist, so specifying that one makes no sense. In both cases you have
klist looking into root user's credential cache and the difference there
is by what your OS defaults are:
- in one case you have KCM as a credential cache collection
- in another case you have kernel keyring as a credential cache
collection
FreeIPA server code does not rely on your root user credential caches at
all. IPA API in httpd is launched with a special configuration that uses
ccaches in pre-defined locations to isolate it from the system-wide
settings.
Below are corresponding snippets for 389-ds instance and for httpd:
$ cat install/share/ds-ipa-env.conf.template
# Installed and maintained by ipa update tools, please do not modify
[Service]
Environment=LC_ALL=C.UTF-8
Environment=KRB5_KTNAME=$KRB5_KTNAME
Environment=KRB5_CLIENT_KTNAME=$KRB5_KTNAME
Environment=KRB5CCNAME=$KRB5CCNAME
$ cat install/share/ipa-httpd.conf.template
# Do not edit. Created by IPA installer.
[Service]
Environment=KRB5CCNAME=$KRB5CC_HTTPD
Environment=GSS_USE_PROXY=yes
Environment=KDCPROXY_CONFIG=$KDCPROXY_CONFIG
Environment=LC_ALL=C.UTF-8
ExecStartPre=$IPA_HTTPD_KDCPROXY
Where KRB5CC_HTTPD is set to '/tmp/krb5cc-httpd' within httpd process
namespace which uses private tmpfs by systemd and for directory server's
instance it is set to '/tmp/krb5cc_<dirsrv_uid>', again, within httpd
process namespace which also uses private tmpfs by systemd.
Both of these isolated from what you are showing. And they also not what
used to store credentials your gssproxy log attempts to find. ;)
Before I proceed with more explanation, when you see something like
below in a ccache with klist:
Valid starting Expires Service principal
1969-12-31 19:00:00 1969-12-31 19:00:00 Encrypted/Credentials/v1@X-GSSPROXY:
it means GSSProxy stored its data in this ccache in encrypted form. Only
GSSProxy daemon can decrypt its content.
Now to your logs:
[CID 15][2022/02/05 16:08:59]: [status] Executing request 6 (GSSX_ACQUIRE_CRED)
from [0x7fdf7c08a880 (176)]
[CID 15][2022/02/05 16:08:59]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for
service "ipa-api", euid: 977,socket: (null)
GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [ ] } input_cred_handle: <Null> add_cred: 0
desired_name: { "[email protected]" { 1 2 840 113554 1 2 2 1 } [ ] [ ] [ ] } time_req:
4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0
acceptor_time_req: 0 )
GSSX_RES_ACQUIRE_CRED( status: { 851968 { 1 2 840 113554 1 2 2 } 2529639107 "Unspecified GSS
failure. Minor code may provide more information" "No credentials cache found" [ ] }
output_cred_handle: <Null> )
In IPA API configuration in httpd we have an arrangement where each user
connecting to IPA API endpoint would have a separate ccache stored by
httpd daemon with credentials encrypted by GSSProxy. This configuration
is enforced through mod_auth_gssapi:
# Protect /ipa and everything below it in webspace with Apache Kerberos auth
<Location "/ipa">
AuthType GSSAPI
AuthName "Kerberos Login"
GssapiUseSessions On
Session On
SessionCookieName ipa_session path=/ipa;httponly;secure;
SessionHeader IPASESSION
# Uncomment the following to have shorter sessions, but beware this may break
# old IPA client tols that incorrectly parse cookies.
# SessionMaxAge 1800
GssapiSessionKey file:$GSSAPI_SESSION_KEY
GssapiImpersonate On
GssapiDelegCcacheDir $IPA_CCACHES
GssapiDelegCcachePerms mode:0660
GssapiDelegCcacheUnique On
GssapiUseS4U2Proxy on
GssapiAllowedMech krb5
Require valid-user
ErrorDocument 401 /ipa/errors/unauthorized.html
Header always append X-Frame-Options DENY
Header always append Content-Security-Policy "frame-ancestors 'none'"
# mod_session always sets two copies of the cookie, and this confuses our
# legacy clients, the unset here works because it ends up unsetting only one
# of the 2 header tables set by mod_session, leaving the other intact
Header unset Set-Cookie
# Disable etag http header. Doesn't work well with mod_deflate
# https://issues.apache.org/bugzilla/show_bug.cgi?id=45023
# Usage of last-modified header and modified-since validator is sufficient.
Header unset ETag
FileETag None
</Location>
In the configuration above mod_auth_gssapi is responsible to generate
ipa_session cookie after a user has been authenticated. Since we rely on
Kerberos extension (S4U2Proxy) to communicate further with LDAP and
other services on behalf of the user, GSSAPI credentials delegation
feature of mod_auth_gssapi is used to create credential caches for each
user in $IPA_CCACHES location (/run/ipa/ccaches). In RHEL 8.4+ we set
'GssapiDelegCcacheUnique On', to have unique credential caches for each
user's session, not just per-user (see
https://pagure.io/freeipa/issue/8589). The stale ccaches then removed by
a script that is run periodically as a systemd timer
(ipa-ccache-sweep.timer which runs ipa-ccache-sweep.service).
When user connecting to IPA API presents valid ipa_session cookie,
its content is unpacked by mod_auth_gssapi to /run/ipa/ccaches/<file> to
represent the ccache with that user's tickets. The cookie itself is
encrypted by mod_auth_gssapi. The content of the ccache is encrypted by
gssproxy.
Basically, this means we have following setup:
- httpd with mod_auth_gssapi runs under 'apache' user
- IPA API in httpd with mod_wsgi runs under 'ipaapi' user
- GSSProxy manages content of the credential caches accessed from both
processes.
---------------
[root@dc ~]# ps axufw|egrep '(httpd|ipaapi)'
root 5999 0.0 0.0 6412 2100 pts/0 S+ 10:26 0:00
\_ grep -E --color=auto (httpd|ipaapi)
root 888 0.0 0.5 26608 16532 ? Ss Feb04 0:07
/usr/sbin/httpd -DFOREGROUND
apache 4698 0.0 0.2 38580 8320 ? S 00:00 0:00 \_
/usr/sbin/httpd -DFOREGROUND
ipaapi 4701 0.0 2.8 207128 84876 ? Sl 00:00 0:03 \_
(wsgi:ipa) -DFOREGROUND
ipaapi 4702 0.0 2.8 272696 86692 ? Sl 00:00 0:03 \_
(wsgi:ipa) -DFOREGROUND
ipaapi 4703 0.0 3.0 274744 90128 ? Sl 00:00 0:03 \_
(wsgi:ipa) -DFOREGROUND
ipaapi 4704 0.0 2.8 272664 84876 ? Sl 00:00 0:03 \_
(wsgi:ipa) -DFOREGROUND
apache 4705 0.0 0.5 1591628 15912 ? Sl 00:00 0:05 \_
/usr/sbin/httpd -DFOREGROUND
apache 4706 0.0 0.6 1460492 20444 ? Sl 00:00 0:05 \_
/usr/sbin/httpd -DFOREGROUND
apache 4707 0.0 0.6 1460488 19756 ? Sl 00:00 0:04 \_
/usr/sbin/httpd -DFOREGROUND
apache 5909 0.0 0.5 1460488 15908 ? Sl 10:20 0:00 \_
/usr/sbin/httpd -DFOREGROUND
---------------
It means ccache generated by GSSAPI operations performed by
either mod_auth_gssapi or IPA API is written under corresponding user
('apache' or 'ipaapi') and need to be readable by either process to allow
gssproxy interposer to send this data back and forth to gssproxy daemon.
To do so, we rely on POSIX ACLs set on /run/ipa/ccaches that allow
apache group to read/write to the files in /run/ipa/ccaches and force
ownership to ipaapi:ipaapi user and group.
Here is an example:
---------------------
[root@dc ~]# ls -la /run/ipa/ccaches/
total 0
drwsrws---+ 2 ipaapi ipaapi 40 Feb 4 11:11 .
drwx--x--x. 3 root root 100 Feb 4 11:12 ..
[root@dc ~]# getfacl /run/ipa/ccaches/
getfacl: Removing leading '/' from absolute path names
# file: run/ipa/ccaches/
# owner: ipaapi
# group: ipaapi
# flags: ss-
user::rwx
group::rwx
group:apache:rwx
mask::rwx
other::---
[root@dc ~]# kinit admin
Password for [email protected]:
[root@dc ~]# ipa ping
-------------------------------------------------------------------------
IPA server version 4.10.0.dev202202031222+git015d5ff89. API version 2.246
-------------------------------------------------------------------------
[root@dc ~]# ls -la /run/ipa/ccaches/
total 12
drwsrws---+ 2 ipaapi ipaapi 60 Feb 6 10:20 .
drwx--x--x. 3 root root 100 Feb 4 11:12 ..
-rw-------. 1 ipaapi ipaapi 8942 Feb 6 10:20 [email protected]
-----------------------
If you look with klist into that ccache, you'll only see encrypted
gssproxy object data:
------------------------
[root@dc ~]# klist -c /run/ipa/ccaches/[email protected]
Ticket cache: FILE:/run/ipa/ccaches/[email protected]
Default principal: [email protected]
Valid starting Expires Service principal
010170 00:00:00 010170 00:00:00 Encrypted/Credentials/v1@X-GSSPROXY:
------------------------
And that is correct. When ipaapi or apache user attempts to access the
ccache through GSSAPI, it will get the request forwarded to GSSProxy via
special gssproxy interposer plugin in GSSAPI. GSSProxy daemon will
decide whether this request is correct and will process it, transparent
to the caller.
What is important here, is that the original requester (one of httpd
daemon processes) still has to have read/write access to the specific
ccache in /run/ipa/ccaches that represents current user.
So if you have problems like in the log below, it means one of the
possible issues:
- /run/ipa/ccaches has wrong setup, not corresponding to what is
shown above
- /run/ipa/ccaches/user@REALM-suffix file is not accesible to httpd
processes, or damaged, making its content not decryptable by GSSProxy
- ipa_session cookie generated by mod_auth_gssapi earlier and received
by httpd process now is encrypted by a different mod_auth_gssapi
session key, rendering its content damaged with regards to what
Kerberos 5 library expects for a ccache
- GSSProxy session key is different to what was used to encrypt ccaches
in /run/ipa/ccaches. This happens on reboot, for example.
The difference between RHEL 8.3 or later and RHEL 8.4 should be in the
name of /run/ipa/ccaches/<ccache> file. Pre-RHEL 8.4 versions used
single file to store session per user, making impossible parallel IPA
API calls under the same user credentials from different machines.
[CID 15][2022/02/05 16:08:59]: [status] Returned buffer 6 (GSSX_ACQUIRE_CRED)
from [0x7fdf7c08a880 (176)]: [0x7fdf7c066260 (176)]
[CID 15][2022/02/05 16:08:59]: [status] Handling query output: 0x7fdf7c066260
(176)
[2022/02/05 16:08:59]: [status] Handling query reply: 0x7fdf7c066260 (176)
[2022/02/05 16:08:59]: [status] Sending data: 0x7fdf7c066260 (176)
[2022/02/05 16:08:59]: [status] Sending data [0x7fdf7c066260 (176)]: successful
write of 176
[2022/02/05 16:08:59]: Client [2022/02/05 16:08:59]: (/usr/sbin/httpd)
[2022/02/05 16:08:59]: connected (fd = 16)[2022/02/05 16:08:59]: (pid = 4268)
(uid = 977) (gid = 973)[2022/02/05 16:08:59]: (context =
system_u:system_r:httpd_t:s0)[2022/02/05 16:08:59]:
[CID 16][2022/02/05 16:08:59]: [status] Handling query input: 0x7fdf7c066260
(176)
[CID 16][2022/02/05 16:08:59]: Connection matched service ipa-api
[CID 16][2022/02/05 16:08:59]: [status] Processing request [0x7fdf7c066260
(176)]
[CID 16][2022/02/05 16:08:59]: [status] Executing request 6 (GSSX_ACQUIRE_CRED)
from [0x7fdf7c066260 (176)]
[CID 16][2022/02/05 16:08:59]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for
service "ipa-api", euid: 977,socket: (null)
GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [ ] } input_cred_handle: <Null> add_cred: 0
desired_name: { "[email protected]" { 1 2 840 113554 1 2 2 1 } [ ] [ ] [ ] } time_req:
4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0
acceptor_time_req: 0 )
GSSX_RES_ACQUIRE_CRED( status: { 851968 { 1 2 840 113554 1 2 2 } 2529639107 "Unspecified GSS
failure. Minor code may provide more information" "No credentials cache found" [ ] }
output_cred_handle: <Null> )
[CID 16][2022/02/05 16:08:59]: [status] Returned buffer 6 (GSSX_ACQUIRE_CRED)
from [0x7fdf7c066260 (176)]: [0x7fdf7c05c9e0 (176)]
[CID 16][2022/02/05 16:08:59]: [status] Handling query output: 0x7fdf7c05c9e0
(176)
[2022/02/05 16:08:59]: [status] Handling query reply: 0x7fdf7c05c9e0 (176)
[2022/02/05 16:08:59]: [status] Sending data: 0x7fdf7c05c9e0 (176)
[2022/02/05 16:08:59]: [status] Sending data [0x7fdf7c05c9e0 (176)]: successful
write of 176
[CID 16][2022/02/05 16:08:59]: [status] Handling query input: 0x7fdf7c05c9e0
(176)
[CID 16][2022/02/05 16:08:59]: Connection matched service ipa-api
[CID 16][2022/02/05 16:08:59]: [status] Processing request [0x7fdf7c05c9e0
(176)]
[CID 16][2022/02/05 16:08:59]: [status] Executing request 6 (GSSX_ACQUIRE_CRED)
from [0x7fdf7c05c9e0 (176)]
[CID 16][2022/02/05 16:08:59]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for
service "ipa-api", euid: 977,socket: (null)
GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [ ] } input_cred_handle: <Null> add_cred: 0
desired_name: { "[email protected]" { 1 2 840 113554 1 2 2 1 } [ ] [ ] [ ] } time_req:
4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0
acceptor_time_req: 0 )
GSSX_RES_ACQUIRE_CRED( status: { 851968 { 1 2 840 113554 1 2 2 } 2529639107 "Unspecified GSS
failure. Minor code may provide more information" "No credentials cache found" [ ] }
output_cred_handle: <Null> )
[CID 16][2022/02/05 16:08:59]: [status] Returned buffer 6 (GSSX_ACQUIRE_CRED)
from [0x7fdf7c05c9e0 (176)]: [0x7fdf7c03f690 (176)]
[CID 16][2022/02/05 16:08:59]: [status] Handling query output: 0x7fdf7c03f690
(176)
[2022/02/05 16:08:59]: [status] Handling query reply: 0x7fdf7c03f690 (176)
[2022/02/05 16:08:59]: [status] Sending data: 0x7fdf7c03f690 (176)
[2022/02/05 16:08:59]: [status] Sending data [0x7fdf7c03f690 (176)]: successful
write of 176
Cheers,
b.
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[email protected]
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[email protected]
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
