Hello list,I have worked on ticket #3222 [1]. The reproducer mentioned on this ticket doesn't work for 100%. After some investigation and discussion with SSSD team I recognized that the issue is caused by memory cache (fast cache).
This memcache saves users and groups. Default timeout is 300 seconds.I prepare simple patch which shows important debug msg. for investigation of mem cache (in this case). And two reproducers, one with memcache and without one.
If we compiled SSSD with 'debug' patch and we run 'group_member_with_memcache.sh' we could see i.e.:
[root@mirach sssd]# group_member_with_memcache.sh && grep '>>>' *.log --------------------- Added user "testuser" --------------------- [...] ----------------------- Added group "testgroup" ----------------------- Group name: testgroup GID: 1703800630 Mon Oct 24 15:20:05 CEST 2016 >>> [A] record not found >>> [B] record not found testgroup:*:1703800630: Group name: testgroup GID: 1703800630 Member users: testuser ------------------------- Number of members added 1 -------------------------ldb: unable to dlopen /usr/lib64/ldb/modules/ldb/memberof.la : /usr/lib64/ldb/modules/ldb/memberof.la: invalid ELF header
Mon Oct 24 15:20:09 CEST 2016 >>> [A] record not found >>> [B] record not found testgroup:*:1703800630:testuser Group name: testgroup GID: 1703800630 --------------------------- Number of members removed 1 ---------------------------ldb: unable to dlopen /usr/lib64/ldb/modules/ldb/memberof.la : /usr/lib64/ldb/modules/ldb/memberof.la: invalid ELF header
Mon Oct 24 15:20:13 CEST 2016 >>> mc record expires at [1477315509] | now [1477315213] >>> [A] MC used testgroup:*:1703800630:testuser ------------------------- Deleted group "testgroup" ------------------------- ----------------------- Deleted user "testuser" -----------------------sssd_nss.log:(Mon Oct 24 15:20:05 2016) [sssd[nss]] [sss_mmap_set_rec_header] (0x0010): >>> MC STORE expiration [1477315505] | now [1477315205] sssd_nss.log:(Mon Oct 24 15:20:05 2016) [sssd[nss]] [sss_mmap_cache_gr_store] (0x0010): >>> MC STORE [testgroup] [300] members [0] sssd_nss.log:(Mon Oct 24 15:20:09 2016) [sssd[nss]] [sss_mmap_set_rec_header] (0x0010): >>> MC STORE expiration [1477315509] | now [1477315209] sssd_nss.log:(Mon Oct 24 15:20:09 2016) [sssd[nss]] [sss_mmap_cache_gr_store] (0x0010): >>> MC STORE [testgroup] [300] members [1]
How we can see the wrong result is just taken from memory cache. If you run 'group_member_without_memcache' it works right. So I suppose that the solution for ticket is simple just adjusting memcache_timeout option on problematic deployments.
If everybody agree I will close ticket as not a bug and write comment about memcache_timeout option.
Regards [1] https://fedorahosted.org/sssd/ticket/3222 -- Petr^4 Čech
>From cd7fd91e0a20740541e0fc10e795e3259e1bc975 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Petr=20=C4=8Cech?= <pc...@redhat.com>
Date: Mon, 24 Oct 2016 15:16:34 +0200
Subject: [PATCH] WIP: debug for t3222
This patch enables debug messages needed for investigation of memory
cache.
---
src/responder/nss/nsssrv_mmap_cache.c | 4 ++++
src/sss_client/nss_group.c | 9 +++++++++
src/sss_client/nss_mc_group.c | 3 +++
3 files changed, 16 insertions(+)
diff --git a/src/responder/nss/nsssrv_mmap_cache.c b/src/responder/nss/nsssrv_mmap_cache.c
index f7f62733941cd3ae3b071d6d54c801f9be1ce800..08f689d5be0020046a62d7118564574a87f405c5 100644
--- a/src/responder/nss/nsssrv_mmap_cache.c
+++ b/src/responder/nss/nsssrv_mmap_cache.c
@@ -643,6 +643,8 @@ static inline void sss_mmap_set_rec_header(struct sss_mc_ctx *mcc,
rec->expire = time(NULL) + ttl;
rec->hash1 = sss_mc_hash(mcc, key1, key1_len);
rec->hash2 = sss_mc_hash(mcc, key2, key2_len);
+
+ DEBUG(SSSDBG_FATAL_FAILURE, ">>> MC STORE expiration [%lu] | now [%lu]\n", rec->expire, time(NULL));
}
static inline void sss_mmap_chain_in_rec(struct sss_mc_ctx *mcc,
@@ -889,6 +891,8 @@ int sss_mmap_cache_gr_store(struct sss_mc_ctx **_mcc,
memcpy(&data->strs[pos], membuf, memsize);
pos += memsize;
+ DEBUG(SSSDBG_FATAL_FAILURE, ">>> MC STORE [%s] [%li] members [%i]\n", name->str, mcc->valid_time_slot, data->members);
+
MC_LOWER_BARRIER(rec);
/* finally chain the rec in the hash table */
diff --git a/src/sss_client/nss_group.c b/src/sss_client/nss_group.c
index 0e686af43aeb84a5938315e3922e9fcf2fef4e83..39c25ad81e78f7b10a4b3699b4cd1260645bd4c5 100644
--- a/src/sss_client/nss_group.c
+++ b/src/sss_client/nss_group.c
@@ -24,6 +24,7 @@
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>
+#include <stdio.h>
#include <stdint.h>
#include <string.h>
#include <stdbool.h>
@@ -421,17 +422,21 @@ enum nss_status _nss_sss_getgrnam_r(const char *name, struct group *result,
switch (ret) {
case 0:
*errnop = 0;
+ printf(">>> [A] MC used\n");
return NSS_STATUS_SUCCESS;
case ERANGE:
*errnop = ERANGE;
+ printf(">>> [A] MC expired\n");
return NSS_STATUS_TRYAGAIN;
case ENOENT:
/* fall through, we need to actively ask the parent
* if no entry is found */
+ printf(">>> [A] record not found\n");
break;
default:
/* if using the mmaped cache failed,
* fall back to socket based comms */
+ printf(">>> [A] default\n");
break;
}
@@ -446,18 +451,22 @@ enum nss_status _nss_sss_getgrnam_r(const char *name, struct group *result,
case 0:
*errnop = 0;
nret = NSS_STATUS_SUCCESS;
+ printf(">>> [B] MC used\n");
goto out;
case ERANGE:
*errnop = ERANGE;
nret = NSS_STATUS_TRYAGAIN;
+ printf(">>> [B] MC expired\n");
goto out;
case ENOENT:
/* fall through, we need to actively ask the parent
* if no entry is found */
+ printf(">>> [B] record not found\n");
break;
default:
/* if using the mmaped cache failed,
* fall back to socket based comms */
+ printf(">>> [B] default\n");
break;
}
diff --git a/src/sss_client/nss_mc_group.c b/src/sss_client/nss_mc_group.c
index aacf59d9fd8b81ea895f4660de08f3e44f0ce645..1b5632d5debf5c0cb6bd948df913ba5fed773aad 100644
--- a/src/sss_client/nss_mc_group.c
+++ b/src/sss_client/nss_mc_group.c
@@ -23,6 +23,7 @@
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
+#include <stdio.h>
#include <stddef.h>
#include <sys/mman.h>
#include <time.h>
@@ -48,8 +49,10 @@ static errno_t sss_nss_mc_parse_result(struct sss_mc_rec *rec,
expire = rec->expire;
if (expire < time(NULL)) {
/* entry is now invalid */
+ printf(">>> mc record expired [%lu] | now [%lu]\n", rec->expire, time(NULL));
return EINVAL;
}
+ printf(">>> mc record expires at [%lu] | now [%lu]\n", rec->expire, time(NULL));
data = (struct sss_mc_grp_data *)rec->data;
--
2.7.4
group_member_with_memcache.sh
Description: application/shellscript
group_member_without_memcache.sh
Description: application/shellscript
_______________________________________________ sssd-devel mailing list -- sssd-devel@lists.fedorahosted.org To unsubscribe send an email to sssd-devel-le...@lists.fedorahosted.org
