Paul B. Henson wrote: > On Sun, Feb 13, 2022 at 08:00:29PM -0800, Paul B. Henson wrote: >> I'm still trying to figure out why my servers sometimes get into a state >> where queries requesting the memberOf attribute take an exceedingly long > > So one of my servers got into this state again: > > Total DISK READ : 89.60 M/s | Total DISK WRITE : 241.97 K/s > [0/9406] > Actual DISK READ: 91.61 M/s | Actual DISK WRITE: 140.50 K/s > TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND > > 430373 be/4 ldap 34.88 M/s 0.00 B/s 0.00 % 96.44 % slapd -d 0 -h > ldap:/// ~dapi:/// -u ldap -g ldap > 430064 be/4 ldap 45.04 M/s 0.00 B/s 0.00 % 94.93 % slapd -d 0 -h > ldap:/// ~dapi:/// -u ldap -g ldap > 430069 be/4 ldap 6.34 M/s 0.00 B/s 0.00 % 8.22 % slapd -d 0 -h > ldap:/// ~dapi:/// -u ldap -g ldap > > There's plenty of free memory: > > ldap-02 ~ # free -m > total used free shared buff/cache > available > Mem: 3901 418 113 0 3368 > 3255 > Swap: 2047 763 1284 > > Just for giggles, I removed all swap: > > total used free shared buff/cache > available > Mem: 3901 730 102 1 3068 > 2949 > Swap: 0 0 0 > > The problem immediately went away. Didn't restart slapd, didn't do anything, > other than remove all swap and force it to use the plethora of memory > it had available. Disk read went back to virtually 0, response time went > back to subsecond. > > I updated vm.swappiness to 1 (it defaulted to 30) and added swap back, I'm > going to see what happens. > > Have no idea what's causing it, but it seems somehow the system and slapd > get into a state where doing memberof queries make it read lmdb pages > from disk rather than keeping them in memory? > Interesting.
Fyi, setting a smaller swappiness will lower the priority of filesystem cache, and should make the OS favor program heap memory instead. So if the OS was swapping out LMDB pages then a lower swappiness should exacerbate the problem. Let us know how things go. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
