Re: [Freeipa-users] SSSD setting memcache_timeout on ipa master
On 2017-04-10 13:23, Jakub Hrozek wrote: [...] This shouldn't be the case with 1.14+ and wasn't in my testing. Did you remove the cache (really remove, not just expire with sss_cache) after you upgraded from 1.13 to 1.14? If yes, can you run some simple systemtap scripts? I did not upgrade from an older version. I experienced the problems with SSSD 1.14. I followed the steps in the performance tuning guide and moved the cache directory into RAM. After that I deleted the directory's content and restarted SSSD. -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] SSSD setting memcache_timeout on ipa master
On Mon, Apr 10, 2017 at 01:07:08PM +0200, Ronald Wimmer wrote: > On 2017-04-10 12:16, Lukas Slebodnik wrote: > > [...] > > sssd_be consumed a lot of CPU and produced a lot of I/O in the sssd cache > > directory. After following > > https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-ipa-ad-trust-deployments/ > > the problems did nod reappear. > > > > Did you try all recommended steps or just few? > > > > Do you know which one was the most useful in your case? > > > > I think the biggest benefit came from moving the sssd cache into RAM. This shouldn't be the case with 1.14+ and wasn't in my testing. Did you remove the cache (really remove, not just expire with sss_cache) after you upgraded from 1.13 to 1.14? If yes, can you run some simple systemtap scripts? -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] SSSD setting memcache_timeout on ipa master
On 2017-04-10 12:16, Lukas Slebodnik wrote: [...] sssd_be consumed a lot of CPU and produced a lot of I/O in the sssd cache directory. After following https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-ipa-ad-trust-deployments/ the problems did nod reappear. Did you try all recommended steps or just few? Do you know which one was the most useful in your case? I think the biggest benefit came from moving the sssd cache into RAM. -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] SSSD setting memcache_timeout on ipa master
On (08/04/17 17:55), Ronald Wimmer wrote: >On 2017-04-08 12:53, Lukas Slebodnik wrote: >> On (04/04/17 09:41), Ronald Wimmer wrote: >> > On 2017-03-31 13:35, Lukas Slebodnik wrote: >> > > On (29/03/17 10:47), Ronald Wimmer wrote: >> > > > Hi, >> > > > >> > > > yesterday I suddenly was unable to use the webinterface of my ipa >> > > > master. SSH >> > > > login (with root user) did not work also. >> > > > >> > > > When I uncommented the setting "memcache_timeout = 600" in the sssd >> > > > config >> > > > file of the master everything seemed to work fine again. (my ipa setup >> > > > has a >> > > > trust to AD) >> > > > >> > > I doubt it had anything to do memcache_timeout. >> > > I would say that restart of sssd helped. But it difficult to say >> > > without log files. either sssd logs or at least /var/log/secure >> > > (journald for pam). >> > You were right. I uncommented the setting and the problem ocurred again. >> > >> Did you find anything suspicious in journald? >> Is sssd_be busy (or any other process)? >> high CPU, IO operations ... >> >> It would be good to know more details. Restarting sssd is not a solution. > >sssd_be consumed a lot of CPU and produced a lot of I/O in the sssd cache >directory. After following >https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-ipa-ad-trust-deployments/ >the problems did nod reappear. > Did you try all recommended steps or just few? Do you know which one was the most useful in your case? LS -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] SSSD setting memcache_timeout on ipa master
On Sat, Apr 08, 2017 at 05:55:00PM +0200, Ronald Wimmer wrote: > On 2017-04-08 12:53, Lukas Slebodnik wrote: > > On (04/04/17 09:41), Ronald Wimmer wrote: > > > On 2017-03-31 13:35, Lukas Slebodnik wrote: > > > > On (29/03/17 10:47), Ronald Wimmer wrote: > > > > > Hi, > > > > > > > > > > yesterday I suddenly was unable to use the webinterface of my ipa > > > > > master. SSH > > > > > login (with root user) did not work also. > > > > > > > > > > When I uncommented the setting "memcache_timeout = 600" in the sssd > > > > > config > > > > > file of the master everything seemed to work fine again. (my ipa > > > > > setup has a > > > > > trust to AD) > > > > > > > > > I doubt it had anything to do memcache_timeout. > > > > I would say that restart of sssd helped. But it difficult to say > > > > without log files. either sssd logs or at least /var/log/secure > > > > (journald for pam). > > > You were right. I uncommented the setting and the problem ocurred again. > > > > > Did you find anything suspicious in journald? > > Is sssd_be busy (or any other process)? > > high CPU, IO operations ... > > > > It would be good to know more details. Restarting sssd is not a solution. > > sssd_be consumed a lot of CPU and produced a lot of I/O in the sssd cache > directory. After following > https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-ipa-ad-trust-deployments/ > the problems did nod reappear. btw even after the performance improvements we did in 1.14 we an issue where even parsing the entries takes too long. What we did in 1.14 was that if the entries didn't change compared to what is already in the cache, then we skipped saving the full entry again just to bump the timestamp. Making the parsing faster is planned for the next version. (btw there was a bug where on upgrade, this new performance improvement didn't take effect for objects that were already cached. Removing the cache is a simple workaround and it's something we should fix soon in the code..) -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] SSSD setting memcache_timeout on ipa master
On 2017-04-08 12:53, Lukas Slebodnik wrote: On (04/04/17 09:41), Ronald Wimmer wrote: On 2017-03-31 13:35, Lukas Slebodnik wrote: On (29/03/17 10:47), Ronald Wimmer wrote: Hi, yesterday I suddenly was unable to use the webinterface of my ipa master. SSH login (with root user) did not work also. When I uncommented the setting "memcache_timeout = 600" in the sssd config file of the master everything seemed to work fine again. (my ipa setup has a trust to AD) I doubt it had anything to do memcache_timeout. I would say that restart of sssd helped. But it difficult to say without log files. either sssd logs or at least /var/log/secure (journald for pam). You were right. I uncommented the setting and the problem ocurred again. Did you find anything suspicious in journald? Is sssd_be busy (or any other process)? high CPU, IO operations ... It would be good to know more details. Restarting sssd is not a solution. sssd_be consumed a lot of CPU and produced a lot of I/O in the sssd cache directory. After following https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-ipa-ad-trust-deployments/ the problems did nod reappear. Regards, Ronald -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] SSSD setting memcache_timeout on ipa master
On (04/04/17 09:41), Ronald Wimmer wrote: >On 2017-03-31 13:35, Lukas Slebodnik wrote: >> On (29/03/17 10:47), Ronald Wimmer wrote: >> > Hi, >> > >> > yesterday I suddenly was unable to use the webinterface of my ipa master. >> > SSH >> > login (with root user) did not work also. >> > >> > When I uncommented the setting "memcache_timeout = 600" in the sssd config >> > file of the master everything seemed to work fine again. (my ipa setup has >> > a >> > trust to AD) >> > >> I doubt it had anything to do memcache_timeout. >> I would say that restart of sssd helped. But it difficult to say >> without log files. either sssd logs or at least /var/log/secure >> (journald for pam). >You were right. I uncommented the setting and the problem ocurred again. > Did you find anything suspicious in journald? Is sssd_be busy (or any other process)? high CPU, IO operations ... It would be good to know more details. Restarting sssd is not a solution. LS -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] SSSD setting memcache_timeout on ipa master
On 2017-03-31 13:35, Lukas Slebodnik wrote: On (29/03/17 10:47), Ronald Wimmer wrote: Hi, yesterday I suddenly was unable to use the webinterface of my ipa master. SSH login (with root user) did not work also. When I uncommented the setting "memcache_timeout = 600" in the sssd config file of the master everything seemed to work fine again. (my ipa setup has a trust to AD) I doubt it had anything to do memcache_timeout. I would say that restart of sssd helped. But it difficult to say without log files. either sssd logs or at least /var/log/secure (journald for pam). You were right. I uncommented the setting and the problem ocurred again. -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] SSSD setting memcache_timeout on ipa master
On (29/03/17 10:47), Ronald Wimmer wrote: >Hi, > >yesterday I suddenly was unable to use the webinterface of my ipa master. SSH >login (with root user) did not work also. > >When I uncommented the setting "memcache_timeout = 600" in the sssd config >file of the master everything seemed to work fine again. (my ipa setup has a >trust to AD) > I doubt it had anything to do memcache_timeout. I would say that restart of sssd helped. But it difficult to say without log files. either sssd logs or at least /var/log/secure (journald for pam). LS -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
