Re: [Freeipa-users] Log filling up a couple of times per day
- Original Message - > Hi Dimitri, > > I can do, we already analyzed it once. > > There is a loadbalancer checking the ldap protocol which seems to be > seen as fail. > > Is there a check I can perform on the ldap ports to see if the service > is available without generating the errors ? If you just need to check ldap i.e. that port 389 is listening: ldapsearch -xLLL -h ldaphost -s base -b "" "objectclass=*" 1.1 That will still log to the directory server access log. > > I will post a snippet later on if you have no clue. > > Thanks, > > Matt > > 2015-03-26 23:01 GMT+01:00 Dmitri Pal : > > On 03/26/2015 05:37 PM, Matt . wrote: > >> > >> Hi Guys, > >> > >> I'm facing every day a fast filling log of: > >> > >> /var/log/krb5kdc.log > >> /var/log/dirsrv/slapd-DOMAIN/access* > >> > >> I need to remove the files and restart ipa. The kerberos log is > >> filling up most, the access logs are quite fast on 100MB and a new one > >> is created. > >> > >> Now I do some LDAP loging/logout per day, servers that chedck if they > >> are registered for SSSD so that it logs it is normal, but I want to > >> get rid of it I guess. > >> > >> I'm throwing out I think about 6GB per day of logs, all loglevels are low. > >> > >> Any idea ? > >> > >> It's 3.x on CentOS 6.6 > >> > >> Any idea ? > >> > >> Thanks Matt > >> > > Do you have some services that are repeatedly doing something kerberos > > related and failing? > > I guess getting a snippet of the kerberos log would give some hint on what > > is it logging all the time. > > > > -- > > Thank you, > > Dmitri Pal > > > > Sr. Engineering Manager IdM portfolio > > Red Hat, Inc. > > > > -- > > Manage your subscription for the Freeipa-users mailing list: > > https://www.redhat.com/mailman/listinfo/freeipa-users > > Go to http://freeipa.org for more info on the project > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Unexpected IPA Crashes
- Original Message - > We have been using FreeIPA since two years and were more than happy. But > since two weeks we are facing unexpected crashed and can not really debug > the strange behaviours. The crashes are definitely not caused by connecting > a new system or changing the LDAP schema heavily. Following IPA is used: > > > > Name : ipa-server > > Arch : x86_64 > > Version : 3.3.3 > > Release : 28.0.1.el7.centos.3 > > Size : 4.1 M > > > > > I have followed the troubleshooting guide > http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#Troubleshooting > and activated logging and activated the core dumping. Do you see the string "segfault" in /var/log/messages or in journalctl? > Unfortunately, I > cannot provide you any core dump, because it is not created after the ipa > servers crashes. I'm sure the dirsrv is causing the problem, because when i > restart the 389, then ipa works fine for a while. Currently I have activated > the replication log level 8192. The error log shows no suspicious error or > any fatal error. Following 389* versions are used: > > > > > Installed Packages > > 389-ds-base.x86_64 1.3.3.1-15.el7_1 @/389-ds-base-1.3.3.1-15.el7_1.x86_64 > > 389-ds-base-debuginfo.x86_64 1.3.1.6-26.el7_0 @base-debuginfo > > > > 389-ds-base-libs.x86_64 1.3.3.1-15.el7_1 > > > > > Can you please provide some hint how I can debug this problem in more detail. > Btw, the ipa infrastructure consist of one master and one replica. The > server was also crashing, when the replica server was turned off. Do you > thing an upgrade would solve the problem as the last resort? > > > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Where do I change the nsslapd-accesslog-level?
- Original Message - > On Tue, May 13, 2014 at 2:26 PM, Richard Megginson > wrote: > > > - Original Message - > > > On Tue, May 13, 2014 at 1:28 PM, Richard Megginson > > > wrote: > > > > > > > - Original Message - > > > > > I am using FreeIPA 3.0.0 on RHEL 6 (ipa-server-3.0.0-37.el6.x86_64). > > > > > > > > > > Where do I change the verbosity of access logging? > > > > > > > > > > > > Why do you need to change the verbosity of access logging? Do you mean > > > > error logging? If so, see http://port389.org/wiki/FAQ#Troubleshooting > > > > > > > > > > I do mean access logging. I want to change it because it's too verbose > > :-) > > > . It's causing high load / iowait on the server. > > > > There isn't a way to change the access log level to make it less verbose. > > You can turn it off completely nsslapd-accesslog-enabled: off > > > > Sorry, you've confused me. Are you saying that "nsslapd-accesslog-level: 4" > is just as verbose as "nsslapd-accesslog-level: 256"? Yes. > Or that there is > literally no way to change the level despite the fact that there are levels? Yes, you can change the level. You can make it much more verbose than it is already. I don't think this is what you want. https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Configuration_Command_and_File_Reference/Core_Server_Configuration_Reference.html#cnconfig-nsslapd_accesslog_level So you could, for example, change the level to 4, and log internal operations. 1) That may be much more verbose than the default of 256 2) That may not be particularly useful to you. If the purpose of changing the access logging level is to reduce the I/O, then no, there is no level which will reduce the verbosity but still give you some sort of useful data in the access log. If you want to reduce the verbosity, but still have some sort of useful information, then you'll have to use the named pipe log script to filter out only those events which are useful to you. > > Cheers > > > > > Note that the access log is buffered, specifically to reduce the I/O load. > > If that buffered load is _still_ too high, then you might want to > > investigate replacing the access log file with a named pipe, then writing a > > small bit of python code to filter out only the events you are interested > > in. See > > https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/using-named-pipe.html > > > > > > > > Based on the link you sent if I crafted an ldif like: > > > > > > dn: cn=config > > > changetype: modify > > > replace: nsslapd-accesslog-level > > > nsslapd-accesslog-level: 4 > > > > > > that would presumably get me what I want. > > > > I don't think so. The error log levels are completely different than the > > access log levels, in that there are no access log levels. > > > > > > > > Does it require a dirsrv restart? > > > > No, but . . . > > > > > > > > Please advise. > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > > > > This doc: > > > > > > > > > > > > > > > > http://docs.fedoraproject.org/en-US/Fedora/18/html/FreeIPA_Guide/server-config.html > > > > > > > > > > discusses turning on global debugging but doesn't help me. The same > > doc > > > > links > > > > > to: > > > > > > > > > > > > > > > > https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/8.2/html/Configuration_and_Command-Line_Tool_Reference/logs-reference.html > > > > > > > > > > which tells me that I need to change the nsslapd-accesslog-level but > > the > > > > link > > > > > on that page is a 404. > > > > > > > > > > So what do I need to do to change the level? I would assume that > > setting > > > > the > > > > > level to 4 would be indicated if 256 is too verbose but can someone > > > > please > > > > > confirm? > > > > > > > > > > I tried looking in the Configuration tab of the admin GUI but I get > > > > thrown: > > > > > > > > > > IPA Error 4204 > > > > > > > > > > limits exceeded for this query > > > > > > > > > > Not sure what's going on there, might be symptomatic of the high > > load the > > > > > server is under due to iowait perhaps... > > > > > > > > > > Thanks! > > > > > > > > > > ___ > > > > > Freeipa-users mailing list > > > > > Freeipa-users@redhat.com > > > > > https://www.redhat.com/mailman/listinfo/freeipa-users > > > > > > > > > > ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Where do I change the nsslapd-accesslog-level?
- Original Message - > On Tue, May 13, 2014 at 1:28 PM, Richard Megginson > wrote: > > > - Original Message - > > > I am using FreeIPA 3.0.0 on RHEL 6 (ipa-server-3.0.0-37.el6.x86_64). > > > > > > Where do I change the verbosity of access logging? > > > > > > Why do you need to change the verbosity of access logging? Do you mean > > error logging? If so, see http://port389.org/wiki/FAQ#Troubleshooting > > > > I do mean access logging. I want to change it because it's too verbose :-) > . It's causing high load / iowait on the server. There isn't a way to change the access log level to make it less verbose. You can turn it off completely nsslapd-accesslog-enabled: off Note that the access log is buffered, specifically to reduce the I/O load. If that buffered load is _still_ too high, then you might want to investigate replacing the access log file with a named pipe, then writing a small bit of python code to filter out only the events you are interested in. See https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/using-named-pipe.html > > Based on the link you sent if I crafted an ldif like: > > dn: cn=config > changetype: modify > replace: nsslapd-accesslog-level > nsslapd-accesslog-level: 4 > > that would presumably get me what I want. I don't think so. The error log levels are completely different than the access log levels, in that there are no access log levels. > > Does it require a dirsrv restart? No, but . . . > > Please advise. > > Thanks! > > > > > > > > > > > This doc: > > > > > > > > http://docs.fedoraproject.org/en-US/Fedora/18/html/FreeIPA_Guide/server-config.html > > > > > > discusses turning on global debugging but doesn't help me. The same doc > > links > > > to: > > > > > > > > https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/8.2/html/Configuration_and_Command-Line_Tool_Reference/logs-reference.html > > > > > > which tells me that I need to change the nsslapd-accesslog-level but the > > link > > > on that page is a 404. > > > > > > So what do I need to do to change the level? I would assume that setting > > the > > > level to 4 would be indicated if 256 is too verbose but can someone > > please > > > confirm? > > > > > > I tried looking in the Configuration tab of the admin GUI but I get > > thrown: > > > > > > IPA Error 4204 > > > > > > limits exceeded for this query > > > > > > Not sure what's going on there, might be symptomatic of the high load the > > > server is under due to iowait perhaps... > > > > > > Thanks! > > > > > > ___ > > > Freeipa-users mailing list > > > Freeipa-users@redhat.com > > > https://www.redhat.com/mailman/listinfo/freeipa-users > > > ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Where do I change the nsslapd-accesslog-level?
- Original Message - > I am using FreeIPA 3.0.0 on RHEL 6 (ipa-server-3.0.0-37.el6.x86_64). > > Where do I change the verbosity of access logging? Why do you need to change the verbosity of access logging? Do you mean error logging? If so, see http://port389.org/wiki/FAQ#Troubleshooting > > This doc: > > http://docs.fedoraproject.org/en-US/Fedora/18/html/FreeIPA_Guide/server-config.html > > discusses turning on global debugging but doesn't help me. The same doc links > to: > > https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/8.2/html/Configuration_and_Command-Line_Tool_Reference/logs-reference.html > > which tells me that I need to change the nsslapd-accesslog-level but the link > on that page is a 404. > > So what do I need to do to change the level? I would assume that setting the > level to 4 would be indicated if 256 is too verbose but can someone please > confirm? > > I tried looking in the Configuration tab of the admin GUI but I get thrown: > > IPA Error 4204 > > limits exceeded for this query > > Not sure what's going on there, might be symptomatic of the high load the > server is under due to iowait perhaps... > > Thanks! > > ___ > Freeipa-users mailing list > Freeipa-users@redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 389-ds memory usage
- Original Message - > On 06/05/2012 05:55 PM, Richard Megginson wrote: > > - Original Message - > >> On Mon, April 23, 2012 20:38, Rich Megginson wrote: > >> > >>> Ok. The current theory is that the memory growth is caused by > >>> the > >>> churn > >>> of entries being added to and removed from the entry cache. It's > >>> not yet known why this growth is > >>> seen. It could be just that the memory is getting fragmented, or > >>> there is a real yet undetected > >>> memory leak. That's why entry cache sizing and monitoring is very > >>> important, to see > >>> if you are churning entries in/out of the cache, and if that is > >>> correlated with the memory growth. > >>> > >> > >> This memory issue is still occuring in the production environment > >> after increasing the max entry > >> cache size to 256MB, and it is impacting performance. See below > >> for > >> an output of the memory usage > >> and the current size and hitratio of the cache on the 3 production > >> IPA servers. > >> > >> How do you suggest moving forward to troubleshoot this issue? > > > > Are you seeing https://fedorahosted.org/389/ticket/386 ? > > > > > > I don't think so. The cache numbers described in this ticket is much > higher than my cache numbers. > > I've set the maxentrycachesize to 256MB, and each IPA server has 8GB > of > memory. There should not have been a consumption of more than 1,8GB > with > the 7 * cachememsize as described in the ticket. > > Beside I don't know where the heavy modify levels would come from. > About > 100 new users we're moved to IPA last week, all having their > passwords > changed, the accounts themselves already existed. But is that > considered > a heavy LDAP modify level? I don't think it matters whether or not the modify level is heavy. If there is a memory leak, a heavy modify level will make it more apparent more quickly. > > There seem to be a memory leak somewhere. How would you recommend > moving > forward? If you think this leak is a completely different issue than ticket 386, please open another ticket. > > > Regards, > Siggi > > > ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 389-ds memory usage
- Original Message - > On Mon, April 23, 2012 20:38, Rich Megginson wrote: > > > Ok. The current theory is that the memory growth is caused by the > > churn > > of entries being added to and removed from the entry cache. It's > > not yet known why this growth is > > seen. It could be just that the memory is getting fragmented, or > > there is a real yet undetected > > memory leak. That's why entry cache sizing and monitoring is very > > important, to see > > if you are churning entries in/out of the cache, and if that is > > correlated with the memory growth. > > > > > This memory issue is still occuring in the production environment > after increasing the max entry > cache size to 256MB, and it is impacting performance. See below for > an output of the memory usage > and the current size and hitratio of the cache on the 3 production > IPA servers. > > How do you suggest moving forward to troubleshoot this issue? Are you seeing https://fedorahosted.org/389/ticket/386 ? > > > > > > Mem: 8050880k total, 7893612k used, 157268k free, 1924k > buffers > Swap: 14811120k total, 3738640k used, 11072480k free,38032k > cached > > dn: cn=monitor,cn=userRoot,cn=ldbm database,cn=plugins,cn=config > entrycachehitratio: 97 > maxentrycachesize: 268435456 > currententrycachesize: 14047758 > currententrycachecount: 3062 > > > > Mem: 8059224k total, 7907904k used, 151320k free, 9060k > buffers > Swap: 14811120k total, 2507796k used, 12303324k free,58104k > cached > > dn: cn=monitor,cn=userRoot,cn=ldbm database,cn=plugins,cn=config > entrycachehitratio: 99 > maxentrycachesize: 268435456 > currententrycachesize: 13963883 > currententrycachecount: 3062 > > > > Mem: 8062240k total, 7932268k used, 129972k free, 864k > buffers > Swap: 2097144k total, 2097144k used,0k free,16788k > cached > > dn: cn=monitor,cn=userRoot,cn=ldbm database,cn=plugins,cn=config > entrycachehitratio: 99 > maxentrycachesize: 268435456 > currententrycachesize: 13809438 > currententrycachecount: 3066 > > > Rgds, > Siggi > > > ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] General status of my FreeIPA servers - is there a method for cleaning them?
- Original Message -
> On Tue, Apr 17, 2012 at 09:26, Rich Megginson
> wrote:
> > On 04/17/2012 07:26 AM, Dan Scott wrote:
> >>
> >> On Fri, Apr 13, 2012 at 17:44, Rich Megginson
> >> wrote:
> >>>
> >>> On 04/13/2012 03:40 PM, Dan Scott wrote:
>
> I cleaned up all the "ruv_compare_ruv: RUV [changelog max RUV]
> does
> not contain element" errors in the logs for each of fileservers
> 1, 2
> and 3. The ldapsearch for
>
>
> '(&(nsuniqueid=---)(objectclass=nstombstone))'
> is still showing entries though. Is that OK?
> >>>
> >>>
> >>> The entry should exist, but the deleted servers should not be
> >>> present in
> >>> the
> >>> nsds50ruv attribute.
> >>
> >> OK, so it's safe to delete replica entries which have
> >> ldap://fileserver4.ecg.mit.edu:389 (fileserver4 is not currently a
> >> replica) but not for the other servers?
> >
> > Yes. Following the CLEANRUV procedure:
> > http://port389.org/wiki/Howto:CLEANRUV
>
> Thanks. I think I'm getting there - removed the tombstones from the
> main directory and the PKI-IPA directory (only one server so far
> though). I still have a few strange entries though:
>
> [root@fileserver1 ~]# ldapsearch -xLLL -D "cn=directory manager" -W
> -b
> dc=ecg,dc=mit,dc=edu
> '(&(nsuniqueid=---)(objectclass=nstombstone))'
> Enter LDAP Password:
> dn:
> nsuniqueid=---,dc=ecg,dc=mit,dc=edu
> objectClass: top
> objectClass: nsTombstone
> objectClass: extensibleobject
> nsds50ruv: {replicageneration} 4e7b746e0004
> nsds50ruv: {replica 6 ldap://fileserver1.ecg.mit.edu:389}
> 4f50e685001d0006
> 4f8d787400020006
> nsds50ruv: {replica 43 ldap://fileserver2.ecg.mit.edu:389}
> 4f88cf450001002b000
> 0 4f8d7814002b
> nsds50ruv: {replica 5 ldap://fileserver3.ecg.mit.edu:389}
> 4f5047ad001d0005
> 4f8d77c30005
> nsds50ruv: {replica 4 ldap://fileserver3.ecg.mit.edu:389}
> nsds50ruv: {replica 9 ldap://fileserver3.ecg.mit.edu:389}
> nsds50ruv: {replica 8 ldap://fileserver3.ecg.mit.edu:389}
> 4f7363d2001d0008
> 4f73640200070008
> dc: ecg
> nsruvReplicaLastModified: {replica 6
> ldap://fileserver1.ecg.mit.edu:389} 4f8d7
> 806
> nsruvReplicaLastModified: {replica 43
> ldap://fileserver2.ecg.mit.edu:389} 4f8d
> 77a6
> nsruvReplicaLastModified: {replica 5
> ldap://fileserver3.ecg.mit.edu:389} 4f8d7
> 756
> nsruvReplicaLastModified: {replica 4
> ldap://fileserver3.ecg.mit.edu:389} 0
> 000
> nsruvReplicaLastModified: {replica 9
> ldap://fileserver3.ecg.mit.edu:389} 0
> 000
> nsruvReplicaLastModified: {replica 8
> ldap://fileserver3.ecg.mit.edu:389} 0
> 000
>
> Is it safe to run CLEANRUV on IDs 4 and 9? That still leaves me with
> 2
> entries for fileserver3. How do I know which one to delete?
Whichever one is the one currently in use.
ldapsearch -xLLL -h fileserver3 -D "cn=directory manager" -W -b cn=config
cn=replica
What is the replica ID? That is the one that is currently in use. You should
be able to safely delete the others.
>
> On my PKI-IPA server, the CLEANRUV task doesn't seem to work. It
> keeps
> re-adding entries after I remove them. I have 3 entries for my
> non-existent fileserver4 - They disappear when I remove them, but
> they
> come back after a few minutes.
Right, because they are being replicated from another master. You will need to
run the CLEANRUV on all masters at the same time.
>
> Thanks,
>
> Dan
>
___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users
