i changed the loggin level to 4 . Modifying nsslapd-accesslog-level But, the hang is still there. though I dont see the sigfault now
On Tue, Aug 23, 2016 at 9:02 PM, Rakesh Rajasekharan < rakesh.rajasekha...@gmail.com> wrote: > My disk was getting filled too fast > > logs under /var/log/dirsrv was coming around 5 gb quickly filling up > > Is there a way to make the logging less verbose > > > > On Tue, Aug 23, 2016 at 6:41 PM, Petr Spacek <pspa...@redhat.com> wrote: > >> On 23.8.2016 15:07, Rakesh Rajasekharan wrote: >> > I was able to fix that may be temporarily... when i checked the >> network.. >> > there was another process that was running and consuming a lot of >> network ( >> > i have no idea who did that. I need to seriously start restricting >> people >> > access to this machine ) >> > >> > after killing that perfomance improved drastically >> > >> > But now, suddenly I started experiencing the same hang. >> > >> > This time , I gert the following error when checked dmesg >> > >> > [ 301.236976] ns-slapd[3124]: segfault at 0 ip 00007f1de416951c sp >> > 00007f1dee1dba70 error 4 in libcos-plugin.so[7f1de4166000+b000] >> > [ 1116.248431] TCP: request_sock_TCP: Possible SYN flooding on port 88. >> > Sending cookies. Check SNMP counters. >> > [11831.397037] ns-slapd[22550]: segfault at 0 ip 00007f533d82251c sp >> > 00007f5347894a70 error 4 in libcos-plugin.so[7f533d81f000+b000] >> > [11832.727989] ns-slapd[22606]: segfault at 0 ip 00007f6231eb951c sp >> > 00007f623bf2ba70 error 4 in libcos-plugin.so[7f6231eb6000+b00 >> >> Okay, this one is serious. The LDAP server crashed. >> >> 1. Make sure all your packages are up-to-date. >> >> Please see >> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html# >> debugging-crashes >> for further instructions how to debug this. >> >> Petr^2 Spacek >> >> > >> > and in /var/log/dirsrv/example-com/errors >> > >> > [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: >> could >> > not delete change record 3291138 (rc: 32) >> > [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: >> could >> > not delete change record 3291139 (rc: 32) >> > [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: >> could >> > not delete change record 3291140 (rc: 32) >> > [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: >> could >> > not delete change record 3291141 (rc: 32) >> > [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: >> could >> > not delete change record 3291142 (rc: 32) >> > [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: >> could >> > not delete change record 3291143 (rc: 32) >> > [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: >> could >> > not delete change record 3291144 (rc: 32) >> > [23/Aug/2016:12:49:36 +0000] DSRetroclPlugin - delete_changerecord: >> could >> > not delete change record 3291145 (rc: 32) >> > [23/Aug/2016:12:49:50 +0000] - Retry count exceeded in delete >> > [23/Aug/2016:12:49:50 +0000] DSRetroclPlugin - delete_changerecord: >> could >> > not delete change record 3292734 (rc: 51) >> > >> > >> > Can i do something about this error.. I treid to restart ipa a couple >> of >> > time but that did not help >> > >> > Thanks >> > Rakesh >> > >> > On Mon, Aug 22, 2016 at 2:27 PM, Petr Spacek <pspa...@redhat.com> >> wrote: >> > >> >> On 19.8.2016 19:32, Rakesh Rajasekharan wrote: >> >>> I am running my set up on AWS cloud, and entropy is low at around 180 >> . >> >>> >> >>> I plan to increase it bu installing haveged . But, would low entropy >> by >> >> any >> >>> chance cause this issue of intermittent hang . >> >>> Also, the hang is mostly observed when registering around 20 clients >> >>> together >> >> >> >> Possibly, I'm not sure. If you want to dig into this, I would do this: >> >> 1. look what process hangs on client (using pstree command or so) >> >> $ pstree >> >> >> >> 2. look to what server and port is the hanging client connected to >> >> $ lsof -p <PID of the hanging process> >> >> >> >> 3. jump to server and see what process is bound to the target port >> >> $ netstat -pn >> >> >> >> 4. see where the process if hanging >> >> $ strace -p <PID of the hanging process> >> >> >> >> I hope it helps. >> >> >> >> Petr^2 Spacek >> >> >> >>> On Fri, Aug 19, 2016 at 7:24 PM, Rakesh Rajasekharan < >> >>> rakesh.rajasekha...@gmail.com> wrote: >> >>> >> >>>> yes there seems to be something thats worrying.. I have faced this >> today >> >>>> as well. >> >>>> There are few hosts around 280 odd left and when i try adding them to >> >> IPA >> >>>> , the slowness begins.. >> >>>> >> >>>> all the ipa commands like ipa user-find.. etc becomes very slow in >> >>>> responding. >> >>>> >> >>>> the SYNC_RECV are not many though just around 80-90 and today that >> was >> >>>> around 20 only >> >>>> >> >>>> >> >>>> I have for now increased tcp_max_syn_backlog to 5000. >> >>>> For now the slowness seems to have gone.. but I will do a try adding >> the >> >>>> clients again tomorrow and see how it goes >> >>>> >> >>>> Thanks >> >>>> Rakesh >> >>>> >> >>>> The issues >> >>>> >> >>>> On Fri, Aug 19, 2016 at 12:58 PM, Petr Spacek <pspa...@redhat.com> >> >> wrote: >> >>>> >> >>>>> On 18.8.2016 17:23, Rakesh Rajasekharan wrote: >> >>>>>> Hi >> >>>>>> >> >>>>>> I am migrating to freeipa from openldap and have around 4000 >> clients >> >>>>>> >> >>>>>> I had openned a another thread on that, but chose to start a new >> one >> >>>>> here >> >>>>>> as its a separate issue >> >>>>>> >> >>>>>> I was able to change the nssslapd-maxdescriptors adding an ldif >> file >> >>>>>> >> >>>>>> cat nsslapd-modify.ldif >> >>>>>> dn: cn=config >> >>>>>> changetype: modify >> >>>>>> replace: nsslapd-maxdescriptors >> >>>>>> nsslapd-maxdescriptors: 17000 >> >>>>>> >> >>>>>> and running the ldapmodify command >> >>>>>> >> >>>>>> I have now started moving clients running an openldap to Freeipa >> and >> >>>>> have >> >>>>>> today moved close to 2000 clients >> >>>>>> >> >>>>>> However, I have noticed that IPA hangs intermittently. >> >>>>>> >> >>>>>> running a kinit admin returns the below error >> >>>>>> kinit: Generic error (see e-text) while getting initial credentials >> >>>>>> >> >>>>>> from the /var/log/messages, I see this entry >> >>>>>> >> >>>>>> prod-ipa-master-int kernel: [104090.315801] TCP: request_sock_TCP: >> >>>>>> Possible SYN flooding on port 88. Sending cookies. Check SNMP >> >> counters. >> >>>>> >> >>>>> I would be worried about this message. Maybe kernel/firewall is >> doing >> >>>>> something fishy behind your back and blocking some connections or >> so. >> >>>>> >> >>>>> Petr^2 Spacek >> >>>>> >> >>>>> >> >>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Started Session >> 4885 >> >> of >> >>>>>> user root. >> >>>>>> Aug 18 13:00:01 prod-ipa-master-int systemd[1]: Starting Session >> 4885 >> >> of >> >>>>>> user root. >> >>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Started Session >> 4886 >> >> of >> >>>>>> user root. >> >>>>>> Aug 18 13:01:01 prod-ipa-master-int systemd[1]: Starting Session >> 4886 >> >> of >> >>>>>> user root. >> >>>>>> Aug 18 13:02:40 prod-ipa-master-int python[28984]: ansible-command >> >>>>> Invoked >> >>>>>> with creates=None executable=None shell=True args= removes=None >> >>>>> warn=True >> >>>>>> chdir=None >> >>>>>> Aug 18 13:04:37 prod-ipa-master-int sssd_be: GSSAPI Error: >> Unspecified >> >>>>> GSS >> >>>>>> failure. Minor code may provide more information (KDC returned >> error >> >>>>>> string: PROCESS_TGS) >> >>>>>> >> >>>>>> Could it be possible that its due to the initial load of adding the >> >>>>> clients >> >>>>>> or is there something else that I need to take care of. >> >> >> > >> >> >> -- >> Petr Spacek @ Red Hat >> > >
-- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
