Re: [Freeipa-users] Process open FD table is full.
Thanks, I can't view the bug either but I'll pass it on in my support case. Erinn, in case it helps my support case # is 00646841. Oh and sorry for the mail formatting, Outlook at work... Regards Johan -Original Message- From: freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Rich Megginson Sent: den 2 november 2012 17:44 To: Erinn Looney-Triggs Cc: FreeIPAUsers Subject: Re: [Freeipa-users] Process open FD table is full. On 11/02/2012 10:41 AM, Erinn Looney-Triggs wrote: On 11/02/12 07:28, Rich Megginson wrote: On 11/02/2012 09:06 AM, Simo Sorce wrote: On Fri, 2012-11-02 at 08:38 +, Johan Sunnerstig wrote: Looks a lot like a problem I have as well. Check out the /proc/xxx/fd directory of the dirsrv process for your IPA realm, in my case it's full of dead pointers to /var/tmp/ldap_xxx where xxx will be the same on one IPA server(I have two in a multi-master setup). These don't clear out until I restart the dirsrv process, so eventually they'll fill up to the FD limit. For now I have a cron job performing a staggered IPA restart on the two servers and a case open with RH, but I haven't gotten any solution yet. This is also RHEL 6.3 by the way, though the problem appeared in 6.2 for me. This looks a memory leak in libkrb5 or dirsrv leaving around so krb context. Those files are replay caches. Rich, can you investigate the use of libkrb5 in dirsrv ? https://bugzilla.redhat.com/show_bug.cgi?id=825863 Simo. Oops missed this, though this is a private bug so I will have to take y'alls word for it being the thing. Sorry about that. It appears to be a problem with either krb5 or selinux, and there is a proposed fix for RHEL 6.4 I hate private bugs. I am going to open a RH support case, just in case that helps in any way. Yes, please. -Erinn ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Process open FD table is full.
Looks a lot like a problem I have as well. Check out the /proc/xxx/fd directory of the dirsrv process for your IPA realm, in my case it's full of dead pointers to /var/tmp/ldap_xxx where xxx will be the same on one IPA server(I have two in a multi-master setup). These don't clear out until I restart the dirsrv process, so eventually they'll fill up to the FD limit. For now I have a cron job performing a staggered IPA restart on the two servers and a case open with RH, but I haven't gotten any solution yet. This is also RHEL 6.3 by the way, though the problem appeared in 6.2 for me. Regards Johan -Original Message- From: freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Erinn Looney-Triggs Sent: den 1 november 2012 23:15 To: FreeIPAUsers Subject: [Freeipa-users] Process open FD table is full. Have any folks run into this: PR_Accept() failed, Netscape Portable Runtime error -5971 (Process open FD table is full.) From the dirsrv logs. It appears that this may have been what killed IPA in total on one server for me last night. I can't turn up anything via Google. After a restart of all the IPA processes everything started working again. I have looked into FD limits on the system and it doesn't seem like that is a likely cause. Found info here: https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Directory_Server/8.2/html/Performance_Tuning_Guide/system-tuning.html This is on a RHEL 6.3 system fully updated. Any ideas? -Erinn ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Process open FD table is full.
On Fri, 2012-11-02 at 08:38 +, Johan Sunnerstig wrote: Looks a lot like a problem I have as well. Check out the /proc/xxx/fd directory of the dirsrv process for your IPA realm, in my case it's full of dead pointers to /var/tmp/ldap_xxx where xxx will be the same on one IPA server(I have two in a multi-master setup). These don't clear out until I restart the dirsrv process, so eventually they'll fill up to the FD limit. For now I have a cron job performing a staggered IPA restart on the two servers and a case open with RH, but I haven't gotten any solution yet. This is also RHEL 6.3 by the way, though the problem appeared in 6.2 for me. This looks a memory leak in libkrb5 or dirsrv leaving around so krb context. Those files are replay caches. Rich, can you investigate the use of libkrb5 in dirsrv ? Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Process open FD table is full.
On 11/02/2012 09:06 AM, Simo Sorce wrote: On Fri, 2012-11-02 at 08:38 +, Johan Sunnerstig wrote: Looks a lot like a problem I have as well. Check out the /proc/xxx/fd directory of the dirsrv process for your IPA realm, in my case it's full of dead pointers to /var/tmp/ldap_xxx where xxx will be the same on one IPA server(I have two in a multi-master setup). These don't clear out until I restart the dirsrv process, so eventually they'll fill up to the FD limit. For now I have a cron job performing a staggered IPA restart on the two servers and a case open with RH, but I haven't gotten any solution yet. This is also RHEL 6.3 by the way, though the problem appeared in 6.2 for me. This looks a memory leak in libkrb5 or dirsrv leaving around so krb context. Those files are replay caches. Rich, can you investigate the use of libkrb5 in dirsrv ? https://bugzilla.redhat.com/show_bug.cgi?id=825863 Simo. ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Process open FD table is full.
On 11/02/12 00:38, Johan Sunnerstig wrote: Looks a lot like a problem I have as well. Check out the /proc/xxx/fd directory of the dirsrv process for your IPA realm, in my case it's full of dead pointers to /var/tmp/ldap_xxx where xxx will be the same on one IPA server(I have two in a multi-master setup). These don't clear out until I restart the dirsrv process, so eventually they'll fill up to the FD limit. For now I have a cron job performing a staggered IPA restart on the two servers and a case open with RH, but I haven't gotten any solution yet. This is also RHEL 6.3 by the way, though the problem appeared in 6.2 for me. Regards Johan -Original Message- From: freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Erinn Looney-Triggs Sent: den 1 november 2012 23:15 To: FreeIPAUsers Subject: [Freeipa-users] Process open FD table is full. Have any folks run into this: PR_Accept() failed, Netscape Portable Runtime error -5971 (Process open FD table is full.) From the dirsrv logs. It appears that this may have been what killed IPA in total on one server for me last night. I can't turn up anything via Google. After a restart of all the IPA processes everything started working again. I have looked into FD limits on the system and it doesn't seem like that is a likely cause. Found info here: https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Directory_Server/8.2/html/Performance_Tuning_Guide/system-tuning.html This is on a RHEL 6.3 system fully updated. Any ideas? -Erinn Spot on! That is exactly what is going on, my second ipa server just died this morning, checked /proc/ out before I restarted, full of dead links. Do they have a bugzilla open for your issue that I could attach to? Or could you give me your case number so I can get RH support to reference it and track it? Thanks again, -Erinn signature.asc Description: OpenPGP digital signature ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Process open FD table is full.
On 11/02/12 07:28, Rich Megginson wrote: On 11/02/2012 09:06 AM, Simo Sorce wrote: On Fri, 2012-11-02 at 08:38 +, Johan Sunnerstig wrote: Looks a lot like a problem I have as well. Check out the /proc/xxx/fd directory of the dirsrv process for your IPA realm, in my case it's full of dead pointers to /var/tmp/ldap_xxx where xxx will be the same on one IPA server(I have two in a multi-master setup). These don't clear out until I restart the dirsrv process, so eventually they'll fill up to the FD limit. For now I have a cron job performing a staggered IPA restart on the two servers and a case open with RH, but I haven't gotten any solution yet. This is also RHEL 6.3 by the way, though the problem appeared in 6.2 for me. This looks a memory leak in libkrb5 or dirsrv leaving around so krb context. Those files are replay caches. Rich, can you investigate the use of libkrb5 in dirsrv ? https://bugzilla.redhat.com/show_bug.cgi?id=825863 Simo. Oops missed this, though this is a private bug so I will have to take y'alls word for it being the thing. I hate private bugs. I am going to open a RH support case, just in case that helps in any way. -Erinn signature.asc Description: OpenPGP digital signature ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Process open FD table is full.
On 11/02/2012 10:41 AM, Erinn Looney-Triggs wrote: On 11/02/12 07:28, Rich Megginson wrote: On 11/02/2012 09:06 AM, Simo Sorce wrote: On Fri, 2012-11-02 at 08:38 +, Johan Sunnerstig wrote: Looks a lot like a problem I have as well. Check out the /proc/xxx/fd directory of the dirsrv process for your IPA realm, in my case it's full of dead pointers to /var/tmp/ldap_xxx where xxx will be the same on one IPA server(I have two in a multi-master setup). These don't clear out until I restart the dirsrv process, so eventually they'll fill up to the FD limit. For now I have a cron job performing a staggered IPA restart on the two servers and a case open with RH, but I haven't gotten any solution yet. This is also RHEL 6.3 by the way, though the problem appeared in 6.2 for me. This looks a memory leak in libkrb5 or dirsrv leaving around so krb context. Those files are replay caches. Rich, can you investigate the use of libkrb5 in dirsrv ? https://bugzilla.redhat.com/show_bug.cgi?id=825863 Simo. Oops missed this, though this is a private bug so I will have to take y'alls word for it being the thing. Sorry about that. It appears to be a problem with either krb5 or selinux, and there is a proposed fix for RHEL 6.4 I hate private bugs. I am going to open a RH support case, just in case that helps in any way. Yes, please. -Erinn ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
[Freeipa-users] Process open FD table is full.
Have any folks run into this: PR_Accept() failed, Netscape Portable Runtime error -5971 (Process open FD table is full.) From the dirsrv logs. It appears that this may have been what killed IPA in total on one server for me last night. I can't turn up anything via Google. After a restart of all the IPA processes everything started working again. I have looked into FD limits on the system and it doesn't seem like that is a likely cause. Found info here: https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Directory_Server/8.2/html/Performance_Tuning_Guide/system-tuning.html This is on a RHEL 6.3 system fully updated. Any ideas? -Erinn signature.asc Description: OpenPGP digital signature ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Process open FD table is full.
On 11/01/2012 04:15 PM, Erinn Looney-Triggs wrote: Have any folks run into this: PR_Accept() failed, Netscape Portable Runtime error -5971 (Process open FD table is full.) From the dirsrv logs. It appears that this may have been what killed IPA in total on one server for me last night. I can't turn up anything via Google. After a restart of all the IPA processes everything started working again. I have looked into FD limits on the system and it doesn't seem like that is a likely cause. Found info here: https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Directory_Server/8.2/html/Performance_Tuning_Guide/system-tuning.html This is on a RHEL 6.3 system fully updated. Any ideas? https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Directory_Server/9.0/html/Performance_Tuning_Guide/system-tuning.html#file-descriptors -Erinn ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Process open FD table is full.
On 11/01/2012 06:57 PM, Erinn Looney-Triggs wrote: On 11/01/12 16:47, Rich Megginson wrote: On 11/01/2012 04:15 PM, Erinn Looney-Triggs wrote: Have any folks run into this: PR_Accept() failed, Netscape Portable Runtime error -5971 (Process open FD table is full.) From the dirsrv logs. It appears that this may have been what killed IPA in total on one server for me last night. I can't turn up anything via Google. After a restart of all the IPA processes everything started working again. I have looked into FD limits on the system and it doesn't seem like that is a likely cause. Found info here: https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Directory_Server/8.2/html/Performance_Tuning_Guide/system-tuning.html This is on a RHEL 6.3 system fully updated. Any ideas? https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Directory_Server/9.0/html/Performance_Tuning_Guide/system-tuning.html#file-descriptors Yeah thanks but to my untrained eye it doesn't exactly fit: erinn@ipa ~ $ cat /proc/sys/fs/file-max 1199770 erinn@ipa ~ $ cat /etc/security/limits.conf dirsrv - nofile 8192 and, session required pam_limits.so in system-auth Did you look at section 3.3.2. Setting Directory Server File Descriptor Values This was all there before and yet I hit on that problem. Am I reading things incorrectly or not understanding them, I have a very small environment (~40 systems) and it doesn't seem like I should be having this problem. That does seem strange. -Erinn ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
