Re: [Samba] Too many open files, Bug 3342
On Sat, Feb 04, 2006 at 04:31:50PM -0800, David Highley wrote: David == David Highley [EMAIL PROTECTED] writes: David We submitted bug 3342 on Dec. 20th, Too many open files. It David has not moved from the new state. So lets try here. David We have Dell Quad Xeon server that we end up rebooting David about every 2-3 weeks when the number of open samba files David sky rockets. The system is a ClearCase server system. We David have been monitoring this system for a few months now and David are not able to pin down a cause to this issue. The system David keeps a pretty constant number of samba open files until David what ever triggers the issue and then the number of file David opens climbs at a steep exponential rate. David RedHat Advanced Server 3 update 3 Linux quinault David 2.4.21-27.0.2.ELsmp #1 SMP Wed Jan 12 23:35:44 EST 2005 David i686 i686 i386 GNU/Linux Samba version=3.0.6-2E I see you are using Samba 3.0.6. There were a lot of bug fixes after that release. I would start with version 3.0.14a. I've been using that version with ClearCase (although with Solaris 8). I would not use 3.0.20anything or even the latest 3.0.21b. It might be useful to see your smb.conf as well as your client MVFS settings. However, ClearCase is not within the scope of this mailing list. David Clients for the most part are Windows XP and Windows 2003 David Enterprise servers. David Still looking clues as to what might be a cause for this David behavior. Our systems people have monitored everything they David can think of to see if we can find a correlation but so far David nothing has turned up. David It is a large, 200+ development environment with NFS and David NAS CX400, storage units on the servers. It is a heavy file David access, large builds parallel builds on multiple David architectures. -- Eric M. Boehm /\ ASCII Ribbon Campaign [EMAIL PROTECTED] \ / No HTML or RTF in mail X No proprietary word-processing Respect Open Standards / \ files in mail -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/listinfo/samba
Re: [Samba] Too many open files, Bug 3342
Eric Boehm wrote: On Sat, Feb 04, 2006 at 04:31:50PM -0800, David Highley wrote: David == David Highley [EMAIL PROTECTED] writes: David We submitted bug 3342 on Dec. 20th, Too many open files. It David has not moved from the new state. So lets try here. David We have Dell Quad Xeon server that we end up rebooting David about every 2-3 weeks when the number of open samba files David sky rockets. The system is a ClearCase server system. We David have been monitoring this system for a few months now and David are not able to pin down a cause to this issue. The system David keeps a pretty constant number of samba open files until David what ever triggers the issue and then the number of file David opens climbs at a steep exponential rate. David RedHat Advanced Server 3 update 3 Linux quinault David 2.4.21-27.0.2.ELsmp #1 SMP Wed Jan 12 23:35:44 EST 2005 David i686 i686 i386 GNU/Linux Samba version=3.0.6-2E I see you are using Samba 3.0.6. There were a lot of bug fixes after that release. I would start with version 3.0.14a. I've been using that version with ClearCase (although with Solaris 8). I would not use 3.0.20anything or even the latest 3.0.21b. It might be useful to see your smb.conf as well as your client MVFS settings. However, ClearCase is not within the scope of this mailing list. We have tuned the client MVFS setting down to 200 for Maximum number of mnodes to kee on the VOB free list and Maximum number of mnodes to keep for cleartext free list. Were not looking for ClearCase support, we believe that this issue maybe a symptom of some other issue but so far we have not been able to make the connection. Because of our operating environment, security, it is very hard to make changes to platform configurations. What information might be available about the samba connections that might lead us to determine what triggers this event? David Clients for the most part are Windows XP and Windows 2003 David Enterprise servers. David Still looking clues as to what might be a cause for this David behavior. Our systems people have monitored everything they David can think of to see if we can find a correlation but so far David nothing has turned up. David It is a large, 200+ development environment with NFS and David NAS CX400, storage units on the servers. It is a heavy file David access, large builds parallel builds on multiple David architectures. -- Eric M. Boehm /\ ASCII Ribbon Campaign [EMAIL PROTECTED] \ / No HTML or RTF in mail X No proprietary word-processing Respect Open Standards / \ files in mail -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/listinfo/samba
Re: [Samba] Too many open files, Bug 3342
On Mon, Feb 06, 2006 at 08:39:34AM -0800, David Highley wrote: David == David Highley [EMAIL PROTECTED] writes: David RedHat Advanced Server 3 update 3 Linux quinault David 2.4.21-27.0.2.ELsmp #1 SMP Wed Jan 12 23:35:44 EST 2005 David i686 i686 i386 GNU/Linux Samba version=3.0.6-2E Eric I see you are using Samba 3.0.6. There were a lot of bug Eric fixes after that release. I would start with version Eric 3.0.14a. I've been using that version with ClearCase Eric (although with Solaris 8). Eric I would not use 3.0.20anything or even the latest 3.0.21b. Eric It might be useful to see your smb.conf as well as your Eric client MVFS settings. However, ClearCase is not within the Eric scope of this mailing list. It might be helpful to see how you've defined the share. Here's a template that I've recommended for ClearCase. # # Duplicate the following block for each share you wish to # define. ClearCase related storage areas must have oplocks = no. For # other applications, you may wish to set oplocks = yes. # [share-name] comment = your-comment-here path= path-to-storage # admin users = # hosts allow = @netgroup, host, ip_address # valid users = @netgroup, userid create mask = 0775 directory mask = 0775 # If guest ok = yes, then anyone will be able to connect # without authentication. If that is the desired configuration, # you may want to make the share read-only (i.e., writeable = no) guest ok= no map archive = no oplocks = no writeable = yes David We have tuned the client MVFS setting down to 200 for David Maximum number of mnodes to kee on the VOB free list and David Maximum number of mnodes to keep for cleartext free list. You might be surprised but these limits are not always observed. I have seen clients open more than 200 files even when MVFS settings are tuned down. Is any of this Java code? That's where I've seen this problem occur (exceeding the mnodes setting). FWIW, we don't tune down MVFS. We have the scaling factor set to 4 and let mnodes be 4500 (VOB free) and 1800 (cleartext free). We've been running this way for years at multiple locations around the globe. We have anywhere from 20-600+ clients at a location. David Were not looking for ClearCase support, we believe that David this issue maybe a symptom of some other issue but so far David we have not been able to make the connection. Because of David our operating environment, security, it is very hard to David make changes to platform configurations. I understand your constraints but given that your environment is having problems, isn't that enough to warrant upgrading? I've looked at the release notes since 3.0.6 and there are lots of fixes for memory leaks and problems in the code that handles files. I think 3.0.14a is your next best bet. 3.0.6 is about 1.5 years old. I would also expect that the Samba team isn't going to be interested in pursuing the bug unless you can reproduce it against the latest version (3.0.21b). Your security setting might be influencing the number of open connections you have (not necessarily the number of file opens). Do you have 'security = domain' or 'security = server' in your smb.conf? David What information might be available about the samba David connections that might lead us to determine what triggers David this event? You might look at the Samba client logs for any strange error messages. Does the problem occur at a predictable time -- such as after a large build? or a particular build? You said that you were using NAS. If that's the case, why aren't you using CIFS through the NAS? You might not even need Samba. Lastly, what are /proc/sys/fs/file-max /proc/sys/fs/file-nr set to? -- Eric M. Boehm /\ ASCII Ribbon Campaign [EMAIL PROTECTED] \ / No HTML or RTF in mail X No proprietary word-processing Respect Open Standards / \ files in mail -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/listinfo/samba
Re: [Samba] Too many open files, Bug 3342
Eric Boehm wrote: On Mon, Feb 06, 2006 at 08:39:34AM -0800, David Highley wrote: David == David Highley [EMAIL PROTECTED] writes: David RedHat Advanced Server 3 update 3 Linux quinault David 2.4.21-27.0.2.ELsmp #1 SMP Wed Jan 12 23:35:44 EST 2005 David i686 i686 i386 GNU/Linux Samba version=3.0.6-2E Eric I see you are using Samba 3.0.6. There were a lot of bug Eric fixes after that release. I would start with version Eric 3.0.14a. I've been using that version with ClearCase Eric (although with Solaris 8). Eric I would not use 3.0.20anything or even the latest 3.0.21b. Eric It might be useful to see your smb.conf as well as your Eric client MVFS settings. However, ClearCase is not within the Eric scope of this mailing list. It might be helpful to see how you've defined the share. [global] workgroup = NW server string = Samba Server security = DOMAIN password server = adc-nw-03 adc-nw-01 log file = /var/log/samba/%m.log max log size = 0 socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192 os level = 0 preferred master = No local master = No domain master = No dns proxy = No ldap ssl = no create mask = 0664 cups options = raw use sendfile = no [export] comment = ClearCase Vobs and Views path = /export valid users = +nw_usawacs read only = No oplocks = No level2 oplocks = No Here's a template that I've recommended for ClearCase. # # Duplicate the following block for each share you wish to # define. ClearCase related storage areas must have oplocks = no. For # other applications, you may wish to set oplocks = yes. # [share-name] comment = your-comment-here path= path-to-storage # admin users = # hosts allow = @netgroup, host, ip_address # valid users = @netgroup, userid create mask = 0775 directory mask = 0775 # If guest ok = yes, then anyone will be able to connect # without authentication. If that is the desired configuration, # you may want to make the share read-only (i.e., writeable = no) guest ok= no map archive = no oplocks = no writeable = yes David We have tuned the client MVFS setting down to 200 for David Maximum number of mnodes to kee on the VOB free list and David Maximum number of mnodes to keep for cleartext free list. You might be surprised but these limits are not always observed. I have seen clients open more than 200 files even when MVFS settings are tuned down. Is any of this Java code? That's where I've seen this problem occur (exceeding the mnodes setting). We do all the Java builds on the Linux platform as it is much faster. We push our C++ builds by running parallel builds and we build all platforms in the same view so we generate a pretty high file system and CPU load. Java builds are run serial of course as ANT is not compatible with parallel building. We also switched to GNU make as clearmake does not throttle across sub make processes when doing parallel builds. Some of our builds on the PCs will have over 1800 threads, 600-700 processes, and 20,000+ file handles open. Our build process has been monitored and IBM has been in and reviewed the ClearCase aspects. FWIW, we don't tune down MVFS. We have the scaling factor set to 4 and let mnodes be 4500 (VOB free) and 1800 (cleartext free). We've been running this way for years at multiple locations around the globe. We have anywhere from 20-600+ clients at a location. Scaling factor is 1. We also remotely monitor the client settings. David Were not looking for ClearCase support, we believe that David this issue maybe a symptom of some other issue but so far David we have not been able to make the connection. Because of David our operating environment, security, it is very hard to David make changes to platform configurations. I understand your constraints but given that your environment is having problems, isn't that enough to warrant upgrading? I've looked at the release notes since 3.0.6 and there are lots of fixes for memory leaks and problems in the code that handles files. I think 3.0.14a is your next best bet. 3.0.6 is about 1.5 years old. I would also expect that the Samba team isn't going to be interested in pursuing the bug unless you can reproduce it against the latest version (3.0.21b). Your security setting might be influencing the number of open connections you have (not necessarily the number of file opens). Do you have 'security = domain' or 'security = server' in your smb.conf? security = DOMAIN David What information might be available about the samba David connections that might lead
[Samba] Too many open files, Bug 3342
We submitted bug 3342 on Dec. 20th, Too many open files. It has not moved from the new state. So lets try here. We have Dell Quad Xeon server that we end up rebooting about every 2-3 weeks when the number of open samba files sky rockets. The system is a ClearCase server system. We have been monitoring this system for a few months now and are not able to pin down a cause to this issue. The system keeps a pretty constant number of samba open files until what ever triggers the issue and then the number of file opens climbs at a steep exponential rate. RedHat Advanced Server 3 update 3 Linux quinault 2.4.21-27.0.2.ELsmp #1 SMP Wed Jan 12 23:35:44 EST 2005 i686 i686 i386 GNU/Linux Samba version=3.0.6-2E Clients for the most part are Windows XP and Windows 2003 Enterprise servers. Still looking clues as to what might be a cause for this behavior. Our systems people have monitored everything they can think of to see if we can find a correlation but so far nothing has turned up. It is a large, 200+ development environment with NFS and NAS CX400, storage units on the servers. It is a heavy file access, large builds parallel builds on multiple architectures. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/listinfo/samba
