Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt
Hello JAB, Thank you for taking the time to respond to this in a very helpful manner... If the SAMBA community does not care about helping someone with a wildly out of date server then they should state that before letting someone join the mailing list. This is a production server on a closed LAN which we don't have the option of upgrading it to RHEL 5.9 or greater in the near future. So with that being said, anyone have any experience with what I am dealing with? Thanks -Original Message- From: Jonathan Buzzard [mailto:jonat...@buzzard.me.uk] Sent: Wednesday, March 06, 2013 9:03 AM To: Joseph, Matthew (EXP) Cc: samba@lists.samba.org Subject: EXTERNAL: Re: [Samba] SAMBA bringing NFS server to a halt On Wed, 2013-03-06 at 06:33 -0500, Joseph, Matthew (EXP) wrote: Hello, We have a Red Hat 5.3 SAMBA 3.0.33-3.7 Server that shares a few directories to 4 other servers. The other servers are Red Hat 5.3 and one Solaris 10 server. Stop right there. Nobody here could care less about someone running a wildly out of date server. There are numerous NFS and Samba fixes in RHEL 5.9 over 5.3 some of which are critical bugs, performance issues and others are ones that make your box open to remote root compromises. Upgrade to RHEL 5.9 and get back if you still have a problem. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt
On 03/06/2013 08:28 AM, Joseph, Matthew (EXP) wrote: Hello JAB, Thank you for taking the time to respond to this in a very helpful manner... If the SAMBA community does not care about helping someone with a wildly out of date server then they should state that before letting someone join the mailing list. Do not ascribe to the whole community the shortcomings of an individuals the volunteers 'his' opinion please. This is a production server on a closed LAN which we don't have the option of upgrading it to RHEL 5.9 or greater in the near future. So with that being said, anyone have any experience with what I am dealing with? Unless you have 15000 servers connected the fact you have that many processes indicates a serious issue with the server or at least one of the clients. Samba creates just 1 single process per client and all its requests are served by that process. If you are seeing multiple processes it means the client is opening multiple connections. That is wrong and indicate there is probably a bug with either server processes crashing, becoming unresponsive or both, or the client misbehaving.. You may want to consider trying playing with the following parameters on your samba server: - deadtime - max connections - keepalive - reset on zero vc You may also want to prevent samba from dumping core if that is activated as it could put pressure on disks and the kernel if too many processes core all at once. HTH, Simo. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt
Hi Matthew, I am not the best person to help you, regrettably, but I do run samba, so perhaps I can say something that is helpful, even if only by accident. What strikes me is the number of smbd processes; do you really have as many as 15000? I would expect most systems to run out of steam before the process list got that long, but I think there is a way of limiting the number of smb processes. I had a brief look at the man page for smb.conf (which seems to reside in /etc/samba, normally), but there is an obscene number of parameters, so I didn't find the relevant one. I'd suggest that you set a reasonable limit, though; when the limit is reached, users won't be able to connect, but the ones that are on will have a decent performance, at least. It surprises me that this should be connected to SVN or Eclipse; unless you have many 1000s of users you shouldn't really get that many smbd processes. A way to get closer to the source of the problem would be to look in the logs (usually in /var/log/samba, or so); there should be one log per connecting system. What I usually do is delete them all and then look at them a shortish while later when they seem to have grown somewhat. I suspect you will see the same message over and over and hopefully that will give you some idea of what is wrong. I hope this will help you; or if not, perhaps it will provoke a better answer from somebody who knows better. /jan From: samba-boun...@lists.samba.org [samba-boun...@lists.samba.org] on behalf of Joseph, Matthew (EXP) [matthew.jos...@lmco.com] Sent: 06 March 2013 13:28 To: Jonathan Buzzard Cc: samba@lists.samba.org Subject: Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt Hello JAB, Thank you for taking the time to respond to this in a very helpful manner... If the SAMBA community does not care about helping someone with a wildly out of date server then they should state that before letting someone join the mailing list. This is a production server on a closed LAN which we don't have the option of upgrading it to RHEL 5.9 or greater in the near future. So with that being said, anyone have any experience with what I am dealing with? Thanks -Original Message- From: Jonathan Buzzard [mailto:jonat...@buzzard.me.uk] Sent: Wednesday, March 06, 2013 9:03 AM To: Joseph, Matthew (EXP) Cc: samba@lists.samba.org Subject: EXTERNAL: Re: [Samba] SAMBA bringing NFS server to a halt On Wed, 2013-03-06 at 06:33 -0500, Joseph, Matthew (EXP) wrote: Hello, We have a Red Hat 5.3 SAMBA 3.0.33-3.7 Server that shares a few directories to 4 other servers. The other servers are Red Hat 5.3 and one Solaris 10 server. Stop right there. Nobody here could care less about someone running a wildly out of date server. There are numerous NFS and Samba fixes in RHEL 5.9 over 5.3 some of which are critical bugs, performance issues and others are ones that make your box open to remote root compromises. Upgrade to RHEL 5.9 and get back if you still have a problem. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt
On Wed, 2013-03-06 at 08:28 -0500, Joseph, Matthew (EXP) wrote: Hello JAB, Thank you for taking the time to respond to this in a very helpful manner... If the SAMBA community does not care about helping someone with a wildly out of date server then they should state that before letting someone join the mailing list. Given you are running RHEL, you should have been over the last four years been reading the security bulletins for RHEL and responding to them appropriately. It should be apparent to any sensible person that the first step would be to check that my distribution does not have fixes for the problems that I am seeing. (hint I am 99% certain it does). This is a production server on a closed LAN which we don't have the option of upgrading it to RHEL 5.9 or greater in the near future. No lan is that closed. That you have no procedure for upgrading the OS on your server which suffers from a number of remote root security holes that require nothing more than a connection to your network is very bad practice. So with that being said, anyone have any experience with what I am dealing with? Read your distro release and security notes. I am 99% certain that this is a known problem that can be fixed by upgrading. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt
A few things aren't clear- - Are Solaris and RHEL servers mounting shares from the primary server as samba clients or NFS clients? - Are people running SVN and Eclipse on Windows or RHEL systems? - Are you using samba to reshare NFS shares? I run a mixed environment of Windows and Linux clients with Solaris servers running samba. The linux clients use NFS (v4 is now the default.) Some of the things I have found are that - It is worth patch solaris to get later version of Samba - if you are using ZFS (not ufs) and you have a complex environment with LDAP and domain trusts.But you really have to test carefully before an upgrade. -Do not use samba to reshare NFS or autofs shares. How are clients checking stuff out from SVN? Via a nfs file share, samba file share, sftp or ssh? I understand the need to maintain stability with a server OS. But I think you do have to plan for an eventual OS upgrade/patch otherwise you end up with a system that you can't get support on. Are you also looking at output of vmstat or iostat ?If disk i/o gets too high, clients may repeat read/write requests which just causes a feedback loop exacerbating the situation.I have seen this with nfs clients. It is like everyone yelling louder to get heard because everyone is yelling. On 03/06/13 08:47, Simo wrote: On 03/06/2013 08:28 AM, Joseph, Matthew (EXP) wrote: Hello JAB, Thank you for taking the time to respond to this in a very helpful manner... If the SAMBA community does not care about helping someone with a wildly out of date server then they should state that before letting someone join the mailing list. Do not ascribe to the whole community the shortcomings of an individuals the volunteers 'his' opinion please. This is a production server on a closed LAN which we don't have the option of upgrading it to RHEL 5.9 or greater in the near future. So with that being said, anyone have any experience with what I am dealing with? Unless you have 15000 servers connected the fact you have that many processes indicates a serious issue with the server or at least one of the clients. Samba creates just 1 single process per client and all its requests are served by that process. If you are seeing multiple processes it means the client is opening multiple connections. That is wrong and indicate there is probably a bug with either server processes crashing, becoming unresponsive or both, or the client misbehaving.. You may want to consider trying playing with the following parameters on your samba server: - deadtime - max connections - keepalive - reset on zero vc You may also want to prevent samba from dumping core if that is activated as it could put pressure on disks and the kernel if too many processes core all at once. HTH, Simo. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt
Presuming you have a RHEL subscription, you should be able to download the ISO's and patches on an internet machine and xfr via sneaker net (USB drive, DVD) to the internal network. You can even set up an internal yum repository. Even with out an internet connection, you still have to consider internal security concerns. With Solaris, you can also download the latest monthly patch cluster (assuming you have a support contract.) This will bring up to samba 3.5.x. or 3.6.x. It also fixes some issues with max group membership, and I recall some mention of kernel and nfs bug fixes. Just make sure you backup all your samba config before patching. On 03/06/13 09:12, Jonathan Buzzard wrote: On Wed, 2013-03-06 at 08:28 -0500, Joseph, Matthew (EXP) wrote: Hello JAB, Thank you for taking the time to respond to this in a very helpful manner... If the SAMBA community does not care about helping someone with a wildly out of date server then they should state that before letting someone join the mailing list. Given you are running RHEL, you should have been over the last four years been reading the security bulletins for RHEL and responding to them appropriately. It should be apparent to any sensible person that the first step would be to check that my distribution does not have fixes for the problems that I am seeing. (hint I am 99% certain it does). This is a production server on a closed LAN which we don't have the option of upgrading it to RHEL 5.9 or greater in the near future. No lan is that closed. That you have no procedure for upgrading the OS on your server which suffers from a number of remote root security holes that require nothing more than a connection to your network is very bad practice. So with that being said, anyone have any experience with what I am dealing with? Read your distro release and security notes. I am 99% certain that this is a known problem that can be fixed by upgrading. JAB. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt
My apologizes Simo, I did not intend with that comment to put down the Samba community as a whole I was just trying to point out a fault with a certain user. I will try fooling around with those options that you listed below and see if any of them remedy my issue. Thanks for taking the time and effort on this issue. Matt -Original Message- From: samba-boun...@lists.samba.org [mailto:samba-boun...@lists.samba.org] On Behalf Of Simo Sent: Wednesday, March 06, 2013 9:47 AM To: samba@lists.samba.org Subject: Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt On 03/06/2013 08:28 AM, Joseph, Matthew (EXP) wrote: Hello JAB, Thank you for taking the time to respond to this in a very helpful manner... If the SAMBA community does not care about helping someone with a wildly out of date server then they should state that before letting someone join the mailing list. Do not ascribe to the whole community the shortcomings of an individuals the volunteers 'his' opinion please. This is a production server on a closed LAN which we don't have the option of upgrading it to RHEL 5.9 or greater in the near future. So with that being said, anyone have any experience with what I am dealing with? Unless you have 15000 servers connected the fact you have that many processes indicates a serious issue with the server or at least one of the clients. Samba creates just 1 single process per client and all its requests are served by that process. If you are seeing multiple processes it means the client is opening multiple connections. That is wrong and indicate there is probably a bug with either server processes crashing, becoming unresponsive or both, or the client misbehaving.. You may want to consider trying playing with the following parameters on your samba server: - deadtime - max connections - keepalive - reset on zero vc You may also want to prevent samba from dumping core if that is activated as it could put pressure on disks and the kernel if too many processes core all at once. HTH, Simo. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt
Hello JAB, You need to understand that installing patches and upgrading servers is not a simple task when it comes to my situation. My first step is to try to figure out if it's a OS fault or if it can be fixed with modifying configurations of the OS or in this case Samba (or my configuration of Samba). You are making a lot of assumptions which is fine if that is what you choose to believe. It is a completely closed LAN with multiple layers of security so let's leave it at that. If the solution is to install patches then it is something I will look into but again that is a long process that I would prefer not to go into if it is not needed for this situation. -Original Message- From: Jonathan Buzzard [mailto:jonat...@buzzard.me.uk] Sent: Wednesday, March 06, 2013 10:12 AM To: Joseph, Matthew (EXP) Cc: samba@lists.samba.org Subject: RE: EXTERNAL: Re: [Samba] SAMBA bringing NFS server to a halt On Wed, 2013-03-06 at 08:28 -0500, Joseph, Matthew (EXP) wrote: Hello JAB, Thank you for taking the time to respond to this in a very helpful manner... If the SAMBA community does not care about helping someone with a wildly out of date server then they should state that before letting someone join the mailing list. Given you are running RHEL, you should have been over the last four years been reading the security bulletins for RHEL and responding to them appropriately. It should be apparent to any sensible person that the first step would be to check that my distribution does not have fixes for the problems that I am seeing. (hint I am 99% certain it does). This is a production server on a closed LAN which we don't have the option of upgrading it to RHEL 5.9 or greater in the near future. No lan is that closed. That you have no procedure for upgrading the OS on your server which suffers from a number of remote root security holes that require nothing more than a connection to your network is very bad practice. So with that being said, anyone have any experience with what I am dealing with? Read your distro release and security notes. I am 99% certain that this is a known problem that can be fixed by upgrading. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt
Matthew, Can you post your smb.conf so we can see if any looks odd. Also when this happens look to see how many network connects you have with netstat. This may require a tcpdump that has been scrubbed of any sensitive data, if possible. Jonn On 03/06/2013 08:27 AM, Joseph, Matthew (EXP) wrote: Hello JAB, You need to understand that installing patches and upgrading servers is not a simple task when it comes to my situation. My first step is to try to figure out if it's a OS fault or if it can be fixed with modifying configurations of the OS or in this case Samba (or my configuration of Samba). You are making a lot of assumptions which is fine if that is what you choose to believe. It is a completely closed LAN with multiple layers of security so let's leave it at that. If the solution is to install patches then it is something I will look into but again that is a long process that I would prefer not to go into if it is not needed for this situation. -Original Message- From: Jonathan Buzzard [mailto:jonat...@buzzard.me.uk] Sent: Wednesday, March 06, 2013 10:12 AM To: Joseph, Matthew (EXP) Cc: samba@lists.samba.org Subject: RE: EXTERNAL: Re: [Samba] SAMBA bringing NFS server to a halt On Wed, 2013-03-06 at 08:28 -0500, Joseph, Matthew (EXP) wrote: Hello JAB, Thank you for taking the time to respond to this in a very helpful manner... If the SAMBA community does not care about helping someone with a wildly out of date server then they should state that before letting someone join the mailing list. Given you are running RHEL, you should have been over the last four years been reading the security bulletins for RHEL and responding to them appropriately. It should be apparent to any sensible person that the first step would be to check that my distribution does not have fixes for the problems that I am seeing. (hint I am 99% certain it does). This is a production server on a closed LAN which we don't have the option of upgrading it to RHEL 5.9 or greater in the near future. No lan is that closed. That you have no procedure for upgrading the OS on your server which suffers from a number of remote root security holes that require nothing more than a connection to your network is very bad practice. So with that being said, anyone have any experience with what I am dealing with? Read your distro release and security notes. I am 99% certain that this is a known problem that can be fixed by upgrading. JAB. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt
On Wed, 2013-03-06 at 08:28 -0500, Joseph, Matthew (EXP) wrote: Hello JAB, Thank you for taking the time to respond to this in a very helpful manner... Actually it is helpful given the limited and insufficient information you provided. The basic problem is you are looking for a magic fix that likely does not exist because you want to keep running an OS that is many revisions out of date and has numerous serious security holes and a whole slew of known problems as a consequence. Where simply keeping your system properly patched has a good chance of eliminating the problem, which would have known had you been reading the release and security bulletins for RHEL5 over the last four years. There is simply too many NFS and Samba issues in RHEL5.3 for it to be remotely reasonable to expect any help trying to debug a setup still running at that level. Consequently a sensible course of action is to upgrade to something recent that does not have a whole bunch of known problems and serious security holes and if the problem still exists then come back with a more detail explanation of your setup. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt
On 03/06/2013 09:46 AM, Jonathan Buzzard wrote: On Wed, 2013-03-06 at 08:28 -0500, Joseph, Matthew (EXP) wrote: Hello JAB, Thank you for taking the time to respond to this in a very helpful manner... Actually it is helpful given the limited and insufficient information you provided. The basic problem is you are looking for a magic fix that likely does not exist because you want to keep running an OS that is many revisions out of date and has numerous serious security holes and a whole slew of known problems as a consequence. Where simply keeping your system properly patched has a good chance of eliminating the problem, which would have known had you been reading the release and security bulletins for RHEL5 over the last four years. There is simply too many NFS and Samba issues in RHEL5.3 for it to be remotely reasonable to expect any help trying to debug a setup still running at that level. Consequently a sensible course of action is to upgrade to something recent that does not have a whole bunch of known problems and serious security holes and if the problem still exists then come back with a more detail explanation of your setup. Jonathan, you are not being helpful here. We all understood you really want Joseph to upgrade, and we all acknowledge that is good practice, but Joseph seem to have constraints he cannot overcome right now. So please stop hammering on this point. If you do not have anything useful to say for his current situation then just ignore this thread and carry on. Simo. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt
On 06.03.2013 15:46, Jonathan Buzzard wrote: Consequently a sensible course of action is to upgrade to something recent I think everybody got your point by now. best regards, Sven -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt
Hey Simo, I modified the entries you listed below and started running a few instances of SVN on the shares and it seems to be holding steady. I'm going to continue testing during the day to see how it does. Looking back on the issue I never noticed the date in which the files were accessed. The Samba clients would be done with a file but the server never clicked in that it should release the files. Like I said I'm going to continue the testing on this to make sure it stays consistent with the current results. Thank you very much for the suggestion. Matt -Original Message- From: samba-boun...@lists.samba.org [mailto:samba-boun...@lists.samba.org] On Behalf Of Simo Sent: Wednesday, March 06, 2013 9:47 AM To: samba@lists.samba.org Subject: Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt On 03/06/2013 08:28 AM, Joseph, Matthew (EXP) wrote: Hello JAB, Thank you for taking the time to respond to this in a very helpful manner... If the SAMBA community does not care about helping someone with a wildly out of date server then they should state that before letting someone join the mailing list. Do not ascribe to the whole community the shortcomings of an individuals the volunteers 'his' opinion please. This is a production server on a closed LAN which we don't have the option of upgrading it to RHEL 5.9 or greater in the near future. So with that being said, anyone have any experience with what I am dealing with? Unless you have 15000 servers connected the fact you have that many processes indicates a serious issue with the server or at least one of the clients. Samba creates just 1 single process per client and all its requests are served by that process. If you are seeing multiple processes it means the client is opening multiple connections. That is wrong and indicate there is probably a bug with either server processes crashing, becoming unresponsive or both, or the client misbehaving.. You may want to consider trying playing with the following parameters on your samba server: - deadtime - max connections - keepalive - reset on zero vc You may also want to prevent samba from dumping core if that is activated as it could put pressure on disks and the kernel if too many processes core all at once. HTH, Simo. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
Re: [Samba] EXTERNAL: Re: SAMBA bringing NFS server to a halt
On Wed, 2013-03-06 at 10:06 -0500, Simo wrote: [SNIP] Jonathan, you are not being helpful here. Actually I am being helpful, given the limited information provided. There are a whole host of issues with Samba and NFS fixed between RHEL5.3 and RHEL5.8/5.9 that are likely to be related to his problem. Trust me I have the scars to prove it. We all understood you really want Joseph to upgrade, and we all acknowledge that is good practice, but Joseph seem to have constraints he cannot overcome right now. Then I believe he won't be able to fix his problem. He might be able to patch over the problem with deadtime and max connections options but that is not really a fix, and won't address the gaping security holes in his setup. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
