Re: internet slowdown
On Wed, 18 Jun 2008 17:23:26 -0400 John Almberg [EMAIL PROTECTED] wrote: Do you have PF installed? pftop is a quick way to see what's happening on the network interface. thanks john that's a good utility! also, thanks bill for the explanation of the ssh problem. we've found the problem we think after a lot of testing and working with the fellow from the cable company. it was our old 10T hubs and nics. we'd put together our server with bits and pieces given to us or bought from the salvation army or value village. it was a homeschooling project for my son and me. it's worked just fine for us until this media blitz that started recently. normally, our most active sites might get 1 hits a month, but what's been happening is we're exceeding that per day. june 11 was 35000+ and yesterday was 44500+ that's what seemed to be bogging everything down. so we bought some 100T switches and nics. we also went to cat6 cabling from cat5 and it looks like we're keeping up with the demand nicely now. now we're thinking about replacing our old 700MHs servers (one has 192M the other 128M), with something more powerful. but that's going to be a subject for another thread. all the assistance here is very much appreciated! -- In friendship, prad ... with you on your journey Towards Freedom http://www.towardsfreedom.com (website) Information, Inspiration, Imagination - truly a site for soaring I's ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: internet slowdown
On Tue, Jun 17, 2008 at 10:05:14PM -0700, prad wrote: On Mon, 16 Jun 2008 08:23:32 -0400 Bill Moran [EMAIL PROTECTED] wrote: Is the console responsive when this happens? when it bogs it takes ages to even ssh into the server. You do realise that this particular symptom usually points to reverse-lookup errors. I would examine your DNS server logs. -- Jonathan Chen [EMAIL PROTECTED] -- Lots of folks confuse bad management with destiny - Kin Hubbard ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: internet slowdown
In response to Jonathan Chen [EMAIL PROTECTED]: On Tue, Jun 17, 2008 at 10:05:14PM -0700, prad wrote: On Mon, 16 Jun 2008 08:23:32 -0400 Bill Moran [EMAIL PROTECTED] wrote: Is the console responsive when this happens? when it bogs it takes ages to even ssh into the server. You do realise that this particular symptom usually points to reverse-lookup errors. I would examine your DNS server logs. It could be, but it depends. I've also seen this problem as a result of high load and sshd taking a long time to spawn a child process because of the load. Prad, once you _are_ logged in, is the server responsive? Or does the sluggishness persist? -- Bill Moran http://www.potentialtech.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: internet slowdown
On Wed, 18 Jun 2008 20:36:54 +1200 Jonathan Chen [EMAIL PROTECTED] wrote: You do realise that this particular symptom usually points to reverse-lookup errors. I would examine your DNS server logs. i will look, but we are using the same named.conf and zone files as we did when we had 6.3 where we had no problems like this. also, would there be such sporadic issues? shouldn't a reverse dns problem be consistently problematic? -- In friendship, prad ... with you on your journey Towards Freedom http://www.towardsfreedom.com (website) Information, Inspiration, Imagination - truly a site for soaring I's ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: internet slowdown
On Wed, 18 Jun 2008 09:16:09 -0400 Bill Moran [EMAIL PROTECTED] wrote: Prad, once you _are_ logged in, is the server responsive? Or does the sluggishness persist? there is no problem once we are in - the cpu is idle 95% and you can move around and do stuff without any slowdown at all. right now, at 11:30am pst we were experiencing very high activity due to a press release done in toronto on canadian horse slaughter from this site: http://defendhorsescanada.org the slowdown is upon us too - but i don't know if it is related or not because the ping times are varying from 100ms to 1000+ms to the same site(s). it's all over the place! what is really weird is that the machine that hosts that site isn't the only one which has slowed down. the other machine which only serves email experiences the same thing (slow ssh connection, long ping times). we are going to bypass the servers by using one of our dynamic ip addresses and see what it is like then. if that computer experiences a slowdown, i think it may suggest that the problem is due to network activity within the cable company. if there is no slowdown, then that seems to point the problem exclusively to our servers. -- In friendship, prad ... with you on your journey Towards Freedom http://www.towardsfreedom.com (website) Information, Inspiration, Imagination - truly a site for soaring I's ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: internet slowdown
In response to prad [EMAIL PROTECTED]: On Wed, 18 Jun 2008 09:16:09 -0400 Bill Moran [EMAIL PROTECTED] wrote: Prad, once you _are_ logged in, is the server responsive? Or does the sluggishness persist? there is no problem once we are in - the cpu is idle 95% and you can move around and do stuff without any slowdown at all. Sounds like DNS, either that or a forking issue. Run ssh with -v to see details of where the process pauses. You can also run sshd in foreground mode on the server to see lots of debugging information. right now, at 11:30am pst we were experiencing very high activity due to a press release done in toronto on canadian horse slaughter from this site: http://defendhorsescanada.org the slowdown is upon us too - but i don't know if it is related or not because the ping times are varying from 100ms to 1000+ms to the same site(s). it's all over the place! Sounds to me like your network is overwhelmed. You need to get some management stuff online -- what is the rate of traffic through each of the interfaces involved? what is really weird is that the machine that hosts that site isn't the only one which has slowed down. the other machine which only serves email experiences the same thing (slow ssh connection, long ping times). I'm confused as to why you think this is related to the machine when there are multiple machines involved? It sure sounds like a network issue, from the description of the symptoms. If you're _absolutely_ sure the problem started occurring with the 7 upgrade, I'd look at the possibility that the NICs you're using aren't as well supported in 7 as they were in 6. Have you verified all the speed/duplex settings are matched? we are going to bypass the servers by using one of our dynamic ip addresses and see what it is like then. if that computer experiences a slowdown, i think it may suggest that the problem is due to network activity within the cable company. if there is no slowdown, then that seems to point the problem exclusively to our servers. That's also a good diagnostic step. -- Bill Moran http://www.potentialtech.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: internet slowdown
On Wed, 18 Jun 2008 15:31:49 -0400 Bill Moran [EMAIL PROTECTED] wrote: what is really weird is that the machine that hosts that site isn't the only one which has slowed down. the other machine which only serves email experiences the same thing (slow ssh connection, long ping times). I'm confused as to why you think this is related to the machine when there are multiple machines involved? i may have stated the case poorly. i thought is was a 7 issue not a machine issue. If you're _absolutely_ sure the problem started occurring with the 7 upgrade, I'd look at the possibility that the NICs you're using aren't as well supported in 7 as they were in 6. Have you verified all the speed/duplex settings are matched? we don't know how to do this, but will try to find out. however, see below, since i no longer think the problem is 7. we are going to bypass the servers by using one of our dynamic ip addresses and see what it is like then. if that computer experiences a slowdown, i think it may suggest that the problem is due to network activity within the cable company. if there is no slowdown, then that seems to point the problem exclusively to our servers. That's also a good diagnostic step. ok i am 99% convinced now that this has nothing to do with freebsd 7 at all. here's why: 1. there are 2 servers involved and they both get affected so as you say, bill It sure sounds like a network issue, from the description of the symptoms. if it were a 7 issue, then there is no reason for them to be affected simultaneously - but if it is a network issue, they would experience the slowdowns together which they do. 2. by-passing the servers produced mirror results. the machine on the dynamic address displayed an identical ping pattern to the same sites as those going through the servers. this would suggest that the problem has nothing to do with our servers. 3. we just found that on more than one instance when there was heavy activity on the website server, the pinging rate was low. this suggests that our servers are more than capable of handling the load and are not slowing our access down at all. 4. though i said we didn't have the problem with 6.3 initially which was true, this doesn't mean that the problem lies with 7 - all it means is that we didn't notice anything wrong with 6.3 while we used it. if this is a network issue from the outside, it may have started recently and merely coincides with our upgrade to 7. also, my son recalls that on rare occasions 6.3 may have acted somewhat slower than usual (though nothing like what we are experiencing now). 5. i don't see why there would be a dns problem since we are sshing from within our local network which the servers are part of. in fact, it is slow even when you use the ip address directly. so my present conclusion is that bottlenecks may be developing in the 'vicinity' of our assigned ip addresses (static and dynamic). these may not even be the fault of our cable company possibly, but i'll check with them anyway again. the only nagging matter though is why sshing in sometimes becomes slow. i will do as you suggest though bill (and jonathan) and produce some network graphs and look at the dns. it will be good to become familiar with these things since part of the reason for setting up our home servers was to gain an education about this stuff. i really appreciate the interest you've shown in our little problem and will follow through on your earlier suggestions as well as any others you may have. -- In friendship, prad ... with you on your journey Towards Freedom http://www.towardsfreedom.com (website) Information, Inspiration, Imagination - truly a site for soaring I's ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: internet slowdown
In response to prad [EMAIL PROTECTED]: 5. i don't see why there would be a dns problem since we are sshing from within our local network which the servers are part of. in fact, it is slow even when you use the ip address directly. Because the ssh server always does DNS lookups on connecting IPs in order to have hostnames to put in the logs. If DNS is sluggish, unresponsive, or configured poorly, it will cause long delays during login. -- Bill Moran http://www.potentialtech.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: internet slowdown
1. there are 2 servers involved and they both get affected so as you say, bill It sure sounds like a network issue, from the description of the symptoms. if it were a 7 issue, then there is no reason for them to be affected simultaneously - but if it is a network issue, they would experience the slowdowns together which they do. Do you have PF installed? pftop is a quick way to see what's happening on the network interface. -- John ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: internet slowdown
On Mon, 16 Jun 2008 08:23:32 -0400 Bill Moran [EMAIL PROTECTED] wrote: Is the console responsive when this happens? when it bogs it takes ages to even ssh into the server. It doesn't sound like a CPU issue to me -- ICMP messages are low enough CPU overhead that the system would have to be pretty badly bogged for CPU starvation to cause problems. agreed - otherwise why would the cpu be idle. What do your network traffic graphs show during the slowdowns? Perhaps some sort of network flood occurring? we are looking at flow-tools unless you have any other suggestions. there has been no problems for 2+ days, but we are going to follow through on this anyways and be prepared. thank you for your ideas bill and i'll let you know if we learn anything. -- In friendship, prad ... with you on your journey Towards Freedom http://www.towardsfreedom.com (website) Information, Inspiration, Imagination - truly a site for soaring I's ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
internet slowdown
we have 2 freebsd7 servers using pf with static ip addresses through our cable company (shaw cable) connecting our internal network to the outside. the servers do websites, email and nameserve (bind94) sometimes one or both become sluggish letting us connect to websites. the ping times become 10+ times as long. pinging the servers from the outside also slows down. whenever we reboot the servers, everything is fine again for several hours and then the speed starts fluctuating and then eventually everything slows down again. (however the several hours is not certain either because i just rebooted and everything slowed down after a few minutes this time). i tried /etc/netstart and that speeded things up for a few pings only. doing a systat shows the cpu to be idle more than 90% of the time. we are presently trying to figure out netstat. we never experienced this issue with freebsd 6.3 which we ran for several months. we've run freebsd7 since the beginning of june and this problem seemed to start showing up within the first week. i am not sure where to look in order to solve this problem - specifically which log files might provide some clues regarding the net. suggestions please? -- In friendship, prad ... with you on your journey Towards Freedom http://www.towardsfreedom.com (website) Information, Inspiration, Imagination - truly a site for soaring I's ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: internet slowdown
In response to prad [EMAIL PROTECTED]: we have 2 freebsd7 servers using pf with static ip addresses through our cable company (shaw cable) connecting our internal network to the outside. the servers do websites, email and nameserve (bind94) sometimes one or both become sluggish letting us connect to websites. the ping times become 10+ times as long. pinging the servers from the outside also slows down. whenever we reboot the servers, everything is fine again for several hours and then the speed starts fluctuating and then eventually everything slows down again. (however the several hours is not certain either because i just rebooted and everything slowed down after a few minutes this time). i tried /etc/netstart and that speeded things up for a few pings only. doing a systat shows the cpu to be idle more than 90% of the time. we are presently trying to figure out netstat. we never experienced this issue with freebsd 6.3 which we ran for several months. we've run freebsd7 since the beginning of june and this problem seemed to start showing up within the first week. i am not sure where to look in order to solve this problem - specifically which log files might provide some clues regarding the net. suggestions please? Is the console responsive when this happens? It doesn't sound like a CPU issue to me -- ICMP messages are low enough CPU overhead that the system would have to be pretty badly bogged for CPU starvation to cause problems. What do your network traffic graphs show during the slowdowns? Perhaps some sort of network flood occurring? -- Bill Moran http://www.potentialtech.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
