UDP flooding / Ethernet issues? WAS Re: named error sending response: not enough free resources
On Thu, Jan 28, 2010 at 12:59 PM, James Smallacombe u...@3.am wrote: To follow up on this: Noticed the issue again this morning, which also was accompanied by latency so high that I could not connect (some pings got through at very high latency). I emailed the provider and they told me that they had my port on their Ether switch set to 10Mbs. They switched it to 100Mbs and only time will tell if that fixes it. Does this sound like it could be the entire cause? I ask because I've maxed out pipes before, but never seen it shut all traffic down this much. One key difference that I forgot to mention is that this server is running TWO instances of named, on two different IPs (for different domains), each running a few hundred zones. Bottom line: Would congestion cause this issue, or would this issue cause congestion? Some updates that may confuse more than inform: I caught this while it was happening yesterday and was able to do a tcpdump. I saw a ton of UDP traffic outbound to one IP that turned out to be a colocated server in Chicago. I put that IP in my ipfw rules and once I blocked any to that IP, it seemed to stop. Since then however, the logs have show the same issue again and there have been a few brief service disruptions. Today's security run output showed this: +(RULE NUMBER) 16054161 131965203420 deny ip from any to (blocked IP) and more alarmingly, this: kernel log messages: +++ /tmp/security.BErFHSS3 2010-01-29 03:09:32.0 -0500 +re0: link state changed to DOWN +re0: link state changed to UP +re0: promiscuous mode enabled +re0: promiscuous mode disabled +re0: promiscuous mode enabled +re0: promiscuous mode disabled +re0: promiscuous mode enabled +re0: promiscuous mode disabled re0 obviously being the Realtek Ethernet driver. The server itself never went down during this time, but the Ethernet did. Is there any DOS type of event that could cause this, or could the root of the problem be an Ethernet hardware or driver issue? Again, it is not clear to me which is the cause and which is the effect. Last bit of info: I just did a: 'tcpdump -n | grep -i udp' and saw a bunch of these, coming up a couple of times per second: 11:31:59.387561 IP (IP REMOVED) (IP REMOVED): NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST Where the source and destination IPs vary, but are NOT one of mine, but DO appear to belong to my colo/dedicated server provider and their customers. Is my server being used to DDOS others? If so, how? TIA, James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UDP flooding / Ethernet issues? WAS Re: named error sending response: not enough free resources
On Fri, Jan 29, 2010 at 10:51 AM, James Smallacombe u...@3.am wrote: Some updates that may confuse more than inform: I caught this while it was happening yesterday and was able to do a tcpdump. I saw a ton of UDP traffic outbound to one IP that turned out to be a colocated server in Chicago. I put that IP in my ipfw rules and once I blocked any to that IP, it seemed to stop. Since then however, the logs have show the same issue again and there have been a few brief service disruptions. Today's security run output showed this: +(RULE NUMBER) 16054161 131965203420 deny ip from any to (blocked IP) and more alarmingly, this: kernel log messages: +++ /tmp/security.BErFHSS3 2010-01-29 03:09:32.0 -0500 +re0: link state changed to DOWN +re0: link state changed to UP +re0: promiscuous mode enabled +re0: promiscuous mode disabled +re0: promiscuous mode enabled +re0: promiscuous mode disabled +re0: promiscuous mode enabled +re0: promiscuous mode disabled re0 obviously being the Realtek Ethernet driver. The server itself never went down during this time, but the Ethernet did. Is there any DOS type of event that could cause this, or could the root of the problem be an Ethernet hardware or driver issue? Again, it is not clear to me which is the cause and which is the effect. Last bit of info: I just did a: 'tcpdump -n | grep -i udp' and saw a bunch of these, coming up a couple of times per second: promiscuous mode entries are caused by tcpdump -- Adam Vande More ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: UDP flooding / Ethernet issues? WAS Re: named error sending response: not enough free resources
Hi-- On Jan 29, 2010, at 8:51 AM, James Smallacombe wrote: On Thu, Jan 28, 2010 at 12:59 PM, James Smallacombe u...@3.am wrote: To follow up on this: Noticed the issue again this morning, which also was accompanied by latency so high that I could not connect (some pings got through at very high latency). I emailed the provider and they told me that they had my port on their Ether switch set to 10Mbs. They switched it to 100Mbs and only time will tell if that fixes it. [ ... ] Today's security run output showed this: +(RULE NUMBER) 16054161 131965203420 deny ip from any to (blocked IP) and more alarmingly, this: kernel log messages: +++ /tmp/security.BErFHSS3 2010-01-29 03:09:32.0 -0500 +re0: link state changed to DOWN +re0: link state changed to UP These are probably from your ISP changing the link speed from 10 to 100Mbs. +re0: promiscuous mode enabled +re0: promiscuous mode disabled +re0: promiscuous mode enabled +re0: promiscuous mode disabled +re0: promiscuous mode enabled +re0: promiscuous mode disabled These are from running tcpdump. re0 obviously being the Realtek Ethernet driver. The server itself never went down during this time, but the Ethernet did. Is there any DOS type of event that could cause this, or could the root of the problem be an Ethernet hardware or driver issue? Again, it is not clear to me which is the cause and which is the effect. Last bit of info: I just did a: 'tcpdump -n | grep -i udp' and saw a bunch of these, coming up a couple of times per second: 11:31:59.387561 IP (IP REMOVED) (IP REMOVED): NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST Where the source and destination IPs vary, but are NOT one of mine, but DO appear to belong to my colo/dedicated server provider and their customers. Is my server being used to DDOS others? If so, how? That is standard Windows NetBIOS over IP traffic. It shouldn't be coming over your link unless your machines are sharing a subnet with someone else's Windows (or Samba) domain. You might discuss this with your ISP and ask them what's up, but failing that, using IPFW rules like this would be prudent: add deny tcp from any 135-139 to any add deny tcp from any to any 135-139 add deny udp from any 135-139 to any add deny udp from any to any 135-139 Regards, -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named error sending response: not enough free resources
On Wed, 27 Jan 2010, Chuck Swiger wrote: Hi-- On Jan 27, 2010, at 1:15 PM, James Smallacombe wrote: Jan 26 21:50:32 host named[667]: client IP REMOVED#57938: error sending response: not enough free resources Jan 26 21:50:32 host named[667]: client IP REMOVED#59830: error sending response: not enough free resources OK, if the nameserver is published / authoritative, then it would be expected to be fielding requests from the Internet at large. To follow up on this: Noticed the issue again this morning, which also was accompanied by latency so high that I could not connect (some pings got through at very high latency). I emailed the provider and they told me that they had my port on their Ether switch set to 10Mbs. They switched it to 100Mbs and only time will tell if that fixes it. Does this sound like it could be the entire cause? I ask because I've maxed out pipes before, but never seen it shut all traffic down this much. One key difference that I forgot to mention is that this server is running TWO instances of named, on two different IPs (for different domains), each running a few hundred zones. Bottom line: Would congestion cause this issue, or would this issue cause congestion? Thanks again! James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named error sending response: not enough free resources
On Thu, Jan 28, 2010 at 12:59 PM, James Smallacombe u...@3.am wrote: To follow up on this: Noticed the issue again this morning, which also was accompanied by latency so high that I could not connect (some pings got through at very high latency). I emailed the provider and they told me that they had my port on their Ether switch set to 10Mbs. They switched it to 100Mbs and only time will tell if that fixes it. Does this sound like it could be the entire cause? I ask because I've maxed out pipes before, but never seen it shut all traffic down this much. One key difference that I forgot to mention is that this server is running TWO instances of named, on two different IPs (for different domains), each running a few hundred zones. Bottom line: Would congestion cause this issue, or would this issue cause congestion? I would guess no, but that guess could easily be wrong. Have you tried turning up the logging to verbosity to get a better idea of what's happening? -- Adam Vande More ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
named error sending response: not enough free resources
NOTE: Please reply off-list as well as I am not subscribed My server (7.2-STABLE) suffered at least two outages Sunday through yesterday after having been up since July (it is a rented dedicated server with my FSBD install). The first time, I was able to log in via remotely, saw a ton of spam apparently abusing a php mail form script (more on that later) filling the /var partition. I purged it, but it still required a reboot as CPU was through the roof. Yesterday morning, I was unable to get into the server at all...pings were very high. I called the provider and got in via KVM over IP. CPU was fine and there wre no full partitions. As I had to catch a flight, I just rebooted it and it was fine. After getting home, I looked in the syslog and see thousands of these: Jan 26 21:50:32 host named[667]: client IP REMOVED#57938: error sending response: not enough free resources Jan 26 21:50:32 host named[667]: client IP REMOVED#59830: error sending response: not enough free resources Some googling on this error found a reference to a possible queue limiting problem in pf/qlimit, but the only firewalling I do is a very basic ipfw setup strictly for bruteblock. I am not even sure if this error caused the outage(s) or was caused by them, let alone a fix or workaround. Appreciate any and all clues, especially if you are familiar with this. TIA! James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named error sending response: not enough free resources
On Jan 27, 2010, at 10:24 AM, James Smallacombe wrote: NOTE: Please reply off-list as well as I am not subscribed OK. In return, please don't cross-post or multi-post the same question to multiple FreeBSD lists. My server (7.2-STABLE) suffered at least two outages Sunday through yesterday after having been up since July (it is a rented dedicated server with my FSBD install). The first time, I was able to log in via remotely, saw a ton of spam apparently abusing a php mail form script (more on that later) filling the /var partition. I purged it, but it still required a reboot as CPU was through the roof. See man pkill for an easier way to terminate processes short of rebooting. Depending on just how badly this PHP script was being taken advantage of and how closely you've been tracking security updates, it's possible that your machine might have been compromised. Yesterday morning, I was unable to get into the server at all...pings were very high. I called the provider and got in via KVM over IP. CPU was fine and there wre no full partitions. As I had to catch a flight, I just rebooted it and it was fine. After getting home, I looked in the syslog and see thousands of these: Jan 26 21:50:32 host named[667]: client IP REMOVED#57938: error sending response: not enough free resources Jan 26 21:50:32 host named[667]: client IP REMOVED#59830: error sending response: not enough free resources Were these client IPs expected to be talking to this machine? It indicates a problem sending UDP traffic; netstat -s output would be informative. You might find that setting options in named.conf to tune the # of outstanding queries will help: clients-per-query 10; max-clients-per-query 20; Doing a tcpdump and examining the queries to see what DNS resources are being requested would also be useful. Regards, -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named error sending response: not enough free resources
On Wed, 27 Jan 2010, Chuck Swiger wrote: On Jan 27, 2010, at 10:24 AM, James Smallacombe wrote: NOTE: Please reply off-list as well as I am not subscribed OK. In return, please don't cross-post or multi-post the same question to multiple FreeBSD lists. I posted to the -isp list a couple of hours earlier, then looked at the archives and noticed zero traffic on that list for the past couple of weeks, so I then posted here. After getting home, I looked in the syslog and see thousands of these: Jan 26 21:50:32 host named[667]: client IP REMOVED#57938: error sending response: not enough free resources Jan 26 21:50:32 host named[667]: client IP REMOVED#59830: error sending response: not enough free resources Were these client IPs expected to be talking to this machine? It This server is authoritative for a few hundred domains, so I would imagine anybody doing a query on any of them would need to talk to it...unless I misunderstand what you mean by talk. indicates a problem sending UDP traffic; netstat -s output would be Unfortunately, I did not have time for netstats or tcpdumps when this was happening and I've not seen this log entry since yesterday evening. informative. You might find that setting options in named.conf to tune the # of outstanding queries will help: clients-per-query 10; max-clients-per-query 20; Thanks, I will look into those. the man page for named.conf doesn't tell you much and my latest cricket book is 3rd edition (only up to BIND 8), so I guess it's time to break down and get the latest. James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named error sending response: not enough free resources
Hi-- On Jan 27, 2010, at 1:15 PM, James Smallacombe wrote: Jan 26 21:50:32 host named[667]: client IP REMOVED#57938: error sending response: not enough free resources Jan 26 21:50:32 host named[667]: client IP REMOVED#59830: error sending response: not enough free resources Were these client IPs expected to be talking to this machine? It This server is authoritative for a few hundred domains, so I would imagine anybody doing a query on any of them would need to talk to it...unless I misunderstand what you mean by talk. OK, if the nameserver is published / authoritative, then it would be expected to be fielding requests from the Internet at large. indicates a problem sending UDP traffic; netstat -s output would be Unfortunately, I did not have time for netstats or tcpdumps when this was happening and I've not seen this log entry since yesterday evening. Unless you rebooted the machine again since the errors were reported, the netstat output would still be relevant. informative. You might find that setting options in named.conf to tune the # of outstanding queries will help: clients-per-query 10; max-clients-per-query 20; Thanks, I will look into those. the man page for named.conf doesn't tell you much and my latest cricket book is 3rd edition (only up to BIND 8), so I guess it's time to break down and get the latest. Good luck -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named error sending response: not enough free resources
On Wed, 27 Jan 2010, Chuck Swiger wrote: On Jan 27, 2010, at 1:15 PM, James Smallacombe wrote: Jan 26 21:50:32 host named[667]: client IP REMOVED#57938: error sending response: not enough free resources indicates a problem sending UDP traffic; netstat -s output would be Unfortunately, I did not have time for netstats or tcpdumps when this was happening and I've not seen this log entry since yesterday evening. Unless you rebooted the machine again since the errors were reported, the netstat output would still be relevant. Ok, I saw this at least once since the last reboot, so here are the tcp and udp portions of the netstat -s: tcp: 31422122 packets sent 23133142 data packets (3473553079 bytes) 314215 data packets (132175418 bytes) retransmitted 6579 data packets unnecessarily retransmitted 11 resends initiated by MTU discovery 5408494 ack-only packets (200066 delayed) 0 URG only packets 1237 window probe packets 868892 window update packets 1713629 control packets 28600984 packets received 17029642 acks (for 3351867346 bytes) 1256410 duplicate acks 73760 acks for unsent data 11363962 packets (548204663 bytes) received in-sequence 184682 completely duplicate packets (16657176 bytes) 2327 old duplicate packets 1468 packets with some dup. data (339128 bytes duped) 334018 out-of-order packets (337877573 bytes) 85687 packets (637782 bytes) of data after window 10 window probes 114047 window update packets 160975 packets received after close 1148 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 9123 discarded due to memory problems 413250 connection requests 1504359 connection accepts 6 bad connection attempts 100 listen queue overflows 186225 ignored RSTs in the windows 1912682 connections established (including accepts) 2050764 connections closed (including 1022550 drops) 1058803 connections updated cached RTT on close 1065370 connections updated cached RTT variance on close 252114 connections updated cached ssthresh on close 3769 embryonic connections dropped 11958433 segments updated rtt (of 11574855 attempts) 285733 retransmit timeouts 12079 connections dropped by rexmit timeout 1884 persist timeouts 4 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 385 keepalive timeouts 345 keepalive probes sent 40 connections dropped by keepalive 2663719 correct ACK header predictions 5996181 correct data packet header predictions 1520655 syncache entries added 58477 retransmitted 26560 dupsyn 20622 dropped 1504359 completed 137 bucket overflow 0 cache overflow 6190 reset 10206 stale 100 aborted 0 badack 47 unreach 0 zone failures 1541277 cookies sent 415 cookies received 21638 SACK recovery episodes 37110 segment rexmits in SACK recovery episodes 51620488 byte rexmits in SACK recovery episodes 240368 SACK options (SACK blocks) received 217836 SACK options (SACK blocks) sent 0 SACK scoreboard overflow udp: 9663633 datagrams received 0 with incomplete header 0 with bad data length field 549 with bad checksum 9609 with no checksum 12092 dropped due to no socket 49230 broadcast/multicast datagrams undelivered 0 dropped due to full socket buffers 0 not for hashed pcb 9601762 delivered 42443353 datagrams output 0 times multicast source filter matched James Smallacombe PlantageNet, Inc. CEO and Janitor u...@3.am http://3.am = ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named: error sending response: not enough free resources
Steve Bertrand wrote: Chris St Denis wrote: Steve Bertrand wrote: What type of device is em1 attached to? Is it a switch or a hub? Is it possible to upgrade this? You should upgrade it to 100 (or 1000) anyways. Does this device show any collisions? This is a dedicated server in a datacenter. I don't know the exact switch specs but it's likely a layer 2/3 managed switch. Probably a 1U catalyst. Do you force 10Mb on your NIC, or do you auto-negotiate that? Perhaps before you pay a higher fee, your colo centre could allow you to connect to a 100Mb port (with perhaps some traffic policing) so you, as a client, could quickly verify if you want to scale up to their next tier without having to spend these up-front costs on troubleshooting this back-asswards. I can upgrade the connection to 100mbps for a small monthly fee. I've left it at 10 because I haven't had a need, but with traffic recently growing, this is probably the problem. Tell the colo that. Tell them you need to test their next tier of service! # mail -s tcpdump output st...@ipv6canada.com /var/log/dns.pcap I don't think this is necessary. If cutting down the http traffic or raising the port speed doesn't fix it, I'll look into further debugging with this. ...one more time, don't attempt to throttle your own traffic to troubleshoot what looks like a throughput bottleneck. Start with the collocation provider. They should, for free, allow you to have a testing period with their next service tier. Hopefully, they can do it without having to swap your Ethernet cable into another device. If it works during the test, then a small 'migration' and monthly upgrade fee would be acceptable (if they choose). Steve The problem was resolved by switching to 100Mbps. It's interesting that bind is all that complains about the bandwidth exhaustion, but I guess it's about my only use of UDP and TCP is better able to handle this kind of issue so doesn't complain. -- Chris St Denis Programmer SmarttNet (www.smartt.com) Ph: 604-473-9700 Ext. 200 --- Smart Internet Solutions For Businesses ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named: error sending response: not enough free resources
This is a dedicated server in a datacenter. I don't know the exact switch specs but it's likely a layer 2/3 managed switch. Probably a 1U catalyst. you mean cisco? there are actually most problematic switches. They don't properly autonegotiate speed and full/half duplex with many network cards. For example card is set to full duplex, cisco to half duplex, or reverse. More funny - even this doesn't help always. the only way to be sure it's fine is to set up speed manually on both sides. in one place i have connectivity from upstream provider that uses cisco switch. They set up speed to 100Mbps and to full duplex on their side, but many NICs does not work with it fine. It works but there are packet losses, or messages showing that card sometimes can't send packet etc. Actually - cheapest RTL8139 works best, digital 21140 or broadcom chips does not. I really wasted a lot of time to discover that cisco really works well with: - another cisco - realtek NICs - some cheapest 5 or 8 port switches ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named: error sending response: not enough free resources
- the network/LAN named tries to sent UDP packet is somehow flooded. Dns is probably fairly busy. It's the primary authorative dns for some busy domains. Is there a setting I can do to increase the limits of UDP packets to keep it from causing problems? it would need to sent 50 (i think) udp packets in burst faster than NIC can send it. unlikely. i'm 90% sure there is some problem with network. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named: error sending response: not enough free resources
On Wednesday 03 June 2009 00:46:20 Wojciech Puchar wrote: named[69750]: client *ip removed*: error sending response: not enough free resources quite misleading message, but the problem is that named want to send UDP packet and get's error from kernel. possible reasons - your firewall rules are the cause - check it. - your network card produce problems (REALLY i have that case) - the network/LAN named tries to sent UDP packet is somehow flooded. - the network card changes from UP to DOWN state at the time of the error See that a lot running local resolver on a wireless-g card and turning on the microwave. -- Mel ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named: error sending response: not enough free resources
possible reasons - your firewall rules are the cause - check it. - your network card produce problems (REALLY i have that case) - the network/LAN named tries to sent UDP packet is somehow flooded. - the network card changes from UP to DOWN state at the time of the error See that a lot running local resolver on a wireless-g card and turning on the microwave. this is extreme case. but card don't need to turn UP and DOWN for long enough for system to get a message. my second case - your network card produce problems (REALLY i have that case) is an example. i had such card that just reported error every some amount of packets. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named: error sending response: not enough free resources
On Wednesday 03 June 2009 11:48:48 Wojciech Puchar wrote: possible reasons - your firewall rules are the cause - check it. - your network card produce problems (REALLY i have that case) - the network/LAN named tries to sent UDP packet is somehow flooded. - the network card changes from UP to DOWN state at the time of the error See that a lot running local resolver on a wireless-g card and turning on the microwave. this is extreme case. Not really. The point is that at the time the network card goes from up to down, named spits out this error. If you log named to a different log file then /var/log/messages, you will not see the relation. The reason for changing UP to DOWN can be from a device operating at the 2.4Ghz band when using wireless-g to someone bumping his elbow into the colo's network cable, driver problems to switch failures, etc etc. -- Mel ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named: error sending response: not enough free resources
Not really. The point is that at the time the network card goes from up to down, named spits out this error. If you log named to a different log file then /var/log/messages, you will not see the relation. The reason for changing this is one reason i always change syslog.conf to configure everything to /var/log/messages. As you said - i see all events in time order. Fortunately i don't use radio networking unless i have no other choice. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
named: error sending response: not enough free resources
I occasionally get named errors like these in my messages log. I've done a lot of searching and have found others with similar problems, but no solutions. named[69750]: client *ip removed*: error sending response: not enough free resources named[69750]: client *ip removed*: error sending response: not enough free resources named[69750]: client *ip removed*: error sending response: not enough free resources named[69750]: client *ip removed*: error sending response: not enough free resources named[69750]: client *ip removed*: error sending response: not enough free resources System isn't particularly heavily loaded. Load averages around 0.5, cpu averages about 90% idle, not swapping much. Other messages on this subject suggest a shortage of mbuffs of an issue with the nic driver (the item I read was complaining about fxp, but I have em) so here is the related info. eureka# uname -a FreeBSD eureka 6.3-RELEASE-p1 FreeBSD 6.3-RELEASE-p1 #1: Mon Feb 25 08:17:08 PST 2008 cstde...@eureka:/usr/obj/usr/src/sys/EUREKA i386 eureka# named -v BIND 9.3.4-P1 eureka# ifconfig em1 em1: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 options=1bRXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING *IPs removed* ether 00:30:48:94:0a:31 media: Ethernet 10baseT/UTP full-duplex status: active eureka# netstat -m 1240/2165/3405 mbufs in use (current/cache/total) 1216/1290/2506/25600 mbuf clusters in use (current/cache/total/max) 1216/150 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 2742K/3121K/5863K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 8/430/6656 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 999635 requests for I/O initiated by sendfile 276104 calls to protocol drain routines How do I fix this? -- Chris St Denis Programmer SmarttNet (www.smartt.com) Ph: 604-473-9700 Ext. 200 --- Smart Internet Solutions For Businesses ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named: error sending response: not enough free resources
lot of searching and have found others with similar problems, but no solutions. named[69750]: client *ip removed*: error sending response: not enough free resources named[69750]: client *ip removed*: error sending response: not enough free resources named[69750]: client *ip removed*: error sending response: not enough free resources named[69750]: client *ip removed*: error sending response: not enough free resources named[69750]: client *ip removed*: error sending response: not enough free resources quite misleading message, but the problem is that named want to send UDP packet and get's error from kernel. possible reasons - your firewall rules are the cause - check it. - your network card produce problems (REALLY i have that case) - the network/LAN named tries to sent UDP packet is somehow flooded. i experienced all 3 cases. last is of course easiest to detect. Other messages on this subject suggest a shortage of mbuffs of an issue with no you are fine with mbufs, memory etc.. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named: error sending response: not enough free resources
Wojciech Puchar wrote: lot of searching and have found others with similar problems, but no solutions. named[69750]: client *ip removed*: error sending response: not enough free resources named[69750]: client *ip removed*: error sending response: not enough free resources named[69750]: client *ip removed*: error sending response: not enough free resources named[69750]: client *ip removed*: error sending response: not enough free resources named[69750]: client *ip removed*: error sending response: not enough free resources quite misleading message, but the problem is that named want to send UDP packet and get's error from kernel. possible reasons - your firewall rules are the cause - check it. Nope eureka# ipfw list 00100 allow ip from any to any via lo0 00200 deny ip from any to 127.0.0.0/8 00300 deny ip from 127.0.0.0/8 to any 65534 allow ip from any to any 65535 deny ip from any to any - your network card produce problems (REALLY i have that case) I have had this kind of error on multiple servers over the years, so i don't think it's a hardware problem. - the network/LAN named tries to sent UDP packet is somehow flooded. Dns is probably fairly busy. It's the primary authorative dns for some busy domains. Is there a setting I can do to increase the limits of UDP packets to keep it from causing problems? The server is approaching it's 10 mbps interface speed during peak hours, I may need to upgrade it to 100mbps. i experienced all 3 cases. last is of course easiest to detect. Other messages on this subject suggest a shortage of mbuffs of an issue with no you are fine with mbufs, memory etc.. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org -- Chris St Denis Programmer SmarttNet (www.smartt.com) Ph: 604-473-9700 Ext. 200 --- Smart Internet Solutions For Businesses ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named: error sending response: not enough free resources
Chris St Denis wrote: Wojciech Puchar wrote: possible reasons - your firewall rules are the cause - check it. Nope eureka# ipfw list - your network card produce problems (REALLY i have that case) I have had this kind of error on multiple servers over the years, so i don't think it's a hardware problem. - the network/LAN named tries to sent UDP packet is somehow flooded. Dns is probably fairly busy. It's the primary authorative dns for some busy domains. Is there a setting I can do to increase the limits of UDP packets to keep it from causing problems? The server is approaching it's 10 mbps interface speed during peak hours, I may need to upgrade it to 100mbps. The 10Mb ceiling (provided by your ifconfig output) could be a damper on this. What type of device is em1 attached to? Is it a switch or a hub? Is it possible to upgrade this? You should upgrade it to 100 (or 1000) anyways. Does this device show any collisions? Can you do the following for a few minutes (until at least the problem is triggered): # tcpdump -n -i em1 proto 17 port 53 -s -w /var/log/dns.pcap ...and then: # mail -s tcpdump output st...@ipv6canada.com /var/log/dns.pcap Is this server a caching recursive server for internal clients, or an authoritative server? What else runs on this box? If you generate further network traffic over the interface, do the log entries pile up faster? What does: # netstat -s -p udp say? I'd focus squarely on the 10Mbps cap first. That should be easy to test and eliminate. Then, once that is rectified, we can find out whether it's an inherent problem with the system. Steve smime.p7s Description: S/MIME Cryptographic Signature
Re: named: error sending response: not enough free resources
Steve Bertrand wrote: Chris St Denis wrote: Wojciech Puchar wrote: possible reasons - your firewall rules are the cause - check it. Nope eureka# ipfw list - your network card produce problems (REALLY i have that case) I have had this kind of error on multiple servers over the years, so i don't think it's a hardware problem. - the network/LAN named tries to sent UDP packet is somehow flooded. Dns is probably fairly busy. It's the primary authorative dns for some busy domains. Is there a setting I can do to increase the limits of UDP packets to keep it from causing problems? The server is approaching it's 10 mbps interface speed during peak hours, I may need to upgrade it to 100mbps. The 10Mb ceiling (provided by your ifconfig output) could be a damper on this. What type of device is em1 attached to? Is it a switch or a hub? Is it possible to upgrade this? You should upgrade it to 100 (or 1000) anyways. Does this device show any collisions? Can you do the following for a few minutes (until at least the problem is triggered): # tcpdump -n -i em1 proto 17 port 53 -s -w /var/log/dns.pcap ...meh, that should be: # tcpdump -n -i em1 proto 17 and port 53 -s -w /var/log/dns.pcap Steve smime.p7s Description: S/MIME Cryptographic Signature
Re: named: error sending response: not enough free resources
Steve Bertrand wrote: Steve Bertrand wrote: Chris St Denis wrote: Wojciech Puchar wrote: possible reasons - your firewall rules are the cause - check it. Nope eureka# ipfw list - your network card produce problems (REALLY i have that case) I have had this kind of error on multiple servers over the years, so i don't think it's a hardware problem. - the network/LAN named tries to sent UDP packet is somehow flooded. Dns is probably fairly busy. It's the primary authorative dns for some busy domains. Is there a setting I can do to increase the limits of UDP packets to keep it from causing problems? The server is approaching it's 10 mbps interface speed during peak hours, I may need to upgrade it to 100mbps. The 10Mb ceiling (provided by your ifconfig output) could be a damper on this. What type of device is em1 attached to? Is it a switch or a hub? Is it possible to upgrade this? You should upgrade it to 100 (or 1000) anyways. Does this device show any collisions? Can you do the following for a few minutes (until at least the problem is triggered): # tcpdump -n -i em1 proto 17 port 53 -s -w /var/log/dns.pcap ...meh, that should be: # tcpdump -n -i em1 proto 17 and port 53 -s -w /var/log/dns.pcap Ok. Perhaps I'm getting too old to try to recollect from memory. One more try (tested ;) pearl# tcpdump -n -i em1 -s 0 -w /var/log/dns.pcap port 53 and proto 17 Sorry! Steve smime.p7s Description: S/MIME Cryptographic Signature
Re: named: error sending response: not enough free resources
Steve Bertrand wrote: Chris St Denis wrote: Wojciech Puchar wrote: possible reasons - your firewall rules are the cause - check it. Nope eureka# ipfw list - your network card produce problems (REALLY i have that case) I have had this kind of error on multiple servers over the years, so i don't think it's a hardware problem. - the network/LAN named tries to sent UDP packet is somehow flooded. Dns is probably fairly busy. It's the primary authorative dns for some busy domains. Is there a setting I can do to increase the limits of UDP packets to keep it from causing problems? The server is approaching it's 10 mbps interface speed during peak hours, I may need to upgrade it to 100mbps. The 10Mb ceiling (provided by your ifconfig output) could be a damper on this. What type of device is em1 attached to? Is it a switch or a hub? Is it possible to upgrade this? You should upgrade it to 100 (or 1000) anyways. Does this device show any collisions? This is a dedicated server in a datacenter. I don't know the exact switch specs but it's likely a layer 2/3 managed switch. Probably a 1U catalyst. I can upgrade the connection to 100mbps for a small monthly fee. I've left it at 10 because I haven't had a need, but with traffic recently growing, this is probably the problem. Can you do the following for a few minutes (until at least the problem is triggered): # tcpdump -n -i em1 proto 17 port 53 -s -w /var/log/dns.pcap ...and then: # mail -s tcpdump output st...@ipv6canada.com /var/log/dns.pcap I don't think this is necessary. If cutting down the http traffic or raising the port speed doesn't fix it, I'll look into further debugging with this. Is this server a caching recursive server for internal clients, or an authoritative server? An authoritative for some moderately busy domains. Also recursive for some jails on this and another server (main recursive is on a private (10.0.0.0/24 on em0) network, and this server predates multi-ip jails) A tcpdump -n -i em1 -s 0 port 53 packets.txt for 1 minute shows eureka# wc -l packets.txt 359 packets.txt So about 350 dns packets a minute, at least in this particular minute. Less than I expected, I guess most is going to the other dns server at the moment. What else runs on this box? Web hosting. Thats where the full 10mbps comes from. If you generate further network traffic over the interface, do the log entries pile up faster? What does: # netstat -s -p udp eureka# netstat -s -p udp udp: 194973570 datagrams received 0 with incomplete header 13 with bad data length field 884 with bad checksum 68521 with no checksum 669174 dropped due to no socket 17 broadcast/multicast datagrams dropped due to no socket 733 dropped due to full socket buffers 0 not for hashed pcb 194302749 delivered 195188906 datagrams output Fyi, if these are since last reboot, this server has been up 381 days. say? I'd focus squarely on the 10Mbps cap first. That should be easy to test and eliminate. Then, once that is rectified, we can find out whether it's an inherent problem with the system. Yes, I'll deal with this, then reply again if the problem is not resolved. Thanks for the suggestions. Steve ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named: error sending response: not enough free resources
On Tue, Jun 2, 2009 at 4:46 PM, Wojciech Puchar woj...@wojtek.tensor.gdynia.pl wrote: lot of searching and have found others with similar problems, but no solutions. named[69750]: client *ip removed*: error sending response: not enough free resources named[69750]: client *ip removed*: error sending response: not enough free resources named[69750]: client *ip removed*: error sending response: not enough free resources named[69750]: client *ip removed*: error sending response: not enough free resources named[69750]: client *ip removed*: error sending response: not enough free resources quite misleading message, but the problem is that named want to send UDP packet and get's error from kernel. possible reasons - your firewall rules are the cause - check it. Not logically. If the firewall were to block it, we would not (except by logging) see any error. The error we're seeing is the inability to send the packet, such as the Tx or Rx buffer in the ethernet card is full and can't store another item in the queue - your network card produce problems (REALLY i have that case) a 1000Mbit card on a 10Mbit link can have problems. A 100/10Mbit card on a 10Mbit doesn't have the same problems. Had that problem in past jobs. - the network/LAN named tries to sent UDP packet is somehow flooded. Given the OPs later response that he's reaching the 10Mbit capacity, this is very likely, and would be the first thing I'd check out. Given that you're unable to send the packets, I'd say the NICs Tx buffer is full due to a network link at capacity, so the NIC driver is returning an error when it tries to queue the packet in the buffers. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named: error sending response: not enough free resources
snip - the network/LAN named tries to sent UDP packet is somehow flooded. Dns is probably fairly busy. It's the primary authorative dns for some busy domains. Is there a setting I can do to increase the limits of UDP packets to keep it from causing problems? /snip If you extend the SOA record to extend the refresh/TTL etc, you can lighten the load, until that refresh hits again. There is no solution other than to bump up the bandwidth, it seems. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: named: error sending response: not enough free resources
Chris St Denis wrote: Steve Bertrand wrote: What type of device is em1 attached to? Is it a switch or a hub? Is it possible to upgrade this? You should upgrade it to 100 (or 1000) anyways. Does this device show any collisions? This is a dedicated server in a datacenter. I don't know the exact switch specs but it's likely a layer 2/3 managed switch. Probably a 1U catalyst. Do you force 10Mb on your NIC, or do you auto-negotiate that? Perhaps before you pay a higher fee, your colo centre could allow you to connect to a 100Mb port (with perhaps some traffic policing) so you, as a client, could quickly verify if you want to scale up to their next tier without having to spend these up-front costs on troubleshooting this back-asswards. I can upgrade the connection to 100mbps for a small monthly fee. I've left it at 10 because I haven't had a need, but with traffic recently growing, this is probably the problem. Tell the colo that. Tell them you need to test their next tier of service! # mail -s tcpdump output st...@ipv6canada.com /var/log/dns.pcap I don't think this is necessary. If cutting down the http traffic or raising the port speed doesn't fix it, I'll look into further debugging with this. ...one more time, don't attempt to throttle your own traffic to troubleshoot what looks like a throughput bottleneck. Start with the collocation provider. They should, for free, allow you to have a testing period with their next service tier. Hopefully, they can do it without having to swap your Ethernet cable into another device. If it works during the test, then a small 'migration' and monthly upgrade fee would be acceptable (if they choose). Steve smime.p7s Description: S/MIME Cryptographic Signature
