RE: Problems with IPFW causing failed DNS and FTP sessions
Well I tried changing them to various numbers up to 180 from 1 and 5 respectively and that didn't help. Anyone else get around all this DNS mess with timeouts? It's causing my mail server to throw errors; host lookup did not complete and not deliver mail. -Original Message- From: owner-freebsd-questi...@freebsd.org [mailto:owner-freebsd-questi...@freebsd.org] On Behalf Of Michael Sierchio Sent: Sunday, March 31, 2013 10:04 PM To: Don O'Neil Cc: freebsd-questions@freebsd.org Subject: Re: Problems with IPFW causing failed DNS and FTP sessions net.inet.ip.fw.dyn_short_lifetime ? net.inet.ip.fw.dyn_udp_lifetime ? You might want to increase these, given the current state of things... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Problems with IPFW causing failed DNS and FTP sessions
Okay, what's your DNS setup? Are you running a recursive cache that contacts the root servers directly? Using your ISP's servers? Etc. As a mitigation step, I tried pointing my caches to 8.8.8.8 and 8.8.4.4. - but it turns out that Google is intentionally blocking (returning NX responses to) many netblocks right now because they contain hosts known to be part of the botnet in the DDOS DNS amplification attack. I'm mirroring the root zone everywhere I have a cache, and it's helping. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
RE: Problems with IPFW causing failed DNS and FTP sessions
My DNS config is pretty generic. I did try putting in the options to stop
recursive lookups, but all that did was cause even more failures (permission
denied lookups, etc...), so I removed that.
Here's my basic config;
options {
directory /etc/namedb;
pid-file/var/run/named/pid;
dump-file /var/dump/named_dump.db;
statistics-file /var/stats/named.stats;
};
zone . {
type hint;
file named.root;
};
I'm not sure the problem is specific to named, but something more systemic
with IPFW like I said, FTP sessions are timing out as well, and when I
turn off IPFW that fixes that problem too.
Is there any way to monitor what IPFW is dropping, by some sort of counters
rather than logging everything, and see what's going on internally to IPFW?
Thanks!
-Original Message-
From: Michael Sierchio [mailto:ku...@tenebras.com]
Sent: Monday, April 01, 2013 7:23 AM
To: Don O'Neil
Cc: freebsd-questions@freebsd.org
Subject: Re: Problems with IPFW causing failed DNS and FTP sessions
Okay, what's your DNS setup? Are you running a recursive cache that
contacts the root servers directly? Using your ISP's servers? Etc.
As a mitigation step, I tried pointing my caches to 8.8.8.8 and
8.8.4.4. - but it turns out that Google is intentionally blocking
(returning NX responses to) many netblocks right now because they
contain hosts known to be part of the botnet in the DDOS DNS
amplification attack.
I'm mirroring the root zone everywhere I have a cache, and it's helping.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Problems with IPFW causing failed DNS and FTP sessions
Hi everyone. recently my server started having issues with DNS and FTP sessions either not resolving or timing out. I've tracked the issue down to IPFW. if I issue a 'sysctl net.inet.ip.fw.enable=0' then my issues go away. I have the basic rules like this for dns; 01160 allow udp from any to any dst-port 53 in keep-state 01161 allow tcp from any to any dst-port 53 in keep-state 01162 allow udp from any to any dst-port 53 out keep-state 01163 allow tcp from any to any dst-port 53 out keep-state When I try an nslookup sometimes they fail, sometimes they get through, even if I change my DNS server to google, my ISP, or even OpenDNS. the firewall seems to be causing the issue. I have about 65 rules in all. Any ideas what could be causing this? My server load is low, usually hovering around .2 How can I look at the actual amount of traffic that the IPFW module is processing and track down potential performance issues? My server isn't pushing much data, only around 4-5 Mbps sustained. Thanks! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Problems with IPFW causing failed DNS and FTP sessions
Hi everyone. recently my server started having issues with DNS and FTP sessions either not resolving or timing out. I've tracked the issue down to IPFW. if I issue a 'sysctl net.inet.ip.fw.enable=0' then my issues go away. I have the basic rules like this for dns; 01160 allow udp from any to any dst-port 53 in keep-state 01161 allow tcp from any to any dst-port 53 in keep-state 01162 allow udp from any to any dst-port 53 out keep-state 01163 allow tcp from any to any dst-port 53 out keep-state When I try an nslookup sometimes they fail, sometimes they get through, even if I change my DNS server to google, my ISP, or even OpenDNS. the firewall seems to be causing the issue. I have about 65 rules in all. Any ideas what could be causing this? My server load is low, usually hovering around .2 How can I look at the actual amount of traffic that the IPFW module is processing and track down potential performance issues? My server isn't pushing much data, only around 4-5 Mbps sustained. Thanks! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Problems with IPFW causing failed DNS and FTP sessions
It would be really helpful if you'd post the ruleset. At first glance, your stateful rules seem rather wrong, unless there's a check-state above. Also, in and out aren't discriminating enough - every packet is seen by the ruleset more than once. You should think in terms of interfaces, direction, etc. Are you doing NAT? Stateful rules with NAT are indeed possible, but subtle. Your problem has nothing to do with server load, and probably everything to do with not-terribly-well-conceived ruleset. Please post yours here. - M On Sun, Mar 31, 2013 at 8:34 PM, Don O'Neil li...@lizardhill.com wrote: Hi everyone. recently my server started having issues with DNS and FTP sessions either not resolving or timing out. I've tracked the issue down to IPFW. if I issue a 'sysctl net.inet.ip.fw.enable=0' then my issues go away. I have the basic rules like this for dns; 01160 allow udp from any to any dst-port 53 in keep-state 01161 allow tcp from any to any dst-port 53 in keep-state 01162 allow udp from any to any dst-port 53 out keep-state 01163 allow tcp from any to any dst-port 53 out keep-state When I try an nslookup sometimes they fail, sometimes they get through, even if I change my DNS server to google, my ISP, or even OpenDNS. the firewall seems to be causing the issue. I have about 65 rules in all. Any ideas what could be causing this? My server load is low, usually hovering around .2 How can I look at the actual amount of traffic that the IPFW module is processing and track down potential performance issues? My server isn't pushing much data, only around 4-5 Mbps sustained. Thanks! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
RE: Problems with IPFW causing failed DNS and FTP sessions
Thanks for the response... here's my full rullset: # ipfw list 00100 check-state 00101 allow tcp from any to any established 00102 allow ip from any to any out keep-state 00103 allow icmp from any to any 00201 allow ip from any to any via lo0 00202 allow ip from any to 127.0.0.0/8 00203 allow ip from 127.0.0.0/8 to any 00204 deny tcp from any to any frag 00301 deny log logamount 50 ip from any to any ipoptions rr 00302 deny log logamount 50 ip from any to any ipoptions ts 00303 deny log logamount 50 ip from any to any ipoptions lsrr 00304 deny log logamount 50 ip from any to any ipoptions ssrr 00305 deny log logamount 50 tcp from any to any tcpflags syn,fin 00306 deny log logamount 50 tcp from any to any tcpflags syn,rst 01110 allow tcp from any to any dst-port 20 in 0 allow tcp from any to any dst-port 20 out 01112 allow tcp from any to any dst-port 21 in 01113 allow tcp from any to any dst-port 21 out 01114 allow tcp from any to any dst-port 990 in 01115 allow tcp from any to any dst-port 990 out 01116 allow udp from any to any dst-port 990 in 01117 allow udp from any to any dst-port 990 out 01118 allow tcp from any to any dst-port 989 in 01119 allow tcp from any to any dst-port 989 out 01120 allow udp from any to any dst-port 989 in 01121 allow udp from any to any dst-port 989 out 01122 allow tcp from any to any dst-port 1024-65000 keep-state 01125 allow tcp from any to any dst-port 22 in 01126 allow tcp from any to any dst-port 22 out 01130 allow tcp from any to any dst-port 25 in 01131 allow tcp from any to any dst-port 25 out 01132 allow tcp from any to any dst-port 587 in 01133 allow tcp from any to any dst-port 587 out 01134 allow tcp from any to any dst-port 2525 in 01135 allow tcp from any to any dst-port 2525 out 01140 allow tcp from any to any dst-port 110 in 01141 allow tcp from any to any dst-port 110 out 01142 allow tcp from any to any dst-port 995 in 01143 allow tcp from any to any dst-port 995 out 01144 allow tcp from any to any dst-port 2110 in 01145 allow tcp from any to any dst-port 2110 out 01150 allow tcp from any to any dst-port 143 in 01151 allow tcp from any to any dst-port 143 out 01152 allow tcp from any to any dst-port 993 in 01153 allow tcp from any to any dst-port 993 out 01160 allow udp from any to any dst-port 53 in keep-state 01161 allow tcp from any to any dst-port 53 in keep-state 01162 allow udp from any to any dst-port 53 out keep-state 01163 allow tcp from any to any dst-port 53 out keep-state 01170 allow tcp from any to any dst-port 80 in 01171 allow tcp from any to any dst-port 80 out 01172 allow tcp from any to any dst-port 443 in 01172 allow tcp from any to any dst-port 443 out 01180 allow tcp from any to any dst-port in 01181 allow tcp from any to any dst-port out 65535 deny ip from any to any I've tried these rules; 01160 allow udp from any to any dst-port 53 in 01161 allow tcp from any to any dst-port 53 in 01162 allow udp from any to any dst-port 53 out 01163 allow tcp from any to any dst-port 53 out Without the keep-state option, and the problem is still persisting... The weird thing is that I've run these rules for a number of years without any issues until just recently. I've checked my interface stats to make sure there aren't a bunch of fragmented packets or errors, and there aren't. I'm not running NAT, it's a publically accessible IP address. -Original Message- From: Michael Sierchio [mailto:ku...@tenebras.com] Sent: Sunday, March 31, 2013 8:58 PM To: Don O'Neil Cc: freebsd-questions@freebsd.org Subject: Re: Problems with IPFW causing failed DNS and FTP sessions It would be really helpful if you'd post the ruleset. At first glance, your stateful rules seem rather wrong, unless there's a check-state above. Also, in and out aren't discriminating enough - every packet is seen by the ruleset more than once. You should think in terms of interfaces, direction, etc. Are you doing NAT? Stateful rules with NAT are indeed possible, but subtle. Your problem has nothing to do with server load, and probably everything to do with not-terribly-well-conceived ruleset. Please post yours here. - M On Sun, Mar 31, 2013 at 8:34 PM, Don O'Neil li...@lizardhill.com wrote: Hi everyone. recently my server started having issues with DNS and FTP sessions either not resolving or timing out. I've tracked the issue down to IPFW. if I issue a 'sysctl net.inet.ip.fw.enable=0' then my issues go away. I have the basic rules like this for dns; 01160 allow udp from any to any dst-port 53 in keep-state 01161 allow tcp from any to any dst-port 53 in keep-state 01162 allow udp from any to any dst-port 53 out keep-state 01163 allow tcp from any to any dst-port 53 out keep-state When I try an nslookup sometimes they fail, sometimes they get through, even if I change my DNS server to google, my ISP, or even OpenDNS. the firewall seems to be causing the issue. I have about 65 rules in all. Any ideas what could
Re: Problems with IPFW causing failed DNS and FTP sessions
Don O'Neil wrote: Hi everyone. recently my server started having issues with DNS and FTP sessions either not resolving or timing out. I've tracked the issue down to IPFW. if I issue a 'sysctl net.inet.ip.fw.enable=0' then my issues go away. [snip] I'm probably not smart enough to be able to help directly with your problem but I'd like to add that there is a snowballing DNS Amplification ddos attack against SpamHaus going on which is spilling over. I was looking at some weird stuff my Suricata was reporting today when I noticed a large majority of it was coming from CloudFlare CDN. They use anycast packet traffic to deflect and diffuse such attacks for their customers. I'm wondering if your box has just been sitting there doing it's thing and you've made zero changes to it so it is essentially 'steady state' and this problem just sort of came up seemingly out of nowhere. Consider a possibility that the cause may be external and what you're seeing is just IPFW's reaction to it. A friend of mine is on a nearby Verizon subnet and he uses their DNS servers. He noticed minimal hiccup while I have my DNS pointed at OpenDNS and it took them almost a day to get their situation under control. Once they did traffic seemed to return to normal, then I noticed Suricata alerting on return traffic in my pf DNS firewall rule. All the traffic Suricata was complaining about was coming from the CloudFlare CDN. I've never seen this before, so I'm not completely certain what to make of it. My hypothesis is OpenDNS subscribed to CloudFlare's protection, and since it is legit return traffic from my DNS server's lookups the firewall never touched it. I would never have noticed if it wasn't for Suricata. I just don't know enough about it all, just that I was having some flaky DNS stalling and hanging and when it seemed like it returned to normal I began to see this weird stuff from CloudFlare CDN on my DNS traffic. Just would like to point out it may be possible your problem is somehow just a reflection of some noise going on outside your box. As for exactly what you might do about it is for smarter people than me. -Mike ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Problems with IPFW causing failed DNS and FTP sessions
I'll give you a more cogent reply tomorrow - if you use keep-state rules, you want to be a little more specific - for tcp, you want allow tcp from X to Y setup keep-state - i.e. you start the stateful rule on packets that have the SYN flag set. There are some other oddities here - I'm guessing that the firewall rules are there to protect this box itself... in which case your stateful rules really need only to consider outbound traffic, and to allow replies. Let me know if that assumption is erroneous. More later. Time for - M On Sun, Mar 31, 2013 at 9:33 PM, Don O'Neil li...@lizardhill.com wrote: Thanks for the response... here's my full rullset: # ipfw list 00100 check-state 00101 allow tcp from any to any established 00102 allow ip from any to any out keep-state 00103 allow icmp from any to any 00201 allow ip from any to any via lo0 00202 allow ip from any to 127.0.0.0/8 00203 allow ip from 127.0.0.0/8 to any 00204 deny tcp from any to any frag 00301 deny log logamount 50 ip from any to any ipoptions rr 00302 deny log logamount 50 ip from any to any ipoptions ts 00303 deny log logamount 50 ip from any to any ipoptions lsrr 00304 deny log logamount 50 ip from any to any ipoptions ssrr 00305 deny log logamount 50 tcp from any to any tcpflags syn,fin 00306 deny log logamount 50 tcp from any to any tcpflags syn,rst 01110 allow tcp from any to any dst-port 20 in 0 allow tcp from any to any dst-port 20 out 01112 allow tcp from any to any dst-port 21 in 01113 allow tcp from any to any dst-port 21 out 01114 allow tcp from any to any dst-port 990 in 01115 allow tcp from any to any dst-port 990 out 01116 allow udp from any to any dst-port 990 in 01117 allow udp from any to any dst-port 990 out 01118 allow tcp from any to any dst-port 989 in 01119 allow tcp from any to any dst-port 989 out 01120 allow udp from any to any dst-port 989 in 01121 allow udp from any to any dst-port 989 out 01122 allow tcp from any to any dst-port 1024-65000 keep-state 01125 allow tcp from any to any dst-port 22 in 01126 allow tcp from any to any dst-port 22 out 01130 allow tcp from any to any dst-port 25 in 01131 allow tcp from any to any dst-port 25 out 01132 allow tcp from any to any dst-port 587 in 01133 allow tcp from any to any dst-port 587 out 01134 allow tcp from any to any dst-port 2525 in 01135 allow tcp from any to any dst-port 2525 out 01140 allow tcp from any to any dst-port 110 in 01141 allow tcp from any to any dst-port 110 out 01142 allow tcp from any to any dst-port 995 in 01143 allow tcp from any to any dst-port 995 out 01144 allow tcp from any to any dst-port 2110 in 01145 allow tcp from any to any dst-port 2110 out 01150 allow tcp from any to any dst-port 143 in 01151 allow tcp from any to any dst-port 143 out 01152 allow tcp from any to any dst-port 993 in 01153 allow tcp from any to any dst-port 993 out 01160 allow udp from any to any dst-port 53 in keep-state 01161 allow tcp from any to any dst-port 53 in keep-state 01162 allow udp from any to any dst-port 53 out keep-state 01163 allow tcp from any to any dst-port 53 out keep-state 01170 allow tcp from any to any dst-port 80 in 01171 allow tcp from any to any dst-port 80 out 01172 allow tcp from any to any dst-port 443 in 01172 allow tcp from any to any dst-port 443 out 01180 allow tcp from any to any dst-port in 01181 allow tcp from any to any dst-port out 65535 deny ip from any to any I've tried these rules; 01160 allow udp from any to any dst-port 53 in 01161 allow tcp from any to any dst-port 53 in 01162 allow udp from any to any dst-port 53 out 01163 allow tcp from any to any dst-port 53 out Without the keep-state option, and the problem is still persisting... The weird thing is that I've run these rules for a number of years without any issues until just recently. I've checked my interface stats to make sure there aren't a bunch of fragmented packets or errors, and there aren't. I'm not running NAT, it's a publically accessible IP address. -Original Message- From: Michael Sierchio [mailto:ku...@tenebras.com] Sent: Sunday, March 31, 2013 8:58 PM To: Don O'Neil Cc: freebsd-questions@freebsd.org Subject: Re: Problems with IPFW causing failed DNS and FTP sessions It would be really helpful if you'd post the ruleset. At first glance, your stateful rules seem rather wrong, unless there's a check-state above. Also, in and out aren't discriminating enough - every packet is seen by the ruleset more than once. You should think in terms of interfaces, direction, etc. Are you doing NAT? Stateful rules with NAT are indeed possible, but subtle. Your problem has nothing to do with server load, and probably everything to do with not-terribly-well-conceived ruleset. Please post yours here. - M On Sun, Mar 31, 2013 at 8:34 PM, Don O'Neil li...@lizardhill.com wrote: Hi everyone. recently my server started having issues with DNS and FTP
Re: Problems with IPFW causing failed DNS and FTP sessions
On Sun, Mar 31, 2013 at 9:39 PM, Michael Powell nightre...@hotmail.com wrote: I'm probably not smart enough to be able to help directly with your problem but I'd like to add that there is a snowballing DNS Amplification ddos attack against SpamHaus going on which is spilling over Yes, this is very much true. The ICANN servers are dropping packets like mad, and many of the .com servers as well. I am mirroring the root zone locally to mitigate. It works to forward DNS to Google's servers (8.8.8.8, 8.8.4.4.) EXCEPT - they are blocking some net blocks (issuing spurious negative responses) because of large numbers of nets with hosts in the botnet participating in the attack. - M ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Problems with IPFW causing failed DNS and FTP sessions
net.inet.ip.fw.dyn_short_lifetime ? net.inet.ip.fw.dyn_udp_lifetime ? You might want to increase these, given the current state of things... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
