Re: [tor-relays] DoS attacks are real (probably)
On Mon, 11 Dec 2017 at 18:07 Felix wrote: > Hi Alex > > Great points. > > > conntrack -L -p tcp --dport 9001 | awk '{print $5}' | sort | uniq -c > | sort -n > > On FreeBSD one can do: > > In packetfilter: > > # play with the numbers but more than 64k per ip if possible > set limit { frags 7, src-nodes 7, states 7, table-entries > 10 } > > table persist > > # 2000 is super high. Rate limit 50 new connects per 5 secs > # overload but not flush > pass in on $if_ext inet proto tcp from any to $relay_ip port $or_port > flags S/SA modulate state (max-src-conn 2000,max-src-conn-rate > 50/5,overload ) > > As cronjob: > > # release block after 10 minutes > pfctl -t blockOR -T expire 600 > > These measures protect your system. IMO for other or future cases we > should keep the clients degree of freedom (researchers / fancy doers) as > high as possible, being not too restrictive. > > > > 1. Open many OR connections (hundreds to thousands) > > 2. Leave open until tor runs out of sockets > > If the ip is saturated for like 2 hours the relay might loose the hsdir > flag. But today there are not enough resources in the game to generate > an issue for the network. > > > > I recommend against > > the blanket approach suggested previously of limiting whole sets of > > /24s, since that may inadvertently block mobile clients and is not > > effective against the current attack. > > Right. In future one could put such loud clients besides useful ips a > let the relays block the usefull. > > > > 2. the connections do not taper off if they are rejected. I banned these > > addresses from accessing Tor, and they continue to make dozens of > > connection attempts every second from each IP address. this means that > > this is probably not a good faith "test" or a misconfigured set of real > > Tor clients, but is instead malicious and using a modified or custom > > client. > > The above rule limits the useless attempts to a certain limit and > recovers after 10 minutes. This protects but gives the 'offender' the > chance to tune his client to a better behaviour (in case he wants it). > > > > 3c. it is almost certainly not real clients using NAT; as far as I know, > > LeaseWeb does not use NAT, and Online.net only uses one-to-one NAT. > > Good point. A general blocking rule should be smart enough to enable NAT > clients anyway ? > > > -- > Cheers, Felix > ___ > tor-relays mailing list > tor-relays@lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays Hi For the ones interested in Linux version, this translates to: -A INPUT -p tcp -m multiport --dports $or_port,$dir_port -m connlimit --connlimit-upto 2000 --connlimit-mask 24 -m hashlimit --hashlimit-upto 10/second --hashlimit-mode srcip --hashlimit-srcmask 16 --hashlimit-name mask24 -j ACCEPT -A INPUT -p tcp -m multiport --dports $or_port,$dir_port -j REJECT ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] DoS attacks are real (probably)
Alex Xu wrote: > Quoting Felix (2017-12-11 17:07:30), as excerpted > > Hi Alex > > > > Great points. > > > > > conntrack -L -p tcp --dport 9001 | awk '{print $5}' | sort | uniq -c > > > | sort -n > > > > On FreeBSD one can do: > > > > yeah, the optimal rule would ban "bad IPs" after some threshold of > connections, like "if one IP makes >1 conn/sec for at least 1 minute ban > for 1 hour" or something. I'm hoping to fix the underlying issue in Tor > so that low-bandwidth attacks like these are less effective. FWIW, the method that Felix posted should also work in DragonflyBSD and NetBSD. It may also work in OpenBSD, but the caveat is that the OpenBSD project has continued to develop its implementation of pf, so I don't know whether Felix's solution still works in OpenBSD. The other three BSDs' pf support has not been synchronized with that of the originating project (OpenBSD) for many years. Perhaps an OpenBSD tor relay operator can comment here on this matter. Scott Bennett, Comm. ASMELG, CFIAG ** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * ** * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * *-- Gov. John Hancock, New York Journal, 28 January 1790 * ** ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] DoS attacks are real (probably)
Hi Alex, > This attack appears to be malicious to me. It seems to work like this: > 1. Open many OR connections (hundreds to thousands) > 2. Leave open until tor runs out of sockets > Tor presently waits for the connections to time out, which takes 3-4.5 > minutes. It should instead more aggressively prune these garbage > connections. https://trac.torproject.org/projects/tor/ticket/19984 > tracks this. This is exactly what we saw as well. After implementing connection limits (thanks again x9p) the problem mostly went away and our relays have been stable since. Thank you for opening the trac ticket. We agree it would be great if this problem could be addressed in the Tor software if possible. In the mean time we should probably be advocating for all relay operators to implement connection limits. Put simply, without those limits, relays are vulnerable to DoS. > Evidence for this attack being malicious and intending to disable Tor is: Agree with all 7 points you listed. We'd also add, there is additional evidence that suggests some of the worst offenders (attacking IPs) are actually orchestrated by a single entity (or perhaps multiple entities working together). There are several commonalities across the infrastructure used for these attacks. We identified and blocked (with iptables DENY) the worst. To be clear, these IPs were not in the consensus, and yes, mostly hosted by LeaseWeb. > The referenced /16 block of guards is *not* part of this attack, and is > simply poorly configured relays. you should not block that set, Completely agree. We haven't blocked anything in the consensus. Thanks. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] DoS attacks are real (probably)
Quoting Felix (2017-12-11 17:07:30), as excerpted > Hi Alex > > Great points. > > > conntrack -L -p tcp --dport 9001 | awk '{print $5}' | sort | uniq -c | > > sort -n > > On FreeBSD one can do: > yeah, the optimal rule would ban "bad IPs" after some threshold of connections, like "if one IP makes >1 conn/sec for at least 1 minute ban for 1 hour" or something. I'm hoping to fix the underlying issue in Tor so that low-bandwidth attacks like these are less effective. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] DoS attacks are real (probably)
I am getting these warnings, not very often, and the exit (restricted) is working well otherwise: "Dec 11 18:07:23.000 [warn] Tried to establish rendezvous on non-OR circuit with purpose Acting as rendevous (pending)" Some posts about this elsewhere hinted this warning could be caused by attacks. I am not seeing attacks otherwise. Gerry -Original Message- From: tor-relays [mailto:tor-relays-boun...@lists.torproject.org] On Behalf Of Felix Sent: 11 December 2017 17:08 To: tor-relays@lists.torproject.org Subject: Re: [tor-relays] DoS attacks are real (probably) Hi Alex Great points. > conntrack -L -p tcp --dport 9001 | awk '{print $5}' | sort | uniq > -c | sort -n On FreeBSD one can do: In packetfilter: # play with the numbers but more than 64k per ip if possible set limit { frags 7, src-nodes 7, states 7, table-entries 10 } table persist # 2000 is super high. Rate limit 50 new connects per 5 secs # overload but not flush pass in on $if_ext inet proto tcp from any to $relay_ip port $or_port flags S/SA modulate state (max-src-conn 2000,max-src-conn-rate 50/5,overload ) As cronjob: # release block after 10 minutes pfctl -t blockOR -T expire 600 These measures protect your system. IMO for other or future cases we should keep the clients degree of freedom (researchers / fancy doers) as high as possible, being not too restrictive. > 1. Open many OR connections (hundreds to thousands) 2. Leave open > until tor runs out of sockets If the ip is saturated for like 2 hours the relay might loose the hsdir flag. But today there are not enough resources in the game to generate an issue for the network. > I recommend against > the blanket approach suggested previously of limiting whole sets of > /24s, since that may inadvertently block mobile clients and is not > effective against the current attack. Right. In future one could put such loud clients besides useful ips a let the relays block the usefull. > 2. the connections do not taper off if they are rejected. I banned > these addresses from accessing Tor, and they continue to make dozens > of connection attempts every second from each IP address. this means > that this is probably not a good faith "test" or a misconfigured set > of real Tor clients, but is instead malicious and using a modified or > custom client. The above rule limits the useless attempts to a certain limit and recovers after 10 minutes. This protects but gives the 'offender' the chance to tune his client to a better behaviour (in case he wants it). > 3c. it is almost certainly not real clients using NAT; as far as I > know, LeaseWeb does not use NAT, and Online.net only uses one-to-one NAT. Good point. A general blocking rule should be smart enough to enable NAT clients anyway ? -- Cheers, Felix ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] DoS attacks are real (probably)
Hi Alex Great points. > conntrack -L -p tcp --dport 9001 | awk '{print $5}' | sort | uniq -c | > sort -n On FreeBSD one can do: In packetfilter: # play with the numbers but more than 64k per ip if possible set limit { frags 7, src-nodes 7, states 7, table-entries 10 } table persist # 2000 is super high. Rate limit 50 new connects per 5 secs # overload but not flush pass in on $if_ext inet proto tcp from any to $relay_ip port $or_port flags S/SA modulate state (max-src-conn 2000,max-src-conn-rate 50/5,overload ) As cronjob: # release block after 10 minutes pfctl -t blockOR -T expire 600 These measures protect your system. IMO for other or future cases we should keep the clients degree of freedom (researchers / fancy doers) as high as possible, being not too restrictive. > 1. Open many OR connections (hundreds to thousands) > 2. Leave open until tor runs out of sockets If the ip is saturated for like 2 hours the relay might loose the hsdir flag. But today there are not enough resources in the game to generate an issue for the network. > I recommend against > the blanket approach suggested previously of limiting whole sets of > /24s, since that may inadvertently block mobile clients and is not > effective against the current attack. Right. In future one could put such loud clients besides useful ips a let the relays block the usefull. > 2. the connections do not taper off if they are rejected. I banned these > addresses from accessing Tor, and they continue to make dozens of > connection attempts every second from each IP address. this means that > this is probably not a good faith "test" or a misconfigured set of real > Tor clients, but is instead malicious and using a modified or custom > client. The above rule limits the useless attempts to a certain limit and recovers after 10 minutes. This protects but gives the 'offender' the chance to tune his client to a better behaviour (in case he wants it). > 3c. it is almost certainly not real clients using NAT; as far as I know, > LeaseWeb does not use NAT, and Online.net only uses one-to-one NAT. Good point. A general blocking rule should be smart enough to enable NAT clients anyway ? -- Cheers, Felix ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] DoS attacks are real (probably)
On Mon, December 11, 2017 1:40 pm, Alex Xu wrote: > tl;dr: run this: > > conntrack -L -p tcp --dport 9001 | awk '{print $5}' | sort | uniq -c | > sort -n Thanks for the detailed analysis. > ignore numbers less than 10. the remaining output should consist of the > following: ... > are not NATed IPs, a high limit is not justified. I recommend against > the blanket approach suggested previously of limiting whole sets of > /24s, since that may inadvertently block mobile clients and is not > effective against the current attack. As mentioned in the previous I agree the approach of /24 connlimit is not a good approach to Exit nodes. But for relays only worked fine for me and others. cheers. -- x9p | PGP : 0x03B50AF5EA4C8D80 / 5135 92C1 AD36 5293 2BDF DDCC 0DFA 74AE 1524 E7EE ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
[tor-relays] DoS attacks are real (probably)
tl;dr: run this: conntrack -L -p tcp --dport 9001 | awk '{print $5}' | sort | uniq -c | sort -n ignore numbers less than 10. the remaining output should consist of the following: 1. your IP 2. LeaseWeb and Online.net IPs (use rDNS and whois) 3. mobile networks block IPs in set 2 from accessing Tor using your firewall software. don't block 1 or you will have problems. don't block 3 or other people may have problems (hopefully more if the Android project gains momentum). also don't block the /16 of guards mentioned. they are causing no measurable harm. my list of set 2 is available upon request to longstanding members (basically if you've done anything on Trac or any tor mailing lists). more information: This attack appears to be malicious to me. It seems to work like this: 1. Open many OR connections (hundreds to thousands) 2. Leave open until tor runs out of sockets Tor presently waits for the connections to time out, which takes 3-4.5 minutes. It should instead more aggressively prune these garbage connections. https://trac.torproject.org/projects/tor/ticket/19984 tracks this. In the interim, this attack is causing real problems, so this approach seems reasonable for now. If you want to be slightly more conservative, instead of blocking them outright, simply limit these IP subnets to a small number of connections. Modern Tor only requires one, but you could do two or three per address, or four or five per /28 or so. Since these are not NATed IPs, a high limit is not justified. I recommend against the blanket approach suggested previously of limiting whole sets of /24s, since that may inadvertently block mobile clients and is not effective against the current attack. As mentioned in the previous paragraph, you should not set DisableOOSCheck 0, as that may wind up killing good sockets instead. again, this is not a good solution, but until that ticket is resolved, it is probably the best that can be done. Evidence for this attack being malicious and intending to disable Tor is: 1. these connections are coming very fast: as shown by others too, dozens of connections per second per IP, and tens of thousands of connections held open. the standard tor software multiplexes circuits over a single TCP connection, so even very high-volume links should only have a single Tor connection, or possibly two if they are very old and must make a separate connection to the DirPort. 2. the connections do not taper off if they are rejected. I banned these addresses from accessing Tor, and they continue to make dozens of connection attempts every second from each IP address. this means that this is probably not a good faith "test" or a misconfigured set of real Tor clients, but is instead malicious and using a modified or custom client. 3. they are spread out over many IPs, *but* only from *two* *hosting* service providers. this means that: 3a. it is likely a single individual or organization, otherwise it would be multiple providers 3b. they are trying to cause as much problems as possible, otherwise they would use one server 3c. it is almost certainly not real clients using NAT; as far as I know, LeaseWeb does not use NAT, and Online.net only uses one-to-one NAT. 4. rDNS is generic. this means they do not care enough to explain their origin. 5. as pointed out, they have not registered themselves in the relay consensus. this means it is probably not a set of high-volume relays that is somehow malfunctioning, or someone conducting a DoS attack through Tor itself. 6. as far as I can tell, these connections do not do anything. they simply remain open, consuming resources until tor times them out. 7. they keep far more connections open than they make. what I mean by this is that they hold open thousands of connections at once but only send less than a hundred connections per second. this supports my theory that it is not a large number of regular tor clients, but is instead some custom client specifically for disruption. additionally: The referenced /16 block of guards is *not* part of this attack, and is simply poorly configured relays. you should not block that set, but instead block the set described above. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays