Re: [tor-onions] DDoS, Single Onion Services and IP Addresses
On 02 Feb (00:23:14), Alec Muffett wrote: > I am not going to pretend that I fully understand the DDoS mitigations yet, > but experience at two jobs has taught me that at least three entire > countries essentially present themselves from behind small numbers of > heavily NATed addresses, so I hope that the mitigations are NAT-friendly. > > ISTR that UAE and Singapore are two such, I forget the third? I've been running the circuit creation mitigation for weeks now in different forms which had much more aggressive threshold in the beginning. At most, my Guard identified 550 ish client address for which I've investigated a bit. They were all from big hosting corp that is dedicatedpanel.com, vultr.com LeaseWeb and Hetzner (the OVH clients were gone at that time). The majority (82%) was Hetzner. Thus so far I would say that it is not impacting that much single countries NATed in some ways or another. This doesn't mean it won't be *especially* when 70% of the network will be rolling out those defenses. We really need to keep a sharp eye on this and adjust accordingly. Cheers! David -- Wbu/qrEyrunjWnT0UyaZiV4x9ISE3TmR5yicMuxsU4E= signature.asc Description: PGP signature ___ tor-onions mailing list tor-onions@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-onions
Re: [tor-onions] DDoS, Single Onion Services and IP Addresses
Hi All,
I think I see the shape of the DDoS mitigations now, and to test my
understanding I'm going to try to recap/quote some of the thread as I
understand it; plus, I'll voice some of the questions which linger at
the back of my head.
Stuff where I am inferring (possibly wrongly) behaviour from context,
is in brackets.
Observation: I had a hard time following whether a bare reference to
"client" in any given explanation, referred to a client-ip-address, or
to one of the tor-daemons ("clients behind an ip-address"), or to
something (eg: curl) actually using tor. So I may have misapprehended
/ got some of this wrong...
# The three defences are...
1) Connection-Limiting: if a makes too many
connections to then
will start to
Question: as above, I am really uncertain whether the circuits are
destroyed because they are attributed to a given ip-address (potential
for collateral damage?) or to a specific overzealous tor-daemon/client
*behind* a given ip-address.
Question: I am not sure whether this mitigation is per-relay, or
whether the status that is being "unfair"
gets somehow propagated to the rest of the Tor network?
2) Rate-Limiting: if a makes connections too
quickly to , when ALSO there are lots of
existing connections , then will
start to ignore that
Question:
3) DoS Mitigation: if a asks to a rendezvous point, it gets
ignored, which thereby kills Tor2web.
Notes: initially this worried me because I misread it as "if a
non-relay CONNECTS to an RP" - which would ban SingleOnions because no
guards - but I am pretty sure of my revised interpretation and I am
fairly certain of why this would be a good idea; but if I can make
this misinterpretation, then others might do, too.
# Re: my situation...
Teor: yep everything is SingleOnion in EOTK by default, not least
because testing onionification of video streams with a bank of
RaspberryPi on a single DSL connection is laggy enough without a
couple of extra hops.
Questions:
The reason I am worried (above) about "notification of bad behaviour
is somehow propagated?" is that I am worried that if I trip a
trigger, am I going to effectively be banned from the entire Tor network
for "a couple of hours", or am I just going to lose ?
Reading the text further down, Teor writes: "...the defence will
trigger. Then both instances will try another relay..." - so I am
pretty sure that this is not a "banhammer" that would kick me off Tor
entirely; instead it's more of a "choke".
# Re: EOTK...
Any given EOTK "worker" currently gets 2x tor daemons provisioned to
it; this is partly to make better use of the compute bandwidth
available on a multiprocessor box, and partly to mitigate exant
problems of Tor daemons going "stale" - having 2x daemons makes it
less likely to lose the entire worker.
Also: I am looking at #24973 "Tor should be more gentle when launching
dozens of circuits at once" - which resonates with my worker pool: I
am running 6x2 = 12 worker-daemons (and that figure may double, soon)
not to mention all the other tor daemons that I am running (eg: TBB,
OnionBalance)
Also: per #24992, I have seen quite a lot of "exceeded launch limit
with 10 intro points in the last 180 seconds"-type messages, recently;
I am not sure why this has been happening because the tor daemons in
question have all been workers running a single HS in each. Perhaps it
was having problems establishing ANY introduction points at all,
because of the DDoS attack, but the failures still counted?
To summarise: all of this "bad" behaviour on my part, explains why I
am/was worried about getting blocked from Tor entirely. :-)
# Re: NYT...
I am still working with NYT to confirm their IP setup on AWS; until
now, one of the selling points of enterprise onion networking for me
has been the capability to keep everything in one/more "NAT enclaves"
which - combined with SingleOnion - mitigates inbound-connection
attacks like flooding, whilst permitting "horizontal scaling" which is
important with a rewriter-proxy like EOTK.
I'm a little bummed that that may no longer be viable / that in order
to do Tor at high bandwidth then your worker fleet will need to have
separate and visible IP addresses, because DDoS attribution; however,
this is an evolving environment so I suppose that we'll adapt.
In the case of AWS this likely means buying "Elastic IPs" for each
worker.
# Speculation...
Aaron wrote: "Because the circuit-creation limit is applied at the
guard, wouldn't this affect hidden sevices instead of single onion
services?"
Teor wrote: "It will only trigger if hundreds of guard-using clients
are behind a single IP address."
This exchange makes me think a couple of things:
a) if my EOTK development farm was using NormalOnions not
SingleOnions, then I might actually be in a worse situation because I
would be more heavily regulated... in fact:
b) I am wondering if might leverage the DDoS mitigations
to knock non-SingleOnion onionsites off-air? The attack would be
Re: [tor-onions] DDoS, Single Onion Services and IP Addresses
On Fri, Feb 02, 2018 at 02:23:24PM +1100, teor wrote: > For IP addresses with 3 or more connections to a single guard, the guard > imposes > a limit of 1 circuit every 3 seconds, with a 90 circuit burst allowance. 3 circuits every 1 seconds, actually. Think of it like a token bucket with a size of 90 circuits and a refill rate of 3 per second. > If it happens, and if they build more than 90 circuits to the same relay, > the defence will trigger. Then both instances will try another relay. I think Tor clients who have all their create cells responded to with destroy cells won't abandon that relay. That is, getting a destroy cell in response to a create cell is not an indication that the relay is broken, so it won't convince us to stop trying that one. That "feature" is actually part of the calculus here, since we want to think very carefully about how our choices shape the behavior of the millions of enthusiastic high-bandwidth Tor clients that are overwhelming the network. > > Because the circuit-creation limit is applied at the guard, wouldn???t this > > affect hidden sevices instead of single onion services? > > It will only trigger if hundreds of guard-using clients are behind a single > IP address. I expect a popular onion service that doesn't use guards and that runs many Tor instances on the same IP address will trigger the defense often: because it doesn't use guards, each new circuit it builds in response to a rendezvous request will pick an entry point at random, and if some of the conversations with clients last for a while, then outgoing connections will accumulate, eventually reaching the threshold for each relay to decide that that address is being unfair. --Roger ___ tor-onions mailing list tor-onions@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-onions
Re: [tor-onions] DDoS, Single Onion Services and IP Addresses
Experimental DoS mitigation is in tor master https://lists.torproject.org/pipermail/tor-relays/2018-January/014357.html Full thread above, some potential affected usage below... https://lists.torproject.org/pipermail/tor-relays/2018-February/014393.html https://lists.torproject.org/pipermail/tor-relays/2018-February/014396.html ___ tor-onions mailing list tor-onions@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-onions
Re: [tor-onions] DDoS, Single Onion Services and IP Addresses
Hi, I didn't give much detail here: >> On 2 Feb 2018, at 11:08, Alec Muffett wrote: >> >> The current limit is 2 connections per IP address. >> This affects single onion services, because they don't use guards. >> >> Can you please make sure that you only have one or two Single Onion >> Services on each outbound IP address? So I'm going to quote from my email to tor-relays earlier today: > Here are the mitigations again: > >> o Major features: >> - Give relays some defenses against the recent network overload. We >> start with three defenses (default parameters in parentheses). >> First: if a single client address makes too many connections >> (>100), hang up on further connections. Second: if a single client >> address makes circuits too quickly (more than 3 per second, with >> an allowed burst of 90) while also having too many connections open >> (3), refuse new create cells for the next while (1-2 hours). > > We could patch clients so they never exceed this number of circuits > by default. But that would penalise large clients that have a dedicated IP > address. For Alec's home development use case, limiting client bursts may be helpful. >> Third: (DoS mitigation) >> if a client asks to establish a rendezvous point to you directly, >> ignore the request. These defenses can be manually controlled >> by new torrc options, but relays will also take guidance from >> consensus parameters, so there's no need to configure anything >> manually. Implements ticket 24902. This shuts down Tor2web, which is a major source of the current load. >> On 2 Feb 2018, at 03:04, David Goulet wrote: >> > On 01 Feb (04:01:10), grarpamp wrote: Applications that use a lot of resources will have to rate-limit themselves. Otherwise, relays will rate-limit them. >>> >>> It's possible if relays figure that stuff by #2 might not be >>> an attack per se, but could be user activities... that relays >>> might push back on that one by... >>> - Seeking significantly higher default values committed >>> - Seeking default action committed as off >>> - Setting similar on their own relays if commits don't >>> work. And by not being default off, it should be prominently >>> documented if #2 affects users activities [1]. >> >> That I agree. We've set up default values for now and they will probably be >> adapted over time so for now this is experimental to see how much we make >> people unhappy (well except for the people doing the DoS ;). >> >> But I do agree that we should document some "real life" use cases that could >> trigger defenses at the relay in some very public way (blog post or wiki) >> before this goes wide in the network. Large amount of tor clienst behind NAT >> is one I have in mind, IPv6 as well... > > We did some analysis when we were choosing these figures. > > It takes a few hundred clients behind an IP address, to have a 50% probability > of 3 clients choosing the same large guard. That's unusual. And if clients > see their > guard timeout, they will move to another guard. > > Here are some other scenarios: > > Peer-to-peer clients like Ricochet, when the user has >90 contacts, but only > when there are hundreds of other clients on the same IP address. > > Any Tor client that doesn't use guards. For example: > > Bridges with multiple users, but only when there are >=3 bridges per outbound > IP address. (This is unlikely, because bridges need their own IPv4 address. > If you have multiple bridges using the default route, and multiple IP > addresses, > set OutboundBindAddress on each bridge.) We will need to consider this issue > when we allow IPv6 bridges without a public IPv4 address. Perhaps an > appropriate solution is to make bridge clients use vanguards. > > Multiple (>=3) Tor2web or single onion services using separate tor instances, > behind a single IP address, making large numbers of circuits. This is a likely > source of our current issues. Alec wrote: > I think the NYT is okay (separate IPs?) I hear Facebook is ok as well. > but if I understand this right, this is going to hamper EOTK development, > since I have ~ 12 worker onions spread over 6 quad-core machines, and then > publish up to 10 additional "service" addresses via OnionBalance ... all > behind my single DSL NAT firewall that protects them from inbound traffic. For IP addresses with 3 or more connections to a single guard, the guard imposes a limit of 1 circuit every 3 seconds, with a 90 circuit burst allowance. Are these single onion services or do they use guards? If they are single onion services, then the probability that 3 of your 13? instances will choose the same relays for their directory, HSDir, intro or rend points is small (< 10%). If it happens, and if they build more than 90 circuits to the same relay, the defence will trigger. Then both instances will try another relay. You should probably limit your circuit construction rate if
Re: [tor-onions] DDoS, Single Onion Services and IP Addresses
I am not going to pretend that I fully understand the DDoS mitigations yet, but experience at two jobs has taught me that at least three entire countries essentially present themselves from behind small numbers of heavily NATed addresses, so I hope that the mitigations are NAT-friendly. ISTR that UAE and Singapore are two such, I forget the third? -a On 2 Feb 2018 00:10, "A. Johnson" wrote: Because the circuit-creation limit is applied at the guard, wouldn’t this affect hidden sevices instead of single onion services? ___ tor-onions mailing list tor-onions@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-onions
Re: [tor-onions] DDoS, Single Onion Services and IP Addresses
Because the circuit-creation limit is applied at the guard, wouldn’t this affect hidden sevices instead of single onion services? Aaron > On Feb 1, 2018, at 7:08 PM, Alec Muffett wrote: > > > The current limit is 2 connections per IP address. > This affects single onion services, because they don't use guards. > > Can you please make sure that you only have one or two Single Onion > Services on each outbound IP address? > > I think the NYT is okay (separate IPs?) but if I understand this right, this > is going to hamper EOTK development, since I have ~ 12 worker onions spread > over 6 quad-core machines, and then publish up to 10 additional "service" > addresses via OnionBalance ... all behind my single DSL NAT firewall that > protects them from inbound traffic. > > Hmmm... > > - a > > ___ > tor-onions mailing list > tor-onions@lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-onions ___ tor-onions mailing list tor-onions@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-onions
Re: [tor-onions] DDoS, Single Onion Services and IP Addresses
The current limit is 2 connections per IP address. This affects single onion services, because they don't use guards. Can you please make sure that you only have one or two Single Onion Services on each outbound IP address? I think the NYT is okay (separate IPs?) but if I understand this right, this is going to hamper EOTK development, since I have ~ 12 worker onions spread over 6 quad-core machines, and then publish up to 10 additional "service" addresses via OnionBalance ... all behind my single DSL NAT firewall that protects them from inbound traffic. Hmmm... - a ___ tor-onions mailing list tor-onions@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-onions
Re: [tor-onions] DDoS, Single Onion Services and IP Addresses
? On Thu, Feb 1, 2018 at 4:35 PM teor wrote: > Hi All, > > Is this list still working? :-) > > One of our new DDoS defences stops Tor clients from building lots of > circuits over multiple connections from the same IP address. > > The current limit is 2 connections per IP address. > This affects single onion services, because they don't use guards. > > Can you please make sure that you only have one or two Single Onion > Services on each outbound IP address? > > If you have multiple IP addresses on your machine, you can set > OutboundBindAddress. Otherwise, Tor will use the default route. > > T > ___ > tor-onions mailing list > tor-onions@lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-onions > ___ tor-onions mailing list tor-onions@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-onions
