Re: [tor-relays] DoS attacks are real (probably)

[Scott Bennett]

Alex Xu  wrote:

> Quoting Felix (2017-12-11 17:07:30), as excerpted
> > Hi Alex
> > 
> > Great points.
> > 
> > > conntrack -L -p tcp --dport 9001 | awk '{print $5}' | sort | uniq -c 
> > > | sort -n
> > 
> > On FreeBSD one can do:
> > 
>
> yeah, the optimal rule would ban "bad IPs" after some threshold of
> connections, like "if one IP makes >1 conn/sec for at least 1 minute ban
> for 1 hour" or something. I'm hoping to fix the underlying issue in Tor
> so that low-bandwidth attacks like these are less effective.

 FWIW, the method that Felix posted should also work in DragonflyBSD
and NetBSD.  It may also work in OpenBSD, but the caveat is that the OpenBSD
project has continued to develop its implementation of pf, so I don't know
whether Felix's solution still works in OpenBSD.  The other three BSDs' pf
support has not been synchronized with that of the originating project
(OpenBSD) for many years.  Perhaps an OpenBSD tor relay operator can comment
here on this matter.


  Scott Bennett, Comm. ASMELG, CFIAG
**
* Internet:   bennett at sdf.org   *xor*   bennett at freeshell.org  *
**
* "A well regulated and disciplined militia, is at all times a good  *
* objection to the introduction of that bane of all free governments *
* -- a standing army."   *
*-- Gov. John Hancock, New York Journal, 28 January 1790 *
**
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] Is your IPv6 relay not Running?

[teor]

Hi,

Since October 2017, a majority of directory authorities check relay IPv6 
ORPorts.

If your relay is configured with an IPv6 ORPort:

ORPort [IPv6]:Port

and it is not reachable over IPv6, it will be excluded from the consensus.
This can happen if the address is wrong, or the IPv6 routing is wrong.

If your relay isn't in the consensus, it will log warnings.

Or, you can check Relay Search at:
https://atlas.torproject.org

If your relay is not Running, you can check the votes for your relay here:
https://consensus-health.torproject.org/consensus-health.html

If all the authorities don't vote for your relay, it has a tor or IPv4 problem.
If a few do vote Running, it probably has an IPv6 problem.
(If you want to be sure, check which authorities support ReachableIPv6
in the consensus health Known Flags table.)


If you have an IPv6 problem, check the address, and check the routing.

Here's what I do to check IPv6 on my relays when I set them up:

From the relay:
ping6 ipv6.google.com

From another IPv6 machine:
ping6 ipv6.google.com
ping6 (the relay's IPv6 address)
telnet (the relay's IPv6 ORPort)

We're working on having better support for IPv6 across Relay Search and
consensus health.

T___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] UbuntuCore stats update

[nusenu]

teor:
>> Chad MILLER:
>>> Torix, that's still true. Snaps restrict syscalls so tightly that switching
>>> users is not possible.
>>
>> Is it possible to start tor with a non-root user directly (without using
>> tor's user parameter to drop privileges)?
> 
> Yes, but you must pre-configure tor's directories with the correct user
> and permissions. Tor has strict requirements for private key security.

Generally speaking tor supports it (FreeBSD does it) but my question was
more towards Chad's tor snap package. Was your answer also for the snap?

thanks,
nusenu

-- 
https://mastodon.social/@nusenu
twitter: @nusenu_



signature.asc
Description: OpenPGP digital signature
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] DoS attacks are real (probably)

[null]

Hi Alex,

> This attack appears to be malicious to me. It seems to work like this:

> 1. Open many OR connections (hundreds to thousands)
> 2. Leave open until tor runs out of sockets

> Tor presently waits for the connections to time out, which takes 3-4.5
> minutes. It should instead more aggressively prune these garbage
> connections. https://trac.torproject.org/projects/tor/ticket/19984
> tracks this.


This is exactly what we saw as well. After implementing connection
limits (thanks again x9p) the problem mostly went away and our relays
have been stable since.

Thank you for opening the trac ticket. We agree it would be great if
this problem could be addressed in the Tor software if possible. In
the mean time we should probably be advocating for all relay operators
to implement connection limits. Put simply, without those limits,
relays are vulnerable to DoS.


> Evidence for this attack being malicious and intending to disable Tor is:


Agree with all 7 points you listed. We'd also add, there is additional
evidence that suggests some of the worst offenders (attacking IPs) are
actually orchestrated by a single entity (or perhaps multiple entities
working together). There are several commonalities across the
infrastructure used for these attacks. We identified and blocked (with
iptables DENY) the worst. To be clear, these IPs were not in the
consensus, and yes, mostly hosted by LeaseWeb.


> The referenced /16 block of guards is *not* part of this attack, and is
> simply poorly configured relays. you should not block that set,


Completely agree. We haven't blocked anything in the consensus.

Thanks.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] DoS attacks are real (probably)

[Alex Xu]

Quoting Felix (2017-12-11 17:07:30), as excerpted
> Hi Alex
> 
> Great points.
> 
> > conntrack -L -p tcp --dport 9001 | awk '{print $5}' | sort | uniq -c | 
> > sort -n
> 
> On FreeBSD one can do:
> 

yeah, the optimal rule would ban "bad IPs" after some threshold of
connections, like "if one IP makes >1 conn/sec for at least 1 minute ban
for 1 hour" or something. I'm hoping to fix the underlying issue in Tor
so that low-bandwidth attacks like these are less effective.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] DoS attacks are real (probably)

[Dr Gerard Bulger]

I am getting these warnings, not very often, and the exit (restricted) is 
working well otherwise:

"Dec 11 18:07:23.000 [warn] Tried to establish rendezvous on non-OR circuit 
with purpose Acting as rendevous (pending)"

Some posts about this elsewhere hinted this warning could be caused by attacks. 
  I am not seeing attacks otherwise.

Gerry

-Original Message-
From: tor-relays [mailto:tor-relays-boun...@lists.torproject.org] On Behalf Of 
Felix
Sent: 11 December 2017 17:08
To: tor-relays@lists.torproject.org
Subject: Re: [tor-relays] DoS attacks are real (probably)

Hi Alex

Great points.

> conntrack -L -p tcp --dport 9001 | awk '{print $5}' | sort | uniq 
> -c | sort -n

On FreeBSD one can do:

In packetfilter:

# play with the numbers but more than 64k per ip if possible set limit { frags 
7, src-nodes 7, states 7, table-entries
10 }

table  persist

# 2000 is super high. Rate limit 50 new connects per 5 secs # overload but not 
flush pass in on $if_ext inet proto tcp from any to $relay_ip port $or_port 
flags S/SA modulate state (max-src-conn 2000,max-src-conn-rate 50/5,overload 
)

As cronjob:

# release block after 10 minutes
pfctl -t blockOR -T expire 600

These measures protect your system. IMO for other or future cases we should 
keep the clients degree of freedom (researchers / fancy doers) as high as 
possible, being not too restrictive.


> 1. Open many OR connections (hundreds to thousands) 2. Leave open 
> until tor runs out of sockets

If the ip is saturated for like 2 hours the relay might loose the hsdir flag. 
But today there are not enough resources in the game to generate an issue for 
the network.


> I recommend against
> the blanket approach suggested previously of limiting whole sets of 
> /24s, since that may inadvertently block mobile clients and is not 
> effective against the current attack.

Right. In future one could put such loud clients besides useful ips a let the 
relays block the usefull.


> 2. the connections do not taper off if they are rejected. I banned 
> these addresses from accessing Tor, and they continue to make dozens 
> of connection attempts every second from each IP address. this means 
> that this is probably not a good faith "test" or a misconfigured set 
> of real Tor clients, but is instead malicious and using a modified or 
> custom client.

The above rule limits the useless attempts to a certain limit and recovers 
after 10 minutes. This protects but gives the 'offender' the chance to tune his 
client to a better behaviour (in case he wants it).


> 3c. it is almost certainly not real clients using NAT; as far as I 
> know, LeaseWeb does not use NAT, and Online.net only uses one-to-one NAT.

Good point. A general blocking rule should be smart enough to enable NAT 
clients anyway ?


--
Cheers, Felix
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] DoS attacks are real (probably)

[Felix]

Hi Alex

Great points.

> conntrack -L -p tcp --dport 9001 | awk '{print $5}' | sort | uniq -c | 
> sort -n

On FreeBSD one can do:

In packetfilter:

# play with the numbers but more than 64k per ip if possible
set limit { frags 7, src-nodes 7, states 7, table-entries
10 }

table  persist

# 2000 is super high. Rate limit 50 new connects per 5 secs
# overload but not flush
pass in on $if_ext inet proto tcp from any to $relay_ip port $or_port
flags S/SA modulate state (max-src-conn 2000,max-src-conn-rate
50/5,overload )

As cronjob:

# release block after 10 minutes
pfctl -t blockOR -T expire 600

These measures protect your system. IMO for other or future cases we
should keep the clients degree of freedom (researchers / fancy doers) as
high as possible, being not too restrictive.


> 1. Open many OR connections (hundreds to thousands)
> 2. Leave open until tor runs out of sockets

If the ip is saturated for like 2 hours the relay might loose the hsdir
flag. But today there are not enough resources in the game to generate
an issue for the network.


> I recommend against
> the blanket approach suggested previously of limiting whole sets of
> /24s, since that may inadvertently block mobile clients and is not
> effective against the current attack.

Right. In future one could put such loud clients besides useful ips a
let the relays block the usefull.


> 2. the connections do not taper off if they are rejected. I banned these
> addresses from accessing Tor, and they continue to make dozens of
> connection attempts every second from each IP address. this means that
> this is probably not a good faith "test" or a misconfigured set of real
> Tor clients, but is instead malicious and using a modified or custom
> client.

The above rule limits the useless attempts to a certain limit and
recovers after 10 minutes. This protects but gives the 'offender' the
chance to tune his client to a better behaviour (in case he wants it).


> 3c. it is almost certainly not real clients using NAT; as far as I know,
> LeaseWeb does not use NAT, and Online.net only uses one-to-one NAT.

Good point. A general blocking rule should be smart enough to enable NAT
clients anyway ?


-- 
Cheers, Felix
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] DoS attacks are real (probably)

[x9p]

On Mon, December 11, 2017 1:40 pm, Alex Xu wrote:
> tl;dr: run this:
>
> conntrack -L -p tcp --dport 9001 | awk '{print $5}' | sort | uniq -c |
> sort -n

Thanks for the detailed analysis.

> ignore numbers less than 10. the remaining output should consist of the
> following:

...

> are not NATed IPs, a high limit is not justified. I recommend against
> the blanket approach suggested previously of limiting whole sets of
> /24s, since that may inadvertently block mobile clients and is not
> effective against the current attack. As mentioned in the previous

I agree the approach of /24 connlimit is not a good approach to Exit
nodes. But for relays only worked fine for me and others.

cheers.

--
x9p | PGP : 0x03B50AF5EA4C8D80 / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE
1524 E7EE

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] DoS attacks are real (probably)

[Alex Xu]

tl;dr: run this:

conntrack -L -p tcp --dport 9001 | awk '{print $5}' | sort | uniq -c | sort 
-n

ignore numbers less than 10. the remaining output should consist of the
following:

1. your IP
2. LeaseWeb and Online.net IPs (use rDNS and whois)
3. mobile networks

block IPs in set 2 from accessing Tor using your firewall software.
don't block 1 or you will have problems. don't block 3 or other people
may have problems (hopefully more if the Android project gains
momentum). also don't block the /16 of guards mentioned. they are
causing no measurable harm. my list of set 2 is available upon request
to longstanding members (basically if you've done anything on Trac or
any tor mailing lists).

more information:

This attack appears to be malicious to me. It seems to work like this:

1. Open many OR connections (hundreds to thousands)
2. Leave open until tor runs out of sockets

Tor presently waits for the connections to time out, which takes 3-4.5
minutes. It should instead more aggressively prune these garbage
connections. https://trac.torproject.org/projects/tor/ticket/19984
tracks this.

In the interim, this attack is causing real problems, so this approach
seems reasonable for now. If you want to be slightly more conservative,
instead of blocking them outright, simply limit these IP subnets to a
small number of connections. Modern Tor only requires one, but you could
do two or three per address, or four or five per /28 or so. Since these
are not NATed IPs, a high limit is not justified. I recommend against
the blanket approach suggested previously of limiting whole sets of
/24s, since that may inadvertently block mobile clients and is not
effective against the current attack. As mentioned in the previous
paragraph, you should not set DisableOOSCheck 0, as that may wind up
killing good sockets instead.

again, this is not a good solution, but until that ticket is resolved,
it is probably the best that can be done.

Evidence for this attack being malicious and intending to disable Tor is:

1. these connections are coming very fast: as shown by others too,
dozens of connections per second per IP, and tens of thousands of
connections held open. the standard tor software multiplexes circuits
over a single TCP connection, so even very high-volume links should only
have a single Tor connection, or possibly two if they are very old and
must make a separate connection to the DirPort.

2. the connections do not taper off if they are rejected. I banned these
addresses from accessing Tor, and they continue to make dozens of
connection attempts every second from each IP address. this means that
this is probably not a good faith "test" or a misconfigured set of real
Tor clients, but is instead malicious and using a modified or custom
client.

3. they are spread out over many IPs, *but* only from *two* *hosting*
service providers. this means that:
3a. it is likely a single individual or organization, otherwise it would
be multiple providers
3b. they are trying to cause as much problems as possible, otherwise
they would use one server
3c. it is almost certainly not real clients using NAT; as far as I know,
LeaseWeb does not use NAT, and Online.net only uses one-to-one NAT.

4. rDNS is generic. this means they do not care enough to explain their
origin.

5. as pointed out, they have not registered themselves in the relay
consensus. this means it is probably not a set of high-volume relays
that is somehow malfunctioning, or someone conducting a DoS attack
through Tor itself.

6. as far as I can tell, these connections do not do anything. they
simply remain open, consuming resources until tor times them out.

7. they keep far more connections open than they make. what I mean by
this is that they hold open thousands of connections at once but only
send less than a hundred connections per second. this supports my theory
that it is not a large number of regular tor clients, but is instead
some custom client specifically for disruption.

additionally:

The referenced /16 block of guards is *not* part of this attack, and is
simply poorly configured relays. you should not block that set, but
instead block the set described above.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays