Re: [tor-relays] connlimit: better to use "DROP" or "REJECT --reject-with tcp-reset"?

[Toralf Förster]

On 01/11/2018 02:10 AM, teor wrote:
> So if you're going to do this, please set a much higher limit than 2.
> I would suggest at least 4, but 10 or more is better.
> 
> You might be able to set it higher if you put a limit on repeated
> connection attempts.
 
The simple approach (allowing 8 syn requests from an address at ORport and at 
DirPort respectively) worked flawlessley for a while - just few dozen/hundreds 
DROPs per hour. Since yesterday however I get > 100K DROPs per hour.

Could a side effect of that traffic be that I lost the Exit flag ?

-- 
Toralf
PGP C4EACDDE 0076E94E



signature.asc
Description: OpenPGP digital signature
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] connlimit: better to use "DROP" or "REJECT --reject-with tcp-reset"?

[teor]

> On 10 Jan 2018, at 16:39, teor  wrote:
> 
> Hi,
> 
> Still having load trouble on your relay?
> Try dropping rapid connection attempts.
> 
>> On 9 Jan 2018, at 16:32, teor  wrote:
>> 
>> I've tried various ways of limiting Tor's RAM and CPU.
>> MaxAdvertisedBandwidth was effective, as was limiting Tor's file
>> descriptors and DisableOOSCheck 1. MaxMemInQueues had a minor impact.
>> 
>> So I decided to use a firewall to limit connections.
>> 
>> ...
>> 
>> So I set up this firewall rule:
>> 
>> /sbin/iptables -A INPUT -p tcp --syn ! --dport 22 -m connlimit 
>> --connlimit-above 100 -j DROP
>> 
>> You should replace 22 with the list of ports you use for SSH and other
>> important connections, just in case.
>> 
>> ...
> 
> This worked well, but Tor was still using a lot of CPU with its OOS checks.
> And it was using 4GB of RAM, which is good, but not sustainable on my machine.
> 
> Today, I added these firewall rules to drop rapid connection attempts
> from the same IP address, even if there are under 100 connections:
> 
> iptables -I INPUT -p tcp --syn ! --dport 22 -m state --state NEW -m recent 
> --set
> iptables -I INPUT -p tcp --syn ! --dport 22 -m state --state NEW -m recent 
> --update --seconds 60 --hitcount 100 -j DROP
> 
> They drop connection attempts after there have been 100 attempts in a minute.
> So if there were 100 clients, that would be 1 connection per client per 
> minute.
> 
> This reduced Tor's CPU usage and OOS warnings within a few minutes.
> I'm hoping RAM will go down over time.
> 
> I made the rules permanent using:
> 
> iptables-save > /etc/iptables/rules.v4
> 
> This might be Debian-specific.

I tried a few configs over the past week.

Now I have:
* MaxMemInQueues 2 GB
* 15000 file descriptors per tor instance
* DisableOOSCheck 0
* A limit of 20 established connections per IP
* A limit of 6 connection attempts per IP per minute

I left this over the weekend, and my relays are stable, and using:
* 3 GB - 6 GB RAM
* 5000 - 11000 file descriptors
* 50 - 120% CPU

They are also not logging too many OOS warnings or other warnings,
apart from the normal "assign to cpuworker failed" and "attempt to
establish rendezvous".

Thanks to everyone for your suggestions in this and other threads.

We are also working on a few different ways to limit the load in Tor.

T

--
Tim / teor

PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n




signature.asc
Description: Message signed with OpenPGP
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] connlimit: better to use "DROP" or "REJECT --reject-with tcp-reset"?

[teor]

> On 12 Jan 2018, at 08:46, Toralf Förster  wrote:
> 
>> On 01/11/2018 02:10 AM, teor wrote:
>> As far as I can tell, this single rule has the same effect:
> Even if " -P INPUT   DROP" is et ?

I think that applying the single rule to the default config,
has the same effect as applying all the rules you specified,
including " -P INPUT DROP".

But I'm not sure, I don't have much iptables experience.

T

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] connlimit: better to use "DROP" or "REJECT --reject-with tcp-reset"?

[Toralf Förster]

On 01/11/2018 02:10 AM, teor wrote:
> As far as I can tell, this single rule has the same effect:
Even if " -P INPUT   DROP" is et ?

-- 
Toralf
PGP C4EACDDE 0076E94E



signature.asc
Description: OpenPGP digital signature
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] connlimit: better to use "DROP" or "REJECT --reject-with tcp-reset"?

[Toralf Förster]

On 01/11/2018 02:10 AM, teor wrote:
> We allow 2 relays per IPv4 address, and each relay makes 1-2 connections
> to each other relay. (Or more, if the connections start failing. This is
> a bug we want to fix.)
> 
> So if you're going to do this, please set a much higher limit than 2.
> I would suggest at least 4, but 10 or more is better.
> 
> You might be able to set it higher if you put a limit on repeated
> connection attempts.
Right - good hint !

-- 
Toralf
PGP C4EACDDE 0076E94E



signature.asc
Description: OpenPGP digital signature
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] connlimit: better to use "DROP" or "REJECT --reject-with tcp-reset"?

[teor]

> On 11 Jan 2018, at 08:10, Toralf Förster  wrote:
> 
> On 01/10/2018 06:39 AM, teor wrote:
>> iptables -I INPUT -p tcp --syn ! --dport 22 -m state --state NEW -m recent 
>> --set
>> iptables -I INPUT -p tcp --syn ! --dport 22 -m state --state NEW -m recent 
>> --update --seconds 60 --hitcount 100 -j DROP
> 
> What's about the following approach;
>  IPT="/sbin/iptables"
> 
>  $IPT -P INPUT   DROP
>  $IPT -P OUTPUT  ACCEPT
>  $IPT -P FORWARD DROP
> 
>  # trust already established connections
>  #
>  $IPT -A INPUT --match conntrack --ctstate ESTABLISHED -j ACCEPT
>  $IPT -A INPUT --match conntrack --ctstate RELATED -j ACCEPT
>  $IPT -A INPUT --match conntrack --ctstate INVALID -j DROP
> 
>  # Tor
>  #
>  for p in 443 80
>  do
>$IPT -A INPUT -p tcp --syn --destination-port $p --match connlimit 
> --connlimit-above 2 --connlimit-mask 32 -j DROP
>$IPT -A INPUT -p tcp --destination-port $p -j ACCEPT
>  done
> 
> 
> Those rules should not prevent clients behind a NAT from accessing the relay 
> as long as the clients do not come in in parallel.

As far as I can tell, this single rule has the same effect:

 # Tor
 #
 for p in 443 80
 do
   $IPT -A INPUT -p tcp --syn --destination-port $p --match connlimit 
--connlimit-above 2 --connlimit-mask 32 -j DROP
 done


> Objections ?

Bridges and hidden services often come in parallel.
As do large client deployments in areas without much IPv4 allocation.

We allow 2 relays per IPv4 address, and each relay makes 1-2 connections
to each other relay. (Or more, if the connections start failing. This is
a bug we want to fix.)

So if you're going to do this, please set a much higher limit than 2.
I would suggest at least 4, but 10 or more is better.

You might be able to set it higher if you put a limit on repeated
connection attempts.

T

--
Tim Wilson-Brown (teor)

teor2345 at gmail dot com
PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n







signature.asc
Description: Message signed with OpenPGP
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] connlimit: better to use "DROP" or "REJECT --reject-with tcp-reset"?

[Toralf Förster]

On 01/10/2018 06:39 AM, teor wrote:
> iptables -I INPUT -p tcp --syn ! --dport 22 -m state --state NEW -m recent 
> --set
> iptables -I INPUT -p tcp --syn ! --dport 22 -m state --state NEW -m recent 
> --update --seconds 60 --hitcount 100 -j DROP

What's about the following approach;
  IPT="/sbin/iptables"
  
  $IPT -P INPUT   DROP
  $IPT -P OUTPUT  ACCEPT
  $IPT -P FORWARD DROP

  # trust already established connections
  #
  $IPT -A INPUT --match conntrack --ctstate ESTABLISHED -j ACCEPT
  $IPT -A INPUT --match conntrack --ctstate RELATED -j ACCEPT
  $IPT -A INPUT --match conntrack --ctstate INVALID -j DROP

  # Tor
  #
  for p in 443 80
  do
$IPT -A INPUT -p tcp --syn --destination-port $p --match connlimit 
--connlimit-above 2 --connlimit-mask 32 -j DROP
$IPT -A INPUT -p tcp --destination-port $p -j ACCEPT
  done
  

Those rules should not prevent clients behind a NAT from accessing the relay as 
long as the clients do not come in in parallel.

Objections ?

-- 
Toralf
PGP C4EACDDE 0076E94E



signature.asc
Description: OpenPGP digital signature
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] connlimit: better to use "DROP" or "REJECT --reject-with tcp-reset"?

[teor]

Hi,

Still having load trouble on your relay?
Try dropping rapid connection attempts.

> On 9 Jan 2018, at 16:32, teor  wrote:
> 
> I've tried various ways of limiting Tor's RAM and CPU.
> MaxAdvertisedBandwidth was effective, as was limiting Tor's file
> descriptors and DisableOOSCheck 1. MaxMemInQueues had a minor impact.
> 
> So I decided to use a firewall to limit connections.
> 
> ...
> 
> So I set up this firewall rule:
> 
> /sbin/iptables -A INPUT -p tcp --syn ! --dport 22 -m connlimit 
> --connlimit-above 100 -j DROP
> 
> You should replace 22 with the list of ports you use for SSH and other
> important connections, just in case.
> 
> And I installed iptables-persistent to save the rules.
> (It might be Debian-specific).
> 
> After I set up the firewall, the connections slowly dropped from 700
> down to around 100. It only affects new connections, so it might take
> a while.

This worked well, but Tor was still using a lot of CPU with its OOS checks.
And it was using 4GB of RAM, which is good, but not sustainable on my machine.

Today, I added these firewall rules to drop rapid connection attempts
from the same IP address, even if there are under 100 connections:

iptables -I INPUT -p tcp --syn ! --dport 22 -m state --state NEW -m recent --set
iptables -I INPUT -p tcp --syn ! --dport 22 -m state --state NEW -m recent 
--update --seconds 60 --hitcount 100 -j DROP

They drop connection attempts after there have been 100 attempts in a minute.
So if there were 100 clients, that would be 1 connection per client per minute.

This reduced Tor's CPU usage and OOS warnings within a few minutes.
I'm hoping RAM will go down over time.

I made the rules permanent using:

iptables-save > /etc/iptables/rules.v4

This might be Debian-specific.

T

--
Tim / teor

PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n




signature.asc
Description: Message signed with OpenPGP
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] connlimit: better to use "DROP" or "REJECT --reject-with tcp-reset"?

[teor]

> On 6 Jan 2018, at 06:05, Zack Weinberg  wrote:
> 
>> On Fri, Jan 5, 2018 at 1:44 PM, tor  wrote:
>> For relay operators using iptables connlimit to mitigate DoS attacks (or 
>> increased load from new clients), is it better for the Tor network to use 
>> "DROP" rules, or should we use something like "REJECT --reject-with 
>> tcp-reset"?
> 
> REJECT is friendlier to clients that are not misbehaving but happen to
> be caught in the crossfire, and to the Internet as a whole.
> 
> I personally think DROP should only ever be used as a desperation
> measure when the DoS load is so high that you can't even afford to
> send RSTs.

I've tried various ways of limiting Tor's RAM and CPU.
MaxAdvertisedBandwidth was effective, as was limiting Tor's file
descriptors and DisableOOSCheck 1. MaxMemInQueues had a minor impact.

So I decided to use a firewall to limit connections.

If I send RST, a Tor client will immediately try another guard.
If I DROP, the Tor client will timeout before trying another guard.

For misbehaving clients, I want to DROP.
For regular clients, I want to RST. (Or let them connect.)

So I want to set a high connection limit, and use DROP.

I see up to 700 connections per IP, and I see normal residential
IP addresses start at around 80. I don't think that clients with more
than 100 connections per IP will get much bandwidth anyway.

I used this command:

netstat -n | grep ESTABLISHED | awk '{print $5}' | cut -d: -f1 | sort | uniq -c 
| sort -rn | head

All the connections over 100 were at Hetzner, OVH, and Leaseweb.
(This command gives /16s for IPv6, but these clients aren't
using IPv6.)

So I set up this firewall rule:

/sbin/iptables -A INPUT -p tcp --syn ! --dport 22 -m connlimit 
--connlimit-above 100 -j DROP

You should replace 22 with the list of ports you use for SSH and other
important connections, just in case.

And I installed iptables-persistent to save the rules.
(It might be Debian-specific).

After I set up the firewall, the connections slowly dropped from 700
down to around 100. It only affects new connections, so it might take
a while.

T


--
Tim / teor

PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n




signature.asc
Description: Message signed with OpenPGP
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] connlimit: better to use "DROP" or "REJECT --reject-with tcp-reset"?

[Zack Weinberg]

On Fri, Jan 5, 2018 at 1:44 PM, tor  wrote:
> For relay operators using iptables connlimit to mitigate DoS attacks (or 
> increased load from new clients), is it better for the Tor network to use 
> "DROP" rules, or should we use something like "REJECT --reject-with 
> tcp-reset"?

REJECT is friendlier to clients that are not misbehaving but happen to
be caught in the crossfire, and to the Internet as a whole.

I personally think DROP should only ever be used as a desperation
measure when the DoS load is so high that you can't even afford to
send RSTs.

zw
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays