Re: [tor-relays] Tor Exit Relay CPU Usage Running at 100% for 1 MB/s on FreeBSD

[Dhalgren Tor]

https://trac.torproject.org/projects/tor/ticket/25461
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Let's increase the amount of exit relays doing DNSSEC validation

[Dhalgren Tor]

Respectfully, I disagree.

https://lists.torproject.org/pipermail/tor-relays/2015-October/007904.html

Thank you for the thought however.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] CPU saturation attack/abuse

[Dhalgren Tor]

Found other ones:  December 24 where egress was much higher then
ingress (but crypto-workers were pegged, not main thread).  December
28 & 29, attack like today.  Feburary 1 & 2, like today with ingress
higher than egress.

In today's and the latter-two above the main event thread was pegged
either continuously or intermittently with ingress higher than egress.

Just looked again and see all threads, crypto-worker and main-event
pegging episodically.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] CPU saturation attack/abuse

[Dhalgren Tor]

On Sun, Mar 4, 2018 at 7:06 PM, Toralf Förster <toralf.foers...@gmx.de> wrote:
> On 03/04/2018 07:41 PM, Dhalgren Tor wrote:
>>  the main event-worker thread
>> going from a normal load level of about 30%/core to 100%/core and
>> staying there for about 30 seconds;
> I do wonder if this is just the normal behaviour when - IIRC correctly - 
> consensus documents are compressed before sending.

No chance whatsoever.  Relay runs for months-on-end never exceeding
40% CPU.  Have seen the same or a similar attack, twice before I
believe under 0.2.9.14.  Just realized the ISP added some bugs to
their data graphs:  in this case _ingress_ traffic is 3-4% higher than
egress (they reversed the labels along with breaking long-term
historical).  Earlier observed a similar attack where _egress_ traffic
was 10-15% higher than ingress traffic.

What's interesting here is the crypto-worker threads are near zero
(normal) in contrast to circuit-extend attacks where the crypto
threads peg at 100%.  Did see one brief, intense crypto-
worker CPU spike today but it's not characteristic of this event in general.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] CPU saturation attack/abuse

[Dhalgren Tor]

Upgraded exit to 0.3.3.3 and now seeing a curious CPU saturation
attack.  Whatever the cause, result is the main event-worker thread
going from a normal load level of about 30%/core to 100%/core and
staying there for about 30 seconds; then CPU consumption declines back
to 30%.  Gradual change on ascent and decent.  Another characteristic
is egress traffic slightly higher than ingress traffic, perhaps 3-4%,
where normally egress and ingress flows match precisly.  Checked
browsing via the node and performance seems fine--no obvious
degradation.  Elevated NTor circuit creation rates as-of the last
heartbeat, from roughly 300k to 700k per-report, but not extreme (at
least in a relative sense since late December).

Anyone else observed this?  Have any idea how the attack works?

Captured a debug-level log of a cycle from normal load to
full-on-attack but won't have time to analyzed it for a couple of
weeks.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Detecting Network Attack [re: exit synflooded]

[Dhalgren Tor]

>Well, it's still going on, and is pretty much ruining Libero :( .  Running 
>CentOS 6, here.
>
>When it's happening it can look like this:
>
># netstat -n | grep -c SYN
>17696

I run a fast exit and can offer some advice:

1) mitigate bug #18580 (also #21394); is a DNS denial-of-service and
could be the problem.  either upgrade to 0.3.2.4+ or edit resolve.conf
per 
https://trac.torproject.org/projects/tor/wiki/doc/DnsResolver#TuningeventdnscomponentofTorDaemon

also check out https://arthuredelstein.net/exits/

2) if you continue to experience excessive outbound scanning SYNs,
I've found that simply enabling connection tracking helps by
implicitly limiting the rate a which connections can originate.  If an
iptables "-m state --state ESTABLISHED,RELATED -j ACCEPT" rule exists
it will turn it on or you could modprobe the module if you don't want
to configure incoming connection rules.

some useful sysctl.conf changes, run sysctl -p after or reboot

# Tor Exit tuning.
net.ipv4.ip_local_port_range = 1025 65535
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_max_tw_buckets = 4194304
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 3
net.netfilter.nf_conntrack_tcp_timeout_established = 1000
net.netfilter.nf_conntrack_checksum = 0

you might see many messages:

kernel: nf_conntrack: table full, dropping packet

which indicates no connection table slots were available for an
outbound connection and that rate limiting is effected

state of connection tracking appears in /proc/net/nf_conntrack
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] About relay size

[Dhalgren Tor]

FWIW I run a 100TB (150mbps) exit and am convinced that this size
provides better quality exit connectivity than the highest ranked
monster bandwidth relays.  The biggest relays attract the greatest
blacklist treatment due to the volume of abuse emanating from them.

As a user of Tor I frequently observe connection attempts commencing
with a handful of top-rated relays, only to work down to a
mid-bandwidth relay such as mine before successfully completing.

For this reason I would like to see the exit selection algorithm bias
somewhat away from relays rated over 5.  Is annoying to wait 30
seconds for the discovery of a usable path.

Hard 100mbps connections will be easier to manage than the 100TB class
of service, as the latter is usually over a 1GB link with a FUP (fair
use policy) associated with it and extravagant surcharges for
exceeding the limit.


>Hi Tor list,
>
>I'm already running a small exit node (100Mbps bandwidth) and I'm ready
>to spend more money on it, so have a question for you guys :
>
>Is it better if I run other small ones (100Mbps too) or only 1 big exit
>relay (1 Gbps) ? What's best for the network stability/security ?
>
>Thanks a lot
>
>IPonU
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] OpenSSL Padding oracle in AES-NI CBC MAC check (CVE-2016-2107)

[Dhalgren Tor]

https://www.openssl.org/news/secadv/20160503.txt

In general I understand that padding oracle attacks are principally a
hazard for browser communications.  Am assuming that updating OpenSSL
for this fix is not an urgent priority for a Tor Relay.

If anyone knows different please comment.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] important DNS tuning for high volume exit relays, fix for Unbound DNS DOS problem

[Dhalgren Tor]

I believe I now understand the cause of exit relay failure when
Unbound is the resolver and GoDaddy null-routes the exit.

Both to prevent this DOS from taking out your relay if Unbound is
running and to maximize DNS performance:

with a local instance of Unbound running /etc/resolv.conf should look like

   options timeout:5 attempts:1 max-inflight:16384 max-timeouts:100
   nameserver 127.0.0.1

with a local instance of 'named' running /etc/resolv.conf should look like

   options timeout:5 attempts:2 max-inflight:16384 max-timeouts:100
   nameserver 127.0.0.1

background material for the above recommendations found at

https://trac.torproject.org/projects/tor/ticket/18580#comment:11
https://unbound.net/pipermail/unbound-users/2016-April/004301.html
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] GoDaddy null-routing Tor Exit IPs

[Dhalgren Tor]

FYI Tor-Relays

GoDaddy AS26496 is null-routing selected Tor Exits, presumably in
response to abuse originating from them.  Know of two thus far

FE67A1BA4EF1D13A617AEFB416CB9E44331B223A 2016/01/26 ashtrayhat3
A0F06C2FADF88D3A39AA3072B406F09D7095AC9E 2016/03/16 Dhalgren

though probably more exist.  Appears to affect faster relays.
Affected relays running 'unbound' for DNS may experience total DNS
request freeze and should switch to 'named' to mitigate the failure.
Null-route seems to be manual and persist for indeterminate time.

Null-routed IPs will not be able to ping or connect to address listed
in the report at
http://www.cidr-report.org/cgi-bin/as-report?v=4=AS26496

Information on how this was discovered can be found at

#18580: exit relay fails with 'unbound' DNS resolver when lots of
requests time-out
https://trac.torproject.org/projects/tor/ticket/18580
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] unbound bogs down strangely, degrading exit relay

[Dhalgren Tor]

Possibly this incident is the result of some malware attempting to use
some sort of domain "fast flux" or DGA algorithm.  Seems improbable
anyone would be dumb enough to try to DDOS GoDaddy DNS using Tor.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] unbound bogs down strangely, degrading exit relay

[Dhalgren Tor]

As with the earlier incident, problem came back within hours of
restarting the daemons.

Was able to figure out what's happening  Operators running 'unbound' take note!

Problem appears to be the result of someone attempting to DDOS a DNS
service, in this case GoDaddy.

Ran

   lsof -Pn -p 

a few times and observed numerous SYN_SENT TCP connections, of of them
to 208.109.255.0/24, where GoDaddy DNS servers are found.  Appears
GoDaddy is rate-limiting or blocking requests from the 'unbound'
instance on the relay IP.

Ran

   unbound-control dump_requestlist

and see a large queue of requests to GoDaddy.  Finally ran

   unbound-control dump_infra >infralst

and see 14000 lines similar to

   208.109.255.26 cycsErvicioSsAS.coM. expired rto 12

indicating a huge number of requests have been made to GoDaddy and
have expired after 120 seconds.

Presently the quantity of requests has fallen off and the exit is
operating fine.  Have alarmed the tell-tale log message.  When it
recurs I expect

   unbound-control purge_requestlist

will mitigate the problem.  Presently looking into configuring
'ratelimit' feature of 'unbound'.  If anyone has already done this
successfully please post to this thread.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] unbound bogs down strangely, degrading exit relay

[Dhalgren Tor]

Problem came back again while I was working on the exit.

unbound-control purge_requestlist

does not help but it appears that

   unbound-control purge_infra
   unbound-control purge_requestlist

will clear up the problem without requiring a daemon restart--at least
temporarily.

Also tried setting

   do-tcp: off

but this did not appear to make a difference.


Seems to me a degenerate interaction between tor's 'eventdns'
subsystem and 'unbound' comes into play when this DNS flood/attack
occurs.  Have an 'info' level log with SafeLogging=0 for a few minutes
where the relay was in the bogged-down state and was failing to
service Tor Browser requests.  If developer is interested in taking a
look at this please contact me directly.

This issue is a PIA and if it continues I'll give up on 'unbound' and
follow the previous operator, switching to bind9 despite the lesser
performance.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] fast exits running 'unbound' should switch to 'named'

[Dhalgren Tor]

Nothing wrong with 'unbound'.  Problem is bug in Tor daemon
interaction with 'unbound' that brings exit effectively offline when
GoDaddy blocks requests from it.  This can happen to any fast exit
anytime.  GoDaddy has been blocking high-volume DNS requesters since
2011, and recent activity by some Tor user enumerating GoDaddy domains
triggers the failure.

On Sat, Mar 19, 2016 at 5:53 AM, Michael McConville <mm...@mykolab.com> wrote:
> Dhalgren Tor wrote:
>> Bug #18580: exit relay fails with 'unbound' DNS resolver when lots of
>> requests time-out
>>
>> https://trac.torproject.org/projects/tor/ticket/18580
>
> I wouldn't go that far. I've been running with Unbound for ~1.5 years
> now (AFAIK) without issue. I've also been told that the code is of
> significantly higher quality and that it's considerably more secure.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] unbound bogs down strangely, degrading exit relay

[Dhalgren Tor]

Hit a repeat of an earlier incident:

https://lists.torproject.org/pipermail/tor-relays/2016-January/008621.html

message from tor daemon is

   Resolved [scrubbed] which was already resolved; ignoring

About 5400 of these messages over 37 hours, during which the relay
dropped down to 30% of usual load and did not work for Tor Browser
when the exit was specifically set.  Unlike the earlier incident,
DNSSEC was disabled.  Probably the IP address for root-server H was
incorrect because it changed in December and 'unbound' was built in
September.  Just installed and configured the current 'named.root'
hints file.  Both "iterator" and "validator" modules were loaded,
though validation was disabled.  Changed the config to load only
"iterator".

Unbound 1.5.4, tor 0.2.6.10.  Running 164 days.

If I see this again I'm planning to attempt purging the 'unbound'
request queue with

   unbound-control dump_requestlist >reqlist.snap

   unbound-control flush_requestlist

and if that doesn't work, bouncing 'unbound' without stopping the 'tor' daemon.

Interesting stats were recorded, starting with the good pre-incident entries:

   Mar 11 05:53
   notice: sendto failed: Invalid argument
   notice: remote address is 0.0.0.1 port 53
   ...repeated 4 times

   Mar 15 17:45
   server stats for thread: 1029423 queries, 415824 answers from
cache, 613599 recursions, 0 prefetch
   server stats for thread: requestlist max 132 avg 22.5692 exceeded 0 jostled 0
   average recursion processing time 4.070100 sec

   Mar 16 05:46
   Resolved [scrubbed] which was already resolved; ignoring

   Mar 16 17:45
   server stats for thread: 1162172 queries, 287662 answers from
cache, 874510 recursions, 0 prefetch
   server stats for thread: requestlist max 421 avg 198.684 exceeded 0 jostled 0
   average recursion processing time 25.833646 sec

   Mar 16 21:08
   notice: sendto failed: Invalid argument
   notice: remote address is 0.0.0.1 port 53
   ...repeated 4 times

   Mar 17 16:52
   notice: sendto failed: Invalid argument
   notice: remote address is 0.0.0.1 port 53
   ...repeated 4 times

   Mar 17 17:45
   server stats for thread: 1078496 queries, 144557 answers from
cache, 933939 recursions, 0 prefetch
   server stats for thread: requestlist max 459 avg 296.668 exceeded 0 jostled 0
   average recursion processing time 40.621863 sec

   service stopped (unbound 1.5.4).

   Mar 17 19:28
   server stats for thread: 53991 queries, 3455 answers from cache,
50536 recursions, 0 prefetch
   server stats for thread: requestlist max 342 avg 290.977 exceeded 0 jostled 0
   average recursion processing time 35.947563 sec

If anyone has seen this or knows anything please comment.  Tried
searching but came up with nothing but thread referenced above.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] unbound bogs down strangely, degrading exit relay

[Dhalgren Tor]

Gave up switched to 'named' and now it's working fine.

Entered BUG: https://trac.torproject.org/projects/tor/ticket/18580

Be advised, anyone running a fast exit with 'unbound' should switch to
using 'named'.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] webiron requesting to block several /24 subnet

[Dhalgren Tor]

FYI Webiron ceased sending these for my relay sometime between 11/24
and today (no reports for 11/25-27).

Possibly this is because I never look at or resolve the reports and
their system eliminates non-responding addresses to avoid listing by
spam honeypots.

If you wish to continue receiving these I suggest marking them
resolved--at least some of time.  In my case the cessation on this
path is desirable since the ISP has an automated system.

Or possibly Webiron has decided to no longer send reports to the
reverse-DNS abuse@ path, in which case this source of intelligence is
lost.

However one can view the Webiron abuse reporting history for an IP on
their web site using the link https://www.webiron.com/abuse_feed/ and
this would also serve as a way to establish if the abuse-desk has
arrived at the optimal approach to Webiron, i.e. ignoring them.


On Mon, Nov 16, 2015 at 11:36 PM, Dhalgren Tor <dhalgren@gmail.com> wrote:
>>. . .I have to understand how my ISP reacts to this kind of things.
>
>>For the moment I will keep a low profile and I will block the
>>mentioned IP range for a month.
>
> Webiron's system sends notifications to both the abusix.org contact
> for the IP and to ab...@base-domain.tld for the reverse-DNS name of
> the relay IP.  So if you can configure abuse@ for the relay domain to
> forward to you, you will see their notices at the same time as the ISP
> abuse desk.  Might be helpful to know about it before they contact you
> and/or to see if they become familiar enough with the notices to
> ignore them.  Automated abuse complaints from other sources do not
> always go to the domain-based address.
>
> http://multirbl.valli.org/
>
> is a handy resource that shows the abuseix.org and abuse.net
> information, as well as how many DNSBLs the relay has racked up.  You
> can change the abuse.net contact but Webiron appears to ignore this
> source and simply construct the abuse@ from the rDNS domain name.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] webiron requesting to block several /24 subnet

[Dhalgren Tor]

>. . .I have to understand how my ISP reacts to this kind of things.

>For the moment I will keep a low profile and I will block the
>mentioned IP range for a month.

Webiron's system sends notifications to both the abusix.org contact
for the IP and to ab...@base-domain.tld for the reverse-DNS name of
the relay IP.  So if you can configure abuse@ for the relay domain to
forward to you, you will see their notices at the same time as the ISP
abuse desk.  Might be helpful to know about it before they contact you
and/or to see if they become familiar enough with the notices to
ignore them.  Automated abuse complaints from other sources do not
always go to the domain-based address.

http://multirbl.valli.org/

is a handy resource that shows the abuseix.org and abuse.net
information, as well as how many DNSBLs the relay has racked up.  You
can change the abuse.net contact but Webiron appears to ignore this
source and simply construct the abuse@ from the rDNS domain name.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] HoneyPot?

[Dhalgren Tor]

Is the end of the month.  Maybe they ran out of bandwidth and will be
back 11/1.  LeaseWeb over-limit rates are terrifying.

BTW the exit policy includes 443.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] webiron requesting to block several /24 subnet

[Dhalgren Tor]

>snake oil service like webiron

A most excellent characterization!

As a sales maneuver WebIron has been grandstanding
for months saying that Tor operators are "unwilling
to cleanup" when they know full-well that tor operators
can not / should not filter traffic due to minor brute-
force login attempts.  For contrast, Fail2ban in 2012
was modified to silently block login attacks without
spamming Tor operators in a reasonable gesture of
politeness.  SSH brute-force attacks have been a
fact-of-life for 20 years.  Hardly anyone thinks
it worth the effort to spam abuse@ desks over it
when a simple rate-limit does the job nicely.

WebIron has become so obnoxious that Spamhaus
placed their domain on the Domain Block List
last Friday

http://www.spamhaus.org/query/dbl?domain=webiron.com

and in addition, the IP of their reporting
system appears on the Spamhaus snow-shoe list
and thereby on the big-bazooka Zen DNSBL.

http://multirbl.valli.org/lookup/23.239.20.29.html

Spamhaus is highly conservative, highly respected
and only  lists egregious  unrepentant abusers.

The only reason to comply is in deference to
the overworked abuse-desk of your ISP if they
do not have an automated system for dealing
with this nonsense.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] LeaseWeb automated abuse notifications

[Dhalgren Tor]

Any exit operators with relays at LeaseWeb who are not enjoying the
new automated abuse-notice system requiring all complaints be acted
upon, send a message directly to the above address.  Have a solution.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] replace BandwidthRate with "traffic control" on busy routers

[Dhalgren Tor]

Routers running at or near a BandwidthRate setting offer terrible
latency and induce connection time-outs.  Refuse all DIR-port requests
with "HTTP/1.0 503 Directory busy, try again later ".

'tc' ingress filter is dramatically better for rate-limiting.

For case and 'tc' example see

final post

 https://lists.torproject.org/pipermail/tor-relays/2015-October/007901.html

initial post

https://lists.torproject.org/pipermail/tor-relays/2015-October/007863.html
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] excessive bandwidth assigned bandwidth-limited exit relay

[Dhalgren Tor]

 Was going to wait a few days before reporting back, but early results
are decisive.

The overload situation continued to worsen over a two-day period, with
consensus weight continuing to rise despite the relay often running in
a state of extreme overload and performing its exit function quite
terribly.

Applied the following to the  server, which has a single network interface:

tc qdisc add dev eth0 handle : ingress
tc filter add dev eth0 parent : protocol ip prio 10 u32 match ip
protocol 6 0xff police rate 163600kbit burst 6mb drop flowid :1

The above two lines put in effect a "traffic control" ingress policing
filter that drops TCP protocol packets in excess of 163.6 MBPS.  This
is 13.6% above the original rate-limit target of 18Mbyte/sec of
payload data.  UDP traffic is not restricted so the rule will not
impact most 'unbound' resolver responses nor 'ntpd' replies.

Statistics for the filter are viewed with the command

tc -s filter show dev eth0 root

With the initial settings the bandwidth filter discarded 0.25% of
incoming TCP packets--in line with what one sees in 'netstat -s'
statistics for a not-overloaded relay.  However the 'tor' daemon went
straight back into the throttling state and, while improving slightly,
continued to operate unacceptably.

Next BandwidthRate was set to parity with the 'tc' filter or
20Mbyte/sec.  Result was a dramatic increase in dropped packets to
1.5% and a dramatic improvement of the responsiveness of the relay.
If fact the exit now runs great.

The bandwidth measurement system now looks like a secondary issue.
The big problem is that Tor daemon bandwidth throttling sucks and
should be avoided in favor of Linux kernel rate-limiting where actual
bandwidth exceeds bandwidth allocated to Tor, whatever the motivation.
TCP is much better at dealing with congestion than the relay internal
limit-rate logic.

The above 'tc' filter is simplest possible manifestation and works
well for an interface dedicated to Tor.  More nuanced rules can be
crafted for situations where Tor traffic coexists with other types
such that a limit will apply only to the Tor traffic.

-

Note the applied 163.6mbps rate-limit is above the 150.0mbps rate that
works out to exactly 100TB per month of data.  The intent is to use
approximately 96% of the 100TB quota and the 'tc' rate will be
gradually fine-tuned to effect this outcome.  BandwidthRate and
BandwidthBurst are are now reset to the 1Gbyte no-effect default
value.

The 6mb buffer/burst chosen is an educated guess (very little
searchable info on this) and equates to 50ms at 1gbps or 300ms at
163mbps.  The number also corresponds with a bit more than what one
might reasonably expect as the per-port egress buffer capacity of a
enterprise-grade switch.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] how important is configuring DNSSEC root trust anchor for 'unbound' running on an exit node?

[Dhalgren Tor]

Spent a few minutes activating the DNSSEC trust-anchor for 'unbound'.

Ran 'dig' on a few signed domains and observed that queries that took
under 50 milliseconds without went to 2000 milliseconds with.

My attitude toward DNSSEC has deteriorated steadily over time and this
finishes it off for me.  It's simply not worth the cost.  Many serious
folk have commented in detail on what a horror show it is.

Disabled it on the exit.

Without DNSSEC, 'unbound' has been reporting:

server stats for thread 0: 1296326 queries, 454942 answers from cache,
841384 recursions, 0 prefetch
server stats for thread 0: requestlist max 112 avg 28.1553 exceeded 0 jostled 0
histogram of recursion processing times
[25%]=0.00737672 median[50%]=0.0492239 [75%]=0.144125
...
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] excessive bandwidth assigned bandwidth-limited exit relay

[Dhalgren Tor]

On 10/2/15, jensm1  wrote:
> You're saying that you're on a 1Gbit/s link, but you are only allowed to
> use 100Mbit/s. Is this averaged over some timescale?

More than 100MB which is 60 TB/month total for both directions.
Is 100 TB/month, a common usage tier.  Has a FUP (fair usage
policy) attached, which means that the bandwidth should be
evenly consumed over accounting interval rather than in
in bursts.

> If so, you could
> try and play around with the 'RelayBandwidthBurst' setting. Increasing
> the Burst might help reduce the queue delay when you're near saturation,
> assuming the traffic is not constant

Would make it worse, not better.  A higher burst rate will
allow the measurement to increase; the average limit
must stay the same regardless.  Best to set burst-max ==
average max.  Tor relays allow some bursting regardless
of the BandwidthRate setting anyway.

>and you're not over-saturated most of the time.

This is the problem and the assumption is not correct.
The link is saturated MOST of the time due to the
overrating.

> I don't know the measuring system, but I doubt that random packet
> dropping with iptables will have a noticeable effect on the measured
> bandwidth, as long as you don't drop enough packets to horribly degrade
> user experience.

NOT RANDOM.  iptables will drop packets above the
configured rate--exactly the same as a switch
port dropping egress packets in excess of 100 MBit,
a common lower-speed attachment type.

At present the bandwidth measurement system
assesses bandwidth-constrained links that drop
above-rate packets more correctly than unconstrained
links where the relay daemon rate-limits via
queuing and protocol flow-control.

To have the relay in question rated properly
the best idea so far is to add an iptables rate
limit to the mix.  Will test keeping the
BandwidthRate setting set slightly below the
iptables limit in case that works better than
an iptables limit alone.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] excessive bandwidth assigned bandwidth-limited exit relay

[Dhalgren Tor]

"So" indeed.  For the time that was under discussion:

cell-stats-end 2015-10-02 00:28:54 (86400 s)
cell-processed-cells 20220,420,72,18,8,4,1,1,1,1
cell-queued-cells 2.00,0.25,0.01,0.00,0.09,0.10,0.02,0.00,0.00,0.00
cell-time-in-queue 203,131,17,7,2832,6198,3014,802,21,26
cell-circuits-per-decile 126717

. . .horrible



On 10/1/15, Yawning Angel <yawn...@schwanenlied.me> wrote:
> On Thu, 1 Oct 2015 19:05:38 +
> Dhalgren Tor <dhalgren@gmail.com> wrote:
>> 3) observing that statistics show elevated cell-queuing delays when
>> the relay has been in the saturated state, e.g.
>>
>> cell-queued-cells 2.59,0.11,0.01,0.00,0.00,0.00,0.00,0.00,0.00,0.00
>> cell-time-in-queue 107,25,3,3,4,3,7,4,1,7
>
> So?
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] excessive bandwidth assigned bandwidth-limited exit relay

[Dhalgren Tor]

On Thu, Oct 1, 2015 at 1:10 PM, Tim Wilson-Brown - teor
 wrote:
>
> Can you help me understand what you think the bug is?

Relay is assigned a consensus weight that is too high w/r/t rate
limit.  Excess weight appears to be due to high quality of TCP/IP
connectivity and low latency of relay.  Result is overloaded relay and
poor end-use latency.

> Is Tor using more bandwidth that the BandwidthRate?

No, but relay is loaded to flat-line maximum and clearly is attracting
too many circuits.

> If so, this is a bug, and should be reported on the Tor Trac.

Not interested in doing that.  Looking for a way to get it work.

If the relay stays overloaded I'll try a packet-dropping IPTABLES rule
to "dirty-up" the connection.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] excessive bandwidth assigned bandwidth-limited exit relay

[Dhalgren Tor]

>Don't cap the speed if you have bandwidth limits. The better way to do it is 
>using AccountingMax in torrc. Just let it run at its full speed less of the 
>time and Tor will enter in hibernation once it has no bandwidth left.

Not possible.  Will violate the FUP (fair use policy) on the account.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] excessive bandwidth assigned bandwidth-limited exit relay

[Dhalgren Tor]

Have a new exit running in an excellent network on a very fast server
with AES-NI.  Server plan is limited to100TB so have set a limit
slightly above this (1800 bytes/sec) thinking that bandwidth would
run 80-90% of the maximum and average to just below the plan limit.
After three days the assigned bandwidth for the relay is going up
instead of moderating--looks like the measurement system has a problem
in this case.  Just dropped the limit to 1650 to stay below 100TB
with maximum load.

A good number appears to be around 65000 to 7, but 98000 was just assigned.

What can be done to extract proper rating from measurement system?
Tried setting TokenBucketRefillInterval to 10 milliseconds for more
exact control but this has not helped.  Should an IPTABLES
packet-dropping limit be established?  Can the rating system be fixed?
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] excessive bandwidth assigned bandwidth-limited exit relay

[Dhalgren Tor]

On Thu, Oct 1, 2015 at 12:55 PM, Tim Wilson-Brown - teor
<teor2...@gmail.com> wrote:
>
> On 1 Oct 2015, at 14:48, Dhalgren Tor <dhalgren@gmail.com> wrote:
>
> A good number appears to be around 65000 to 7, but 98000 was just
> assigned.
>
>
> Since I don’t have your relay fingerprint, I don’t know where you got this
> figure from.

FP is Dhalgren A0F06C2FADF88D3A39AA3072B406F09D7095AC9E

> Is it the consensus weight?

yes, this refers to the consensus weight.

> The consensus weight is a unitless figure. It does not represent a number in
> bytes per second.

yes, know that; the 65-70k ideal number is based on what similar,
properly loaded relays have.  Relay was assigned 73k yesterday and was
close to, but still above a proper load

> If so, that is a bug.

Does seem like a bug.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] excessive bandwidth assigned bandwidth-limited exit relay

[Dhalgren Tor]

On Thu, Oct 1, 2015 at 12:59 PM, s7r  wrote:
> Ouch, that's wrong.

I have it correct.  You are mistaken.

See https://www.torproject.org/docs/tor-manual.html.en

and read it closely.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] excessive bandwidth assigned bandwidth-limited exit relay

[Dhalgren Tor]

On Thu, Oct 1, 2015 at 1:33 PM, Tim Wilson-Brown - teor
<teor2...@gmail.com> wrote:
>
> On 1 Oct 2015, at 15:22, Dhalgren Tor <dhalgren@gmail.com> wrote:
>
> If the relay stays overloaded I'll try a packet-dropping IPTABLES rule
> to "dirty-up" the connection.
>
> Please reduce your BandwidthRate until your relay load is what you want it
> to be, or wait until the bandwidth authorities notice your relay is
> overloaded and reduce its consensus weight.

You are willfully missing the entire point here.

HAVE set the bandwidth limit and the measurement system is overrating
the relay despite this.

> Dropping packets using IPTABLES will actually increase your relay’s load due
> to retransmits, and degrade the performance of clients which use your relay.
> An IPTABLES rule would also degrade the performance of the overall Tor
> network due to these same retransmits.

I will give it a couple of days to normalize, but if the relay remains
overrated I will proceed with a IPTABLES packet-dropping rate limit.

The point is to have the measurement system set a rating that gives a
80-90% load for active circuits.  At that load no packets will be
discarded for regular Tor users.  The packet-dropping rate limit will
only impact measurement connections.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] excessive bandwidth assigned bandwidth-limited exit relay

[Dhalgren Tor]

>Maybe use this:
>
>MaxAdvertisedBandwidth

This setting causes the relay to limit the self-meausre value
published in the descriptor.  Has no effect on the measurement system.
Would be helpful if it did.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] excessive bandwidth assigned bandwidth-limited exit relay

[Dhalgren Tor]

This relay appears to have the same problem:

sofia

https://atlas.torproject.org/#details/7BB160A8F54BD74F3DA5F2CE701E8772B841859D


On Thu, Oct 1, 2015 at 12:33 PM, Dhalgren Tor <dhalgren@gmail.com> wrote:
> Have a new exit running in an excellent network on a very fast server
> with AES-NI.  Server plan is limited to100TB so have set a limit
> slightly above this (1800 bytes/sec) thinking that bandwidth would
> run 80-90% of the maximum and average to just below the plan limit.
> After three days the assigned bandwidth for the relay is going up
> instead of moderating--looks like the measurement system has a problem
> in this case.  Just dropped the limit to 1650 to stay below 100TB
> with maximum load.
>
> What can be done to extract proper rating from measurement system?
> . . .  Can the rating system be fixed?
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] excessive bandwidth assigned bandwidth-limited exit relay

[Dhalgren Tor]

On Thu, Oct 1, 2015 at 5:12 PM, Moritz Bartl <mor...@torservers.net> wrote:
> On 10/01/2015 06:28 PM, Dhalgren Tor wrote:
>> This relay appears to have the same problem:
>> sofia
>> https://atlas.torproject.org/#details/7BB160A8F54BD74F3DA5F2CE701E8772B841859D
>
> This is one of ours, and works just fine and the way it's supposed to?

Certainly 'sofia' is working well enough, but it's clearly spending
much if it's time at or somewhat above the configured rate-limit in
terms of load.  This is sub-optimal for end-user latency because the
relay delays traffic to enforce the rate limit.  On this relay
BandwidthBurst is unconfigured and perhaps setting it to the same
value as BandwidthRate will cause the authorities to slightly lower
the rating and eliminate the saturated state.

> Your 1800 is quite near the 16.5 MByte/s it is currently pushing
> since you must have changed something on Sept 26/27, so I don't really
> see the issue.

You are overlooking TCP/IP protocol bytes which add between 5 and 13%
to the data and are considered billable traffic by providers.  At 18M
it's solidly over 100TB, at 16.5M it will consume 97TB in 31 days.

>As said before in this thread, the consensus weight is a
> unitless unit that is relative to the rest of the network and of no
> 'external significance'.

YES I understand this.  Nowhere do I say I expect the consensus weight
to correspond directly to BandwidthRate.  What I SAID is that ,based
on comparative observation, the Dhalgren relay should be rated around
65000 to effect an approximate 90% utilization of the 18M limit.  THIS
is supposedly the intended design objective of the bandwidth
allocation system.

> If my quick calculation isn't off, 1800 gives you 42.4TB per
> direction, which means your relay will stay below the projected 100TB limit.

Add TCP/IP overhead.  I am looking at the service provider bandwidth
consumption graph when determining the setting as well as including
TCP/IP overhead in calculations.

> How exactly do you determine that you see "too many connections"? Do you
> have any errors in the Tor log?

I determine this by

1) watching the service provider bandwidth graph

2) watching the output of "SETEVENTS BW" on a control channel and
observing that every sample shows the relay is flat-line saturated at
BandwidthRate.

3) observing that statistics show elevated cell-queuing delays when
the relay has been in the saturated state, e.g.

cell-queued-cells 2.59,0.11,0.01,0.00,0.00,0.00,0.00,0.00,0.00,0.00
cell-time-in-queue 107,25,3,3,4,3,7,4,1,7

4) explicitly browsing through the relay utilizing "SETCONF
ExitNodes=" and observing that latency is at minimum degraded and is
sometimes terrible when the relay is overrated/saturated, while on the
other hand latency is extraordinarily good when the relay is not in a
saturated / rate-limited state.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] excessive bandwidth assigned bandwidth-limited exit relay

[Dhalgren Tor]

On Thu, Oct 1, 2015 at 7:45 PM, Steve Snyder  wrote:
>
> Another consumer of bandwidth is name resolution, if this is an exit node.  
> And the traffic incurred by the resolutions is not reflected in the relay 
> statistics.
>
> An exit node that allocates 100% of it's bandwidth to relaying traffic will 
> starve the resolver, and vice versa.

Absolutely true where physical bandwidth is the limiting factor.

However please note this thread/topic is explicitly in regard to
relays that have unconfined gigabit Ethernet connectivity in excellent
capacity networks, but that must limit bandwidth consumption in order
to avoid billing-plan overuse charges.

Loss of DNS resolver traffic is not a concern here.

In this specific case it appears that the Tor bandwidth allocation
system "over rates" subject relays to the point where the relays will
internally apply rate-limit throttling, thereby degrading end-user
latency.  This is not optimal and is undesirable.

As stated earlier in this thread, if the consensus weight of the relay
in question does not moderate within a few days and unless a better
idea materializes, an IPTABLES packet-dropping rate limit will be
applied.  This will cause the under-utilized gigabit connection to
behave similarly to a heavily utilized lower-bandwidth connection.
This should result in a lower authority rating that does not saturate
the relay, while still making use of the intended amount of bandwidth.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] excessive bandwidth assigned bandwidth-limited exit relay

[Dhalgren Tor]

On Thu, Oct 1, 2015 at 10:17 PM, Yawning Angel  wrote:
>
> Using IP tables to drop packets also is going to add queuing delays
> since cwnd will get decreased in response to the loss (CUBIC uses beta
> of 0.2 IIRC).

Unfortunately true.  Empirical arrival to a better result is the idea.

When saturated and rate-limiting, the relay sometimes
is so bad  connections time-out.  Consistent though
less-than-amazing performance is better than erratic
sometimes-failing performance IMO.  Tried a few
exit relays that appear to be limited by 100MB physical links
and have consensus weights around 60k (i.e. TCP congestion
control is at work though the load is not excessive) and
they function much better.

> It may be less queuing delay (note: write() completes the moment data
> is in the outgoing buffer kernel side, so it may not be as apparent,
> and is somewhat harder to measure), and it's your relay so I don't care
> what you do, so do whatever you think works.
>
> That said, placing an emphasis on unit-less quantities generated by a
> measurement system that is currently held together by duct tape,
> string, and chewing gum seems rather pointless and counter
>productive.

Really?

The consensus weight has a precise and predictable
effect on the amount of traffic directed to the relay.  So gaming
the measuring system for a weight that yields the best-possible
user experience is not "pointless."  I am paying for this.

Does seem the system generating the measurements has
problem and if someone can look at this issue that would
seem "productive."

Still interested in hearing "a better idea."
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] excessive bandwidth assigned bandwidth-limited exit relay

[Dhalgren Tor]

On Thu, Oct 1, 2015 at 11:41 PM, Tim Wilson-Brown - teor
 wrote:
>
> We could modify the *Bandwidth* options to take TCP overhead into account.

Not practical.  TCP/IP overhead varies greatly.  I have a guard that
averages 5% while the exit does 10% when saturated and more
when running in good balance.  Depends on the TCP packet
payload sizes and header options, which can  differ from
connection to connection.

> Alternately, we could modify the documentation to explicitly state that TCP
> overhead and name resolution on exits (and perhaps other overheads?) *isn’t*
> taken into account by those options. This would inform relay operators to
> take the TCP and DNS overheads for their particular setup into account when
> configuring the *Bandwidth* options, if the overhead is significant for
> them.

A good idea.  Easy to overlook even for the experienced
and a simple reminder goes a long way.

> You suggested TCP overhead was 5%-13%, I can include that in the manual.
> Do we know what fraction of exit traffic is DNS requests?

Seems relatively miniscule though I haven't dug into it.
Some resolver queries go over TCP while most are UDP.
'unbound' works so well I haven't spent much time with
it.  Have 'unbound' writing statistics to syslogd and
if I see anything useful will append it to this thread.
Some relay operators configure ISP DNS where
the cache resides on another machine.

> Are there any other overheads/additional traffic we should
> note while updating the manual?

Think that covers it.  Operator 'ssh' is nothing, relay traffic
the 1000 lb Gorilla.

> (Or would would you suggest that we update the code? I’m not sure how much
> this actually helps, as, once deployed to all relays, the consensus weights
> for all relays that set a *Bandwidth* options would come out slightly lower,
> and other relays without *Bandwidth* options set would take up the load.)

Per above, impossible to figure.  Better to remind
folks when they refer to the documentation.

What I do is have a script that writes 'ifconfig ethX' to a file
once a day, in sync with the service-provider accounting time-
zone.  Have an 'awk' script that calculates bytes for each day
--is very precise including all traffic and overheads.
The script subtracts-out MAC headers (using packet counts)
since the 14 bytes does not hit the WAN and is not billable.

Perhaps passing mention of 'ifconfig' statistics in the
manual is worthwhile.  'ip -s link show eth0X' truncates
byte counters (on some distros anyway) and is useless.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] how important is configuring DNSSEC root trust anchor for 'unbound' running on an exit node?

[Dhalgren Tor]

Is it important to configure the DNSSEC trust-anchor for an instance
of 'unbound' running on an exit node?  I put a lot of work into
setting up a new exit and want to take a break, but just noticed this
item. 'unbound' was built from source rather than installed from a
distribution, so this step must be performed manually.  The relay
resides in the high-quality German LeaseWeb network and the risk of
DNS mischief appears low.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays