Re: [tor-relays] Exit relay operators please help test #2667 branch

[Roger Dingledine]

On Fri, Jan 29, 2021 at 12:34:28AM +0100, nusenu wrote:
> If dir auths (some or all) are willing to share (privately or publicly) the 
> distribution of
> attack load (frequency, bandwidth, ...) by exit source IP in total or 
> relative values
> I can correlate this data to strengthen a hypothesis that malicious/suspicious
> exits are involved to a greater extend than well-known long term exits.

I'll send you out-of-band a little snapshot of requests from relay
IP addresses -- 160k requests over a 24 minute period from yesterday
early evening.

At one point later in the evening I was getting several tens of millions
of requests per hour. That's when I started to realize that exit relay
operators were probably seeing this increased load too.

> That could mean that they are not (exclusively) attacking via but _from_ 
> servers that also happen to
> run tor exits. 

Well, there are definitely other addresses -- the overload from last
week was non-relay addresses, and that's still going.

It's possible that there are exits that are generating more than their
"fair" share of requests. I didn't see that pattern obviously happening,
and confirming it would be complicated by the fact that some relays
probably have less or more congestion, which would cause the attacks to
be more efficient or less efficient through them.

We had a long debugging session in #tor-dev on irc last night, and there
will be more of those as we proceed. We've found a bunch of short-term
fragile distinguishers, which we could use to block the "bad" traffic
right now, but which wouldn't hold up if the bad traffic adapts a bit.

More broadly, we're trying to walk the fine line between doing our
analysis and patches in public (yay transparency), vs being aware
that whoever is doing this is probably reading these threads too. The
destination we want is that we have defenses that are robust to the
attacker knowing about them, but things will be a bit bumpy as we get
to that destination.

I'm also trying to make sure everybody continues to think about the
privacy side -- the directory authorities or fallbackdirs don't know
what paths clients build, or what destinations they reach with them,
but they can know at what timestamps some IP addresses seemed to be using
Tor. And like most things, that information is better private by default.

> From another angle this is an interesting precedence
> because the tor project uses it's access to protect dir auths
> from exit relays. Why is that interesting? Because no one else
> that gets attacked via exit relays has that "luxury" to "filter"
> it at the "source" (exits).

Actually, the #2667 patch protects all relays from exit relays. That is,
exit relays will decline to exit to known ORPorts or DirPorts of any
relay. There are two benefits here: (a) people can't do circuit-level
amplification attacks (happy to elaborate on these once the defense is
more in place), and (b) people can't create directory requests which
blend with the directory requests that the relay itself does.

These two issues are Tor-specific, and the second one is an especially
good argument I think, because the relay is reserving for itself the
ability to make its dir connections in a way that the destination can
know that the relay is the one making the connection. (Another option
would be to add more authentication to the connection, but most ways of
doing that are heavier-weight, not lighter-weight.)

--Roger

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Metrics Error: staledesc

[Roger Dingledine]

On Thu, Jan 28, 2021 at 07:00:45PM +0100, li...@for-privacy.net wrote:
> Metrics showed my relay offline. But my Tor daemon is running normally.
> Then I saw _many_ relays suddenly have flag: staledesc
> ?
> 
> https://metrics.torproject.org/rs.html#search/flag:staledesc

Yep. The reason that happens is that the directory authorities are
receiving too many dirport connections from exit relays, but the exit
relays use a dirport connection to post their own descriptor.

So if we don't handle all of the dirport attempts, then we end up not
receiving some of the descriptor publish attempts.

I'm thinking that this part will still work out though, for two reasons.

One is that if *any* of the dir auths receive the descriptor, then they
will mention it in their next vote, and the other dir auths will learn
about it from that vote and ask for a copy.

And two is that relays watch to see if they are still listed in the
consensus, and if they're not then they try more often to upload a
new descriptor.

So yes, we are making an effort to make sure there is at least one dir
auth that will be good at receiving descriptor publishes.

Some small fraction of relays are expected to get the StaleDesc flag in
normal network operation, because there is an unfortunate interaction
between how relays publish a new descriptor "every 18 hours or when
something important changes", but dir auths ignore new descriptors if
they are too close in time or other characteristics to one that they
already have. So for example there is a known bad interaction where you
restart your relay, and the relay publishes a new descriptor because
it doesn't know that it just published one earlier, but then the dir
auths discard that new descriptor because they already have the old one,
and then your relay waits 18 hours to create a new one.

For much more backstory, see
https://gitlab.torproject.org/tpo/core/tor/-/issues/1810
https://gitlab.torproject.org/tpo/core/tor/-/issues/2479
https://gitlab.torproject.org/tpo/core/tor/-/issues/3327
https://gitweb.torproject.org/torspec.git/tree/proposals/293-know-when-to-publish.txt

But I guess the other way to look at it is: the StaleDesc flag is a
*feature*, to let your relay know that it has fallen into this edge case
so it can take steps to recover.

> https://metrics.torproject.org/rs.html#details/5D84900DBE6D6365684A9675B81A68ACE9577A68

This relay looks genuinely down.

--Roger

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Exit relay operators please help test #2667 branch

[nusenu]

Unless you already ruled out that hypothesis by looking at the attack
distribution by source IP:

If dir auths (some or all) are willing to share (privately or publicly) the 
distribution of
attack load (frequency, bandwidth, ...) by exit source IP in total or relative 
values
I can correlate this data to strengthen a hypothesis that malicious/suspicious
exits are involved to a greater extend than well-known long term exits.
That could mean that they are not (exclusively) attacking via but _from_ 
servers that also happen to
run tor exits. 


>From another angle this is an interesting precedence
because the tor project uses it's access to protect dir auths
from exit relays. Why is that interesting? Because no one else
that gets attacked via exit relays has that "luxury" to "filter"
it at the "source" (exits).


-- 
https://nusenu.github.io
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Exit relay operators please help test #2667 branch

[Chris Dagdigian]

Have a tor exit running in the US ; fingerprint is 
3DE567C1350C0E858C6147AECB06EA9B3EAF3261 and OR address is 
71.174.105.126:9001


Just built and launched the ticket-2667 branch; came up as:

[notice] Tor 0.4.6.0-alpha-dev running on Linux with Libevent 
2.0.21-stable, OpenSSL 1.0.1t, Zlib 1.2.8, Liblzma N/A, Libzstd N/A and 
Glibc 2.19 as libc.


I'll monitor the log notices but feel free to probe to test. Thanks!





Roger Dingledine 
January 28, 2021 at 1:40 AM
Hello friendly relay operators,

Another day, another weird thing with the Tor network. This time we
have some jerk bombing the directory authorities with directory fetches,
and doing it via exits:
https://lists.torproject.org/pipermail/network-health/2021-January/000661.html

The network is mostly holding together, but I wouldn't say it is pretty.

One of the long-term fixes will be ticket #2667:
https://gitlab.torproject.org/tpo/core/tor/-/issues/2667
where exit relays refuse to let users connect back into the Tor network.

David and I made a branch this evening that implements #2667, and it
could use some testing. If you're comfortable building your exit relay
from a git branch, please do, and let us know how it goes. It is the
"ticket2667" branch on either
https://git.torproject.org/user/arma/tor
or
https://gitlab.torproject.org/arma/tor/

And if your relay is currently using 100% cpu and/or way more bandwidth
than usual, you might be especially excited to try out this patch. :)

When the defense triggers, you will see an info-level log line like
"%s tried to connect back to a known relay address. Closing."
(where %s is the destination, so don't get upset at them. :)

You can let us know how it's going either by mail just to me, or by a
reply on the list, whichever you prefer. Once we know that you're running
the branch, we can also probe your relay remotely to verify that it is
correctly refusing those connections.

Thanks!
--Roger

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays


___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] Metrics Error: staledesc

[niftybunny]

“Normal”, half of my relays are also shown as offline and 1/3 of my relays are 
not even known by https://consensus-health.torproject.org/consensus-health.html 
 right now.

just wait a little bit until everything is back to normal.

whatever the new normal is normal nowadays.

> On 28. Jan 2021, at 19:00, li...@for-privacy.net wrote:
> 
> Metrics showed my relay offline. But my Tor daemon is running normally.
> Then I saw _many_ relays suddenly have flag: staledesc
> ?
> 
> https://metrics.torproject.org/rs.html#search/flag:staledesc
> https://metrics.torproject.org/rs.html#details/5D84900DBE6D6365684A9675B81A68ACE9577A68
> 
> 
> --
> ╰_╯ Ciao Marco!
> 
> Debian GNU/Linux
> 
> It's free software and it gives you freedom!
> ___
> tor-relays mailing list
> tor-relays@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays



signature.asc
Description: Message signed with OpenPGP
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] Metrics Error: staledesc

[lists]

Metrics showed my relay offline. But my Tor daemon is running normally.
Then I saw _many_ relays suddenly have flag: staledesc
?

https://metrics.torproject.org/rs.html#search/flag:staledesc
https://metrics.torproject.org/rs.html#details/5D84900DBE6D6365684A9675B81A68ACE9577A68


--
╰_╯ Ciao Marco!

Debian GNU/Linux

It's free software and it gives you freedom!
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays