Re: [tor-dev] [tor-project] Intent to Minimise Effort: Fallback Directory Mirrors

[teor]

> On 9 Jan 2018, at 14:23, Tom Ritter  wrote:
> 
>> On 8 January 2018 at 20:56, teor  wrote:
>> Add a torrc option and descriptor line to opt-in as a FallbackDir [4]
> 
> Setting a config entry is easy and requires no thought. It's easy to
> set without understanding the requirements or implications. Getting a
> personal email and request for one's relay to keep a stable IP lends a
> lot of gravity to the importance of it.
> 
> Are you worried about that?

My experience is that personal emails don't actually make much
difference. Operators try hard to keep things stable, but sometimes
their provider makes changes, or their circumstances change.

Here's what I've tried in the past:
* I sent personal emails the first few times,
* I sent list emails,
* I sent no emails at all, because we wanted to get seized relays
   off the list quickly.

Every time, we rebuilt the list after 6-12 months.
So this time, we only selected fallbacks that had the same address for
at least 3 months. I'm hoping that makes a difference.

Even if it doesn't change fallback stability, it's still good to tell people.

I would be happy if someone else sent personal emails to:
* potential operators of high-bandwidth relays
* every fallback operator when there is a new list
* any fallback operator when their relay details change, or their
   fallback goes down, asking them to fix it if they can

I think a bot or a human could do a very good job of these things.
(But I'm not that human.)

Hiro, did you want to write an email bot?
Or anyone else?
(The bot would need to work out how to deal with obfuscated email
addresses. We might need a human to code this part, and a list of
any particularly difficult ones.)

> Work-wise, the worst thing that would happen is that the list just
> needs to be regenerated more frequently - I think?

I want to reduce the time it takes to rebuild the list.
It takes days right now. If I'm going to keep on doing it, I'd like
to cut that down to hours.

The features I suggested let me do that.
And they make it easier for others to rebuild the list.
(I don't want to be the only one who can rebuild the list.)

And they don't stop people sending personal emails.
They just mean that the process doesn't rely on them.

> But user-wise
> people on those older versions will have significantly increased
> startup times if only 20% of their FallbackDirs are working...

Tor is designed to bootstrap if 0% of the fallbacks and 50% of the
authorities are working.

Recent tor versions wait 6 seconds before trying an authority.
(Older versions try an authority and a fallback straight away.)
They will try 5 fallbacks and 3 authorities in the first 30 seconds.

And we backport the new fallback list to all supported versions.

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [tor-project] Intent to Minimise Effort: Fallback Directory Mirrors

[Tom Ritter]

On 8 January 2018 at 20:56, teor  wrote:
> Add a torrc option and descriptor line to opt-in as a FallbackDir [4]

Setting a config entry is easy and requires no thought. It's easy to
set without understanding the requirements or implications. Getting a
personal email and request for one's relay to keep a stable IP lends a
lot of gravity to the importance of it.

Are you worried about that?

Work-wise, the worst thing that would happen is that the list just
needs to be regenerated more frequently - I think? But user-wise
people on those older versions will have significantly increased
startup times if only 20% of their FallbackDirs are working...

-tom
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [tor-project] Intent to Minimise Effort: Fallback Directory Mirrors

[teor]

Hi all,

I'd like to move this thread to the tor-dev list. I made a list of
Core Tor tickets at the end of this email. Let's talk about them
on tor-dev.

Here's some background:

Rebuilding the fallback list takes a lot of effort!

So I'm picking up this tor-project thread while the latest fallback rebuild
is fresh in my mind.

The rebuild took a day or two of my time for regular changes.
And another day or two for once-off changes.

We made a few changes to streamline the process:
* atagar and I wrote scripts that make it easier to generate fallback lines,
  and get operator contact details [0]
* pastly, atagar and irl helped out with code reviews and list format changes,
  which should make the list easier to read and parse
* pastly and nickm helped out with fallback list updates and backport

Having this help meant that I had time to make the process better.
Thanks!

But there's still more we can do:

> On 29 Aug 2017, at 11:12, teor  wrote:
> 
>> On 28 Aug 2017, at 18:15, Linus Nordberg  wrote:
>> 
>> teor  wrote
>> Sat, 26 Aug 2017 22:39:52 +1000:
>> 
>>> So I can't see any reasons to keep running opt-ins.
>> 
>> Are we listing non-exits?
> 
> Yes. Non-exits and under-used exits are the best fallback directory
> mirrors, because they are under less load.
> 
>> I suppose those might see a radical change in
>> their "IP reputation" when they end up on the blocklists after the next
>> ransomware attack. Exits are already in the lists.
> 
> Yes, there is a higher risk that a fallback will end up on one of these
> lists. But all public Tor relays are at risk of ending up on a list like
> this.

Because of the higher risk, we will continue to ask relay operators to
opt-in [1]. So we need to find people to help with this process every
6-12 months. And make it more efficient.

>> On 28 Aug 2017, at 19:34, Silvia [Hiro]  wrote:
>> 
>> On 26/08/17 15:44, nusenu wrote:
> use ContactInfo
> as a way to collect the intention of relay operators to opt-in/opt-out
> of the fallback directory mirrors list.
 This is a good idea, *if* we still think an opt-in is necessary.
>>> 
>>> Isn't that way to collect operator intentions also useful in opt-out
>>> mode? (instead of continuing to use emails) Or how do you plan to
>>> collect the opt-out info?
>> 
>> Why are emails preferred over other methods? Another options would be
>> wrapping a script that ops can run to create a cyberpunk trac ticket and
>> opt-out. This is probably easier than merging emails. But maybe we do
>> not want to handle this through trac.
>> 
>> I'd be happy to help w/ that eventually, and the fallback script.
> 
> This is a good idea.
> 
> At the moment, I take emails sent to me or tor-relays, and turn them
> into trac comments or trac tickets. If we put it directly on trac,
> anyone could handle the ticket.

The most time-consuming part of the rebuild is the opt-in process.

Here's what we do at the moment: [2]

0. Send an email to tor-relays to ask operators to opt-in
1. Run a script to find fallbacks whose details have changed, and email
   tor-relays to ask operators if their details are permanent

Here are the most time-consuming steps:

2a. Receive personal emails and emails to the list with fallback details
2b. Generate an updated fallback line for each relay [0]
2c. Update the whitelist

And these steps are quick:

3. Run the update script to generate a new list
4. Write a changes file
5. Announce the new list
6. Backport the list to all supported Tor versions

Here are some things I'd like to change:

Just have relay fingerprints in the fallback whitelist [3]

The fallback script checks relay details, but it can just rely on Onionoo
to say when the address last changed.

This removes step 1 and step 2b.


Add a torrc option and descriptor line to opt-in as a FallbackDir [4]

This is inspired by nusenu's "ContactInfo" idea, and Hiro's "bot" idea.
Having a descriptor line and torrc option gets us a more reliable
implementation, and it's easier for relay operators to use. And
there's no need for an email bot or trac bot.

This removes step 2a and step 2c, for tor versions with this feature.
For tor versions without this feature, we can use the simpler fingerprint
list.


Make the hard-coded authorities into a separate include file with a standard 
format [5]

We'll also change the fallback file format (again, sorry!) to make
them consistent. This makes it easier for stem, metrics-lib, and
other Tor implementations to use the same authorities and fallbacks.

This makes step 5 easier (for libraries that use fallbacks).


Let's discuss these changes on the tor-dev list!

T

[0]: 
https://gitweb.torproject.org/tor.git/tree/scripts/maint/generateFallbackDirLine.py
 
https://gitweb.torproject.org/tor.git/tree/scripts/maint/lookupFallbackDirContact.py
[1]: https://trac.torproject.org/projects/tor/ticket/24789
[2]: