Re: [tor-dev] Proposal XXX: FlashFlow: A Secure Speed Test for Tor (Parent Proposal)

[teor]

> On 16 May 2020, at 16:05, Mike Perry  wrote:
> 
>> On 4/23/20 1:48 PM, Matt Traudt wrote:
>> 
>> 5.4 Other Changes/Investigations/Ideas
>> 
>> - How can FlashFlow data be used in a way that doesn't lead to poor
>>  load balancing given the following items that lead to non-uniform
>>  client behavior:
>>- Guards that high-traffic HSs choose (for 3 months at a time)
>>- Guard vs middle flag allocation issues
>>- New Guard nodes (Guardfraction)
>>- Exit policies other than default/all
>>- Directory activity
>>- Total onion service activity
>>- Super long-lived circuits
>> - What is the explanation for dennis.jackson's scary graphs in this [2]
>>  ticket?  Was it because of the speed test? Why? Will FlashFlow produce
>>  the same behavior?
> 
> It will also be wise to provide a way for relays to signify that they
> are on the same machine. I bet concurrent machine deployments are one of
> the top contributors to the long tail of bad perf we saw caused by the
> Flashflow experiment[2]. If flashflow measures each such relay as having
> the full link capacity instead of a shared fraction, this is obviously
> going to result in overload on those relays, leading to a long tail of
> bad perf when they are chosen and are also overloaded. It is unlikely
> that we can deploy a FlashFlow that has this long tail perf problem
> without fixing this and related balancing issues (though hopefully most
> will be smoothed over by sbws).
> 
> This is a little tricky, because we might not want rogue relays joining
> each others "machines" (similar to the Family problem), but for testing
> something as simple as how MyFamily works would be great. Ideally,
> though, relays would ask or detect that they are concurrently running in
> nearby IP space and either warn the operator to set the flag, or set it
> automatically.
> 
> We actually have this work included in a future performance funding
> proposal, but the timeline on that getting approved (or even rejected)
> is so far out that we should figure out a way to do this before that,
> especially if Flashflow development is going to begin soon.

We could assume that relays on the same IPv4 /24 or IPv6 /48 share a
network link, and re-do the experiment.

Then we could tweak the network size based on those results. We'd
need to compromise between "false sharing" and "missed sharing".

Then individual operators could fine-tune that initial heuristic using the
"same network link" config.

(This is similar to how MyFamily works: Tor assumes that relays in the
same IPv4 /16 and IPv6 /32 have the same network operator. Then
individual relay operators can declare extra families using MyFamily.)

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Deprecating Tor Protocol Versions

[teor]

Hi David,

> On 15 May 2020, at 20:53, David Goulet  wrote:
> 
> On 15 May (13:58:06), teor wrote:
>> 
>> Nick and I were talking about how we remove legacy features in tor,
>> and their corresponding subprotocol versions.
>> 
>> Here is a list of the current subprotocol versions:
>> https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt#n2049
>> 
>> Here's a recent protocol version proposal, which deals with
>> recommending and requiring new features:
>> https://gitweb.torproject.org/torspec.git/tree/proposals/303-protover-removal-policy.txt
>> 
>> But we don't have a similar proposal for removing support for older
>> protocol versions from the tor codebase.
>> 
>> For an example of what that proposal could look like, see our proposal
>> for deprecating consensus methods:
>> https://gitweb.torproject.org/torspec.git/tree/proposals/290-deprecate-consensus-methods.txt
>> 
>> Here's the original conversation Nick and I had:
>> https://github.com/torproject/tor/pull/1874#discussion_r423713579
>> 
>> But after reading our consensus methods deprecation proposal, I've
>> changed my mind. I think that we should check for "protocol N, and
>> any later version" by default.
> 
> I agree that this is the right approach imo as well.
> 
>> That's what we do for consensus methods, and it seems to work well.
>> We can drop the earliest consensus methods, and recent tor versions
>> just keep working.
>> 
>> If we need an incompatible change, we can make it another protocol
>> version, and recommend then require support for it.
>> 
>> So here's an edited version of my notes on that ticket:
>> 
>> There are a few instances of ">=" and "=" confusion in protocol
>> versions. We should try to fix them all.
>> 
>> It only matters when we remove protocol versions. We haven't
>> really specified, tested, or exercised this functionality in
>> practice. And so our reviewers lack experience. (And when we did
>> discover a need for it with NSS and LinkAuth, it was more complicated
>> than we expected.)
>> 
>> I'd like to see a proposal that tells us how to check future protocol
>> versions as they are added. Along with a migration plan for disabling
>> protocol versions.
>> 
>> So let's also open a ticket to check for "any future version".
>> We should replace all "=" checks with ">=". Let's make sure we check
>> all the places where we use protocol versions, even if they don't
>> have a summary flag.
>> 
>> Overall, I think it would be helpful if future protocol versions were
>> orthogonal. Or if they depend on earlier features, that dependency
>> should be clearly documented. (For example, Relay=3 IPv6 extends
>> depends on Relay=2 EXTEND2 cells. So if we were checking EXTEND2 cell
>> support, it would be Relay=2 or Relay=3.)
> 
> At the moment, they do depend between each other last time I had that
> discussion with Nick. As in the later in your example.
> 
> Which means that supporting protocol version with a ">=" is consistent with
> our "non-written expectations" that we have now.

That's how most protocol versions work right now.

The following protocol versions are exceptions:
* Linkall Link protocols introduce incompatible changes, but a shared
  Link protocol is dynamically negotiated at runtime
* LinkAuth=3  might not support LinkAuth=1 (RSA link authentication)
* Padding=1   accidentally enabled for relays that don't support
  circuit-level padding

Link is a special case because it's the lowest-level protocol, and negotiated
at runtime. (And using an unknown feature terminates the connection.)

LinkAuth became incompatible when we introduced NSS support. But a better
design might have been:
* LinkAuth=1 RSA only
* LinkAuth=3 ed25519 and RSA
* LinkAuth=4 ed25519 only

Padding was a mistake :-(

> So if I understand correctly, we'll need to add a new protocol version let say
> N+1 in order to deprecate anything <= N ?
> 
> As an example, Relay=4 could mean "deprecate Relay=2 and use only Relay=3"
> 
> I'm +1 on this if iiuc.

Yes, I think that's what we should do in future.

So we should:
* write a proposal or spec update, and
* change the current code to use "protocol_list_supports_protocol_or_later()
  in most cases.

I'm going to wait for asn or nickm to weigh in, before opening tickets.

T


signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Deprecating Tor Protocol Versions

[teor]

Hi all,

Nick and I were talking about how we remove legacy features in tor,
and their corresponding subprotocol versions.

Here is a list of the current subprotocol versions:
https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt#n2049

Here's a recent protocol version proposal, which deals with
recommending and requiring new features:
https://gitweb.torproject.org/torspec.git/tree/proposals/303-protover-removal-policy.txt

But we don't have a similar proposal for removing support for older
protocol versions from the tor codebase.

For an example of what that proposal could look like, see our proposal
for deprecating consensus methods:
https://gitweb.torproject.org/torspec.git/tree/proposals/290-deprecate-consensus-methods.txt

Here's the original conversation Nick and I had:
https://github.com/torproject/tor/pull/1874#discussion_r423713579

But after reading our consensus methods deprecation proposal, I've
changed my mind. I think that we should check for "protocol N, and
any later version" by default.

That's what we do for consensus methods, and it seems to work well.
We can drop the earliest consensus methods, and recent tor versions
just keep working.

If we need an incompatible change, we can make it another protocol
version, and recommend then require support for it.

So here's an edited version of my notes on that ticket:

There are a few instances of ">=" and "=" confusion in protocol
versions. We should try to fix them all.

It only matters when we remove protocol versions. We haven't
really specified, tested, or exercised this functionality in
practice. And so our reviewers lack experience. (And when we did
discover a need for it with NSS and LinkAuth, it was more complicated
than we expected.)

I'd like to see a proposal that tells us how to check future protocol
versions as they are added. Along with a migration plan for disabling
protocol versions.

So let's also open a ticket to check for "any future version".
We should replace all "=" checks with ">=". Let's make sure we check
all the places where we use protocol versions, even if they don't
have a summary flag.

Overall, I think it would be helpful if future protocol versions were
orthogonal. Or if they depend on earlier features, that dependency
should be clearly documented. (For example, Relay=3 IPv6 extends
depends on Relay=2 EXTEND2 cells. So if we were checking EXTEND2 cell
support, it would be Relay=2 or Relay=3.)

T

--
teor
--



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Chutney code refactoring

[teor]

Hi Caitlin,

> On 15 May 2020, at 09:59, Nick Mathewson  wrote:
> 
> On Thu, May 14, 2020 at 7:04 PM c  wrote:
>> 
>>> On Sat, 18 Apr 2020 11:02:03 +
>>> c  wrote:
>>> 
>>> I came across Node.specialize() which does not seem to be called
>>> elsewhere, and I cannot guess at its purpose.
>> 
>> I ran vulture (a static analyzer for dead code), trimmed the output to
>> omit things I know were being used (some functions/attributes are used
>> solely in Python 2), and found these unused names:
>> 
>> lib/chutney/Templating.py:280: unused function 'getUpdateTime' (60% 
>> confidence)
>> lib/chutney/TorNet.py:439: unused function 'specialize' (60% confidence)
>> lib/chutney/TorNet.py:658: unused function '_getFreeVars' (60% confidence)
>> lib/chutney/TorNet.py:1232: unused function 'isBootstrapped' (60% confidence)
>> lib/chutney/TorNet.py:1799: unused function 'isInExpectedDirInfoDocs' (60% 
>> confidence)
>> lib/chutney/TorNet.py:2151: unused function 'configure' (60% confidence)
>> lib/chutney/TorNet.py:2191: unused function 'restart' (60% confidence)
>> lib/chutney/TorNet.py:2286: unused function 'wait_for_bootstrap' (60% 
>> confidence)
>> lib/chutney/Traffic.py:335: unused attribute 'am_closing' (60% confidence)
>> lib/chutney/Traffic.py:345: unused attribute 'am_closing' (60% confidence)
>> lib/chutney/Traffic.py:400: unused attribute 'pending_close' (60% confidence)
>> lib/chutney/Traffic.py:406: unused attribute 'dot_repetitions' (60% 
>> confidence)
>> 
>> Aside from isBootstrapped() which we discussed previously and are
>> likely going to use, is there any code that stands out as unnecessary
>> or dead?
> 
> Hm. Of these:
> 
> isInExpectedDirInfoDocs looks like it might once have done something useful.
> 
> configure and restart and wait_for_bootstrap are all in use; they are
> invoked by name, from the command line, at the end of runConfigFile,
> where it says `return getattr(network, verb)()`.
> 
> I think the rest are likely to be unused.

You could try deleting functions, and then running tor's
"make test-network-all". If that test still passes, then the code is probably
unused (in practice).

Chutney's CI does a few more tests for different tor versions, but I'm pretty
sure they all use the same code.

T



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 320: Removing TAP usage from v2 onion services

[teor]

Hi Nick,

> On 14 May 2020, at 00:09, David Goulet  wrote:
> 
> On 11 May (16:47:53), Nick Mathewson wrote:
> 
>> ```
>> Filename: 320-tap-out-again.md
>> Title: Removing TAP usage from v2 onion services
>> Author: Nick Mathewson
>> Created: 11 May 2020
>> Status: Open
>> ```
>> 
>> (This proposal is part of the Walking Onions spec project.  It updates
>> proposal 245.)
>> 
>> # Removing TAP from v2 onion services
>> 
>> As we implement walking onions, we're faced with a problem: what to do
>> with TAP keys?  They are bulky and insecure, and not used for anything
>> besides v2 onion services.  Keeping them in SNIPs would consume
>> bandwidth, and keeping them indirectly would consume complexity.  It
>> would be nicer to remove TAP keys entirely.
>> 
>> But although v2 onion services are obsolescent and their
>> cryptographic parameters are disturbing, we do not want to drop
>> support for them as part of the Walking Onions migration.  If we did
>> so, then we would force some users to choose between Walking Onions
>> and v2 onion services, which we do not want to do.
> 
> I haven't read the entire proposal so I won't comment on its technical aspect.
> I was reading and got here and that made me very uncertain about the whole
> proposal itself.
> 
> I will propose that we revisit the overall idea of changing v2 here.
> 
> I personally think this is the wrong approach. Onion services v2 should be
> deprecated as in removed from the network instead of being offered as a choice
> to the users.
> 
> We haven't properly done a deprecation path yet for v2 primarly due to our
> lack of time to do so. But at this point in time, where the network is 100%
> composed of relays supporting v3 now (which took 3+ years to get there), it is
> time for v2 to not be presented as a choice anymore.
> 
> It is a codebase that is barely maintained, no new features are being added to
> it and thus moving it to ntor means another at least 3 years of network
> migration. This would mean a major new feature in that deprecated code base...
> 
> So thus, I personally will argue that moving v2 to ntor is really not the
> right thing to do. Onion service v2 are, at this point in time, _dangerous_
> choice for the users.

I agree that we shouldn't support old features forever. And it seems unwise
to spend development effort just to migrate away from TAP, when we could
instead spend that time migrating away from TAP and v2 onion services.
(And reducing our dependency on SHA1 and RSA keys.)

Strategically, it also seems unwise to carry v2 onion services, TAP
handshakes, RSA relay keys and signatures, and SHA1 into walking onions.

But it's hard to make these kinds of decisions without approximate
timeframes.

How long would it take to migrate away from v2 onion services?

How long would it take to introduce walking onions?

If we decide to modify v2 onion services, how long would that migration
take? And what's the final plan to end the modified v2 onion services?

T



___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 320: Removing TAP usage from v2 onion services

[teor]

Hi Nick,

> On 12 May 2020, at 06:48, Nick Mathewson  wrote:
> 
> ## Migration steps
> 
> First, we implement all of the above, but set it to be disabled by
> default.  We use torrc fields to selectively enable them for testing
> purposes, to make sure they work.

Can you expand on the testing plan here?

One of the risks with multi-year migration projects is that unrelated
changes break the migration, and we don't notice.

For example, you might need to create a chutney network for each
stage, and run them on every pull request and merge. In our current
CI, that's 30-60 seconds of extra time per network, or 2-4 extra
minutes of CI time.

If you need to test different combinations of features for each stage,
let's try to do that in the same networks. Otherwise, the testing matrix
expands out very quickly.

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 319: RELAY_FRAGMENT cells

[teor]

Hi Nick,

> On 12 May 2020, at 06:47, Nick Mathewson  wrote:
> 
> In this proposal,
> we allow the following cell types to be fragmented: EXTEND2, EXTENDED2,
> INTRODUCE1, INTRODUCE2, RENDEZVOUS.  Any party receiving a command that they
> believe should not be fragmented should close the circuit.

Do you mean RENDEZVOUS1 and RENDEZVOUS2 here?

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 318: Limit protover values to 0-63

[teor]

Hi Nick,

> On 12 May 2020, at 06:47, Nick Mathewson  wrote:
> 
> ```
> Filename: 318-limit-protovers.md
> Title: Limit protover values to 0-63.
> Author: Nick Mathewson
> Created: 11 May 2020
> Status: Open
> ```
> 
> # Limit protover values to 0-63.
> 
> I propose that we no longer accept protover values higher than 63,
> so that they can all fit nicely into 64-bit fields.
> 
> (This proposal is part of the Walking Onions spec project.)
> 
> ## Motivation
> 
> Doing this will simplify our implementations and our protocols.
> Right now, an efficient protover implementation needs to use ranges
> to represent possible protocol versions, and needs workarounds to
> prevent an attacker from constructing a protover line that would
> consume too much memory.  With Walking Onions, we need lists of
> protocol versions to be represented in an extremely compact format,
> which also would benefit from a limited set of possible versions.

If you'd like to make the format even more compact, you could also
lower these limits:

const MAX_PROTOCOLS_TO_EXPAND: usize = 1 << 16;
const MAX_PROTOCOL_NAME_LENGTH: usize = 100;

In particular:
* now that protocols can only have 63 versions, we can directly
   limit the maximum number of protocol names. (Rather than
   indirectly limiting them via the expansion limit.) We currently have
   12 protocol names, so a suitable limit might be 50.
* depending on the parsing algorithm, we might not need an
   expansion limit any more. If we do, we should set the limit to
  (max versions)*(max names) or lower.
* our current protocol names are all 4-9 characters, so we could
   lower the name limit to 30.

That way, the maximum length of a compact list would be:
8 bytes per protocol version * 50 protocols = 400 bytes of versions
30 bytes per protocol name * 50 protocols = 1500 bytes of names

Hmm, maybe that's a bit high, and we should drop some of those
limits even further.

> …
> 
> Even if we did someday need to implement higher protocol
> versions, we could simply add a new subprotocol name instead.  For
> example, instead of "HSIntro=64", we could say "HSIntro2=1".

If we drop the name limit, it should be high enough to support new
protocols, and extensions to existing protocols.

> ## Migration
> 
> Immediately, authorities should begin rejecting relays with protocol
> versions above 63.  (There are no such relays in the consensus right
> now.)
> 
> Once this change is deployed to a majority of authorities, we can
> remove support in other Tor environments for protocol versions
> above 63.

As you said in your previous email, bridge clients can start rejecting
descriptors along with regular clients. Even though they get their
descriptors directly from bridges.

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [RFC] Proposal: A First Take at PoW Over Introduction Circuits

[teor]

Hi tevador,

> On 9 May 2020, at 06:43, tevador  wrote:
>> For our dynamic PoW system to work, we will need to be able to compare PoW
>> tokens with each other. To do so we define a function:
>>unsigned effort(uint8_t *token)
>> which takes as its argument a hash output token, and returns the number of
>> leading zero bits on it.
> 
> This definition makes the effort exponential, i.e. the computational resources
> required to reach one notch higher effort increase by a factor of 2 each time.
> 
> I suggest to use linear effort defined as the quotient of dividing a bitstring
> of 1s by the hash value:
> 
> == Example A:
> 
>  effort(0001100010101101) =  / 0001100010101101
> 
> or in decimal:
> 
>  effort(6317) = 1048575 / 6317 = 165.
> 
> This definition of effort has the advantage of directly expressing the 
> expected
> number of hashes that the client had to calculate to reach the effort.
> 
> With the exponential definition, we could have an equivalent linear effort of
> either 128 (7 leading zeroes) or 256 (8 leading zeroes), while the linear
> definition provides smoother classification of PoW results.

There are two possible issues with this design:

Division is expensive on some platforms, including ARM-based devices.
But there might be a way to calculate an approximate value without division.
(For example, bit shifts, or multiplying by an inverse.) Or we could calculate
the maximum value once, and then re-use it.

Is it still possible to express the full range of difficulties? Is that 
expression
reasonably compact?

Some advantages of this exponential distribution are:
* spurious results can be filtered using a single instruction (a bit mask),
* the expected effort is quick and easy to calculate,
* the effort can be expressed in a compact way.

Maybe we don't need some of these properties, and a linear design would
be fine.

But if we do, we could change the exponent to the square or cube root of
two. There would be a smoother distribution, but a wider range, and the
checks would still be reasonably fast.

T


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal Idea: IPv6-Only Exit Relays

[teor]

Hi all,

The network team is making it easier to set up a dual-stack Tor relay.

We're currently working on:
* IPv6 reachability self-tests
* IPv6 address auto-detection
* IPv6 relay statistics

We're also thinking about the next IPv6 project. One possibility is
adding IPv6-only exits to the Tor network.

We'd need to solve two technical problems:
1. stop requiring IPv4 in the consensus and circuit code
2. avoiding delays when using an IPv6-only exit for an IPv4-only site

And one deployment problem:
3. Get more dual-stack middle relays (50% - 75%)

Here's a quick sketch of a solution to the IPv4-only site issue:

Clients open two exit streams for every site. One of those streams
must use an exit with IPv4 support. The client uses whichever
stream succeeds first.

Just like the "happy eyeballs" protocol, there is a delay between
launching the two streams:
https://www.rfc-editor.org/rfc/rfc8305.txt

Here are some benefits of a universal change:
* code and protocol consistency
* makes traffic analysis harder
* also helps with other exit failures, for a better user experience

Here are some drawbacks:
* extra load

We can change the delay to manage the tradeoff between load,
user experience, and traffic analysis resistance. Making the delay
longer reduces the load, but makes the user experience worse.

Traffic analysis resistance is a bit more complex. Consistency
across different clients is important, but we also don't want to
create obvious traffic patterns. So there are also some benefits
to randomisation.

If I get time, I'd like to turn these ideas into a proposal.

T

-- 
teor
--

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Tor Relay wont connect to private IP address

[teor]

Hi Eli,

> On 29 Apr 2020, at 07:40, Eli Vakrat  wrote:
> 
> So thanks to teor's insightful response yesterday I decided to try to run a 
> second tor relay (my middle node) on my private network.
> 
> Unfortunately, I can't do it with Chutney because my python client is running 
> on a windows machine. But I do have 3 machines at my disposal:
> 
> 1. A windows machine (the python client)
> 2. A mac (the guard node)
> 3. Another mac (the middle node)

Can you run chutney on one of your macs?
You don't need a separate machine for each Tor relay.

If you want to reach your chutney tors from another machine on your local 
network, you'll need to set this environmental variable:
CHUTNEY_LISTEN_ADDRESS=(IPv4 of your Mac)

See:
https://github.com/torproject/chutney/blob/master/README#L65

> When my guard node tries to connect to my middle node after receiving from 
> the client a RELAY_EXTEND cell, the guard node logs the following error:
> 
> Apr 28 17:00:31.000 [info] circuit_extend: Client asked me to extend to a 
> private address
> 
> So regarding this, I have two questions:
> 
> 1. Is there a way for me to change something in my torrc file to override 
> this error and allow my relay to extend to private IP addresses?
> 
> My torrc is currently configured as such (Notice I put some place holders for 
> the drectories and for the ip address tha aren't actually whats written 
> there):
> ContactInfo em...@example.com
> ControlPort 9051
> DataDirectory 
> ExitPolicy reject *:*
> ExitRelay 0
> GeoIPFile 
> GeoIPv6File 
> Log notice file /notice.log
> Log debug file /debug.log
> Log warn file /warn.log
> Nickname vtoria
> ORPort 443 NoAdvertise
> ORPort Relay.Public.IP.Example:443 NoListen 
> SafeLogging 0
> ExtendAllowPrivateAddresses 1

That's odd. ExtendAllowPrivateAddresses is set to 1, which is what you want.

Are you sure your relay is using this torrc?
Check the path of the torrc in the logs.
Try restarting the relay.

> 2. Would there maybe be a better way to run this private tor network (without 
> chutney)?

There are two alternatives:
1. Chutney does a lot of the work for you, so I'd recommend using it, if you 
can.
2. You can also set up relays on private IP addresses, and set 
"PublishServerDescriptor 0", so they don't publish their descriptors to the Tor 
directory authorities.

If you're not publishing descriptors, you can have as many relays as you like 
on the same IP address.

T

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Relay Truncated Cell Example

[teor]

Hi Eli,

>> On 28 Apr 2020, at 11:47, Eli Vakrat  wrote:
> But then I looked at the debug log of my guard node (remember that my guard 
> node is on my local machine) and it said that it had received a DESTROY Cell 
> back from the middle node and was passing on a RELAY_TRUNCATED cell to me:
> 
> Apr 26 16:11:03.014 [debug] command_process_destroy_cell: Received for circID 
> 2297363203. <--this is the circuit ID between my gaurd node and the 
> middle node
> Apr 26 16:11:03.014 [debug] command_process_destroy_cell: Delivering 
> 'truncated' back.
> Apr 26 16:11:03.014 [debug] relay_send_command_from_edge_: delivering 9 cell 
> backward.
> 
> To my understanding what this log means is that some part of the EXTEND cell 
> I sent to the middle node was wrong or malformed and because of this when the 
> middle node tried to extend the circuit, an error occurred, and the circuit 
> needed to be torn down.
> 
> This is very weird because when I send an EXTEND cell that is meant for my 
> guard node (meaning I want to extend the circuit from one hop to two hops) 
> everything works fine, and I can even successfully derive the shared key 
> material for the middle node.

It looks like there are at least two bugs here:
* encryption to the guard node
* decryption from the guard node

> So I have several questions regarding this:
> 
> 1. First of all, I didn't quite understand the exact format of a 
> RELAY_TRUNCATED cell. Does it contain a relay cell command + recognized+field 
> +digest and so on? or is it just a single octet that immediately follows the 
> cell command field? If some could show me an example of the cell, it would be 
> much appreciated... 

It's a relay cell, so it contains all the relay cell fields:
https://github.com/torproject/torspec/blob/master/tor-spec.txt#L1551
Then the relay cell's "data" contains a single byte "reason" field.

I agree it's confusing, I've fixed it in the spec.

> 2. What are some common errors that would make an OR drop a RELAY EXTEND 
> cell? I thought maybe it was a problem with my TAP handshake data, but after 
> extensive checking that doesn't seem to be the case.

Have you tried running the middle relay on your machine as well?

You can run a whole tor network on your machine using chutney:
https://github.com/torproject/chutney/blob/master/README#L257
(If you're using Windows, Chutney won't work, because directory authorities 
don't work on Windows.)

Try:
CHUTNEY_STOP_TIME=-1 chutney/tools/test-network.sh
Then extend to the relays on 127.0.0.1.

It looks like you can't encrypt cells to the guard.

Have you tried sending a RELAY_DROP cell to the guard?
That's a good way to test encryption.

It also looks like you can't decrypt cells from the guard.

What's the data in the first CREATED cell from the guard?
Should it be all-zero? (Check the tor source code.)

Have you tried opening a stream to the guard node?
That's a good way to test decryption.

Most guards are directory caches, so you should be able to open a  
RELAY_BEGINDIR stream, and download something small. Like the relay's 
descriptor.

> 3. If someone could describe the exact steps of extending a circuit to a 
> third node, it would greatly help me to make sure that I didn't miss a step 
> or do something wrong.

Here's the spec for multi-hop circuits:
https://github.com/torproject/torspec/blob/master/tor-spec.txt#L1290

It's hard to know what steps you're missing, without more information. Feel 
free to share your code, or ask questions about any confusing parts of the spec.

T


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 316: FlashFlow: A Secure Speed Test for Tor (Parent Proposal)

[teor]

Hi Matt,

Thanks for the quick response!

I've trimmed the conversation to the comments that need further
discussion.

> On 25 Apr 2020, at 06:46, Matt Traudt  wrote:
> 
> On 4/23/20 21:05, teor wrote:
>> ...
>> 
>>> - msm_duration  [1 byte]
>> 
>> What are the minimum and maximum valid values for this field?
>> 1..255 ?
>> 
>> Do we want to limit measurements to 4 minutes at a protocol level?
>> 
>> In general, protocols should make invalid states impossible to represent.
>> But do we want a 4 minute hard limit here?
>> 
> 
> This document suggests a measurement duration of 30 seconds. We see no
> reason to ever go above 1 minute. If there's a byte to spare, then sure
> let's make this a uint16.

I've thought about this a bit more, and from a user experience perspective,
we also want a 30 second limit. (Most users will give up on a slow
connection after 30 seconds.)

So as long as there is a documented limit in the protocol, we should be
fine with 2 bytes.

>>> second is the number of seconds since the measurement began. MSM_BG
>>> cells are sent once per second from the relay to the FlashFlow
>>> coordinator. The first cell will have this set to 1, and each
>>> subsequent cell will increment it by one. sent_bg_bytes is the number of
>>> background traffic bytes sent in the last second (since the last MSM_BG
>>> cell). recv_bg_bytes is the same but for received bytes.
>>> 
>>> The payload of MSM_ERR cells:
>>> 
>>> - err_code [1 byte]
>>> - err_str  [possibly zero-len null-terminated string]
>> 
>> We don't have strings in any other tor protocol cells.
>> 
>> If you need extensible error information, can I suggest using
>> ext-type-length-value fields:
>> 
>> N_EXTENSIONS [1 byte]
>> N_EXTENSIONS times:
>>EXT_FIELD_TYPE [1 byte]
>>EXT_FIELD_LEN  [1 byte]
>>EXT_FIELD  [EXT_FIELD_LEN bytes]
>> 
>> https://gitweb.torproject.org/torspec.git/tree/rend-spec-v3.txt#n1518
>> 
>> If strings are necessary, please specify a character encoding
>> (ASCII or UTF-8), and an allowed set of characters.
>> 
>> If we don't whitelist characters, we risk logging terminal escape
>> sequences, or other arbitrary data.
> 
> I seem to remember strings used between directory authority /directory
> mirror relays and clients to communicate certain errors (clock skew?),
> but what's probably reality is a *code* is communicated and what I'm
> thinking of is merely the Tor client interpreting the code for logging
> purposes.

There are two sources for clock skew warnings:
* A binary time field in NETINFO cells
* A HTTP header on directory documents

The header is text, but it's very structured, and at a different
protocol layer.

In another part of the directory protocol, when authorities reject a
relay descriptor upload, they send a rejection reason to the relay.
That's unstructured text in a HTTP response. (But we do escape it before
logging.)

> Regardless, we probably don't really need a string. It occurs to me we
> might want *something* that carries more information than a code; for
> example, a MSM_ERR cell with a code stating "I'm refusing to be measured
> because I've been measured too recently" would benefit from a field
> stating either time till measurement allowed again or time since last
> measurement.

Yes, I think a code and ext-type-length-value fields for any additional
info would work here.

>>> At this point, the
>>> relay
>>> 
>>> - Starts a repeating 1s timer on which it will report the amount of
>>>   background traffic to the coordinator over the coordinator's
>>>   connection.
>>> - Enters "measurement mode" and limits the amount of background
>>>   traffic it handles according to the torrc option/consensus
>>>   parameter.
>>> 
>>> The relay decrypts and echos back all MSM_ECHO cells it receives on
>>> measurement connections
>> 
>> Are MSM_ECHO cells relay cells?
>> How much of the relay protocol does the measurer implement?
>> 
>> The references to decrypting cells suggest that MSM_ECHO cells are
>> relay (circuit-level) cells. But earlier sections suggest that they are
>> link cells.
>> 
>> If they are link cells, what key material is used for decryption?
>> How do the measurer and relay agree on this key material?
>> 
>> If they are relay cells, do they use the ntor handshake?
>> https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt#n1132
>> 
> 
> MEASURE cells are cells like CREATE, CREATED, RELAY, etc.

In

Re: [tor-dev] Building a privacy-preserving "contact tracing" app

[teor]

Hi Adrien,

> On 22 Apr 2020, at 16:35, Adrien Luxey  wrote:
> 
> The French state is making a glosing about the "privacy-preserving", 
> "anonymous" contact tracing app they are developing with Inria (national 
> informatics research agency). You can check about the protocol proposal, 
> ROBERT, here: https://github.com/ROBERT-proximity-tracing/documents (in 
> English!)
> 
> As you would expect, the proposal is not privacy-preserving unless you 
> believe the State would never ever misbehave, e.g. link IP address to 
> identity with the help of ISPs etc. There is some relevant criticism here: 
> https://github.com/ROBERT-proximity-tracing/documents/issues/6

Not just the French state, lots of them are doing it :-(

> I'd like to propose a really private "contact tracing" counter-proposal, 
> which would use Tor's onion services for sender-receiver anonymity. Not that 
> I am a proponent of the idea, but we need to come up with alternatives in the 
> debate.

There are a few decent privacy-preserving contact protocols.

I'd give you links, but I can't find them right now.

(Search engines are doing weird things to covid searches, and I can't remember 
my friends' twitter handles. Oops!)

> My question is: would the Tor community agree on having StopCovid's traffic 
> go through its net? In my proposal, I would only broadcast a message from a 
> declared COVID-positive person to all its recent contacts; the proximity 
> sensing would remain Bluetooth. Still, deployed at the scale of a country, 
> it's a lot of traffic.

How much traffic will the app use?
How many users?

The Tor network currently handles about 70 Gigabits per second of user traffic. 
This graph shows the traffic for all relays. There are usually 3 relays in each 
user circuit:
https://metrics.torproject.org/bandwidth.html

Tor currently has a few million active users:
https://metrics.torproject.org/userstats-relay-country.html

I'm sure we'd love to help. But maybe the Tor network can't scale to hundreds 
of millions of people using an app?

T






signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal XXX: FlashFlow: A Secure Speed Test for Tor (Parent Proposal)

[teor]

Hi,

Thanks for this proposal!

I'm looking forward to more secure bandwidth measurements on the Tor
network.

Overall, this proposal looks good.

But I'm particularly concerned about any communication between
bandwidth coordinators. Our general principle is that directory
authorities should be independent, and we should minimise their
communication and dependencies. This principle also extends to
bandwidth authorities.

For FlashFlow, here are some specific reasons to avoid bandwidth
coordinator communication:
* it adds complexity to the protocol
* it adds an additional failure mode: failure of coordinator
  communication
  * if the communication is required, this failure mode becomes a
denial of service vulnerability
  * if the communication is optional, the failure could activate a
less-tested fallback mode, and change coordinator behaviour
* it adds a class of additional bugs: coordinator miscommunication,
  including race conditions
* it adds a class of additional security vulnerabilities, via
  coordinator communication
* it adds additional coordinator configuration, which must stay
  synchronised. There's two ways to sync config:
  * in the consensus: the coordinator IP addresses are public, or
  * privately: the configs easily get out of sync

There's also some information missing from the proposal, I'll point it
out as part of this review.

> On 24 Apr 2020, at 04:48, Matt Traudt  wrote:
> 
> Filename: xxx-flashflow.txt
> Title: FlashFlow: A Secure Speed Test for Tor (Parent Proposal)
> Author: Matthew Traudt, Aaron Johnson, Rob Jansen, Mike Perry
> Created: 23 April 2020
> Status: Draft
> 
> ...
> 
> 3.1.2 New Cell Types
> 
> FlashFlow will introduce a new cell command MEASURE.
> 
> The payload of each MEASURE cell consists of:
> 
>  Measure command [1 byte]
>  Length  [2 bytes]
>  Data[Length-3 bytes]
> 
> The measure commands are:
> 
>  0 -- MSM_PARAMS[forward]
>  1 -- MSM_PARAMS_OK [backward]
>  2 -- MSM_ECHO  [forward and backward]
>  3 -- MSM_BG[backward]
>  4 -- MSM_ERR   [forward and backward]

Readability note:

"MSM" is a standard abbreviation for "mainstream media".

A standard abbreviation for measurement is "MEAS":
https://www.abbreviations.com/abbreviation/MEASurement

> ...
> 
> 3.1.3 Pre-Measurement Handshaking/Starting a Measurement
> 
> The coordinator connects to the target relay and sends it a MSM_PARAMS
> cell.

How much of the tor link protocol does the coordinator implement?

Currently, tor requires the following cells:
* VERSIONS
* NETINFO

https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt#n542

> If the target is unwilling to be measured at this time or if the
> coordinator didn't use a TLS certificate that the target trusts, it
> responds with an error cell and closes the connection. Otherwise it
> checks that the parameters of the measurement are acceptable (e.g. the
> version is acceptable, the duration isn't too long, etc.). If the
> target is happy, it sends a MSM_PARAMS_OK, otherwise it sends a MSM_ERR
> and closes the connection.
> 
> Upon learning the IP addresses of the measurers from the coordinator in
> the MSM_PARAMS cell, the target whitelists their IPs in its DoS
> detection subsystem until the measurement ends (successfully or
> otherwise), at which point the whitelist is cleared.
> 
> Upon receiving a MSM_PARAMS_OK from the target, the coordinator will
> instruct the measurers to open their TCP connections with the target. If
> the coordinator or any measurer receives a MSM_ERR, it reports the error
> to the coordinator and considers the measurement a failure. It is also a
> failure if any measurer is unable to open at least half of its TCP
> connections with the target.
> 
> The payload of MSM_PARAMS cells [XXX more may need to be added]:
> 
>  - version   [1 byte]

What are the minimum and maximum valid values for this field?
0..255 ?
1..255 ?

Tor uses a standard ext-type-length-value format for new cell
fields, rather than parsing them based on a version field.

It may still be useful to have a version field for information
purposes. (And to workaround bugs in older versions.) Normally, we'd use
Tor Relay protocol versions, but the coordinators and measurers aren't in
the consensus.

Here's an example of the ext-type-length-value format:

 N_EXTENSIONS [1 byte]
 N_EXTENSIONS times:
EXT_FIELD_TYPE [1 byte]
EXT_FIELD_LEN  [1 byte]
EXT_FIELD  [EXT_FIELD_LEN bytes]

https://gitweb.torproject.org/torspec.git/tree/rend-spec-v3.txt#n1518

You'd want an extension in each measurer info, and also one at the end
of the cell, for any new general fields.

Tor already knows how to parse these fields, because they are used for
v3 onion services.

>  - msm_duration  [1 byte]

What are the minimum and maximum valid values for this field?
1..255 ?

Do we want to limit measurements to 4 minutes at a protocol level?

In general, protocols should make invalid states 

Re: [tor-dev] Proposal 315: Updating the list of fields required in directory documents

[teor]

Hi Nick,

This proposal is missing the "bridge" case.

Bridges are more complicated, because we have at least
3 kinds of bridges:
* bridges distributed by BridgeDB
* bridges distributed with apps (such as Tor Browser)
* private bridges

Bridge option transitions are also more complicated, because clients
download bridge descriptors directly from their configured bridges.

T

> On 24 Apr 2020, at 00:45, Nick Mathewson  wrote:
> 
> Filename: 315-update-dir-required-fields.txt
> Title: Updating the list of fields required in directory documents
> Author: Nick Mathewson
> Created: 23 April 2020
> Status: Open
> 
> 1. Introduction
> 
>   When we add a new field to a directory document, we must at first
>   describe it as "optional", since older Tor implementations will
>   not generate it.  When those implementations are obsolete and
>   unsupported, however, we can safely describe those fields as
>   "required", since they are always included in practice.
> 
>   Making fields required is not just a matter of bookkeeping: it
>   helps prevent bugs in two ways.  First, it simplifies our code.
>   Second, it makes our code's requirements match our assumptions
>   about the network.
> 
>   Here I'll describe a general policy for making fields required
>   when LTS versions become unsupported, and include a list of
>   fields that should become required today.
> 
>   This document does not require to us to make all optional fields
>   required -- only those which we intend that all Tor instances
>   should always generate and expect.
> 
>   When we speak of making a field "required", we are talking about
>   describing it as "required" in dir-spec.txt, so that any document
>   missing that field is no longer considered well-formed.
> 
> 2. When fields should become required
> 
>   We have three relevant kinds of directory documents: those
>   generated by relays, those generated by authorities, and those
>   generated by onion services.
> 
>   Relays generate extrainfo documents and routerdesc documents.
>   For these, we can safely make a field required when it is always
>   generated by all relay versions that the authorities allow to
>   join the network.  To avoid partitioning, authorities should
>   start requiring the field before any relays or clients do.
> 
>   (If a relay field indicates the presence of a now-required
>   feature, then instead of making the field mandatory, we may
>   change the semantics so that the field is assumed to be
>   present. Later we can remove the option.)
> 
>   Authorities generate authority certificates, votes, consensus
>   documents, and microdescriptors.  For these, we can safely make a
>   field required once all authorities are generating it, and we are
>   confident that we do not plan to downgrade those authorities.
> 
>   Onion services generate service descriptors.  Because of the risk
>   of partitioning attacks, we should not make features in service
>   descriptors required without a phased process, described in the
>   following section.
> 
> 2.1. Phased addition of onion service descriptor changes
> 
>   Phase one: we add client and service support for the new field,
>   but have this support disabled by default. By default, services
>   should not generate the new field, and clients should not parse
>   it when it is present.  This behavior is controlled by a pair of
>   network parameters.  (If the feature is at all complex, the
>   network parameters should describe a _minimum version_ that
>   should enable the feature, so that we can later enable it only in
>   the versions where the feature is not buggy.)
> 
>   During this phase, we can manually override the defaults on
>   particular clients and services to test the new field.
> 
>   Phase two: authorities use the network parameters to enable the
>   client support and the service support.  They should only do this
>   once enough clients and services have upgraded to a version that
>   supports the feature.
> 
>   Phase three: once all versions that support the feature are
>   obsolete and unsupported, the feature may be marked as required
>   in the specifications, and the network parameters ignored.
> 
>   Phase four: once all versions that used the network parameters
>   are obsolete and unsupported, authorities may stop including
>   those parameters in their votes.
> 
> 3. Directory fields that should become required.
> 
>   These fields in router descriptors should become required:
>  * identity-ed25519
>  * master-key-ed25519
>  * onion-key-crosscert
>  * ntor-onion-key
>  * ntor-onion-key-crosscert
>  * router-sig-ed25519
>  * proto
> 
>   These fields in router descriptors should become "assumed present":
>  * hidden-service-dir
> 
>   These fields in extra-info documents should become required:
>  * identity-ed25519
>  * router-sig-ed25519
> 
>   The following fields in microdescriptors should become
>   required:
>  * ntor-onion-key
> 
>   The 

Re: [tor-dev] Fallback Directory Handover

[teor]

> On 22 Apr 2020, at 12:27, Ian Goldberg  wrote:
> 
> On Wed, Apr 22, 2020 at 11:56:54AM +1000, teor wrote:
>> a bad fallback guard can continue to manipulate its client's view of
>> the network
> 
> This is only true to the extent that the fallback guard can choose which
> of three still-valid consensuses to give to the client, right?

Not quite.

Clients tolerate recently-expired consensuses for some operations, up
to 72 hours in some cases.

When I last checked, TAILS set its system clock off the date in the
consensus it receives.

Clients also download authority certificates from fallback directory
mirrors. I think that's the whole trust path from the hard-coded
authority fingerprints, to the certificates, and then a valid consensus.

Since clients use an ORPort connection to download consensuses,
a malicious fallback directory mirror can also provide them with:
* the wrong date (triggering a clock skew warning)
* the wrong external IP address (not used for much)
* malicious directory documents
  * note that decompression and some parsing happens before the
 signature checks
* slow transfer speeds (like slowloris)

Using multiple fallbacks mitigates most of these issues.

T



___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Fallback Directory Handover

[teor]

Hi all,

Here's a summary of the current state of fallback directory mirrors.

Overall Design

This repository contains a list of potential fallback directory mirrors
(a fallback "offer list"), and a script that checks each mirror for speed
and reliability:
https://gitweb.torproject.org/fallback-scripts.git/

There is a "Fallback Scripts" component for tickets:
https://trac.torproject.org/projects/tor/query?status=!closed=Core+Tor%2FFallback+Scripts

The fallback system is designed to gracefully degrade as fallback
directory mirrors fail. Failures shift load to directory authorities,
and cause brief delays during client bootstrap.

We expect the system to operate well, even if all the fallbacks have
failed. But we try to keep the fallback failure rate below 20-30%.
When the failure rate gets too high, we rebuild the fallback list.

Regular Tasks

This ticket is the parent ticket for the next fallback rebuild:
https://trac.torproject.org/projects/tor/ticket/30971

This ticket contains the "offer list" changes that relay operators have
requested. I usually commit them all at once, but you should feel free
to do them incrementally:
https://trac.torproject.org/projects/tor/ticket/30972

Sometimes, we don't have enough relays on the offer list, and we have
to ask relay operators to opt-in to the list. Ideally, we want at least 100
fallbacks, we usually have between 120-160.

Future Work

It's hard to verify changes to the offer list. Changes are usually sent by
email or through trac tickets. There's no reliable trust path from the
relay key to the email or ticket.

The opt-in process is also a manual process. It can be time-consuming.

To resolve these issues, I had planned to add a signed fallback offer line
to relay descriptors:
https://trac.torproject.org/projects/tor/ticket/24839

Instead of checking the list in the fallback-scripts repository, the script
can check relay descriptors instead. (Or check both, during the transition
period.)

Unresolved Issues

Fallbacks eventually see the entire set of clients. Clients that are active 
all the time may only ever contact one fallback. (Clients re-use the same
fallback for authority keys, and then switch to the consensus as soon as
possible.) But clients whose consensuses have expired will choose new
fallbacks at random.

Ideally, clients should select fallback (and maybe authority) guards.
That is, they should retry previously-selected fallbacks. There are some
tradeoffs here: a bad fallback guard can continue to manipulate its
client's view of the network. We can avoid this issue by selecting multiple
fallback guards.

Clients will need persistent state to remember their guards, so transient
systems like TAILS won't benefit from this change.

T

-- 
teor
--

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Chutney tests fail for tor/maint-0.3.5 (bug #33677)

[teor]

Hi Caitlin,

> On 1 Apr 2020, at 18:58, c  wrote:
> 
> On Mon, 30 Mar 2020 13:22:21 +1000
> teor  wrote:
> 
>> Check for onion service descriptor uploads:
>> https://trac.torproject.org/projects/tor/ticket/33609
>> 
>> Someone else is working on the microdescriptor changes at the moment.
>> 
>> Would you like to start working on the onion service descriptor changes?
> 
> Sure, it would give me an opportunity to learn about onion descriptors
> in more detail. I will get started on it.
> 
> But first, looking at it more, I think my struggle with understanding
> what to do stems from my unfamiliarity with the Chutney codebase (first
> I have heard of the tool was with this project, even though commits
> date back to 2011).

Chutney is a custom tool that we use to integration test tor networks.

> I need to make sure I understand #33609
> sufficiently:
> 
> - Is the requested functionality only for Chutney or will Tor
>  potentially need any changes to allow for HS verification?

The required messages are already in tor's onion service logs, so I don't
think that tor will need any changes.

I tried to describe the changes in detail on the ticket:
https://trac.torproject.org/projects/tor/ticket/33609#comment:5

Please let me know if you have any further questions. We're talking in a
lot of detail now, so let's continue on the ticket. That way, any
reviewers can also see the conversation.

> - So I know where to begin looking in the codebase, the ticket wants us
>  to "check each onion service log" -- is this referring to Tor log
>  output (such as the instances chutney spawns), chutney-specific logs,
>  or something else entirely?

The info-level log output of the tor instances that chutney spawns.

> - For "check v2 and v3 onion services" -- check if they've propagated
>  the network?

Check if v2 and v3 onion services have uploaded their descriptors.

> - For "call it an extra 200% 'bootstrap' stage" -- again is this
>  chutney-specific? I only know bootstrapping percentage from Tor
>  notice-level logging and obviously it only goes up to 100%, so I'm
>  wondering if "200%" is a magic number here or something arbitrary.

It's an arbitrary number, greater than 100%, so we can integrate it
with the existing bootstrap checks. (But that might not be necessary.)

> From this and the parent #33050 it doesn't seem to me like the request
> is very clear.

You're right, the ticket contains my rough notes and hints. I didn't
know what level of detail people would need.

> I am reading proposals 311-313 after sending this
> message so maybe I can come across some answers to my
> questions/confusions via the proposals themselves. The proposals will
> probably give me a better idea of the work I am in for overall, too,
> and perhaps I should have come across them sooner.

The proposals might help, but they are mainly focused on tor changes,
not chutney changes.

> I figure it is wise
> regardless to ask for clarification here and read while I wait for
> feedback. Efficiency and all :)

Please feel free to ask further questions on this list. But let's try
to have detailed discussions on the relevant tickets, so reviewers can
see the conversation.

Thanks!

T



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Chutney tests fail for tor/maint-0.3.5 (bug #33677)

[teor]

Hi,

> On 28 Mar 2020, at 21:42, c  wrote:
> 
> For the Outreachy.org internship project I decided to take on bug
> #33677 [1].

Thanks for applying for Outreachy with us.

> I followed steps to run `make test-network-all` both on tor master and
> maint-0.3.5 branches, using Chutney master branch. I tested before and
> after tweaking value of HS_WAIT_FOR_UNCHECKED_DIR_INFO just to ensure
> my testing environment was sane; all tests passed (except for two that
> were skipped on master: mixed+hs-v23 and mixed+hs-v23-ipv6, and one
> on 0.3.5: mixed+hs-v2, I assume this is expected though) on the first
> go, on both branches of Tor.
> 
> Tests pass on Tor master after setting HS_WAIT_FOR_UNCHECKED_DIR_INFO
> to 0, but on maint-0.3.5 I've run tests a few times (both with and
> without CHUTNEY_DEBUG set) each time with a different outcome. Some of
> the tests which fail: basic-min [2], single-onion-v23 [3],
> bridges+ipv6-min [4], and ipv6-exit-min [5]. I have excerpted logs at
> the bottom of this message, narrowed down to test failures; let me know
> if I need to provide full logs or additional information.
> 
> I'll keep digging on this issue; I just wanted to show my current
> progress and share what I hope to be useful information.

Thank you for letting us know what you've done so far. And for providing
comprehensive logs.

This ticket is a cleanup ticket, after some other changes have been made.
I've edited the ticket description to make that clearer.

So we expect that the chutney tests will fail with
HS_WAIT_FOR_UNCHECKED_DIR_INFO = 0, until the following changes
have been merged:

Check for relay microdescriptor downloads:
 https://trac.torproject.org/projects/tor/ticket/33428

Check for onion service descriptor uploads:
https://trac.torproject.org/projects/tor/ticket/33609

Someone else is working on the microdescriptor changes at the moment.

Would you like to start working on the onion service descriptor changes?

T

> [1]: 
> "Stop waiting a set time for onion service descriptors"
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Walking onions status update: week 3 notes

[teor]

Hi Nick,

This is from week 3:

> On 30 Mar 2020, at 07:16, Nick Mathewson  wrote:
> 
>> There's a faster calculation and more efficient encoding for
>> Weighted, because there's a common factor in each POS()
>> calculation:
>> (1 << 32) / total_weight
>> 
>> If we remove that factor, we end up with a much simpler algorithm,
>> that's also more flexible:
>> 
>> Algorithm: Expanding a "Weighted" indexspec.
>> 
>> 
>> Each weighted indexspec also has a multiplier, which may vary
>> 
>> between indexspecs.
>> 
>> 
>> Let total_weight = SUM(indexspec.index_weights)
>> 
>> Verify total_weight * multiplier <= UINT64_MAX.
>> 
>> 
>> Let total_so_far = 0.
>> 
>> Let result_idx = {} (an empty mapping).
>> 
>> Define POS(b) = b * multiplier
>> 
>> 
>> For 0 <= i < LEN(indexspec.indexweights):
>> 
>>   Let w = indexspec.indexweights[i].
>> 
>>   Let lo = POS(total_so_far).
>> 
>>   Let total_so_far = total_so_far + w.
>> 
>>   Let hi = POS(total_so_far) - 1.
>> 
>>   Append (lo, hi) => i to result_idx.
>> 
>> 
>> Return result_idx.
> 
> I'm not so sure about this one -- see the note below about the total value.
> 
>> If multiplier is large, then the weights can be quite small.
>> 
>> (Small weights sacrifice some accuracy.) And if the multiplier
>> 
>> is 1, you effectively have a "Raw" index.
>> 
>> 
>> If you make those changes, you should be able to use a similar
>> process to expand all the different index types. (After multiplying,
>> truncating, or hashing, you either end up with a delta, or an
>> absolute position. You can turn deltas into absolute positions,
>> and then feed them all into the same base algorithm.)
>> 
>> There are also a few things I think might be bugs:
>> 
>> Was there meant to be a constraint that the Weighted total is
>> UINT64_MAX? Or close to UINT64_MAX?
> 
> The actual intention is that it has to be " = UINT32_MAX", not" <=
> UINT32_MAX".  Remember that the the client picks its next relay by
> specifying a random index value in range 0..UINT32_MAX, and it expects
> that the relay to that it is extending to _will_ have a snip that
> covers that range.
> 
>> The fixed parameters don't make much sense otherwise.

Hmm, ok, then if we use multiplication, there are going to be some issues
with the cumulative error from truncating the original division.

The error won't be too big, on average, it's N/2, worst case, it's N. (Where
N is the number of indexes.) That's a tiny amount for large weights, but a
huge amount for small weights. (Assuming that we continue to vote for
relays with weights 1-100.)

But divisions also have some cumulative error.

Once you've revised these sections, let's review?

Only doing one division on the authorities is very appealing.

T

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Walking onions status update: week 4 notes

[teor]

Hi Nick,

> On 30 Mar 2020, at 09:18, Nick Mathewson  wrote:
> 
> Walking Onions: week 4 update
> 
> …
> 
> == Next steps
> 
> In this coming week I plan to try to wrap up section 3 on voting and
> section 5 on extending circuits.  I'm going to go back to the start
> of section 2 and start revising all I've written so far for
> consistency.

Some typos, ambiguities, and questions:

In the client and relay root documents:

require-versions should be require-protos

In the voting description:

It seems weird to suddenly add a ".txt" extension to tor's
directory URLs. No other directory URL has an extension.
(Apart from DirPortFrontPage.)

This sentence fragment is ambiguous:
"more than half of two" for example, means "at least 1",
not "one or more"
I suggest:
"more than half of two" for example, means "greater than 1",
not "one or more"

N_PRESENT is a typo here:
The lowest integer that is greater than half of N_FIELD.
Equivalent to CEIL( (N_PRESENT+1)/2 ).

Do we need a list of MDs, one for each consensus method range?
   ? md_literal : LiteralMD,

Measured is a uint, not a bool:
   ? weight : {
  bw : uint,
  ? measured : bool,
  ? unmeasured : bool
   },
https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n2432

In the DirPort link specifiers:

IPv6 addresses MUST be in square brackets.
(To avoid port/address confusion.)

> [EXTENDING] 
> https://github.com/nmathewson/walking-onions-wip/blob/master/specs/05-extending-circuits.md
> 
> [VOTING]  
> https://github.com/nmathewson/walking-onions-wip/blob/master/specs/02-voting-and-authorities.md
> 
> [WIDE_EVERYTHING]
> https://github.com/nmathewson/walking-onions-wip/blob/master/other-proposals/xxx-wide-everything.md

T


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Walking onions status update: week 3 notes

[teor]

Hi Nick,

> On 21 Mar 2020, at 05:38, Nick Mathewson  wrote:
> 
> Walking Onions: week 3 update
> 
> As you might recall, the SNIPs need to be nice and small, but they
> need to be completely self-contained.  The ENDIVE, on the other
> hand, needs to be diff-friendly, and compressible.  Relays need to
> derive all the SNIPs from the ENDIVE.  The ENDIVE and SNIP formats
> that I worked on during last week [SNIPFMT] was designed to meet these
> criteria.
> 
> This week, I wrote up the algorithms to extract signed SNIPs from an
> ENDIVE.  This is in section 4 [RECONST] of the specs in my
> work-in-progress repository [GITREPO] This was mainly straightforward
> consequence from the work I did last week, but there were a few
> things that turned up.
> 
> One trickier aspect of this work is that we want to move most of the
> decision-making to the authorities, and have the relays simply do an
> expansion algorithm.  Updating all the relays would be slow, so relays
> need to be able to handle new index types and data types without
> fully understanding them.  That means that we need to specify an
> algorithm that can work flexibly with many future kinds of data.
> 
> Another tricky aspect is that the data which the relays need to put
> into SNIPs all needs to be signed, so all of that data needs to come
> out the same when relays calculate it as the when the authorities
> calculate it. Thus, all the algorithms for building Merkle trees and
> indices and SNIPRouterData bodies need to be deterministic.
> 
> I think that I got all of this relatively right in [RECONST], but
> there are probably some mistakes hiding in there.
> 
> I'm hoping that there is a cleaner alternative to the weighted-index
> reconstruction algorithm -- the one that's there right now puts a
> lot of requirements on the input data, in order to avoid
> floating-point operations.

There's a shorter encoding for Raw:

for each [i, pos1, pos2] in index_ranges:
  w = pos2 - pos1
  j  = the index of pos1 among all sorted pos1s
  new_encoding = [i, j, w]

[i, j, w] is an efficient encoding if index_ranges is sparse compared
with ENDIVERouterData, because:
* j has the same cardinality as I
* w is smaller than pos1 and pos2

If index_ranges is dense, there may be a more efficient encoding:
  add missing i with weight 0
  drop j

With this encoding, you can drop a few of the constraints.

There's a faster calculation and more efficient encoding for
Weighted, because there's a common factor in each POS()
calculation:
(1 << 32) / total_weight

If we remove that factor, we end up with a much simpler algorithm,
that's also more flexible:

Algorithm: Expanding a "Weighted" indexspec.

Each weighted indexspec also has a multiplier, which may vary
between indexspecs.

Let total_weight = SUM(indexspec.index_weights)
Verify total_weight * multiplier <= UINT64_MAX.

Let total_so_far = 0.
Let result_idx = {} (an empty mapping).
Define POS(b) = b * multiplier

For 0 <= i < LEN(indexspec.indexweights):
   Let w = indexspec.indexweights[i].
   Let lo = POS(total_so_far).
   Let total_so_far = total_so_far + w.
   Let hi = POS(total_so_far) - 1.
   Append (lo, hi) => i to result_idx.

Return result_idx.

If multiplier is large, then the weights can be quite small.
(Small weights sacrifice some accuracy.) And if the multiplier
is 1, you effectively have a "Raw" index.

If you make those changes, you should be able to use a similar
process to expand all the different index types. (After multiplying,
truncating, or hashing, you either end up with a delta, or an
absolute position. You can turn deltas into absolute positions,
and then feed them all into the same base algorithm.)

There are also a few things I think might be bugs:

Was there meant to be a constraint that the Weighted total is
UINT64_MAX? Or close to UINT64_MAX?
The fixed parameters don't make much sense otherwise.

I think v2 and v3 onion services assign descriptors to the next
highest HSDir RSA id (or ed25519 hash). But
INDEX_FROM_RING_KEYS() uses the relay input as the lowest value.

There is no next member for the last member in
INDEX_FROM_RING_KEYS(), but the code asks for one.
(Perhaps there are some typos here that make the code hard to
understand.)

We'll need special treatment for ring wrapping. (That is, 0xFF… is not
a real upper limit, it actually means, "use the lowest relay".)

It's weird to call a middle value a "suffix", but "infix" is also a bit of an
unusual word.

There are also a bunch of typos, let me know when you're ready for
copy-editing.

T___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Walking Onions status update: week 2 notes

[teor]

Hi Nick,

> On 20 Mar 2020, at 23:01, Nick Mathewson  wrote:
> 
> On Wed, Mar 18, 2020 at 11:21 AM teor  wrote:
>> 
>>> On 14 Mar 2020, at 14:44, teor  wrote:
>>> 
>>>> * As I work, I'm identifying other issues in tor that stand in
>>>>   the way of a good efficient walking onion implementation that
>>>>   will require other follow-up work.  This week I ran into a
>>>>   need for non-TAP-based v2 hidden services, and a need for a
>>>>   more efficient family encoding.  I'm keeping track of these
>>>>   in my outline file.
>> 
>> Here's another issue you might want to consider:
>> 
>> Currently, new relays get in the consensus as soon as:
>>  * they post their descriptors, and
>>  * a majority of authorities can contact their ORPorts.
>> 
>> That means clients and relays waste a whole bunch of bandwidth
>> downloading consensus info and descriptors for relays with very
>> low weights.
>> 
>> Instead, we could have two documents:
> 
> Thanks for this! This dovetails nicely with some of the voting design
> work I'm up to right now.

It would be great to have a protocol that doesn't depend on:
* time synchronisation
* big documents
* one-shot updates
* absolute consistency

We've already made vote timing a bit more robust in the tor master
branch, by ignoring late votes:
https://trac.torproject.org/projects/tor/ticket/4631

Here's a few other tweaks that might help:

Tor halves the consensus interval when there is no fresh consensus.
But changing the interval makes tor's code much more complex.
Instead, let's have a fixed consensus interval. And make it long
enough for efficiency, but short enough to recover from a failed
consensus.

Let's support vote diffs, as well as consensus diffs. Vote diffs
don't help when posting votes. But when requesting votes, authorities
can include hashes of votes they already have. That way, authorities
that are under heavy load are more likely to get all the votes.

We could increase the time that authorities have to fetch votes, and
make them retry fetches every few minutes.

We could do consistency checks on smaller shards, so that a
consistency failure in any one document does not break the entire
consensus.

We could create a shard for each supported consensus method (like we
do microdescriptors). That way, a consistency failure in any one
consensus method does not break the entire consensus.

We could make shards valid for a longer time, so that if the
replacement shard does not reach consensus, the older one is used.

Then, the final documents are a combination of all the consistent
shards, using the highest consistent consensus method. (Much like
the current microdesc consensus.)

Once we've made some of those changes, then some other changes
become plausible:

Let's make votes valid for exactly 2 voting periods, and use the
latest available vote from each authority.

Currently, each consensus can have one of two inputs from each
authority:
* the current vote, or
* no vote.
If a majority of authorities don't vote, then the consensus fails.
(And if enough bandwidth authorities don't vote, then measured
bandwidths fail.)

If there are up to two valid votes during a voting period, then
each consensus can have one of three inputs from each authority:
* the current vote, or
* the vote before the current vote, or
* no vote.
Having 3 possible choices is slightly worse than having 2 choices.

But with the changes above, authorities are more likely to have the
latest vote from each other authority. And having similar votes
will be enough for most of the shards to be consistent, for most
consensus methods.

If a majority of authorities don't have any valid votes, then the
consensus fails. But that's much less likely when there are two
valid votes at any one time.

We could also make each authority construct its own merkle root(s),
and allow the N most popular/recent roots on the network.
(Or equivalently, allow roots with current signatures from M
authorities.)

We could split votes into shards as well, and make authorities
exchange them like they exchange relay descriptors?
(When they see a reference to a new vote shard, they try to
download it from all other authorities.)

We'd need extra monitoring, to make sure that diffs, authorities,
shards, consensus methods, or latest votes aren't consistently
broken.

Maybe there's a few more steps we could take, and then we'd have
a voting protocol that doesn't require strict time
synchronisation. Where updates just happen as authorities make
them available, rather than all at once.

T




signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Walking Onions status update: week 2 notes

[teor]

Hi Nick,

> On 14 Mar 2020, at 14:44, teor  wrote:
> 
>>  * As I work, I'm identifying other issues in tor that stand in
>>the way of a good efficient walking onion implementation that
>>will require other follow-up work.  This week I ran into a
>>need for non-TAP-based v2 hidden services, and a need for a
>>more efficient family encoding.  I'm keeping track of these
>>in my outline file.

Here's another issue you might want to consider:

Currently, new relays get in the consensus as soon as:
  * they post their descriptors, and
  * a majority of authorities can contact their ORPorts.

That means clients and relays waste a whole bunch of bandwidth
downloading consensus info and descriptors for relays with very
low weights.

Instead, we could have two documents:

  * a "potential relays" document, that's used by the authorities,
bandwidth scanner, and any other relay test infrastructure
(perhaps exit scanners, sybil scanners, and other tools), and

  * a "useful relays" document, that contains good relays with
reasonable weights.

Let's think about this kind of efficiency as part of walking
onions.

We might even be able to make this change before walking onions, by:
  * making sbws and other tools use the ns consensus
(I think most tools already use the ns consensus), and
  * adding a new consensus method, which requires a minimum consensus
weight (or consensus weight fraction) for relays in the microdesc
consensus.

Since relays and clients use the microdesc consensus, low-weight
relays would disappear from the network. But they would still be in
the ns consensus.

We might find some interesting bugs in tor though. We never quite
got rid of all the old code that uses the ns consensus and full
descriptors.

T



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Lets give every circuit its own exit IP?

[teor]

Hi,

> On 8 Mar 2020, at 08:36, nusenu  wrote:
> 
> Would it help to write a short proposal to move this forward?

Yes, proposals help us know what to implement, when we have time.

The proposal can be short, but it needs to describe the feature in
enough detail, that a developer could implement it:
https://gitweb.torproject.org/torspec.git/tree/proposals/001-process.txt#n39

> Would there be someone to actually implement it?

If there's a good proposal, some volunteer or staff member may decide
to implement it, when they have time.

We can also include proposals in grant applications. (It's much harder
to include an idea in a grant application, we don't have enough details.)

Perhaps it might fit into some of our existing anti-censorship grants.
But that's something for the anti-censorship team to decide.
Again, a proposal makes that decision much easier.

> According to nickm:
> "This wouldn't be too hard, actually." [1]
> 
> As more platforms (i.e. youtube) are more strictly blocking IPs with bad
> reputation this would be a crucial feature to make the internet more 
> accessible
> to Tor users.

Yes, I agree.

Unfortunately, there are lots of really useful and important things
we can do. And never enough people to do them.

But a proposal is a good step forward.

> [1] https://trac.torproject.org/projects/tor/ticket/3847#comment:6

T




signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Auto close old tor GitHub pull requests?

[teor]

Hi all,

I'd like to use a GitHub bot to automatically close old tor and
torspec pull requests.

The Network Team currently uses GitHub to review pull requests
(but we use Trac for task tracking).

Some GitHub users won't contribute to a project with lots of open
pull requests, because they think it's been abandoned. (Or they
think we're bad at merging PRs.)

But lots of our PRs get left open because we've squashed and
merged the PR, and GitHub doesn't recognise the squashed
commits, so GitHub doesn't auto-close the PR.

I'm proposing closing PRs with no activity for 60 days, but I
think anywhere between 30 and 180 days would work.
(Most of our backports happen within 60 days.)

For more details, see:
https://trac.torproject.org/projects/tor/ticket/33629

T

-- 
teor
--

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Walking Onions status update: week 2 notes

[teor]

Hi Nick,

I'm interested in following along with Walking Onions, but I might
drop out when the relay IPv6 work gets busy.

I'm not sure how you'd like feedback, so I'm going to try to put it
in emails, or in pull requests.

(I made one comment on a git commit in walking-onions-wip, but
I'm not sure if you see those, so I'll repeat it here.)

> On 14 Mar 2020, at 03:52, Nick Mathewson  wrote:
> 
> This week, I worked specifying the nitty-gritty of the SNIP and
> ENDIVE document formats.  I used the CBOR meta-format [CBOR] to
> build them, and the CDDL specification language [CDDL] to specify
> what they should contain.
> 
> As before, I've been working in a git repository at [GITHUB]; you
> can see the document I've been focusing on this week at
> [SNIPFMT].  (That's the thing to read if you want to send me
> patches for my grammar.)

I'm not sure if you've got to exit ports yet, but here's one possible
way to partition ports:
* choose large partitions so that all exits support all ports in the
  partition
* choose smaller categories so that most exits support most ports
  in the partition
* ignore small partitions, they're bad for client privacy anyway

For example, you might end up with:
* web (80 & 443)
* interactive (SSH, IRC, etc.)
* bulk (torrent, etc.)
* default exit policy
* reduced exit policy

I'm not sure if we will want separate categories for IPv4-only
and dual-stack policies. We can probably ignore IPv6-only
policies for the moment, but we should think about them in
future.

> There were a few neat things to do here:
> 
>   * I had to define SNIPs so that clients and relays can be
> mostly agnostic about whether we're using a merkle tree or a
> bunch of signatures.
> 
>   * I had to define a binary diff format so that relays can keep
> on downloading diffs between ENDIVE documents. (Clients don't
> download ENDIVEs).  I did a quick prototype of how to output
> this format, using python's difflib.

Can we make the OrigBytesCmdId use start and length?
length may be shorter than end, and it will never be longer.

If we are doing chunk-based encoding, we could make start
relative to the last position in the original file. But that would
mean no back-tracking, which means we can't use some
more sophisticated diff algorithms.

>   * To make ENDIVE diffs as efficient as possible, it's important
> not to transmit data that changes in every ENDIVE.  To this
> end, I've specified ENDIVEs so that the most volatile parts
> (Merkle trees and index ranges) are recomputed on the relay
> side.  I still need to specify how these re-computations work,
> but I'm pretty sure I got the formats right.
> 
> Doing this calculation should save relays a bunch of
> bandwidth each hour, but cost some implementation complexity.
> I'm going to have to come back to this choice going forward
> to see whether it's worth it.
> 
>   * Some object types are naturally extensible, some aren't.  I've
> tried to err on the size of letting us expand important
> things in the future, and using maps (key->value mappings)
> for object that are particularly important.
> 
> In CBOR, small integers are encoded with a little less space
> than small strings.  To that end, I'm specifying the use of
> small integers for dictionary keys that need to be encoded
> briefly, and strings for non-tor and experimental extensions.
> 
>   * This is a fine opportunity to re-think how we handle document
> liveness.  Right now, consensus directories have an official
> liveness interval on them, but parties that rely on
> consensuses tolerate larger variance than is specified in the
> consensus.  Instead of that approach, the usable lifetime of
> each object is now specified in the object, and is ultimately
> controlled by the authorities.  This gives the directory
> authorities more ability to work around network tolerance
> issues.
> 
> Having large lifetime tolerances in the context of walking
> onions is a little risky: it opens us up to an attack where
> a hostile relay holds multiple ENDIVEs, and decides which one
> to use when responding to a request.  I think we can address this
> attack, however, by making sure that SNIPs have a published
> time in them, and that this time moves monotonically forward.

If the issue is having multiple valid ENDIVEs, then authorities could
also put a cap on the number of concurrently valid ENDIVEs.

There are two simple schemes to implement a cap:
* set a longer interval for rebuilding all ENDIVEs
  (the cap is the rebuild interval, divided by the validity interval)
* refuse to sign a new SNIP for a relay that's rapidly changing
  (or equivalently, leave that relay out of the next ENDIVE)

Both these schemes also limit the amount of bandwidth used
for a relay that's rapidly changing details.

>   * As I work, I'm identifying other issues in tor that stand in
> the way of a 

Re: [tor-dev] interested in working on a ticket

[teor]

Hi Amir,

> On 23 Feb 2020, at 10:28, indigo omega  wrote:
> 
> I am a new contributor interested in working on ticket 13184: 
> https://trac.torproject.org/projects/tor/ticket/13184 .
> The ticket says that it was assigned 8 months ago, but then deleted? I am 
> having trouble understanding if this ticket is under current development or 
> not. Should I start working on it and submit it for review, or should I email 
> someone, perhaps a maintainer, to see if someone is actively working on it. 
> Any advice would be much appreciated.

I don't think anyone is actively working on Torsocks right now.

I have cc'd David to confirm. David is the Torsocks maintainer.

T

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] TOR Browser Circuit Selection of Exit Nodes

[teor]

Hi,

Thanks for your interest in Tor's path selection algorithm.

Some of my colleagues are working on "vanguards", which
significantly changes path selection. I think this is their
latest proposal:
https://gitweb.torproject.org/torspec.git/tree/proposals/292-mesh-vanguards.txt

I'll let them share any details they feel are helpful.

See also my specific answers inline below:

> On 20 Feb 2020, at 06:20, Vianney Gomezgil Yaspik  wrote:
> 
> A group of students at Johns Hopkins University and I have been analyzing the 
> circuit selection algorithm for TOR´s browser EXIT nodes. It is an 
> exploratory project trying to discover how do the EXIT nodes are selected 
> every time the circuit changes. 
> 
> So far, we have discovered that the time that the browser is accessed, 
> calendar date,
> physical location, use of a bridge (or not), and the entry node do not change 
> the pattern of the EXIT nodes.

The set of Tor exits changes over time, so the calendar
date will change Tor's path selection slightly.

Similarly, Tor clients try to avoid choosing paths that
are within the same network, or all controlled by the
same operator. So guard selection does have a slight
impact on the chosen exit.

For more details, see tor's path spec, particularly the
constraints section:
https://gitweb.torproject.org/torspec.git/tree/path-spec.txt#n230

> Moreover, than approximately 80% of all exit nodes come from 10 specific 
> countries, even though these 10 countries only account for approximately 50% 
> of all available exit nodes. 

How are you counting exit nodes?

Tor uses the bandwidth weights in the consensus, to
weight its random selection of exit nodes:
https://gitweb.torproject.org/torspec.git/tree/path-spec.txt#n92

These weights are limited by:
* any operator-configured bandwidth limits,
and scaled using:
* the relay's own observed bandwidth usage, and
* the capacities measured by the 6 Tor bandwidth
  authorities.

> All of which has led us to conclude that the selection of the EXIT nodes in 
> the TOR browser is not random. We would like to further explore however, what 
> are the factors that determine which relay becomes the exit node every time a 
> circuit is changed. Is there anyone that we could speak to or that could give 
> us further insight as to how the selection of the exit node in TOR´s circuit 
> works?

Have you looked at the destination port?
Tor tried to select exits that will allow the requested
port.

Are you aware of preemptive circuits?
https://gitweb.torproject.org/torspec.git/tree/path-spec.txt#n147

If you're mainly measuring preemptive circuits, you'll
see fairly consistent behaviour. These circuits have
fewer constraints, because they need to be suitable
for general use.

That's probably enough to get you started, please
let us know if you have more questions.

T___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Making Tor's CI Faster

[teor]

Hi,

Want to help make Tor's CI go faster?

As part of the Sponsor 55 IPv6 work, I need to run Chutney IPv6 tests
on every Tor commit, as part of Tor's CI.

We're currently running these tests with fast_finish, because:
* Travis CI only has IPv6 on macOS
* macOS Chutney tests take 20-45 minutes in Travis CI
But fast_finish requires allow_failure.

We can't have allow_failure on code that's we're modifying every day,
so I'm going to make the Travis IPv6 Chutney job mandatory.

To speed up CI after this change, I'm going to make a new IPv6-only
test-network target, and run it in the macOS CI:
https://trac.torproject.org/projects/tor/ticket/33280

I've also identified one redundant Travis CI job that we can delete:
https://trac.torproject.org/projects/tor/ticket/33195#comment:3

Overall, these changes should reduce total CI runtime by 20-35
minutes. Due to parallelism, the wall clock time should reduce by
10-35 minutes. We can squeeze the most benefit from the parallelism
by sorting jobs in speed order (longest first):
https://trac.torproject.org/projects/tor/ticket/33194

If you have any other ideas for making Tor's CI go faster, please let
me know.

Slow CI is a particular issue for backports, because each 0.3.5
backport updates 9 branches:
  * {maint,release}-{0.3.5,0.4.1,0.4.2,0.4.3} and
  * master.

T

--
teor
--



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 313: Relay IPv6 Statistics

[teor]

Hi Karsten, Nick,

Thanks for your feedback!

I've removed the sections without your comments, to keep this email short.

> On 10 Feb 2020, at 20:49, Karsten Loesing  wrote:
> 
>> On 2020-02-10 07:36, teor wrote:
> 
> I'm including some comments below.
> 
>> Here is an initial draft of Proposal 313: Relay IPv6 Statistics.
>> 
>> This proposal includes:
>> * logging the number of IPv6 relays in the consensus, and
>> * relays publishing IPv6 connection and consumed bandwidth statistics.
>> 
>> This is the third of 3 proposals:
>> * Proposal 311: Relay IPv6 Reachability
>> * Proposal 312: Automatic Relay IPv6 Addresses
>> * Proposal 313: Relay IPv6 Statistics
>> 
>> ...
>> 
>> The full text is included below, and it is also available as a GitHub
>> pull request:
>> https://github.com/torproject/torspec/pull/108
>> 
>> The related tickets are #33159 (proposal) and #33051 and #33052
>> (implementation):
>> https://trac.torproject.org/projects/tor/ticket/33159
>> https://trac.torproject.org/projects/tor/ticket/33051
>> https://trac.torproject.org/projects/tor/ticket/33052
>> 
>> ...
>> 
>> Filename: 313-relay-ipv6-stats.txt
>> Title: Relay IPv6 Statistics
>> Author: teor
>> Created: 10-February-2020
>> Status: Draft
>> Ticket: #33159
>> 
>> ...
>> 
>> 3. Logging IPv6 Relays in the Consensus
>> 
>>   We propose that relays (and bridges) log:
>> * the number of relays, and
>> * the consensus weight fraction of relays,
>>   in the consensus that:
>> * have an IPv6 ORPort,
>> * support IPv6 reachability checks, and
>> * support IPv6 clients.

> On 11 Feb 2020, at 01:21, Nick Mathewson  wrote:
> 
> I don't understand the motivation behind doing this in the Tor code,
> since it's not something that relay operators need to know about or
> take action on.  To me, it seems more like something do do as part of
> metrics than in Tor per se.

I agree, we don't need these logs in tor. These calculations are
medium-term, and some of them only apply to Sponsor 55.

Also, as Karsten said, "Usable Guards" definition doesn't match Onionoo,
so these calculations really don't belong in metrics, either.

I've modified this section so we just do these calculations in a script:
https://github.com/torproject/torspec/pull/108/commits/91356f5db02b6a62afa3061278872b8d607db7ea

>>   In order to test these changes, and provide easy access to these
>>   statistics, we propose implementing a script that:
>> * downloads a consensus, and
>> * calculates and reports these statistics.
>> 
>>   As well as the statistics listed above, this script should also report the
>>   following relay statistic:
>> * support IPv6 reachability checks and IPv6 clients.
>> 
>>   The following consensus weight fractions should divide by the total
>>   consensus weight:
>> * have an IPv6 ORPort (all relays have an IPv4 ORPort), and
>> * support IPv6 reachability checks (all relays support IPv4 
>> reachability).
>> 
>>   The following consensus weight fractions should divide by the
>>   "usable Guard" consensus weight:
>> * support IPv6 clients, and
>> * support IPv6 reachability checks and IPv6 clients.
>> 
>>   "Usable Guards" have the Guard flag, but do not have the Exit flag. If the
>>   Guard also has the BadExit flag, the Exit flag should be ignored.
> 
> This definition is different from the one we're using in Onionoo for
> computing the "guard probability". There we include a relay with the
> Guard flag, regardless of whether it has the Exit and/or BadExit flag.
> Not sure if this matters and which definition is more useful, I just
> wanted to point out that they're different.

The Onionoo definition is long-term, see Nick's explanation:

> On 11 Feb 2020, at 01:21, Nick Mathewson  wrote:
> 
> It seems to me that this rule should depend on the Wgd
> bandwidth-weights value ("Weight for Guard+Exit-flagged nodes in the
> guard Position"), right?  (Right now that is zero, and I don't expect
> it to change.)

You're right, I've made that check part of the script design:
https://github.com/torproject/torspec/pull/108/commits/91356f5db02b6a62afa3061278872b8d607db7ea

Since I mainly expect to use the script for Sponsor 55 in 2020, I don't
propose a design for other values of Wgd. The script should just warn.
(These warnings might happen in chutney networks.)

>>   We propose that these logs happen whenever tor:
>> * receives a consensus from a directory server, or
>> * loads a live, 

[tor-dev] Proposal 313: Relay IPv6 Statistics

[teor]

Hi,

Here is an initial draft of Proposal 313: Relay IPv6 Statistics.

This proposal includes:
* logging the number of IPv6 relays in the consensus, and
* relays publishing IPv6 connection and consumed bandwidth statistics.

This is the third of 3 proposals:
* Proposal 311: Relay IPv6 Reachability
* Proposal 312: Automatic Relay IPv6 Addresses
* Proposal 313: Relay IPv6 Statistics

I revised proposals 311 and 312 last week, and merged them to torspec as
drafts.

There are still some TODO items in the proposal, about:
* safely collecting these new statistics on bridges, and
* getting accurate IPv6 connection statistics.
If you know about tor's statistics, please give us some feedback!

The full text is included below, and it is also available as a GitHub
pull request:
https://github.com/torproject/torspec/pull/108

The related tickets are #33159 (proposal) and #33051 and #33052
(implementation):
https://trac.torproject.org/projects/tor/ticket/33159
https://trac.torproject.org/projects/tor/ticket/33051
https://trac.torproject.org/projects/tor/ticket/33052

Please feel free to reply on this list, or via GitHub pull request
comments.

Filename: 313-relay-ipv6-stats.txt
Title: Relay IPv6 Statistics
Author: teor
Created: 10-February-2020
Status: Draft
Ticket: #33159

0. Abstract

   We propose that Tor relays (and bridges) should log the number of relays in
   the consensus that support IPv6 extends, and IPv6 client connections.

   We also propose that Tor relays (and bridges) should collect statistics on
   IPv6 connections and consumed bandwidth. Like tor's existing connection
   and consumed bandwidth statistics, these new IPv6 statistics will be
   published in each relay's extra-info descriptor.

1. Introduction

   Tor relays (and bridges) can accept IPv6 client connections via their
   ORPort. But current versions of tor need to have an explicitly configured
   IPv6 address (see [Proposal 312: Relay Auto IPv6 Address]), and they don't
   perform IPv6 reachability self-checks (see
   [Proposal 311: Relay IPv6 Reachability]).

   As we implement these new IPv6 features in tor, we want to monitor their
   impact on the IPv6 connections and bandwidth in the tor network.

   Tor developers also need to know how many relays support these new IPv6
   features, so they can test tor's IPv6 reachability checks. (In particular,
   see section 4.3.1 in [Proposal 311: Relay IPv6 Reachability]:  Refusing to
   Publish the Descriptor.)

2. Scope

   This proposal modifies Tor's behaviour as follows:

   Relays, bridges, and directory authorities log the number of relays that
   support IPv6 clients, and IPv6 relay reachability checks. They also log the
   corresponding consensus weight fractions.

   As an optional change, tor clients may also log this information.

   Relays, bridges, and directory authorities collect statistics on:
 * IPv6 connections, and
 * IPv6 consumed bandwidth.
   The design of these statistics will be based on tor's existing connection
   and consumed bandwidth statistics.

   Tor's existing consumed bandwidth statistics truncate their totals to the
   nearest kilobyte. The existing connection statistics do not perform any
   binning.

   We do not proposed to add any extra noise or binning to these statistics.
   Instead, we expect to leave these changes until we have a consistent
   privacy-preserving statistics framwework for tor. As an example of this
   kind of framework, see
   [Proposal 288: Privacy-Preserving Stats with Privcount (Shamir version)].

   We avoid:
 * splitting connection statistics into clients and relays, and
 * collecting circuit statistics.
   These statistics are more sensitive, so we want to implement
   privacy-preserving statistics, before we consider adding them.

   Throughout this proposal, "relays" includes directory authorities, except
   where they are specifically excluded. "relays" does not include bridges,
   except where they are specifically included. (The first mention of "relays"
   in each section should specifically exclude or include these other roles.)

   Tor clients do not collect any statistics for public reporting. Therefore,
   clients are out of scope in this proposal. (Except for some optional changes
   to client logs, where they log the number of IPv6 relays in the consensus).

   When this proposal describes Tor's current behaviour, it covers all
   supported Tor versions (0.3.5.7 to 0.4.2.5), as of January 2020, except
   where another version is specifically mentioned.

3. Logging IPv6 Relays in the Consensus

   We propose that relays (and bridges) log:
 * the number of relays, and
 * the consensus weight fraction of relays,
   in the consensus that:
 * have an IPv6 ORPort,
 * support IPv6 reachability checks, and
 * support IPv6 clients.

   In order to test these changes, and provide easy access to these
   statistics, we propose implementing a script that

Re: [tor-dev] Proposal 312: Automatic Relay IPv6 Addresses

[teor]

Hi,

> On 4 Feb 2020, at 07:17, s7r  wrote:
> 
> teor wrote:
>> Hi s7r,
>> 
>> Thanks for bringing up IPv6 address privacy extensions.
>> 
>>> On 30 Jan 2020, at 02:19, s7r  wrote:
>>> 
>> 
>> I read RFCs 4941 and 3041, looked at the tor directory spec, and did some
>> analysis:
>> * tor clients get new relay addresses within 4.5 hours
>> * IPv6 privacy extensions rotate addresses every day (by default)
>> * IPv6 privacy extensions remove old addresses after a week (by default)
>> 
>> (And applications have to opt-in to IPv6 privacy extensions addresses,
>> by default, according to the RFC.)
>> 
>> Therefore, I don't think tor relays or clients will be affected by relays
>> using IPv6 privacy extensions.
>> 
>> See my detailed analysis here:
>> https://github.com/torproject/torspec/pull/105/files#diff-28c992d72bedaa9378a4f3627afb8694R816
> 
> Thanks for looking into it!
> 
> I agree with your analysis fully. However, I just think it would be
> better if we mention in proposal 312 explicitly that Tor should try hard
> to get an IPv6 address that has the desired state, and use that. It is
> true that this is different on each operating system, but the operating
> systems we most care about should be pretty trivial to patch for this
> change.
> 
> IPv6 addresses have multiple states. We simply request for one that has
> state `public` and not `temporary`.
> (https://tools.ietf.org/html/rfc3484).

I've made this change to the proposal, using similar wording to what
you wrote in your previous email:

https://github.com/torproject/torspec/pull/105/commits/ca8dfa983ad78d67f68a6f0fff0a9dd90ec8c5f2

Here's why I didn't make it a mandatory change:

Tor supports address detection using DNS, and detection of the public
address of any NAT box between tor and the internet. Therefore, the
address tor publishes in its descriptor may not be on the local
machine.

Here are the relevant address detection methods:

1. Address hostname
2. ORPort hostname
(method 3 only uses local information)
4. auto hostname
5. auto dir header

Therefore, it's not possible for tor to reliably find out the IPv6
address privacy extensions status of these addresses.

(Some operators also use sandboxes that block the relevant APIs.)

> In the current form of this proposal, it looks kind of optional ("We
> propose this optional change, to improve...").

IPv6 privacy extensions are an optional change for Sponsor 55.

Sponsor 55 covers relay IPv6 reachability checks, address auto-detection,
and basic IPv6 statistics. It's a small sponsor, so we need to focus on
the changes that are required to achieve these goals.

We'll choose which optional changes we implement and test, after all the
required changes are implemented and tested.

As I wrote to Nick:

>>> I'm trying to keep a clear distinction in this proposal, to keep the
>>> sponsor 55 scope manageable. So I am keeping different sections for:
>>>  * required changes: changes that we must make to achieve the objectives
>>>  of sponsor 55
>>>  * optional changes: good ideas that we can implement if we have time left
>>>  in sponsor 55, or in future IPv6 work

> sbws of course should account relays per IPv6 prefix, and not per
> address. Usually we should be able to determine if an address is in the
> same /64 IPv6 subnet and not reset the bandwidth measurement because
> most probably it is the same relay. A /64 is standard, however there are
> ISPs that do now follow the standard in assigning /64 to end users and
> sometimes assign /112 or strange things like that. So this can become
> complicated again. Which is why it is more simple to always ask for a
> `public` IPv6 address and ignore `temporary` ones. I think it's simpler
> and more efficient than changing sbws.

I don't think we'll be back porting any code changes that make relays
avoid IPv6 privacy extensions addresses.

Also, any IPv6 privacy extensions code probably won't work on all of our
supported platforms:
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/SupportedPlatforms

So sbws will have to support relays which use IPv6 privacy extensions.

(sbws changes are out of scope for Sponsor 55, but we're looking for other
funding.)

> # NOT DIRECTLY RELATED TO PROPOSAL 312 SECTION #
> These privacy extensions IPv6 addresses might be good for outgoing IPv6
> exit connections, like changing per circuit or per destination to get
> rid of captchas and blacklists, but this is something different.

Feel free to open a ticket for this feature.

> Our internal DoS defense subsystem should also treat prefixes instead of
> addresses, because right now with a client with a /64 public IP

Re: [tor-dev] Proposal 312: Automatic Relay IPv6 Addresses

[teor]

Hi Nick,

Thanks so much for your review!

I've made most of the changes you've suggested, you can see the latest
version of the proposal here:
https://github.com/torproject/torspec/pull/105/files

I've also made changes in response to s7r's feedback about IPv6 privacy
extensions. Since sending the draft, I've also noticed some missing
information, and some things that could be explained better.

I'm trying to keep a clear distinction in this proposal, to keep the
sponsor 55 scope manageable. So I am keeping different sections for:
  * required changes: changes that we must make to achieve the objectives
  of sponsor 55
  * optional changes: good ideas that we can implement if we have time left
  in sponsor 55, or in future IPv6 work

There's some flexibility, particularly if we decide that an optional
change is the fastest way to get something implemented.

I'll send another full text to the list when all the reviews are done.

I've also responded to your comments below individually:

> On 31 Jan 2020, at 08:08, Nick Mathewson  wrote:
> 
> On Wed, Jan 29, 2020 at 9:04 AM teor  wrote:
> 
> Hello again!  This looks like another fine proposal.  I'm leaving
> comments inline, and clipping sections that I'm not commenting on.
> 
>> Filename: 312-relay-auto-ipv6-addr.txt
>> Title: Tor Relays Automatically Find Their IPv6 Address
>> Author: teor
>> Created: 28-January-2020
>> Status: Draft
>> Ticket: #33073
>> 
>> 0. Abstract
>> 
>>   We propose that Tor relays (and bridges) should automatically find their
>>   IPv6 address, and use it to publish an IPv6 ORPort. For some relays to find
>>   their IPv6 address, they may need to fetch some directory documents from
>>   directory authorities over IPv6. (For anonymity reasons, bridges are unable
>>   to fetch directory documents over IPv6, until clients start to do so.)
>> 
>> 1. Introduction
>> 
>>   Tor relays (and bridges) currently find their IPv4 address, and use it as
>>   their ORPort and DirPort address when publishing their descriptor. But
>>   relays and bridges do not automatically find their IPv6 address.
> 
> At the beginning of this document, we should be a bit more clear about
> which address specifically we're trying to find.  If we wanted _some_
> address, or if NAT and firewalls didn't exist, we could just open a
> socket, call getsockname(), and be done with it.  What we are looking
> for specifically is an address that we can advertise to the rest of
> the world in our server descriptor.  [I know you know this, but we
> should say so.]

Thanks, that's a great reminder. I've explained how we use the detected
IP addresses, and what kind of addresses we are looking for.

I've left a detailed description of how we ignore private addresses, and
use other methods, to later in the proposal.

Please see this commit:
https://github.com/torproject/torspec/pull/105/commits/dff29ec0424a31147c040a8d8b6724df4d2dfc25

> [...]
>> 3. Finding Relay IPv6 Addresses
>> 
>>   We propose that tor relays (and bridges) automatically find their IPv6
>>   address, and use it to publish an IPv6 ORPort.
>> 
>>   For some relays to find their IPv6 address, they may need to fetch some
>>   directory documents from directory authorities over IPv6. (For anonymity
>>   reasons, bridges are unable to fetch directory documents over IPv6, until
>>   clients start to do so.)
>> 
>> 3.1. Current Relay IPv4 Address Implementation
>> 
>>   Currently, all relays (and bridges) must have an IPv4 address. IPv6
>>   addresses are optional for relays.
>> 
>>   Tor currently tries to find relay IPv4 addresses in this order:
>> 1. the Address torrc option
>> 2. the address of the hostname (resolved using DNS, if needed)
>> 3. a local interface address
>>(by making a self-connected socket, if needed)
>> 4. an address reported by a directory server (using X-Your-Address-Is)
> 
> Any server, or only an authority?  Over any connection, or only an
> authenticated one?

Thanks for picking up on this inconsistency.

In the current implementation, when a tor relay doesn't know its address, it
tried to fetch from directory authorities. But it will believe an addresses from
any directory server.

Relays try to use DirPorts, even if they don't know their own IP address. But if
they select a directory mirror that only has an ORPort, they will use its 
ORPort.

I've fixed this particular inconsistency in this commit:
https://github.com/torproject/torspec/pull/105/commits/dff29ec0424a31147c040a8d8b6724df4d2dfc25

Ideally, relays should use authenticated directory authorities to discover their
addresses. But it's not a simple change,

Re: [tor-dev] Proposal 312: Automatic Relay IPv6 Addresses

[teor]

Hi s7r,

Thanks for bringing up IPv6 address privacy extensions.

> On 30 Jan 2020, at 02:19, s7r  wrote:
> 
> There is another RFC (older), that is in use by Debian apparently:
> RFC3041. https://tools.ietf.org/rfc/rfc3041.txt
> 
> From:
> https://manpages.debian.org/buster/iproute2/ip-address.8.en.html
> see `mngtmpaddr`
> 
> RFC4941 is newer and with some improvements, however it does not mention
> its purpose is to update / deprecate RFC3041. Actually it mentions the
> differences / improvements.
> 
> Anyway, the point still fully stands, I just thought both RFCs should be
> mentioned. Bottom line still is temporary (expiring) but otherwise
> public and reachable IPv6 addresses in relay descriptor.
> 
> s7r wrote:
>> 
>> 
>> By briefly reviewing I've noticed something important is missing that
>> should be a part of this proposal.
>> 
>> I am not sure under which section it should go under. I guess `3.2.2.
>> Use the Advertised ORPort IPv4 and IPv6 Addresses`, or maybe it's
>> important enough that we should make its own section.
>> 
>> In IPv6, besides publicly routable and non-publicly routable addresses
>> (fe80:// etc.) which are documented in the proposal, we have temporary
>> IPv6 addresses coming from Privacy extensions or RFC4941 IPv6 addresses.
>> 
>> https://tools.ietf.org/rfc/rfc4941.txt
>> 
>> These addresses are publicly routable, they can appear as reachable from
>> the directory authorities or from directory data fetches, but they have
>> limited lifetime and change over time. I am not sure if one  such
>> address becomes deprecated if already in use (say by Tor), as the RFC
>> states MAY _if not in use by applications  or upper layers_:
>> 
>>   "As an optional optimization, an implementation MAY remove a
>>   deprecated temporary address that is not in use by applications or
>>   upper layers as detailed in Section 6."
>> 
>> But since this is implementation dependent, we cannot be sure about the
>> behavior across different platforms that relays might run on.
>> 
>> It is up to the operating system if such addresses are used or not. In
>> Debian they are disabled by default net.ipv6.conf.eth0.use_tempaddr=0
>> (unless some desktop packages that use network manager with different
>> settings change it). In Windows (at least Windows 10) apparently they
>> are enabled by default.
>> 
>> The question is, do we want such addresses in relay descriptors? I think
>> such addresses will behave similar to dynamic IPv4 addresses, or even
>> worse since these ones really change when they want, not just when we
>> disconnect and reconnect the network interface. So maybe Tor should
>> detect such behavior and log an error or something?
>> 
>> Actually I'll setup a vm this weekend and give it a native, static /64
>> IPv6 prefix, enable privacy extension to use temporary addresses and
>> spin up a Tor process on it. Then disconnect the internet a couple of
>> times and see how it behaves, how often it changes.
>> 
>> What do you think?

I read RFCs 4941 and 3041, looked at the tor directory spec, and did some
analysis:
* tor clients get new relay addresses within 4.5 hours
* IPv6 privacy extensions rotate addresses every day (by default)
* IPv6 privacy extensions remove old addresses after a week (by default)

(And applications have to opt-in to IPv6 privacy extensions addresses,
by default, according to the RFC.)

Therefore, I don't think tor relays or clients will be affected by relays
using IPv6 privacy extensions.

See my detailed analysis here:
https://github.com/torproject/torspec/pull/105/files#diff-28c992d72bedaa9378a4f3627afb8694R816

(I still have to revise proposal 312 based on Nick's review, I hope to do
that today or tomorrow.)

T


signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 312: Automatic Relay IPv6 Addresses

[teor]

Hi,

Here is an initial draft of Proposal 312: Automatic Relay IPv6 Addresses.

This proposal includes:
 * relay auto IPv6 addresses, and
 * relay auto IPv6 ORPorts.

This is the second of 3 proposals:
* Proposal 311: Relay IPv6 Reachability
* Proposal 312: Automatic Relay IPv6 Addresses
* Proposal 313: Relay IPv6 Statistics
(I haven't written the final one yet.)

I also want to make some minor changes to Proposal 306, so that bridge
IPv6 behaviour stays in sync with client IPv6 behaviour. (See section
7 of this proposal for details.)

There is still one TODO item in the proposal, about Tor's current
behaviour. If you know the answer, please let me know.

The full text is included below, and it is also available as a GitHub
pull request:
https://github.com/torproject/torspec/pull/105

The related tickets are #33073 (proposal) and #5940 (implementation):
https://trac.torproject.org/projects/tor/ticket/33073
https://trac.torproject.org/projects/tor/ticket/5940

Please feel free to reply on this list, or via GitHub pull request
comments.

Filename: 312-relay-auto-ipv6-addr.txt
Title: Tor Relays Automatically Find Their IPv6 Address
Author: teor
Created: 28-January-2020
Status: Draft
Ticket: #33073

0. Abstract

   We propose that Tor relays (and bridges) should automatically find their
   IPv6 address, and use it to publish an IPv6 ORPort. For some relays to find
   their IPv6 address, they may need to fetch some directory documents from
   directory authorities over IPv6. (For anonymity reasons, bridges are unable
   to fetch directory documents over IPv6, until clients start to do so.)

1. Introduction

   Tor relays (and bridges) currently find their IPv4 address, and use it as
   their ORPort and DirPort address when publishing their descriptor. But
   relays and bridges do not automatically find their IPv6 address.

   However, relay operators can manually configure an ORPort with an IPv6
   address, and that ORPort is published in their descriptor in an "or-address"
   line (see [Tor Directory Protocol]).

   Many relay operators don't know their relay's IPv4 or IPv6 addresses. So
   they rely on Tor's IPv4 auto-detection, and don't configure an IPv6
   address. When operators do configure an IPv6 address, it's easy for them to
   make mistakes. IPv6 ORPort issues are a significant source of relay
   operator support requests.

   Implementing IPv6 address auto-detection, and IPv6 ORPort reachability
   checks (see [Proposal 311: Relay IPv6 Reachability]) will increase the
   number of working IPv6-capable relays in the tor network.

2. Scope

   This proposal modifies Tor's behaviour as follows:

   Relays, bridges, and directory authorities:
 * automatically find their IPv6 address, and
 * for consistency between IPv4 and IPv6 detection:
   * start using IPv4 ORPort and DirPort for IPv4 address detection, and
   * re-order IPv4 address detection methods.

   Relays and directory authorities (but not bridges):
 * fetch some directory documents over IPv6.
   For anonymity reasons, bridges are unable to fetch directory documents over
   IPv6, until clients start to do so. (See
   [Proposal 306: Client Auto IPv6 Connections].)

   This proposal makes a small, optional change to existing client behaviour:
 * clients also check IPv6 addresses when rotating TLS keys for new
   networks.
   In addition to the changes to IPv4 address resolution, most of which won't
   affect clients. (Because they do not set Address, ORPort, or DirPort.)

   Throughout this proposal, "relays" includes directory authorities, except
   where they are specifically excluded. "relays" does not include bridges,
   except where they are specifically included. (The first mention of "relays"
   in each section should specifically exclude or include these other roles.)

   When this proposal describes Tor's current behaviour, it covers all
   supported Tor versions (0.3.5.7 to 0.4.2.5), as of January 2020, except
   where another version is specifically mentioned.

3. Finding Relay IPv6 Addresses

   We propose that tor relays (and bridges) automatically find their IPv6
   address, and use it to publish an IPv6 ORPort.

   For some relays to find their IPv6 address, they may need to fetch some
   directory documents from directory authorities over IPv6. (For anonymity
   reasons, bridges are unable to fetch directory documents over IPv6, until
   clients start to do so.)

3.1. Current Relay IPv4 Address Implementation

   Currently, all relays (and bridges) must have an IPv4 address. IPv6
   addresses are optional for relays.

   Tor currently tries to find relay IPv4 addresses in this order:
 1. the Address torrc option
 2. the address of the hostname (resolved using DNS, if needed)
 3. a local interface address
(by making a self-connected socket, if needed)
 4. an address reported by a directory server (using X-Your-Address-Is)

   When using 

Re: [tor-dev] Prop 311: Relay IPv6 Reachability

[teor]

Hi Nick,

Thanks for your proposal review.

I've made most of the changes you've suggested, you can see the latest
version of the proposal here:
https://github.com/torproject/torspec/pull/103/files

I'll send another full text to the list when all the reviews are done.

I've also responded to your comments below individually:

> On 25 Jan 2020, at 06:09, Nick Mathewson  wrote:
> 
> On Fri, Jan 24, 2020 at 1:31 AM teor  wrote:
>> 
>> 
>> Here is an initial draft of Proposal 311: Relay IPv6 Reachability.
>> 
>> This proposal includes:
>>  * relay IPv6 ORPort extends, and
>>  * relay IPv6 ORPort reachability checks.
>> 
>> This is the first of 3 proposals:
>> * Proposal 311: Relay IPv6 Reachability
>> * Proposal 312: Automatic Relay IPv6 Addresses
>> * Proposal 313: Relay IPv6 Statistics
>> (I haven't written the others yet!)
>> 
>> I also want to make some minor changes to Proposal 306, so that bridge
>> IPv6 behaviour stays in sync with client IPv6 behaviour.
>> 
>> The full text is included below, and it is also available as a GitHub
>> pull request:
>> https://github.com/torproject/torspec/pull/103
>> 
>> The related tickets are #24404 (proposal) and #24403 (implementation):
>> https://trac.torproject.org/projects/tor/ticket/24404
>> https://trac.torproject.org/projects/tor/ticket/24403
>> 
>> Please feel free to reply on this list, or via GitHub pull request
>> comments.
> 
> This proposal seems like a good start: It's solid, readable, and I
> think it would work.  There are few places where I have questions or
> suggested clarifications, and a few where I think we should consider
> other designs as well.
> 
> I'm going to only quote the parts I'm commenting on.
> 
>> 3.2.1. Making IPv6 ORPort Extend Connections
>> 
>>   Relays can make a new connection over IPv6 when:
>> * there is no existing authenticated connection to the requested relay,
>>   and
>> * the extend cell contains an IPv6 ORPort.
> 
> Also we should probably have the condition, "The relay itself supports
> IPv6, or thinks that it does"?

Thanks, I included that condition later in the proposal, but missed it here:
https://github.com/torproject/torspec/pull/103/commits/9204293f9b1e106ca73c4726703c3560f81cdeb8

>>   If these conditions are satisfied, and the extend cell also contains an
>>   IPv4 ORPort, we propose that the relay choose between an IPv4 and an IPv6
>>   connection at random.
> 
> [...]
>> 3.3. Alternative Extend Designs
>> 
>>   We briefly mention some potential extend designs, and the reasons that
>>   they were not used in this proposal.
>> 
>>   (Some designs may be proposed for future Tor versions, but are not 
>> necessary
>>   at this time.)
> 
> Let me sketch another alternative design here:
> 
> Suppose that a relay's extend cell contains the IPv4 address and the
> IPv6 address in their _preferred order_.  So if the party generating
> the extend cell would prefer an IPv4 connection, it puts the IPv4
> addess first; if it would prefer an IPv6 connection, it puts the IPv6
> address first.
> 
> The relay that receives the extend cell could respond in several ways:
>   * One possibility (similar to your 3.2.1) is to choose at random,
> with a higher probability given to the first option.
>   * One possibility (similar to your 3.3.1) is to try the first, and
> then try the second if the first one fails.
> 
> I think this scheme has some advantage, in that it lets the
> self-testing relay say "please try IPv6 if you can" or "please try
> IPv4 if you can" in a reliable way, and lets us migrate from the
> current behavior to the 3.3.1 behavior down the road.

I've added this alternate design as section 3.3.3, and renumbered the
previous section 3.3.3 to 3.4:
https://github.com/torproject/torspec/pull/103/commits/654bcb16815c0c57d5c551de53a79116a6d0188a

Apart from relay reachability self-tests, I'm not sure if there are any
other use cases where preferring IPv4 or IPv6 extends is necessary. Most
clients shouldn't care, they just want to get to an exit safely. (And
most clients can't care, because existing connections may be used, and
they may be over IPv4 or IPv6.)

But I wanted to include this design in the proposal, in case we find that
it is necessary during testing.

> [...]
>> 4.3. Refuse to Publish Descriptor if IPv6 ORPort is Unreachable
>> 
>>   If an IPv6 ORPort reachability check fails, relays (and bridges) should log
>>   a warning.
>> 
>>   In addition, if IPv6ORPort reachability checks are reliable, based on the
>>   number of relays in the network that sup

[tor-dev] Prop 311: Relay IPv6 Reachability

[teor]

Hi,

Here is an initial draft of Proposal 311: Relay IPv6 Reachability.

This proposal includes:
  * relay IPv6 ORPort extends, and
  * relay IPv6 ORPort reachability checks.

This is the first of 3 proposals:
* Proposal 311: Relay IPv6 Reachability
* Proposal 312: Automatic Relay IPv6 Addresses
* Proposal 313: Relay IPv6 Statistics
(I haven't written the others yet!)

I also want to make some minor changes to Proposal 306, so that bridge
IPv6 behaviour stays in sync with client IPv6 behaviour.

There are still a few TODO items in the proposal, mostly about Tor's
current behaviour. If you know the answers, please let me know.

The full text is included below, and it is also available as a GitHub
pull request:
https://github.com/torproject/torspec/pull/103

The related tickets are #24404 (proposal) and #24403 (implementation):
https://trac.torproject.org/projects/tor/ticket/24404
https://trac.torproject.org/projects/tor/ticket/24403

Please feel free to reply on this list, or via GitHub pull request
comments.

Filename: 311-relay-ipv6-reachability.txt
Title: Tor Relay IPv6 Reachability
Author: teor
Created: 22-January-2020
Status: Draft
Ticket: #24404

0. Abstract

   We propose that Tor relays and bridges should check the reachability of
   their IPv6 ORPort, before publishing their descriptor. To check IPv6 ORPort
   reachability, relays and bridges need to be able to extend circuits via
   other relays, and back to their own IPv6 ORPort.

1. Introduction

   Tor relays and bridges currently check the reachability of their IPv4
   ORPort and DirPort before publishing them in their descriptor. But relays
   and bridges do not test the reachability of their IPv6 ORPorts.

   However, Directory Authorities make direct connections to relay IPv4 and
   IPv6 ORPorts, to test each relay's reachability. Once a relay has been
   confirmed as reachable by a majority of authorities, it is included in the
   consensus. (Currently, 6 out of 9 Directory Authorities perform IPv4 and
   IPv6 reachability checks. The others just check IPv4.)

   The Bridge authority makes direct connections to bridge IPv4 ORPorts, to
   test each bridge's reachability. Depending on its configuration, it may also
   test IPv6 ORPorts. Once a bridge has been confirmed as reachable by the
   bridge authority, it is included in the bridge networkstatus used by
   BridgeDB.

   Many relay and bridge operators don't know when their relay's IPv6 ORPort is
   unreachable. They may not find out until they check [Relay Search], or
   their traffic may drop. For new operators, it might just look like Tor
   simply isn't working, or it isn't using much traffic. IPv6 ORPort issues
   are a significant source of relay and bridge operator support requests.

   Implementing IPv6 ORPort reachability checks will provide immediate, direct
   feedback to operators in the relay or bridge's logs. It also enables future
   work, such as automatically discovering relay and bridge addresses for IPv6
   ORPorts (see [Proposal 312]).

2. Scope

   This proposal modifies Tor's behaviour as follows:

   Relays:
   * circuit extension,
   * OR connections for circuit extension,
   * reachability testing.

   Bridges:
   * reachability testing only.

   This proposal does not change client behaviour.

   When this proposal describes Tor's current behaviour, it covers all
   supported Tor versions (0.3.5.7 to 0.4.2.5), as of January 2020.

3. Allow Relay IPv6 Extends

   To check IPv6 ORPort reachability, relays and bridges need to be able to
   extend circuits via other relays, and back to their own IPv6 ORPort.

   We propose that relays start to extend some circuits over IPv6 connections.
   We do not propose any changes to bridge extend behaviour.

3.1. Current IPv6 ORPort Implementation

   Currently, all relays and bridges must have an IPv4 ORPort. IPv6 ORPorts
   are optional for relays and bridges.

   Tor supports making direct IPv6 OR connections:
 * from directory authorities to relay ORPorts,
 * from the bridge authority to bridge ORPorts,
 * from clients to relay and bridge ORPorts.

   Tor relays and bridges accept IPv6 ORPort connections. But IPv6 ORPorts are
   not currently included in extend requests to other relays. And even if an
   extend cell contains an IPv6 ORPort, bridges and relays will not extend
   via an IPv6 connection to another relay.

   Instead, relays will extend circuits:
 * Using an existing authenticated connection to the requested relay
   (which is typically over IPv4), or
 * Over a new connection via the IPv4 ORPort in an extend cell.

   If a relay receives an extend cell that only contains an IPv6 ORPort, the
   extend typically fails.

3.2. Relays Extend to IPv6 ORPorts

   We propose that relays make some connections via the IPv6 ORPorts in
   extend cells.

   Relays will extend circuits:
 * using an existing authenticated connection to the requested relay
   (which may be over IPv4 or IPv6

Re: [tor-dev] Updates to Prop306: A Tor Implementation of IPv6 Happy Eyeballs

[teor]

ptions.)

   We should also allow falling back to the alternate address. For this,
   a design will be given in Section 3.1.

3.1. TCP Connection to Preferred Address On First TCP Success

   In this design, we will connect via TCP to the first preferred address.
   On a failure or after a 250 msec delay, we attempt to connect via TCP to
   the alternate address. On a success, Tor attempts to authenticate and
   closes the other connection.

   This design is close to RFC 8305 and is similar to how Happy Eyeballs
   is implemented in a web browser.

3.2. Handling Connection Successes And Failures

   Should a connection to a entry node succeed and is authenticated via TLS,
   we can then use the connection. In this case, we should cancel all other
   connection timers and in-progress connections. Cancelling the timers is
   necessary so we don't attempt new unnecessary connections when our
   existing connection is successful, preventing denial-of-service risks.

   However, if we fail all available and allowed connections, we should tell
   the rest of Tor that the connection has failed. This is so we can attempt
   another entry node.

3.3. Connection Attempt Delays

   As mentioned in [TEOR-P306-REP], initially, clients should prefer IPv4
   by default. The Connection Attempt Delay, or delay between IPv4 and IPv6
   connections should be 250 msec. This is to avoid the overhead from tunneled
   IPv6 connections.

   The Connection Attempt Delay should not be dynamically adjusted, as it adds
   privacy risks. This value should be fixed, and could be manually adjusted
   using this torrc option or consensus parameter:

 * ConnectionAttemptDelay N [msec|second]

   The Minimum and Maximum Connection Attempt Delays should also not be
   dynamically adjusted for privacy reasons. The Minimum should be fixed at
   10 msec as per RFC 8305. But the maximum should be higher than the RFC 8305
   recommendation of 2 seconds. For Tor, we should make this timeout value 30
   seconds to match Tor's existing timeout.

   We need to make it possible for users to set the Maximum Connection Attempt
   Delay value higher for slower and higher-latency networks such as dial-up
   and satellite.

4. Option Changes

   As we enable IPv6-enabled clients to connect out of the box, we should
   adjust the default options to enable IPv6 while not breaking IPv4-only
   clients.

   The new default options should be:

* ClientUseIPv4 1 (to enable IPv4)

* ClientUseIPv6 1 (to enable IPv6)

* ClientPreferIPv6ORPort 0 (for load-balancing reasons so we don't
  overload IPv6-only guards)

* ConnectionAttemptDelay 250 msec (the recommended delay between IPv4
  and IPv6, as per RFC 8305)

   One thing to note is that clients should be able to connect with the above
   options on IPv4-only, dual-stack, and IPv6-only networks, and they should
   also work if ClientPreferIPv6ORPort is 1. But we shouldn't expect
   IPv4 or IPv6 to work if ClientUseIPv4 or ClientUseIPv6 is set to 0.

   When the majority of clients and relay are IPv6-capable, we could set the
   default value of ClientPreferIPv6ORPort to 1, in order to take advantage
   of IPv6. We could add a ClientPreferIPv6ORPort consensus parameter, so we
   can make this change network-wide.

5. Relay Statistics

   Entry nodes could measure the following statistics for both IPv4 and IPv6:

 * Number of successful connections

 * Number of extra Prop306 connections (unsuccessful or cancelled)
   * Client closes the connection before completing TLS
   * Client closes the connection before sending any circuit or data cells

 * Number of client and relay connections
   * We can distinguish between authenticated (relay, authority
 reachability) and unauthenticated (client, bridge) connections

   Should we implement Section 5:

 * We can send this information to the directory authorities using relay
   extra-info descriptors

 * We should consider the privacy implications of these statistics, and
   how much noise we need to add to them

 * We can include these statistics in the Heartbeat logs

6. Initial Feasibility Testing

   We should test this proposal with the following scenarios:

* Different combinations of values for the options ClientUseIPv4,
  ClientUseIPv6, and ClientPreferIPv6ORPort on IPv4-only, IPv6-only,
  and dual-stack connections

* Dual-stack connections of different technologies, including
  high-bandwidth and low-latency (e.g. FTTH), moderate-bandwidth and
  moderate-latency (e.g. DSL, LTE), and high-latency and low-bandwidth
  (e.g. satellite, dial-up) to see if Prop306 is reliable and feasible

7. Minimum Viable Prop306 Product

   The mimumum viable product for Prop306 must include the following:

* The address handling, bootstrap, and entry guard changes described in
  Section 2. (Single Onion Services are optional, Bridge Clients are out
  of scope. 

Re: [tor-dev] Updates to Prop306: A Tor Implementation of IPv6 Happy Eyeballs

[teor]

Hi Neel,

> On 17 Dec 2019, at 09:37, Neel Chauhan  wrote:
> 
> Hi tor-dev@ mailing list,
> 
> Sorry for the many-months delay in updating Prop306.
> 
> I have updated Prop306, which is the IPv6 Happy Eyeballs proposal.
> 
> The GitHub PR is here: https://github.com/torproject/torspec/pull/98
> 
> The Trac ticket is here: https://trac.torproject.org/projects/tor/ticket/29801
> 
> Some of the older discussion on Prop306 can be seen on the thread here: 
> https://lists.torproject.org/pipermail/tor-dev/2019-August/013959.html
> 
> Could some of you please review this proposal?

Overall, I think the proposal is too complicated. It doesn't tell us what
changes are necessary, important, and optional. So it will be hard to
review and implement.

I have asked for these changes a few times now, but I can't see them in
your pull request:

> On 1 Aug 2019, at 10:36, teor  wrote:
> 
>> On 30 Jul 2019, at 03:11, Neel Chauhan  wrote:
>> 
>> Just a reminder that this proposal (Prop306) needs to be reviewed: 
>> https://github.com/torproject/torspec/pull/87
> 
> I can't find these changes that I requested to the proposal:
> 
>>> On 14 Jul 2019, at 02:47, teor  wrote:
>>> 
>>> I have some overall comments on the proposal:
>>> 
>>> 1. The proposal is very large now. Let's add an intro section that splits 
>>> the changes into:
> 
> These parts of the proposal exist, but they are not in separate sections:
> 
>>> * initial feasability testing (for initial developer review)
>>> * minimum viable product (for testing in Tor Browser Alpha)
>>> * parameter tuning (for performance, load, and user experience)
>>> * optional features, and how we will know if we need them (if users 
>>> experience particular bugs)
> 
> These parts of the proposal do not exist:
> 
>>> * relay statistics (for ongoing monitoring)
> 
> ...
> 
> I don't see any deleted text:
> 
>>> 3. Each revision of this proposal has added text. Is there any text that is 
>>> redundant or not essential? Can we make it shorter?
> 
> David suggested we delete one section, and I suggested we keep some of it, 
> but delete most of it. ...
> 
> Is there anything else you think we can delete?

How can we help you make these changes?

They are really important.

T




signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Onion DoS: Killing rendezvous circuits over the application layer

[teor]

Hi,

There's also another negative we haven't considered:

> On 3 Dec 2019, at 00:16, George Kadianakis  wrote:
> 
> Negatives:
> 
> a) It's a dirty hotfix that blends the networking layers and might be annoying
>   to maintain in the long-term.
> 
> b) It only works for HTTP (and without SSL?).

c) We'll need to make sure that this defence can't be triggered accidentally,
   (or maliciously via request or response content), otherwise it turns into
   another way of triggering a DoS.

For example, if we searched for a custom string anywhere in the data stream,
then any page documenting that string would be unavailable.

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Practracker regen in #30381

[teor]

Hi,

> On 28 Nov 2019, at 15:43, teor  wrote:
> 
>> On 27 Nov 2019, at 22:34, George Kadianakis  wrote:
>> 
>> teor  writes:
>> 
>>> It looks like you regenerated the whole practracker file in #30381:
>>> https://trac.torproject.org/projects/tor/ticket/30381
>>> https://github.com/torproject/tor/commit/53ac9a9a91a8f2ab45c75550456716074911e685#diff-9fd3400f062c4541d79881e199fd9e1f
>>> 
>>> But we usually just add exceptions for the files that we modified.
>>> 
>>> When we do a full regeneration, we lose a whole lot of warnings that
>>> tell us where our code quality is getting worse.
>>> 
>>> Do you mind if I revert the unrelated changes?
>>> 
>> 
>> No problem either! Sorry for the trouble.

I had a think about this overnight.

Since we've been adjusting exceptions constantly, I don't think reverting
to a particular instant in time is actually helpful.

practracker is meant to help us improve our code quality, and maintain
that code quality once it has been improved.

Here's how we're doing that right now:
1. For large changes, require new or modified code to use best practices
2. Allow small changes, without requiring big refactors

I think that's ok, but let's talk about how we could make it better at
the next roadmap / retrospective?

T

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Practracker regen in #30381

[teor]

Hi,

> On 27 Nov 2019, at 22:34, George Kadianakis  wrote:
> 
> teor  writes:
> 
>> It looks like you regenerated the whole practracker file in #30381:
>> https://trac.torproject.org/projects/tor/ticket/30381
>> https://github.com/torproject/tor/commit/53ac9a9a91a8f2ab45c75550456716074911e685#diff-9fd3400f062c4541d79881e199fd9e1f
>> 
>> But we usually just add exceptions for the files that we modified.
>> 
>> When we do a full regeneration, we lose a whole lot of warnings that
>> tell us where our code quality is getting worse.
>> 
>> Do you mind if I revert the unrelated changes?
>> 
> 
> No problem either! Sorry for the trouble.
> 
> FWIW, what happened there is that when I need to rebase an old dev
> branch to master (because of revisions etc.), there are almost always
> multiple conflicts with practracker. Resolving these manually is very
> annoying (they are many and confusing), so sometimes I have ditched the
> exceptions.txt completely and just regenerated practracker exceptions
> from scratch. That's what happened in that case. If someone has a tip
> for this situation, it would be cool :)

I think the best way to fix this issue is to stop creating code that
requires practracker exceptions :-)

If that's not possible, we should:
1. use the master copy of practracker/exceptions.txt
2. re-run practracker
3. ignore warnings
4. fix any new errors

T

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Raising exceptions in add_event_listener() threads (was Re: HSv3 descriptor work in stem)

[teor]

Hi George,

> On 28 Nov 2019, at 02:04, George Kadianakis  wrote:
> 
> Hello Damian (and list),
> 
> here is another question about an issue I have encountered while
> developing onionbalance v3.
> 
> In particular, I'm fetching HS descriptors using HSFETCH and then adding
> an add_event_listener() event to a function that does the descriptor
> parsing and handling as follows:
> 
>controller.add_event_listener(handle_new_desc_content_event, 
> EventType.HS_DESC_CONTENT)
> 
> The problem is that the handle_new_desc_content_event() callback has
> grown to a non-trivial size and complexity, since it needs to parse the
> descriptor, put it in the right places, and tick off the right
> checkboxes.
> 
> Since its size has increased, so has the number of bugs and errors that
> are appearing during development. The problem is that because the
> callback is running on a separate thread (?) any errors and exceptions
> that get raised in that thread never surface to the my console and hence
> I don't see them. This means that I need to do very tedious printf
> debugging to find the exact place and type of error everytime something
> happens.
> 
> What's the proper way to do debugging and development in callbacks like
> that? Is there a way to make the exceptions float back to the main
> thread or something? Or any other tips?

In general, there are two ways to avoid exceptions disappearing in python
threads, a code change, and a design change.

Code Change

The code change catches all exceptions in a thread, you should be able to
do it right now. (Or Damian could implement it in stem, every time a thread
is launched.)

1. Wrap all code called in a thread in a try/catch block
2. Catch every exception
3. Log it
4. Stop the thread/process/...

The code looks a bit like this:

try:
run_thread()
except e:
print("Error in thread")
log_error()
# Pick one of these actions
# Terminate the process immediately
os._exit(1)
# Re-raise the error - does this terminate the thread?
raise
# Stop the thread
return 1

You'll want a log_error() function like this one:
https://github.com/privcount/privcount/blob/master/privcount/log.py#L19

Another alternative is to join() threads when they terminate, and re-raise
the exception in the main thread. But the code is a lot more complex:
https://stackoverflow.com/a/6874161

Design Change

The design change moves all the work to the main thread, and just stores data/
sets a flag from other threads. That's not always possible, but it's usually a
good idea to limit work in threads, because it avoids races, delays, and
deadlocks.

In this case, you could store the data in the fetch thread, and then do the
processing the next time the main thread tries to access that data, or anything
related to it.

T

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Practracker regen in #30381

[teor]

Hi George, David,

It looks like you regenerated the whole practracker file in #30381:
https://trac.torproject.org/projects/tor/ticket/30381
https://github.com/torproject/tor/commit/53ac9a9a91a8f2ab45c75550456716074911e685#diff-9fd3400f062c4541d79881e199fd9e1f

But we usually just add exceptions for the files that we modified.

When we do a full regeneration, we lose a whole lot of warnings that
tell us where our code quality is getting worse.

Do you mind if I revert the unrelated changes?

T

-- 
teor
--

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] reproducible builds for Android tor daemon

[teor]

Hi,

> On 12 Sep 2019, at 20:50, Hans-Christoph Steiner  
> wrote:
> 
> Then that work
> will hopefully be extended into sharing tor between apps, e.g. letting
> Briar, Tor Browser, etc share the tor SOCKS proxy to other apps that
> want to use it. That would happen via Android mechanisms like Intents to
> manage the discovery of SOCKS ports.

It's not always safe to have apps share Tor: a malicious website in one app
can use various caches to discover activity in other apps. And there may
be similar data leaks in other shared data structures or network
connections.

How do these data leaks affect your use cases?

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Putting onion services behind a third-party TCP proxy

[teor]

> On 20 Aug 2019, at 13:31, Pop Chunhapanya  wrote:
> 
> Hi Tim,
> 
>> 
>> TCPProxy protocol host:port
>> 
>> Tor will use the given protocol to make all its OR (SSL) connections through 
>> a TCP proxy on host:port, rather than connecting directly to servers. You 
>> may want to set FascistFirewall to restrict the set of ports you might try 
>> to connect to, if your proxy only allows connecting to certain ports. There 
>> is no equivalent option for directory connections, because all Tor client 
>> versions that support this option download directory documents via OR 
>> connections.
>> 
>> The only protocol supported right now 'haproxy'. This option is only for 
>> clients. (Default: none)
>> 
> 
> The other point that I want to make is that haproxy has 2 versions. I think 
> it's better to also put the version number in the protocol name like 
> 'haproxy1'.
> However I saw you already used 'haproxy' in the HiddenServiceExportCircuitID 
> option.

I would be happy with "haproxy" and "haproxy2".

But minimal patches are good - let's not implement features that no-one is 
using.

T___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] TBB Memory Allocator choice fingerprint implications

[teor]

Hi,

> On 18 Aug 2019, at 08:35, Shawn Webb  wrote:
> 
> Having the heap implementation selectable at runtime would enable
> users to make the determination for themselves, while also making
> future integration efforts easier through modularization/abstraction
> APIs (I'm making a silly, naive, and likely wrong, assumption that such
> APIs don't already exist.)
> 
> I hope I'm not coming off as "hey, do this." I'm just thinking out
> loud in an admittedly naive fashion.
> 
> Anyone have any thoughts?

Allowing users to select the allocator at runtime would split the anonymity 
set, *if* the allocator was detectable remotely.

This is why we try to use safe defaults, and avoid user-selected options. When 
we do give users options, we create a few well-defined settings (like the 
security slider).

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Putting onion services behind a third-party TCP proxy

[teor]

> On 16 Aug 2019, at 04:52, Pop Chunhapanya  wrote:
> 
> Hi Tim,
> 
>> The only protocol supported right now 'haproxy'. This option is only for 
>> clients. (Default: none)
> 
> I think TCPProxy option is more generic than HTTPSProxy, Socks4Proxy and 
> Socks5Proxy. Why don't we also allow https, socks4, and socks5 instead of 
> just haproxy?

That's possible, but it's not required as part of the patch.

If we wanted a fully generic option, we should probably call it something like
ORConnectionProxy or OutboundORProxy.

I'd like to see what Nick thinks when he's back next week.

T___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [prop305] Introduction Point Behavior

[teor]

Hi David,

> On 15 Aug 2019, at 22:56, David Goulet  wrote:
> 
> I'm leaning towards not closing the circuit and falling back on the consensus
> parameters.

Using the consensus parameters seems like a good thing to do.

We can say "valid parameters override the consensus parameters. Invalid
parameters are ignored, and the circuit uses the consensus parameters."

> And at some point in time, we'll be able to implement the
> INTRO_ESTABLISHED response. In the meantime, there is little chances that tor
> vanilla start sending bad values since they are validated from the torrc file.

Do we have a protocol warning for invalid parameters? If we do, we should be
able to detect issues as they happen.

I don't think there's an easy way of detecting these kinds of bugs in chutney.
But maybe we want to set consensus parameters and torrc parameters on one
of the chutney networks?

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Putting onion services behind a third-party TCP proxy

[teor]

Hi Haxxpop,

> On 15 Aug 2019, at 16:53, Pop Chunhapanya  wrote:
> 
> 
>>> So I'm thinking putting the tor daemon behind some third party TCP proxy 
>>> that will protect me from this kind of DDoS attack.
>>> 
>>> What do you think if I want to implement a feature that forward all the 
>>> onion service traffic to the TCP proxy before going to the Tor network?
>>> 
>>> The protocol that I'm thinking is TCP Proxy Protocol [1]
>>> 
>>> [1] https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
>> 
>> You could try the existing HTTPSProxy torrc option?
>> 
>> HTTPSProxy host[:port]
>> Tor will make all its OR (SSL) connections through this host:port (or 
>> host:443 if port is not specified), via HTTP CONNECT rather than connecting 
>> directly to servers. You may want to set FascistFirewall to restrict the set 
>> of ports you might try to connect to, if your HTTPS proxy only allows 
>> connecting to certain ports.
>> 
>> 
>> 
>> Tor also allows an intelligent firewall to filter circuits using a field in 
>> haproxy protocol format, see HiddenServiceExportCircuitID for details. But 
>> you probably won't need this advanced feature.
>> 
> 
> I feel that HTTPSProxy is too expensive. As far as I know, it needs to do (1) 
> tcp handshake, (2) tls handshake, and (3) http connect. If I can use haproxy, 
> it would be just one tcp handshake.
> 
> Could I propose another option for haproxy?
> I can do it myself. You just review and merge :)

Sure, I suggest you use this torrc option format:

TCPProxy protocol host:port

Tor will use the given protocol to make all its OR (SSL) connections through a 
TCP proxy on host:port, rather than connecting directly to servers. You may 
want to set FascistFirewall to restrict the set of ports you might try to 
connect to, if your proxy only allows connecting to certain ports. There is no 
equivalent option for directory connections, because all Tor client versions 
that support this option download directory documents via OR connections.

The only protocol supported right now 'haproxy'. This option is only for 
clients. (Default: none)

The haproxy protocol works in the following way: when the feature is enabled, 
the Tor process will write a header line on every outbound connection. The 
header is in the following format:
   "PROXY" [ "TCP4" | "TCP6" ] SourceIPAddress DestinationIPAddress SourcePort 
DestinationPort "\r\n"
There is a single space after each item, except for the last item, which is 
followed by a CRLF.

After parsing a correctly-formatted PROXY line, the haproxy connects to 
DestinationIPAddress:DestinationPort, and forwards all subsequent data to the 
destination. Any data sent by the destination is forwarded by haproxy to the 
Tor client.

The HAProxy version 1 proxy protocol is described in detail at 
https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

I don't think you'll need to implement a TCPProxyAuthenticator option.

T___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Putting onion services behind a third-party TCP proxy

[teor]

Hi,

> On 15 Aug 2019, at 05:10, Pop Chunhapanya  wrote:
> 
> When deploying an onion service, I noticed some problem that the ip address 
> of my machine that runs tor daemon is exposed to the Tor network which is 
> vulnerable to the DDoS attack if someone knows my ip address.

You can reject all inbound connections to your onion service using a simple 
firewall rule. Onion services are tor clients: they only make outbound 
connections.

> So I'm thinking putting the tor daemon behind some third party TCP proxy that 
> will protect me from this kind of DDoS attack.
> 
> What do you think if I want to implement a feature that forward all the onion 
> service traffic to the TCP proxy before going to the Tor network?
> 
> The protocol that I'm thinking is TCP Proxy Protocol [1]
> 
> [1] https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

You could try the existing HTTPSProxy torrc option?

HTTPSProxy host[:port]
Tor will make all its OR (SSL) connections through this host:port (or host:443 
if port is not specified), via HTTP CONNECT rather than connecting directly to 
servers. You may want to set FascistFirewall to restrict the set of ports you 
might try to connect to, if your HTTPS proxy only allows connecting to certain 
ports.



Tor also allows an intelligent firewall to filter circuits using a field in 
haproxy protocol format, see HiddenServiceExportCircuitID for details. But you 
probably won't need this advanced feature.

T

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] New Proposal 306: A Tor Implementation of IPv6 Happy Eyeballs

[teor]

Hi Nick,

> On 3 Aug 2019, at 01:25, Nick Mathewson  wrote:
> 
>> On Tue, Jun 25, 2019 at 9:24 PM  wrote:
>> 
>> Hi tor-dev@ mailing list,
>> 
>> I have a new proposal: A Tor Implementation of IPv6 Happy Eyeballs
>> 
>> This is to implement Tor IPv6 Happy Eyeballs and acts as an alternative
>> to Prop299 as requested here:
>> https://trac.torproject.org/projects/tor/ticket/29801
>> 
>> The GitHub pull request is here:
>> https://github.com/torproject/torspec/pull/87
>> 
>> Thank You,
> 
> Hi, Neel!  Thanks for working on this; I believe it's come a long way
> in the last month!
> 
> Here are a few questions based on the current PR.
> 
> * We need to revise the "relay selection changes" to match the
> vocabulary of guard-spec.txt.  It's easy to say "select at least one
> relay with an ipv6 address", but it's not trivial to do so in
> practice.

On the pull request, I suggested that we make sure that at least one
of the three primary guards has IPv6. (This change might also place
a similar IPv6 requirement on the larger sets of guards chosen by
clients. Is there a nice Venn diagram of all the guard sets?)

Here's one way we could implement an IPv6 guard requirement:

When choosing the last primary guard (or rotating any primary guard),
if there are no IPv6 primary guards, pass a new flag CRN_NEED_IPV6
to router_choose_random_node().

CRN_NEED_IPV6 can be implemented like CRN_PREF_ADDR, but with
a hard-coded preference for IPv6.

/* On clients, only provide nodes that satisfy ClientPreferIPv6OR */
  CRN_PREF_ADDR = 1<<7,

(I cover the non-guard cases below.)

> (Also, do we do this always, or do we do this only when we
> think we can connect to ipv6 addresses?)

Happy eyeballs does not and should not require the client to guess IPv6
reachability. Tor can't reliably get that information, because the results of
OS network APIs may be unreliable, unavailable, or incorrect. (And past
connectivity is not a reliable guide to future connectivity, particularly on
mobile.)

If we want to try to guess, that's an optimisation, which belongs in the
"optional optimisations" section of the proposal.

> * We also need to think about what this algorithm means in terms of
> guard-spec.txt's data structures.  Does it mean that each connection
> to a guard is replaced with two?  Does it mean that some of the
> reachability variables are replaced by two?

I would prefer a new low-level network module that takes an IPv4 and
IPv6 address for each connection request, and reports success if
either address succeeds. And failure if both fail.

(Note that IPv4-only, dual-stack, and IPv6-only are all valid address
combinations. Relays, authorities, and fallbacks are IPv4 or dual stack,
bridge lines are currently IPv4-only or IPv6-only, and v3 single onion
service rendezvous direct connections can be all three.)

This design would have a minimal impact on existing guard data
structures and guard code.

I'd like to put any other guard changes in the "optional optimisations"
section of the proposal. Unless we are sure that they are essential.

> * The proposal considers TCP success vs authentication success as
> indicating that a connection has succeeded. There is a good
> alternative that reduces CPU load, however.  The TLS handshake has
> multiple phases, and the expensive CPU stuff all happens after we
> receive a ServerHello message.  If we treat an incoming ServerHello as
> meaning that the connection will be successful, we can avoid most
> wasted handshakes.

Sounds sensible. Let's use the ServerHellos as the minimal viable product
for merging and release in an alpha. So this feature belongs in the
"minimal viable product" section of the proposal.

Initial feasibility testing can just use TCP connections though.

> [This would definitely not handle the problem where one of a server's
> addresses is correct but the other address is a different server
> entirely, but I hope we can catch that earlier in data flow, possibly
> at the authorities.]

Authority IPv4 or IPv6 reachability checks should catch this issue and
mark the relayMas not Running. (And therefore it won't be in the client's
consensus 2-4 hours after the bad address is in the descriptor or on the
machine.)

IPv4 reachability checks on relays should also catch most IPv4
misconfigurations.

We also have a funding proposal to do IPv6 reachability checks on relays,
which will catch IPv6 misconfigurations before relays upload their
descriptors.

> * The 1.5 second delay, and associated other hardcore times, should be
> a network parameter, transmitted in the consensus.  1.5 seconds can be
> the default, but we will want to keep the ability to tune it later on.

David and I suggested this change on the pull request.

> * For pluggable transports, do we want to manage this process
> ourselves, or delegate the decisions to the PT?  Each option has its
> own benefits and risks.

I suggested on the proposal pull request that we make changing
bridge lines (including pluggable 

Re: [tor-dev] New Proposal 306: A Tor Implementation of IPv6 Happy Eyeballs

[teor]

Hi,

> On 2 Aug 2019, at 13:02, NOC  wrote:
> 
> That would imply that guard relays run at 100% capacity which i can't confirm 
> for any of my guards.

I have run guards at 100% before, but it's not ideal.

Generally, Tor load-balances traffic to make sure that clients get consistently 
good bandwidth and latency. Adding significant extra load to some relays makes 
it hard to achieve this goal. In particular, 100% relay usage can significantly 
increase latency.

> Also seeing decline in usage would maybe give a incentive for some relay 
> operators to configure IPv6 in their torrc,

But IPv4-only relays *will* see a decrease in traffic, and IPv6 relays will see 
an increase.

Sending 33% of client traffic to the 20% of guards that have IPv6 will cause an 
increase in usage for IPv6 relays (and a decrease for IPv4 relays). Even if we 
check the relay's IPv6 address after its IPv4 address, the traffic is still 
going to that relay.

Tor clients on IPv6-only networks will also add extra to IPv6 relays. These 
clients don't work now, so we don't know how many there are.

Once we've tested IPv6 in at least one stable release, we will have a better 
idea of the load balancing impacts of IPv6. And we can think about changing the 
address order.

Another reason that we want to check IPv4 first is that is preserves tor's 
current behaviour. Adding an extra attempt at IPv6 once IPv4 has failed, is a 
smaller change that is unlikely to have any negative impacts.

(Our last attempt at adding automatic client IPv6 didn't work, and we had to 
disable it in Tor Browser alpha.)

> because there are thousands of relays which do have IPv6 connectivity, but 
> the operator simply didn't configure it because it is not as easy as putting 
> "ORPort 9001" in your torrc and it listens on IPv4 and IPv6.
> 
> Sadly ORPort 9001 gives you a working relay which listens only on IPv4, so i 
> would say the lack of IPv6 enabled relays is a problem which is created by 
> how the config works

>> We are working on a funding proposal that will increase the number of IPv6 
>> relays by automatically detecting, testing, and using IPv6 addresses.

> and that a default torrc file loses no word about IPv6.

You're right, I opened a ticket to add IPv6 ORPorts to the example torrcs:
https://trac.torproject.org/projects/tor/ticket/31320

> Beside that if i look at https://metrics.torproject.org/bandwidth-flags.html 
> the Tor network could lose their entire IPv4 only guards and still work, but 
> after that guards would indeed run at 100% capacity.

See my answer above: 100% capacity is not good for the network or clients.

T___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] New Proposal 306: A Tor Implementation of IPv6 Happy Eyeballs

[teor]

Hi,

> On 2 Aug 2019, at 09:20, NOC  wrote:
> 
> I see this staying longer with IPv4 longer than we should also problematic, 
> we are at the point that there are providers out there who do have more 
> clients than IPv4 space which results in having them making carrier grade 
> NAT. Some of them have the problem that their NAT gear gets maxed out at peak 
> hours which results in heavy packet loss and very slow speeds, while their 
> IPv6 connection is perfectly fine. That will not get better, i guess it will 
> get worse in the future. So i would also prefer to use IPv6 if it works 
> better.

Currently, Tor clients don't use IPv6 unless they are specifically configured 
to use it. Some apps (OnionBrowser) use the OS network APIs to automatically 
configure Tor, but most don't.

This proposal makes sure that Tor clients try IPv4, then try IPv6 after a short 
delay. If either works, the client will connect to the Tor network.

At this stage, only 20% of guards support IPv6. But we are going to make sure 
at least one of the three client primary guards has IPv6. Ensuring at least one 
IPv6 client guard will increase traffic to IPv6 guards by up to 1.7x, which 
could cause load balancing issues.

So we need to counter this load imbalance by trying IPv6 after IPv4.

Once 33% of non-exit guards support IPv6, we can reduce the delay, or try IPv6 
first at random. Once 67% of non-exit guards support IPv6, we can try IPv6 
first.

We are working on a funding proposal that will increase the number of IPv6 
relays by automatically detecting, testing, and using IPv6 addresses.

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] New Proposal 306: A Tor Implementation of IPv6 Happy Eyeballs

[teor]

Hi Neel,

> On 30 Jul 2019, at 03:11, Neel Chauhan  wrote:
> 
> Just a reminder that this proposal (Prop306) needs to be reviewed: 
> https://github.com/torproject/torspec/pull/87

I can't find these changes that I requested to the proposal:

>> On 14 Jul 2019, at 02:47, teor  wrote:
>> 
>> I have some overall comments on the proposal:
>> 
>> 1. The proposal is very large now. Let's add an intro section that splits 
>> the changes into:

These parts of the proposal exist, but they are not in separate sections:

>>  * initial feasability testing (for initial developer review)
>>  * minimum viable product (for testing in Tor Browser Alpha)
>>  * parameter tuning (for performance, load, and user experience)
>>  * optional features, and how we will know if we need them (if users 
>> experience particular bugs)

These parts of the proposal do not exist:

>>  * relay statistics (for ongoing monitoring)

This modification is not in the current proposal:

>> 2. Bridges can't be configured with an IPv4 and an IPv6 address in tor, 
>> because tor only accepts one IP address per bridge. Let's be clear that 
>> bridges are out of scope. (Tor already attempts to connect to all? its 
>> configured bridges.)

I added some suggest text to make bridges out of scope for this proposal.

I don't see any deleted text:

>> 3. Each revision of this proposal has added text. Is there any text that is 
>> redundant or not essential? Can we make it shorter?


David suggested we delete one section, and I suggested we keep some of it, but 
delete most of it. I also suggested we move some alternative options to an 
appendix.

Is there anything else you think we can delete?

T



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] New Proposal 306: A Tor Implementation of IPv6 Happy Eyeballs

[teor]

Hi,

On July 11, 2019 12:37:03 AM UTC, n...@neelc.org wrote:
>I'm really sorry about the delay in responding to your review. I was 
>busy with an internship (unrelated to Tor, but still related to 
>security) and was out a lot in my "free time".
>
>I have implemented your requested changes and the GitHub PR is here: 
>https://github.com/torproject/torspec/pull/87
>
>Hopefully I have not missed anything.
>
>Most of these changes you (Iain and Teor) suggested sound good. I'm not
>
>a huge fan of preferring IPv4 in the case of tunneled IPv6 connections 
>(reason: we stay with IPv4 longer than we should), but understand why 
>you have it (reason: better network performance) and have added this 
>change anyways.

Thanks for these revisions.

I have some overall comments on the proposal:

1. The proposal is very large now. Let's add an intro section that splits the 
changes into:
  * initial feasability testing (for initial developer review)
  * minimum viable product (for testing in Tor Browser Alpha)
  * parameter tuning (for performance, load, and user experience)
  * relay statistics (for ongoing monitoring)
  * optional features, and how we will know if we need them (if users 
experience particular bugs)

2. Bridges can't be configured with an IPv4 and an IPv6 address in tor, because 
tor only accepts one IP address per bridge. Let's be clear that bridges are out 
of scope. (Tor already attempts to connect to all? its configured bridges.)

3. Each revision of this proposal has added text. Is there any text that is 
redundant or not essential? Can we make it shorter?

After these revisions, I will do a final review. I hope we can get another tor 
developer to also do a final review.

T

--
teor
--
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] resolving DNS TXT records?

[teor]

On July 11, 2019 1:59:00 PM UTC, nusenu  wrote:
>Hi,
>as far as I can tell there is no way for a tor client to ask an exit to
>resolve
>a given TXT record and to provide the answer for it.
>
>Just wanted to make sure there is even no hack around that limitation.

A few exits allow connections to TCP port 53, so you can do arbitrary DNS 
queries to any DNS server using those exits.

>https://gitweb.torproject.org/torspec.git/tree/proposals/219-expanded-dns.txt

T

--
teor
--
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] exitmap/RESOLVE control command limitations

[teor]

On July 9, 2019 8:55:00 AM UTC, nusenu  wrote:
>Hi,
>
>I noticed some unexpected answers in exitmap's [1] dnsenum results
>and suspected that this has todo with IPv4 vs. IPv6.
>
>First I looked at [2] and found that it only lists IPv4 and hostnames
>as possible answers but then I realized that exitmap might not be using
>the RESOLVE command?

Exitmap is using tor's RESOLVE extension to SOCKS:
https://gitweb.torproject.org/torspec.git/tree/socks-extensions.txt#n49

>> def resolve(self, domain):
>> """
>> Resolve the given domain using Tor's SOCKS1 resolution
>extension.
>> """
>> 
>> domain_len = len(domain)
>> if domain_len > 255:
>> raise error.SOCKSv5Error("Domain must not be longer than
>255 "
>>  "characters, but %d given." %
>domain_len)
>> 
>> # Tor defines a new command value, \x0f, that is used for
>domain
>> # resolution.
>> 
>> self._send_all("\x05\xf0\x00\x03%s%s%s" %
>>  (chr(domain_len), domain, "\x00\x00"))

Exitmap uses the SOCKS 5, resolve, DNS command:
See page 4 of https://www.ietf.org/rfc/rfc1928.txt

>> 
>> resp = self._recv_all(10)
>> if resp[:2] != "\x05\x00":
>> raise error.SOCKSv5Error("Invalid server response: 0x%s"
>%
>>  resp[1].encode("hex"))

Resolve can return an IPv4 or IPv6 response, but Exitmap ignores the address 
type, and turns the first 4 bytes of the response into an IPv4 address.

>> return socket.inet_ntoa(resp[4:8])
>
>
>Does Tor's SOCKS resolution extension support IPv6 answers
>or does it only attempt A records?

If it gets both IPv4 and IPv6, I think it will prefer IPv4.

Try testing with ipv6.google.com, which only has an IPv6 address.

>I'm aiming to resolve a hostname and would like to get 
>the IPv4 and if available the IPv6 address.

I don't know how you can reliably get the IPv6 address over SOCKS, when the 
site has an IPv4 address.

Try using the controller RESOLVE command and ADDRMAP event, which supports IPv6:

Address = ip4-address / ip6-address / hostname

If that doesn't work, we might need to make some changes to tor, or fix some 
bugs.

>thanks,
>nusenu
>
>
>[1] https://github.com/NullHypothesis/exitmap
>[2]
>https://gitweb.torproject.org/torspec.git/tree/control-spec.txt#n1349


--
teor
--
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] New Proposal 306: A Tor Implementation of IPv6 Happy Eyeballs

[teor]

thenticated
>> Connection
>> 
>> * Section 3.1.3: TCP Connection to Preferred Address On First TCP
> Success
>> 
>> 3.1.1. First Successful Authentication
>> 
>> In this design, Tor will first connect to the preferred address and
>> attempt to authenticate. After a 1.5 second delay, Tor will connect
>> to the alternate address and try to authenticate. On the first
>> successful authenticated connection, we close the other connection.
>> 
>> This design places the least connection load on the network, but
>> might add extra TLS load.
> 
> The delay seems arbitrary. OnionPerf collects data on latency in the Tor
> network, and could be used to inform better timing choices for the best
> end user performance (the happiest eyeballs).

The 1.5 second delay is based on Onionperf data, and we should reference
the Onionperf figures in the proposal.

See my previous review of an earlier draft of this proposal:

>> On 26 Jun 2019, at 13:33, teor  wrote:
> 
>>> 
>>> Depending on their location, most tor clients authenticate to the first
>>> hop within 0.5-1.5 seconds. So I suggest we use a 1.5 second delay:
>>> https://metrics.torproject.org/onionperf-buildtimes.html
>>> 
>>> In RFC 8305, the default delay is 250 milliseconds, and the maximum
>>> delay is 2 seconds. So 1.5 seconds is reasonable for TLS and tor link
>>> authentication.
>>> https://tools.ietf.org/html/rfc8305#section-8
>>> 
>>> (This delay will mainly affect initial bootstrap, because all of Tor's
>>> other connections are pre-emptive, or re-used.)
>>> 
>>> A small number of clients may do wasted authentication.
>>> That's ok. Tor already does multiple bootstrap and guard connections.


> If we choose to take this route, we should open new connections with a
> timeout of ~250ms, and only change the condition for deciding which is
> the connection we will use.

Tor already does multiple bootstrap and guard connections over IPv4, so
I'm not sure exactly what design you're proposing. Can you give me an
example?

>> 3.1.2. TCP Connection to Preferred Address On First Authenticated
> Connection
>> 
>> This design attempts a TCP connection to a preferred address. On a
>> failure or a 250 ms delay, we try the alternative address.
>> 
>> On the first successful TCP connection Tor attempts to authenticate
>> immediately. On the authentication failure, or a 1.5 second delay,
>> Tor closes the other connection.

Neel, that's not what I wrote in my last email:

>> On 26 Jun 2019, at 13:33, teor  wrote:
>>> 
>>> 1. Tor connects to the preferred address and tries to authenticate.
>>>On failure, or after a 1.5 second delay, it connects to the alternate 
>>> address
>>>and tries to authenticate.
>>>On the first successful authentication, it closes the other connection.

A small number of clients will take longer than 1.5 seconds to
authenticate. So we should only close a connection when the other
connection to the relay successfully authenticates.

>> This design is the most reliable for clients, but increases the
>> connection load on dual-stack guards and authorities.
> 
> Creating TCP connections is not a huge issue,

That's not true: Tor's last connection level denial of service event
was November 2017 - February 2018. And there are occasional connection
spikes on authorities and fallbacks.

These connection DoSes need to be mentioned in the proposal.

> and we should be racing
> the connections with the ~250ms timeout anyway. All the designs will
> have this issue.

I'm not sure exactly what issue you're referring to?

>> 3.1.3. TCP Connection to Preferred Address On First TCP Success
>> 
>> In this design, we will connect via TCP to the first preferred
>> address. On a failure or after a 250 ms delay, we attempt to connect
>> via TCP to the alternate address. On a success, Tor attempts to
>> authenticate and closes the other connection.
>> 
>> This design is the closest to RFC 8305 and is similar to how Happy
>> Eyeballs is implemented in a web browser.
> 
> This is probably also the "simplest" to implement, as it means that the
> happy eyeballs algorithm is contained to the socket handling code.
> 
> I don't believe that requiring authentication to complete is going to do
> anything more than generate load on relays. Either the packet loss is
> high enough that the three way handshake fails, or there is low packet
> loss. I don't think the case where requiring an additional few packets
> make it through helps you choose a better connection is going to be that
> common.

Middleboxes

[tor-dev] Changed Network Team Meeting Times

[teor]

Hi all,

We've moved the first network team meeting of each month to Wednesday
2300 UTC. (It used to be Tuesday.) We have also moved the weekly
"patch party" to Wednesday 2300 UTC in the other weeks of the month.

The regular team meeting time is still the same, Monday 1700 UTC.

Here is our meeting schedule for July:
* Wednesday 3 July 2300 UTC - Changed Day!
* Monday 8 July at 1700 UTC
* (In-person meeting 12-14 July)
* Monday 22 July 1700 UTC
* Monday 29 July 1700 UTC

T

--
teor
--



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Per-peer stream isolation for Bitcoin clients

[teor]

Hi Jeremy,

> On 28 Jun 2019, at 06:35, Jeremy Rand  wrote:
> 
> 2. Per-peer stream isolation prevents a single exit relay from feeding
> the user a chain that's not the longest chain, so it's desirable from a
> Bitcoin security point of view.

Tor itself uses 3 directory guards to make sure that one guard can't
feed it bad data.

If you use one exit, and it happens to be malicious, all your peers are
accessed via that bad exit.

But if you use 8 exits, then it's very unlikely that all of them will be
malicious in exactly the same way. But it makes it 8x more likely that
any one of your exits will be malicious.

It's also worth using multiple circuits for redundancy, so a single exit
can't deny service all 8 of your peers. (As noted on the pull request.)

So it's a tradeoff, based on the protocol. For example:
* If you were mainly worried about exploits, then you'd want one exit.
* If you are worried about short chains or denial of service, then you'd
  want many exits.

If you don't know which one to choose, follow Tor Browser, which uses
many exits.

(The right way to deal with exploits is to fix those bugs, and use coding
practices that make them less likely.)

> 3. Per-peer stream isolation would mean more potential for one of the
> circuits being deanonymizable, via traffic analysis etc.  It's not clear
> to me whether this amount of increased circuits is harmful, or how it
> compares to other common usage of Tor such as Tor Browser (which uses
> first-party stream isolation, so a user with a lot of tabs open may very
> well have 8 or more circuits in use at once).

It's not quite that simple:

If you send traffic from all 8 peers over one circuit, then a pattern might
be visible on you client to guard, and exit to internet connections. That
pattern makes correlation easier.

Using 8 circuits splits the traffic pattern across 8 different exit locations
on the internet, so correlation is harder.

I'd follow Tor Browser on this one, too.

> 4. Per-peer stream isolation puts more load on the Tor network.  It's
> not clear to me whether this increased load (8 circuits instead of 1) is
> so much that it's harmful.

Every Tor client tries to keep 6+ pre-emptive circuits open, so that new
requests have lower latency. So you're really looking at doubling your
circuit load, not multiplying it by 8.

Once the circuits are built, there's not much extra load splitting the
streams across 8 circuits, because it's already split into streams and
cells within the protocol.

And you're right, Tor Browser can use lots more than 8 circuits, so
I wouldn't worry about it.

Do you know how much load Bitcoin places on the Tor network?

If it's a lot, one good answer is to encourage users to run relays,
or to donate to organisations that run relays. (Or donate to Tor,
so we can make the network more efficient.)

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] New Proposal 306: A Tor Implementation of IPv6 Happy Eyeballs

[teor]

Hi Neel,

> On 27 Jun 2019, at 11:10, n...@neelc.org wrote:
> 
> Thank you so much for your feedback. I have pushed your changes (and revert 
> my changes to 000-index.txt) as fixup commits to the GitHub PR.
> 
> Could you please review the new changes?

I'm going to wait a week or two for other people to provide feedback on this 
thread.

T

>> On 2019-06-25 23:33, teor wrote:
>> Hi Neel,
>> Thanks for this proposal.
>>> On 26 Jun 2019, at 11:15, n...@neelc.org wrote:
>>> I have a new proposal: A Tor Implementation of IPv6 Happy Eyeballs
>>> This is to implement Tor IPv6 Happy Eyeballs and acts as an
>>> alternative to Prop299 as requested here:
>>> https://trac.torproject.org/projects/tor/ticket/29801
>>> The GitHub pull request is here:
>>> https://github.com/torproject/torspec/pull/87
>> Here's the proposal content, with my comments:
>>> Filename: 306-ipv6-happy-eyeballs.txt
>>> Title: A Tor Implementation of IPv6 Happy Eyeballs
>>> Author: Neel Chauhan
>>> Created: 25-Jun-2019
>> Supercedes: 299
>>> Status: Open
>>> Ticket: https://trac.torproject.org/projects/tor/ticket/29801
>>> 1. Introduction
>>> As IPv4 address space becomes scarce, ISPs and organizations will
>>> deploy
>>> IPv6 in their networks. Right now, Tor clients connect to guards
>>> using
>>> IPv4 connectivity by default.
>>> When networks first transition to IPv6, both IPv4 and IPv6 will
>>> be enabled
>>> on most networks in a so-called "dual-stack" configuration. This
>>> is to not
>>> break existing IPv4-only applications while enabling IPv6
>>> connectivity.
>>> However, IPv6 connectivity may be unreliable and clients should
>>> be able
>>> to connect to the guard using the most reliable technology,
>>> whether IPv4
>>> or IPv6.
>>> In ticket #27490, we introduced the option ClientAutoIPv6ORPort
>>> which
>>> lets a client randomly choose between IPv4 or IPv6. However, this
>>> random decision does not take into account unreliable
>>> connectivity
>>> or falling back to the competing IP version should one be
>>> unreliable
>>> or unavailable.
>>> One way to select between IPv4 and IPv6 on a dual-stack network
>>> is a
>>> so-called "Happy Eyeballs" algorithm as per RFC 8305. In one, a
>>> client
>>> attempts an IP family, whether IPv4 or IPv6. Should it work, the
>>> client
>>> sticks with the working IP family. Otherwise, the client attempts
>>> the
>>> opposing version. This means if a dual-stack client has both IPv4
>>> and
>>> IPv6, and IPv6 is unreliable, the client uses IPv4, and vice
>>> versa.
>>> In Proposal 299, we have attempted a IP fallback mechanism using
>>> failure
>>> counters and preferring IPv4 and IPv6 based on the state of the
>>> counters.
>>> However, Prop299 was not standard Happy Eyeballs and an
>>> alternative,
>>> standards-compliant proposal was requested in [P299-TRAC] to
>>> avoid issues
>>> from complexity caused by randomness.
>>> This proposal describes a Tor implementation of Happy Eyeballs
>>> and is
>>> intended as a successor to Proposal 299.
>>> 2. Address Selection
>>> To be able to handle Happy Eyeballs in Tor, we will need to
>>> modify the
>>> data structures used for connections to guards, namely the extend
>>> info
>>> structure.
>>> The extend info structure should contain both an IPv4 and an IPv6
>>> address.
>>> This will allow us to try IPv4 and the IPv6 addresses should both
>>> be
>>> available on a relay and the client is dual-stack.
>>> When parsing relay descriptors and filling in the extend info
>>> data
>>> structure, we need to fill in both the IPv4 and IPv6 address if
>>> they both
>>> are available. If only one family is available for a relay (IPv4
>>> or IPv6),
>>> we should fill in the address for available family and leave the
>>> opposing
>>> family null.
>> When we implement this feature in tor, it would be a good idea to call
>> the
>> two addresses "preferred" and "alternate" address. With this design,
>> the low-level connection code doesn't have to know about reachable
>> addresses, or IPv4/IPv6 preferences. It just has to try them in order.
>>> 3. Connecting To A Relay
>>> When a client connects to a gu

Re: [tor-dev] New Proposal 306: A Tor Implementation of IPv6 Happy Eyeballs

[teor]

ting using the other address
> 
>should one be available and allowed, and the first attempted version
> 
>fails. This should be higher than most client's successful TLS
> 
>authentication time. I propose that the timer is 15 seconds. The reason
> 
>for this is to accommodate high-latency connections such as dial-up and
> 
>satellite.
> 
In the worst case scenario, users see Tor Browser hang for 15 seconds
before it makes a successful connection. That's not acceptable.

Depending on their location, most tor clients authenticate to the first
hop within 0.5-1.5 seconds. So I suggest we use a 1.5 second delay:
https://metrics.torproject.org/onionperf-buildtimes.html

In RFC 8305, the default delay is 250 milliseconds, and the maximum
delay is 2 seconds. So 1.5 seconds is reasonable for TLS and tor link
authentication.
https://tools.ietf.org/html/rfc8305#section-8

(This delay will mainly affect initial bootstrap, because all of Tor's
other connections are pre-emptive, or re-used.)

A small number of clients may do wasted authentication.
That's ok. Tor already does multiple bootstrap and guard connections.

We have talked about this design in the team over the last few months.
Our key insights are that:
* TCP connections are cheap, but TLS is expensive
* most failed TCP connections fail immediately in the kernel, some
  fail quickly with a response from the router, and others are blackholed
  and time out
* it's unlikely that a client will fail to authenticate to a relay over one
  IP version, but succeed over the other IP version, because the directory
  authorities authenticate to each relay when they check reachability
* some censorship systems only break authentication over IPv4,
  but they are rare

So here are some alternative designs:

1. Tor connects to the preferred address and tries to authenticate.
   On failure, or after a 1.5 second delay, it connects to the alternate address
   and tries to authenticate.
   On the first successful authentication, it closes the other connection.

This design places the least connection load on the network, but might add
a bit of extra TLS load.

2. Tor connects via TCP to the preferred address.
   On failure, or after a 250 ms delay, it connects via TCP to the alternate
   address.
   On the first TCP success, tor attempts to authenticate immediately.
   On authentication failure, or after a 1.5 s delay, tor attempts to
   authenticate over the second TCP connection.
   On the first successful authentication, it closes the other connection.

This design is the most reliable for clients, but it also puts a bit more
connection load on dual-stack guards and authorities.

3. Tor connects via TCP to the preferred address.
   On failure, or after a 250ms delay, it connects via TCP to the alternate
   address.
   On the first TCP success, tor attempts to authenticate, and closes the
   other connection.

This design looks similar to a web browser's implementation of Happy
Eyeballs, because it closely follows the RFC. That might help hide tor
from censors. It adds some extra connection load, but no extra TLS load.

I suggest that we put all 3 alternative designs in the proposal, but start
by implementing and testing alternative 1.

When we implement this code, let's put the happy eyeballs part in a
separate module, as much as possible. That helps us review the code,
and make sure it has good test coverage. It also stops existing files and
functions getting too big.
> 4. Handling Connection Successes And Failures
> 
> 
> 
>Should a connection to a guard succeed and is authenticated via TLS, we
> 
>can then use the connection. In this case, we should cancel all other
> 
>connection timers and in-progress connections. Cancelling the timers is
> 
>so we don't attempt new unnecessary connections when our existing
> 
>connection is successful, preventing denial-of-service risks.
> 
> 
> 
>However, if we fail all available and allowed connections, we should tell
> 
>the rest of Tor that the connection has failed. This is so we can attempt
> 
>another guard relay.
> 
> 
> 
> 5. Acknowledgments
> 
> 
> 
>Thank you so much to teor for the discussion of the happy eyeballs 
> proposal.
> 
>I wouldn't have been able to do this has it not been for your help.
> 
> 
> 
> 6. Appendix
> 
>[P299-TRAC]: https://trac.torproject.org/projects/tor/ticket/29801
> 

T___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Shortcomings of the pluggable transports specification?

[teor]

Hi Philipp,

> On 13 Jun 2019, at 09:41, Philipp Winter  wrote:
> 
> We are working on improving Tor's pluggable transports specification:
> 
> 
> The goal is to make the spec useful to more people and fix issues that
> have accumulated over the years.  For more context, have a look at the
> following ticket, which we use to coordinate this effort:
> 
> 
> Before changing the spec, we need to understand its shortcomings and
> what issues implementers have run into.  For those of you who have
> experience with the spec -- either Tor's version 1.0 or version 2.1
> maintained by pluggabletransports.info -- please let us know:
> 
> * What version of the PT specification and what library implementation
>  (if any) are you using?

Yawning, David, and I pointed out a bunch of issues in PT 2.0 and 2.1:

https://lists.torproject.org/pipermail/tor-dev/2017-June/012332.html

Most of these issues were present in PT 1.0, some of them were newly
introduced in 2.0.

Some of the issues were caused by tor's limited PT interface, which
we've improved recently.

Some were also caused by confusion over whether the application or the
transport should take responsibility for certain features.

I've also opened some trac tickets over the past few years. I assume
they've been triaged, and someone has an overview of which changes we
have wanted to make in the past.

> * What has your experience been with the PT specification?

I am very confused by all the different specifications and implementations.

> * How would you improve the specification?

Start by defining a scope for pluggable transports. Ruthlessly limit the
specification to a programming language-agnostic interface. Allow for
extensions to the core specification, which can be included once they are
in active use.

Document language-specific APIs separately. Or create a language-agnostic
API using some kind of binding generator. (If that results in a functional
and usable interface.)

T



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 305: ESTABLISH_INTRO Cell DoS Defense Extension

[teor]

Hi,

> On 12 Jun 2019, at 22:39, George Kadianakis  wrote:
> 
> David Goulet  writes:
> 
>> Filename: 305-establish-intro-dos-defense-extention.txt
>> Title: ESTABLISH_INTRO Cell DoS Defense Extension
>> Author: David Goulet, George Kadianakis
>> Created: 06-June-2019
>> Status: Draft
>> 
> 
> Thanks for this proposal, it's most excellent and an essential building
> block for future work on intro point related defences.

+1

>> 
>>   We propose a new EXT_FIELD_TYPE value:
>> 
>>  [01] -- DOS_PARAMETERS.
>> 
>>  If this flag is set, the extension should be used by the
>>  introduction point to learn what values the denial of service
>>  subsystem should be using.
>> 
> 
> Perhaps we can name it "rate-limiting parameters"? But no strong opinion.
> 
>>   The EXT_FIELD content format is:
>> 
>>  N_PARAMS[1 byte]
>>  N_PARAMS times:
>> PARAM_TYPE  [1 byte]
>> PARAM_VALUE [8 byte]
>> 
>>   The PARAM_TYPE proposed values are:
>> 
>>  [01] -- DOS_INTRODUCE2_RATE_PER_SEC
>>  The rate per second of INTRODUCE2 cell relayed to the service.
>> 
>>  [02] -- DOS_INTRODUCE2_BURST_PER_SEC
>>  The burst per second of INTRODUCE2 cell relayed to the service.
>> 
>>   The PARAM_VALUE size is 8 bytes in order to accomodate 64bit values
>>   (uint64_t). It MUST match the specified limit for the following PARAM_TYPE:
>> 
>>  [01] -- Min: 0, Max: INT_MAX
>>  [02] -- Min: 0, Max: INT_MAX

This is ambiguous:
* the value is 8 bytes long
* the length of the maximum is unspecified: is it 4 bytes, 8 bytes, signed, or
  unsigned?
* the torrc default is unsigned 4 bytes: UINT32_MAX

> How would this new addition to the cell impact the size of the cell? How
> much free space do we have for additional features to this cell (e.g. to
> do the PoW stuff of the other thread)?
> 
>>   A value of 0 means the defense is disabled which has precedence over the
>>   network wide consensus parameter.

Let's say "any value has precedence over the network wide consensus
parameter". Otherwise it's unclear if 0 is a special value or not.

>>   In this case, if the rate per second is set to 0 (param 0x01) then the
>>   burst value should be ignored. And vice-versa, if the burst value is 0,
>>   then the rate value should be ignored. In other words, setting one single
>>   parameter to 0 disables the INTRODUCE2 rate limiting defense.

What happens if burst is less than rate?

> I think it could be cool to add a discussion section where we introduce
> a new cell from the intro to the service which informs the service that
> rate limiting limits have been hit. So that there is a way for the
> service to get feedback that it's under attack or capped by
> limits. Otherwise, there is simply no way to learn it.
> 
> This can be a later feature fwiw.
> 
>> 3. Protocol Version
>> 
>>   We introduce a new protocol version in order for onion service that wants
>>   to specifically select introduction points supporting this new extension.
>>   But also, it should be used to know when to send this extension or not.
>> 
>>   The new version for the "HSIntro" protocol is:
>> 
>>  "5" -- support ESTABLISH_INTRO cell DoS parameters extension for onion
>> service version 3 only.
>> 
>> 4. Configuration Options
>> 
>>   We also propose new torrc options in order for the operator to control
>>   those values passed through the ESTABLISH_INTRO cell.
>> 
>>  "HiddenServiceEnableIntroDoSDefense 0|1"
>> 
>> If this option is set to 1, the onion service will always send to the
>> introduction point denial of service defense parameters

if the intro point protocol supports them

>> regardless of
>> what the consensus enables it or not. The value will be taken from

* values will be taken from

the HiddenServiceEnableIntroDoSRatePerSec and
HiddenServiceEnableIntroDoSBurstPerSec torrc options, then

>> the consensus and if not present, the default values will be used.
>> (Default: 0)
>> 
>>  "HiddenServiceEnableIntroDoSRatePerSec N sec"
>> 
>> Controls the introduce rate per second the introduction point should
>> impose on the introduction circuit.
>> (Default: 25, Min: 0, Max: 4294967295)

Doesn't the default come from the consensus, and then the hard-coded
default?

Also see my notes about ambiguous size/signed maximums above.

>>  "HiddenServiceEnableIntroDoSBurstPerSec N sec"
>> 
>> Controls the introduce burst per second the introduction point should
>> impose on the introduction circuit.
>> (Default: 200, Min: 0, Max: 4294967295)

Doesn't the default come from the consensus, and then the hard-coded
default?

Also see my notes about ambiguous size/signed maximums above.

>>   They respectively control the parameter type 0x01 and 0x02 in the
>>   ESTABLISH_INTRO cell detailed in section 2.
>> 
>>   The default values of the rate and burst are taken from 

Re: [tor-dev] Dealing with critical sbws tickets

[teor]

Hi,

> On 4 Jun 2019, at 15:29, juga  wrote:
> 
> teor:
>> Hi juga,
>> 
>> I read your meeting notes from this week's network team meeting:
>> 
>> juga(offline):
>>   Week of 05/20 (planned)
>>  - Add Tor version to the bandwidth file (#30196)
>>   Week of 05/20 (actual)
>>   Week of 06/03 (plan)
>>  - Continue with #30406: Refactor header constants in sbws to
>> use Stem's one
>> 
>> For the next few weeks, can you focus on fixing critical sbws bugs,
>> and helping with authority deployments?
> 
> Yes, i'll do my best with the little time i've to continue with sbws.

Thanks!

Please let us know if you need help.

Until we get more sbws funding, we won't have much time to spend
on sbws features (like extra diagnostic information). But we can
spend a bit of time fixing critical sbws bugs.

>> Here's what I think we could do:
>> 
>> I would like us to deploy sbws to 3/6 bandwidth authorities some time
>> in June. We can do this deployment as soon as another directory
>> authority operator is ready.
> 
> There's another authority operator ready. If you think we don't need to
> fix any bug before a 3rd directory authority runs sbws, i can tell them
> to start running sbws.

Yes please!

>> 
>> To deploy more than 3 sbws instances, we need to fix these critical sbws
>> bugs:
>> 
>> We need sbws to generate bandwidth lines for all relays with results,
>> even if they are not Running in the sbws tor client's current consensus.
>> https://trac.torproject.org/projects/tor/ticket/29710#comment:13
>> 
>> We need sbws to use MaxAdvertisedBandwidth from the latest descriptors:
>> https://trac.torproject.org/projects/tor/ticket/30733
>> 
>> We also need to look for any more critical bugs in sbws. Here are some
>> ways we can check for bugs:
>> 
>> We need to check if all sbws instances exclude some relays, to help us find
>> any more bugs in sbws:
>> https://trac.torproject.org/projects/tor/ticket/30735
>> 
>> 90% of sbws measurement attempts fail. But these are internal errors, not
>> network errors. So it looks like sbws has a relay selection bug:
>> https://trac.torproject.org/projects/tor/ticket/30719#comment:2
>> 
>> After we do these tasks, we can deploy sbws to 4 bandwidth authorities.
>> 
>> What do you think?
>> 
> I'll look at all these bugs more in detail.
> How many directory authorities would we like to be running sbws (after
> those bugs are fixed) by which date?.

I think switching one per month is a good idea:

https://lists.torproject.org/pipermail/tor-dev/2019-June/013869.html

> BTW, longclaw's sbws did not have network for ~1 days (which for sure
> has affected to some metrics), i should have documented that somewhere,
> not sure there's a better place for that than trac.

I'm not really sure. Sending a quick email to tor-dev might be a good idea.

T___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Metrics for evaluating sbws vs torflow? (was: Raising AuthDirMaxServersPerAddr to 4)

[teor]

Hi,

> On 4 Jun 2019, at 12:54, Mike Perry  wrote:
> 
> teor:
>> 
>> 
>>> On 4 Jun 2019, at 06:20, Mike Perry  wrote:
>>> 
>>> Mike Perry:
>>>> teor:
>>>>> I have an alternative proposal:
>>>>> 
>>>>> Let's deploy sbws to half the bandwidth authorities, wait 2 weeks, and
>>>>> see if exit bandwidths improve.
> 
> Yes, that makes sense. A minimal version of this could be: don't do the
> swapping back and forth, just add sbws and replace torflow scanners one
> by one. As we do this, we could just keep a record of the metrics over
> the votes and consensus during this time, and compare how the metrics
> look for the sbws vs torflow votes vs the consensus, over time.
> 
> What were you thinking for the timeframe for the complete transition to
> sbws?

longclaw has been running sbws for a while.
bastet started running it mid-May.
We can transition a third directory authority any time we like.

We need to keep 3 torflow instances until we fix these 4 critical sbws bugs:
https://lists.torproject.org/pipermail/tor-dev/2019-June/013867.html

After those bugs are fixed, we could transition one per month.

moria1 will need to install python 3 to run sbws, I don't know how long that 
will take.

Maybe September to December 2019?

T___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Raising AuthDirMaxServersPerAddr to 4?

[teor]

Hi Mike,

> On 4 Jun 2019, at 06:20, Mike Perry  wrote:
> 
> Mike Perry:
>> teor:
>>> I have an alternative proposal:
>>> 
>>> Let's deploy sbws to half the bandwidth authorities, wait 2 weeks, and
>>> see if exit bandwidths improve.
>>> 
>>> We should measure the impact of this change using the tor-scaling
>>> measurement criteria. (And we should make sure it doesn't conflict
>>> with any other tor-scaling changes.)
>> 
>> I like this plan. To tightly control for emergent effects of all-sbws vs
>> all-torflow, ideally we'd switch back and forth between all-sbws and
>> all-torflow on a synchronized schedule, but this requires getting enough
>> measurement instances of sbws and torflow for authorities to choose
>> either the sbw file, or the torflow file, on some schedule. May be
>> tricky to coordinate, but it would be the most rigorous way to do this.
>> 
>> We could do a version of this based on votes/bwfiles alone, without
>> making dirauths toggle back and forth. However, this would not capture
>> emergent effects (such as quicker bwadjustments in sbws due to decisions
>> to pair relays with faster ones during measurement). Still, even
>> comparing just votes would be better than nothing.

I don't know how possible this is: we would need two independent network
connections per bandwidth scanner, one for sbws, and one for torflow.

(Running two scanners on the same connection means that they compete
for bandwidth. Perhaps we could use Tor's BandwidthRate to share the
bandwidth.)

I also don't know how many authority operators are able to run sbws:
Roger might be stuck on Python 2.

And I don't know how often they will be able to switch configs.

Let's make some detailed plans with the dirauth list.

>> For this experiment, my metric of choice would be "Per-Relay Spare
>> Network Capacity CDF" (see
>> https://trac.torproject.org/projects/tor/wiki/org/roadmaps/CoreTor/PerformanceExperiments#MetricsDefinitions),
>> for both the overall consensus, and every authority's vote. It would
>> also be useful to generate separate flag breakdowns of this CDF (ie
>> produce separate CDFs for Guard-only, Middle-only, Exit-only, and
>> Guard+Exit-only relays).
>> 
>> In this way, we have graphs of how the votes and the consensus
>> distribution of the difference between self-reported and measured values
>> across the network. 
> 
> Arg, I misspoke here. The metric from that performance experiment page
> is the difference between peak observed bandwidth and bw history. This
> will still be interesting to measure load balancing effects, but it does
> not directly involve the measured values.. We may also want a metric
> that directly compares properties of the measured vs advertised values.
> See below.
> 
>> We should be able to pinpoint any major
>> disagreements in how relays are measured compared to their self-reported
>> values with these metrics. (In the past, karsten produced very similar
>> sets of CDFs of just the measured values per vote when we were updating
>> bwauths, and we compared the shape of the measured CDF, but I think
>> graphing the difference is more comprehensive).
>> 
>> We should also keep an eye on CDF-DL and the failure rainbow metrics, as
>> they may be indirectly affected by improvements/regressions in load
>> balancing, but I think the distribution of "spare capacity" is the first
>> order metric we want.

Yes, I agree: some idea of client bandwidth and latency is important.

>> Do you like these metrics? Do you think we should be using different
>> ones? Should we try a few different metrics and see what makes sense
>> based on the results?
> As additional metrics, we could do the CDFs of the ratio of measured bw
> to advertised bw, and/or the metrics Karsten produced using just
> measured bw. (I can't still find the ticket where those were graphed
> during previous torflow updates, though).
> 
> These metrics would be pretty unique to torflow/sbws experiments, but if
> we have enough of those in the pipeline (such as changes to the scaling
> factor), they may be worth tracking over time.

If we get funding for sbws experiments, we can definitely tweak the sbws
scaling parameters, and do some experiments.

At the moment, I'd like to focus on fixing critical sbws issues, deploying
sbws, and making sure it works at least as well as torflow.

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Raising AuthDirMaxServersPerAddr to 4?

[teor]

Hi,

> On 3 Jun 2019, at 17:48, Roger Dingledine  wrote:
> 
>> On Sun, Jun 02, 2019 at 10:43:14PM +1000, teor wrote:
>> Let's deploy sbws to half the bandwidth authorities, wait 2 weeks, and
>> see if exit bandwidths improve.
>> 
>> We should measure the impact of this change using the tor-scaling
>> measurement criteria. (And we should make sure it doesn't conflict
>> with any other tor-scaling changes.)
> 
> Rolling out more sbws measurers sounds good to me.
> 
> But, maybe I haven't been following, but isn't the first plan for sbws
> to replace torflow but have identical behavior? And then we can work on
> changing it to have better behavior?

No, we fixed some obvious torflow bugs and design flaws.

Here are some details:

Let's talk engineering tradeoffs.

sbws had a few conflicting goals:
* create a modern bandwidth scanner implementation
* produce results that are similar to torflow
* be ready to deploy in 2019

Here's how we resolved those tradeoffs:
* use modern designs, libraries, and protocols when building sbws
* compare sbws results against torflow, and identify any issues:
  * when torflow is obviously wrong, do something better in sbws
  * when sbws is obviously wrong, log a bug against sbws, and triage it
  * when the results differ by a small amount, accept that difference

See these tickets for more details:
https://trac.torproject.org/projects/tor/ticket/27339
https://trac.torproject.org/projects/tor/ticket/27107

Here are some network health checks we are doing as we deploy sbws:
https://sbws.readthedocs.io/en/latest/monitoring_bandwidth.html

Here are some FAQs about the design, and the bandwidth file spec:
https://sbws.readthedocs.io/en/latest/faq.html
https://gitweb.torproject.org/torspec.git/tree/bandwidth-file-spec.txt

It would be great to have more design documentation, but keeping that
documentation up to date is a lot of work. And we needed to deliver
working code, too.

> I ask because in that case switching to more sbws measurers should not
> cause the exit bandwidths to improve, until we then change the measurers
> to measure better.

One of the design flaws that we fixed was torflow's "scanner partitions".

Relays can get stuck in a slow torflow scanner partition, and never improve
their measurements.

But in sbws, each relay is measured against a random faster relay.
sbws tries to choose relays that are at least 2x faster than the target.

So some stuck relay bandwidths should improve under sbws, as long as
we have enough sbws instances (about half, I think).

That said, there are still some bugs in sbws. Some of those bugs were
copied from torflow. Others are new bugs. sbws has detailed diagnostics
that will help us chase down and fix these bugs.

And we can also make design changes. But let's stabilise sbws first, and
fix any high-impact bugs.

T___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Raising AuthDirMaxServersPerAddr to 4?

[teor]

Hi all,

This is an email about an alternative proposal:

Let's deploy sbws to some more bandwidth authorities.

>> Do we have funding to continue to improve the bandwidth measurement
>> infrastructure? Or to maintain it?
>> 
>> If we don't have any grants in the pipeline, now would be a good time to
>> start some.
> 
> Agreed.
> 
> sbws was always intended (as far as I recall) to be a bandaid to make
> the torflow approach more maintainable, while we continue to await
> research on better-but-still-workable approaches. I hear the NRL folks
> have another design they've been working on that sounds promising.

There were a bunch of bugs in sbws that seemed to be excluding some
relays. So we stopped deploying sbws to any more bandwidth authorities.

In March and April, I said that we should block further deployments.
But I did some more analysis today, and I don't think those bugs are
actually blockers.

In #29710, it looked like sbws was missing about 1000 relays. But
it turns out that those relays aren't actually Running:
https://trac.torproject.org/projects/tor/ticket/29710#comment:13

In #30719, 90% of sbws measurement attempts fail. But these are
internal errors, not network errors. So it looks like it's a relay
selection bug in sbws:
https://trac.torproject.org/projects/tor/ticket/30719#comment:2

So I have an alternative proposal:

Let's deploy sbws to half the bandwidth authorities, wait 2 weeks, and
see if exit bandwidths improve.

We should measure the impact of this change using the tor-scaling
measurement criteria. (And we should make sure it doesn't conflict
with any other tor-scaling changes.)


If we do decide to change AuthDirMaxServersPerAddr, let's work out how
many new relays would be added to the consensus straight away. There
shouldn't be too many, but let's double-check.

T

--
teor
--


signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Detailed relay diagnostics for Re: Raising AuthDirMaxServersPerAddr to 4?

[teor]

Hi,

Here are some detailed diagnostics.

My overall conclusion is: there isn't much bandwidth left on that exit.

> On Sun, Jun 02, 2019 at 01:30:18PM +1000, teor wrote:
>> Which bandwidth authorities are limiting the consensus weight of these
>> relays? Where are they located?
> 
> The one in question is in Sweden:
> https://metrics.torproject.org/rs.html#details/D5F2C65F4131A1468D5B67A8838A9B7ED8C049E2
> 
> It has votes of:
> w Bandwidth=1 Measured=65200
> w Bandwidth=1 Measured=7
> w Bandwidth=1 Measured=74200
> w Bandwidth=1 Measured=77000
> w Bandwidth=1 Measured=99400
> w Bandwidth=1 Measured=102000
> 
> and it currently reports a self-measured peak at 56MBytes/s.
> 
> So one could interpret the current bwauths as saying that it is
> a bit above average compared to other 56MByte/s relays. Maybe that's
> because the other 56MByte/s relays got better lately, or maybe that's
> because there's less overall traffic on the network, but my guess is
> it's because it's stuck in that rut because the bwauths are not good
> at realizing it could go a lot faster.

Well, it's not a simple geographical bias. That's the most common
measurement issue we see. The closest bwauth has the median measurement,
and the North American bwauths are evenly distributed above and below
the median.

Interestingly, sbws measures just slightly above the median, so this
also isn't an instance of torflow's "stuck in a partition" bug.

It would be nice to have some evidence that the relay is stuck, rather
than just slow, poorly connected, or variable.

The Relays Search bandwidth history shows that both relays on that
machine vary a lot:
https://metrics.torproject.org/rs.html#details/D5F2C65F4131A1468D5B67A8838A9B7ED8C049E2
https://metrics.torproject.org/rs.html#details/6B37261F1248DA6E6BB924161F8D7B019BC3B5B4

But it doesn't tell us *why* they vary.

>> Are the relays' observed bandwidths limiting their consensus weight?
> 
> bandwidth 8960 10240 55999620
> 
> So it looks like no.

I'm sorry, my question was poorly phrased.

The observed bandwidth is part of the torflow/sbws scaling algorithm,
so it's always limiting the consensus weight.

In this case, if the relay observed more bandwidth, it would get about
1.3x that bandwidth as its consensus weight.

>> If the relays are being measured by longclaw's sbws instance, we should
>> also look at their detailed measurement diagnostics.
> 
> Looks like yes, it is measured:
> 
> w Bandwidth=1 Measured=78000
> 
> I look forward to hearing about these detailed measurement diagnostics. :)

We wrote a spec to answer all^ your questions:
https://gitweb.torproject.org/torspec.git/tree/bandwidth-file-spec.txt

^ except for these undocumented fields:
https://trac.torproject.org/projects/tor/ticket/30726


Here are some of the diagnostics from the latest bandwidth file:

> 1559468088
> version=1.4.0
> earliest_bandwidth=2019-05-28T09:35:16
> file_created=2019-06-02T09:35:04
> generator_started=2019-05-19T14:04:34
> latest_bandwidth=2019-06-02T09:34:48

sbws has been running for a few weeks, and its still measuring.

> number_consensus_relays=6552
> number_eligible_relays=6302
> percent_eligible_relays=96

It's measuring 96% of Running relays.

> recent_measurement_attempt_count=329137
> recent_measurement_failure_count=30

It has a 90% measurement failure rate, which is way too high:
https://trac.torproject.org/projects/tor/ticket/30719

But it's still measuring 96% of Running relays, so this bug might
not be as much of a blocker as we thought.

> recent_measurements_excluded_error_count=892
> recent_measurements_excluded_few_count=647
> recent_measurements_excluded_near_count=232
> recent_measurements_excluded_old_count=0

1-4% of measurements are excluded for various reasons. We think
that's normal. But it's hard to check, because torflow has
limited diagnostics.

> software=sbws
> software_version=1.1.0
> time_to_report_half_network=224554

2.6 days is quite a long time to measure half the network.
Probably due to #30719.


And here are the diagnostics for that relay, split over a few lines:

> bw=7700

This is the vote measured bandwidth.

> bw_mean=803269 bw_median=805104

This is the raw measured bandwidth, 784 KBytes/s.
This is a *lot* lower than the observed bandwidth of 56 MBytes/s.

The most likely explanation is that the relay doesn't have much
bandwidth left over.

But maybe this sbws instance needs more bandwidth. If we fixed #30719,
there might be a lot more sbws bandwidth for successful measurements.

> consensus_bandwidth=7500 consensus_bandwidth_is_unmeasured=False

This is the consensus measured bandwidth in the sbws client's consensus,
converted from scaled-kilobytes to scaled-bytes.

> desc_bw_avg=8960 desc_bw_bur=10240

This

Re: [tor-dev] Raising AuthDirMaxServersPerAddr to 4?

[teor]

Hi,

> On 2 Jun 2019, at 18:21, Roger Dingledine  wrote:
> 
> On Sun, Jun 02, 2019 at 01:30:18PM +1000, teor wrote:
>> Which bandwidth authorities are limiting the consensus weight of these
>> relays? Where are they located?
> 
> The one in question is in Sweden:
> https://metrics.torproject.org/rs.html#details/D5F2C65F4131A1468D5B67A8838A9B7ED8C049E2

In your first email, you said:

>>> He used to push an average of 500mbit on his exit relay, but then the
>>> HSDir DoS flatlined his relay for a while (!), and now, perhaps due to
>>> the bwauth variability, his exit relay only recovered to maybe 200mbit.
>>> He is running a second exit relay on that IP address, but also perhaps
>>> due to the bwauth variability, it hasn't attracted much attention either.

The relay's recent history is a bit more complicated. This fingerprint has
only been around since October 2018. It pushed 500 mbit from November 2018
to January 2019, then failed from January to March 2019. Its other bandwidths
are about the same as they are now, at around 250 mbit.

And there are two exits on this machine. Here's the other one:

https://metrics.torproject.org/rs.html#details/6B37261F1248DA6E6BB924161F8D7B019BC3B5B4

It's RelayBandwidthRate-limited to 15 MBytes/s, so the operator's first
step should be to remove this limit, and wait a week for the bandwidths
to stabilise.

The first exit is also rate-limited to 85 MBytes/s. It might be a good
idea to remove both limits at the same time.

Tor is only using about 50% of advertised exit bandwidth right now. These
particular exits are using 35% and 60% of their bandwidth limits.

So I don't see anything unusual happening here.

Can you hold off on your proposed changes until we see what happens after
the bandwidth limits are removed?

I'll send a separate detailed email with the sbws diagnostics.

T



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Release: obfs4proxy-0.0.10

[teor]

Hi,

On 5 May 2019, at 07:02, Steve Snyder  wrote:
> On 5/4/19 12:26 PM, David Fifield wrote:
>> On Sat, May 04, 2019 at 03:27:53PM +, Yawning Angel wrote:
>>> On 5/3/19 1:48 PM, Steve Snyder wrote:
 FYI, obfs4proxy no longer recognizes address:port in this form:
 
ServerTransportListenAddr obfs4 [000.000.000.000]:443
 
 Note the square brackets. Tor 0.3.5.8 / 0.4.0.5 still parses this
 syntax, and obfs4proxy used to too. As of 0.0.10 it no longer does.
>>> 
>>> Odd.  None of that code, both in obfs4proxy and goptlib, has changed for
>>> years.  I'll look at it when I have a moment.
>> 
>> Might be this?
>> 
>> tor_addr_parse is overly permissive
>> https://bugs.torproject.org/23082
> 
> Well, if that was supposed be a general fix then bug 23082 wasn't fixed
> very well. ORPort is still OK with the square bracket syntax while
> ServerTransportListenAddr is not.

You're right: tor's internal lookup function has a similar bug, which
we didn't fix in 23082. Tor uses separate functions for IP address parsing
and hostname lookups, and they handle IPv4 addresses in square brackets
differently.

I opened a ticket for this issue, and submitted a fix which uses the same
address parser on all IP addresses:
https://trac.torproject.org/projects/tor/ticket/30721

We won't backport this fix, because it changes config parsing. So it will
probably end up in tor 0.4.2.1-alpha and later.

T



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Raising AuthDirMaxServersPerAddr to 4?

[teor]

Hi,

> On 2 Jun 2019, at 13:30, teor  wrote:
> 
> On 2 Jun 2019, at 05:22, Roger Dingledine  wrote:
>> 
>> I've been talking to a longtime exit relay operator, who is in the
>> odd position of having a good 1gbit network connection, but only one
>> IP address.
>> 
>> He used to push an average of 500mbit on his exit relay, but then the
>> HSDir DoS flatlined his relay for a while (!), and now, perhaps due to
>> the bwauth variability, his exit relay only recovered to maybe 200mbit.
>> He is running a second exit relay on that IP address, but also perhaps
>> due to the bwauth variability, it hasn't attracted much attention either.
> 
> I'd like to confirm the problem before we make major network changes.
> (And I'd like to know how widespread it is.)
> 
> Which bandwidth authorities are limiting the consensus weight of these
> relays? Where are they located?
> 
> Are the relays' observed bandwidths limiting their consensus weight?
> 
> Here's how the operator can find out:
> https://trac.torproject.org/projects/tor/wiki/doc/MyRelayIsSlow#TorNetworkLimits
> 
> If the relays are being measured by longclaw's sbws instance, we should
> also look at their detailed measurement diagnostics.
> 
> longclaw's bandwidth file is available at:
> http://199.58.81.140/tor/status-vote/next/bandwidth

For example, this relay is limited by Comcast's poor peering to MIT and
Europe. We've spoken to a few Comcast relay operators with similar issues.

https://lists.torproject.org/pipermail/tor-relays/2019-June/017376.html

Adding more tor instances on networks like Comcast would only slow down
Tor.

>> The real answer is to fix the bandwidth measurement infrastructure.
> 
> Do we have funding to continue to improve the bandwidth measurement
> infrastructure? Or to maintain it?
> 
> If we don't have any grants in the pipeline, now would be a good time to
> start some.

I wrote to the grants team about bandwidth authority funding.

T


signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Raising AuthDirMaxServersPerAddr to 4?

[teor]

Hi all,

> On 2 Jun 2019, at 05:22, Roger Dingledine  wrote:
> 
> I've been talking to a longtime exit relay operator, who is in the
> odd position of having a good 1gbit network connection, but only one
> IP address.
> 
> He used to push an average of 500mbit on his exit relay, but then the
> HSDir DoS flatlined his relay for a while (!), and now, perhaps due to
> the bwauth variability, his exit relay only recovered to maybe 200mbit.
> He is running a second exit relay on that IP address, but also perhaps
> due to the bwauth variability, it hasn't attracted much attention either.

I'd like to confirm the problem before we make major network changes.
(And I'd like to know how widespread it is.)

Which bandwidth authorities are limiting the consensus weight of these
relays? Where are they located?

Are the relays' observed bandwidths limiting their consensus weight?

Here's how the operator can find out:
https://trac.torproject.org/projects/tor/wiki/doc/MyRelayIsSlow#TorNetworkLimits

If the relays are being measured by longclaw's sbws instance, we should
also look at their detailed measurement diagnostics.

longclaw's bandwidth file is available at:
http://199.58.81.140/tor/status-vote/next/bandwidth

> The real answer is to fix the bandwidth measurement infrastructure.

Do we have funding to continue to improve the bandwidth measurement
infrastructure? Or to maintain it?

If we don't have any grants in the pipeline, now would be a good time to
start some.

> But
> while we're patiently waiting for progress there, I've been thinking
> to raise moria1's AuthDirMaxServersPerAddr to 4, i.e. to allow 4 relays
> per IP address onto the network.
> 
> I don't think it would significantly increase our risk due to Sybil
> attacks, whereas there is a clear benefit in terms of some more 100's
> of mbits of good exit relay capacity.
> 
> I will propose this change to the dir-auth list in a bit, but here is
> your chance to point out surprising impacts that I haven't thought of.

Splitting bandwidth between multiple relays has privacy implications,
because traffic is easier to track between instances.

It also increases the size of the consensus.

So we should choose a value for AuthDirMaxServersPerAddr that is
a compromise between these competing goals.

Why is 4 better than 3 or 5?

T___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Should we allow matrix.org users on tor IRC channels?

[teor]

Hi all,

JeremyRand has asked us to allow matrix.org users on the tor IRC channels on 
irc.oftc.net.

matrix.org users are rejected at the moment, because they are not registered 
with OFTC.

For details, see:
https://trac.torproject.org/projects/tor/ticket/30717

T

-- 
teor
--

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Merger and Mainline Handovers

[teor]

> On 31 May 2019, at 02:26, Nick Mathewson  wrote:
> 
>> On Wed, Apr 17, 2019 at 5:13 AM teor  wrote:
>> When does 0.4.0 stop being mainline?
>> 
>> It looks like people aren't merging backports to 0.4.0 any more.
>> That's probably a good idea: we should minimise release candidate changes.
>> 
>> When should I start doing 0.4.0 merges as part of the backports?
> 
> Just realized I hadn't answered this.  Here's my suggestion, and
> please let me know if you disagree: I think that once we declare that
> 0.4.1 is stable, 0.4.0 merges count as backports. How does that sound?

That makes sense to me.

If there are any tickets where mainline mergers have missed an 0.4.0 merge,
I can do them when I do the backports to 0.3.5 and earlier.

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Onion Service - Intropoint DoS Defenses

[teor]

Hi,

> On 30 May 2019, at 23:49, David Goulet  wrote:
> 
> Over the normal 3 intro points a service has, it means 150 introduction
> per-second are allowed with a burst of 600 in total. Or in other words, 150
> clients can reach the service every second up to a burst of 600 at once. This
> probably will ring alarms bell for very popular services that probably gets
> 1000+ users a second so please check next section.

Do we know how many introduce cells are sent to popular services?

How can the operators of these services find out their current introduce rate?

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Next Network Team Meeting Times

[teor]

Hi,

Monday 27 May is a US public holiday, so we've changed the next two
meeting times:

May schedule:
* Tuesday 28 May at 2300 UTC (Monday is a US public holiday)

June schedule:
* Monday 3 June at 1700 UTC (The 28 May meeting was at the patch party time)

We're back to our regular schedule after that:
* Monday 10 June at 1700 UTC
* Monday 17 June at 1700 UTC
* Monday 24 June at 1700 UTC

We meet in the #tor-meeting channel on irc.oftc.net.

T

--
teor
--



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] TrackTor - GUI Based Implementation for Monitoring Tor

[teor]

Hi Harsh,

> On 12 May 2019, at 20:57, Harsh Gandhi <2015ucp1...@mnit.ac.in> wrote:
> 
> We are under-graduate students from Malaviya National Institute of Technology 
> currently pursuing Computer Science & Engineering.
>  
> We have developed a GUI Based Tool for monitoring TOR (inspired by Nyx). We 
> have packaged and uploaded the same on,
> 
> PyPI - https://pypi.org/project/TrackTor/
> 
> Please do give it a try and suggest for improvements.

Looks interesting!

If your software works on tor relays, you could also write to 
tor-rel...@lists.torproject.org and ask relay operators if they want to try it.

Using "Tor" in the name of the software might confuse people:

"Please don't use Tor in your product name or domain name. Instead, find a name 
that will accurately identify your products or services. … our goal is to make 
sure that people aren't confused about whether your product or project is made 
or endorsed by The Tor Project."

https://2019.www.torproject.org/docs/trademark-faq.html.en

Would you mind choosing a different name?

> Also, please let us know on how can we incorporate/integrate this with The 
> Tor Project?

Tor is a community with an supporting non-profit organisation. We make 
software, run relays, and help people be private, secure, and uncensored on the 
internet.

Did you want to join the Tor community?
Keep working on this software, and let us know how you go.
That's how lots of people become part of the community.

Did you want us to recommend your software?
We try to recommend software that is well-maintained, has multiple users, and 
community support. But that takes time.

How long do you expect to maintain it?

T___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Network Health Monitoring

[teor]

Hi,

> On 15 May 2019, at 22:40, David Goulet  wrote:
> 
>> On 08 May (13:27:31), Iain Learmonth wrote:
>> Hi All,
>> 
>> I'm working on #28322 to improve the monitoring of Tor Metrics services,
>> but this also has the side effect of monitoring network health. For
>> example, we'd like to know when Onionoo messes up and starts reporting
>> zero relays, but we also get to learn for free in the same check how
>> many relays we have and alert if that number does something weird.
>> 
>> What would be the most useful checks to add here?
>> 
>> * Range of expected total relays
>> * Range of expected relays with Guard flag
>> * Range of expected relays with Exit flag
>> * Range of expected consensus weight in each position
> 
> For all of them, what could be reported is if a large fraction disappears all
> the sudden.
> 
> Loosing for instance 500 relays at once is something worth our attention imo.
> Same goes with Exit relays... if we drop from 900 to 500, it is scary.
> 
> For the consensus weight, I would report the outliers. Maybe someone is gaming
> us and so a HUGE values compared to our top usual 10 means something is up.
> 
> As what are the good values, I don't know but I think you can probably figure
> out the average relay we loose/gain every day and scale that like 3 times for
> a warning?

Maybe it's also worth checking how many times each rule would trigger in the
past year?

If the statistics are normally distributed, you could use 4 standard deviations,
so that each rule (falsely) triggers about once a year.

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 302: Hiding onion service clients using WTF-PAD

[teor]

> On 21 May 2019, at 00:35, George Kadianakis  wrote:
> 
> Tom Ritter  writes:
> 
>>> On Thu, 16 May 2019 at 11:20, George Kadianakis  
>>> wrote:
>>>3) Duration of Activity ("DoA")
>>> 
>>>  The USENIX paper uses the period of time during which circuits send and
>>>  receive cells to distinguish circuit types. For example, client-side
>>>  introduction circuits are really short lived, wheras service-side
>>>  introduction circuits are very long lived. OTOH, rendezvous circuits 
>>> have
>>>  the same median lifetime as general Tor circuits which is 10 minutes.
>>> 
>>>  We use WTF-PAD to destroy this feature of client-side introduction
>>>  circuits by setting a special WTF-PAD option, which keeps the circuits
>>>  open for 10 minutes completely mimicking the DoA of general Tor 
>>> circuits.
>> 
>> 10 minutes exactly; or a median of 10 minutes?  Wouldn't 10 minutes
>> exactly be a near-perfect distinguisher? And if it's a median of 10
>> minutes, do we know if it follows a normal distribution/what is the
>> shape of the distribution to mimic?
>> 
> 
> Oops, you are right, Tom.
> 
> It's not 10 minutes exactly. The right thing to say is that it's a median
> of 10 minutes, altho I'm not entirely sure of the exact distribution.
> 
> These circuits basically now follow the MaxCircuitDirtiness
> configuration like general circuits, and it gets orchestrated by
> circuit_expire_old_circuits_clientside(). Not sure if it's in a spec
> somewhere.
> 
> I will update the spec soon with the fix. Thanks!

If I understand correctly, Tor's circuits close about 10 minutes after
the last time they handled traffic.

So that's a *minimum* of 10 minutes. And probably a *median* of
slightly more than 10 minutes, if the user is web browsing.

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposing sbws changes

[teor]

Hi,

We've talked about sbws a bit over the last few days.

I'm going to quote some things I wrote in a private email, with some
minor edits. (I would quote the whole thread, but I don't have
permission.)

We also discussed sbws in our team meeting today. Because we are in
lots of different timezones, only some people were there. I'll try
to summarise the meeting discussion as well.

Juga wrote some questions in the team meeting pad. I'll try to answer
their questions here.

>> On 22 Apr 2019, at 14:54, teor  wrote:
>> 
>> We finished our first working version of sbws in March, and deployed
>> it to a directory authority. We're now working on deploying it to a
>> few more directory authorities:
>> https://trac.torproject.org/projects/tor/ticket/29290
>> 
>> We're also working on archiving and analysing the bandwidth files
>> produced by sbws and Torflow:
>> https://trac.torproject.org/projects/tor/ticket/21378
>> 
>> During this work, we've discovered some missing sbws features:
>> https://trac.torproject.org/projects/tor/ticket/30255
>> 
>> We need a better process for proposing and reviewing sbws changes.
>> 
>> At the moment, I am spending a lot of time reviewing and designing
>> sbws changes. And that's not sustainable. We need a process that works
>> when I go on leave, or get busy with other tasks.


> I wrote, privately:
> 
> Our problem is that we keep making lots of changes to sbws. But we don't
> have anyone managing those changes. That makes them high risk.
> 
> We need to assign another paid staff member to do some design and planning
> work on sbws.

> 
> We need to slow down the change process, so sbws becomes stable software.


In the team meeting, we agreed to block sbws merges until I get back from
leave at the end of May. We can make exceptions for critical bug fixes.

In general, we also need to slow down changes to sbws, to manage our
team's workload. sbws is not funded right now, so we need to spend less
time on it.


>> I suggest that we use the tor proposals process:
>> https://gitweb.torproject.org/torspec.git/tree/proposals/001-process.txt
>> 
>> We can submit small changes as diffs to the bandwidth file spec:
>> https://gitweb.torproject.org/torspec.git/tree/bandwidth-file-spec.txt
>> 
>> But large changes, controversial changes, and changes with multiple
>> competing options should have their own proposal. Then, once we decide
>> what to do, we can integrate those changes into the spec.


> Our priority for sbws is maintaining stable software. That's more important
> than writing and merging features quickly.
> 
> We need to talk about sbws changes on tor-dev, so that other people
> can get involved. We can't expect people to watch all the sbws tickets.
> 
> If we keep doing high-risk changes with no feedback, then we will never
> be able to deploy sbws on more than 2 directory authorities.


Juga wrote on the network-team meeting pad:

> - Question 1: teor or network-team: which on do you think has higher
>   priority, #29710 or #30255?
> 
> - Question 2: teor or network-team: longclaw's sbws 1.1.0 has been
>   running for more than 1 week, do we want to start
>   running sbws 1.1.0 on basted or should #29710 and/or
>   #30255 be solved first?


For #29710 sbws reports 6200 relays, 1000 fewer than Torflow's 7200

We should deploy sbws 1.1.0 to bastet. Then wait 5-10 days for the
measurements to be stable.

Comparing longclaw and bastet will help us diagnose #29710:
* Does bastet also report 1000 fewer relays than Torflow?
* Do bastet and longclaw exclude the same relays?
* What else do the results tell us?

We can write and test changes for #29710, and I will review them at
the end of May. (These changes are high risk.)


For #30255 Add additional bandwidth file headers in sbws 1.2

We can add extra headers to sbws. Anyone can review them. But I'd
like to double-check the design before we merge. (These changes are
low risk.)

We talked about how to structure these pull requests on the ticket.
We wanted the changes to be easy to review and merge.
But my advice was not the best advice.

Here is some better advice:

Put the commits in one pull request, in this order:
1. We want to be able to add each header using 1-2 lines of code.
   So we need to refactor the header code. Add commits for this
   refactor.
2. Add one new header in each commit. If a group of headers makes
   more sense together, put their commits next to each other.



We will do sbws reviews as a low priority, so we can manage team
workload. That means that it might take us a few weeks to review them.

But please test them while you are waiting for a review.

I hope that helps!

T

--
teor

[tor-dev] PrivCount Status

[teor]

Hi,

Nick asked me to send a status email about PrivCount, before I go on
leave for a few weeks.

Plan

We want to add the following counters to a PrivCount Proof of Concept:
* check counters (zero, relay count, time in seconds)
* consumed bandwidth

Nick also suggested adding connection counts. That seems like a good
counter, but we want to make sure we do bandwidth in the first release,
because it's a high-risk statistic.


Status

In March and April, I deferred PrivCount tasks to work on chutney for
one of our other sponsors.

I also delayed these tasks, because I was waiting for #29017 and #29018
to merge:
* #29017 PaddingStatistics should be disabled when ExtraInfoStatistics is 0
* #29018 Make all statistics depend on ExtraInfoStatistics


Tickets

The top-level ticket is:

PrivCount proof of concept with existing statistics
https://trac.torproject.org/projects/tor/ticket/27908

I was mainly working on code for these tickets:

PrivCount proof of concept: implement check counters
https://trac.torproject.org/projects/tor/ticket/29004

PrivCount proof of concept: implement consumed bandwidth counters
https://trac.torproject.org/projects/tor/ticket/29005

Make relays report bandwidth usage more often in test networks
https://trac.torproject.org/projects/tor/ticket/29019


Code

I have incomplete branches for #29004, #29005, and #29019 here:
https://github.com/teor2345/tor/tree/ticket29004-wip
https://github.com/teor2345/tor/tree/ticket29005
https://github.com/teor2345/tor/tree/ticket29019

I think all the necessary code is present in these branches.
(But maybe it's not???)

But it needs some cleanup:
* rebase on to the current master,
* put the commits on the right branches
* make sure it does what these tickets say it should do

I'm happy to do that after I come back from leave.
I am also happy if Nick wants to clean up this code.

See also my previous email about BridgeDB and PrivCount. Maybe we can
save ourselves some effort by using PrivCount's obfuscation on
BridgeDB's statistics.

T

--
teor
--



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Export BridgeDB usage statistics

[teor]

Hi Philipp, Karsten,

> On 24 Apr 2019, at 10:50, Philipp Winter  wrote:
> 
> I'm working on , which will make
> BridgeDB export usage statistics.  I would like these statistics to be
> public, privacy-preserving, and -- ideally -- added to Tor Metrics.  I
> wanted to hear your thoughts on 1) what statistics we should collect,
> 2) how we can collect these statistics safely, and 3) what format these
> statistics should have.
> 
> Broadly speaking, these statistics should answer the following
> questions:
> 
>  * How many requests does BridgeDB see per day?
>  * What obfuscation protocols are the most popular?
>  * What bridge distribution mechanisms are the most popular?
>  * From what countries do we see the most bridge requests?
>  * How many BridgeDB requests fail and succeed, respectively?
>  * How many requests does BridgeDB see from Yahoo/Gmail/Riseup?
>  * How many HTTPS requests are coming from proxies?
>  * How many requests are suspicious, and likely issued by bots?
> 
> Each request to BridgeDB carries with it some information, which allows
> us to answer the above questions.  I suggest that we collect the
> following:
> 
>  * The distribution mechanism.  Currently, this is HTTPS, email, or
>Moat.
> 
>  * The requested transport.  Currently this is vanilla, fte, obfs3,
>obfs4, or scramblesuit.
> 
>  * The request's origin.  For Moat and HTTPS, it's the two-letter
>country code, e.g., IT for Italy.  For email, it's the user's email
>domain (Gmail, Yahoo, or Riseup).
> 
>  * Whether the request was successful or unsuccessful, i.e., resulted
>in BridgeDB handing out bridges or not.
> 
>  * Whether the request was issued by a user or a bot.
>David suggested heuristics that would allow us to estimate if a
>request came from a bot:
> I like these
>suggestions but I'm not sure yet how to encode them -- it's more
>complex than a simple binary flag.
> 
> The combination of these statistics results in ~16,800 buckets (3
> mechanisms * 5 transports * ~280 ISO country codes * 2 success states *
> 2 bot states).  We only need to export statistics with non-empty
> buckets.  To protect users whose request is the only one in a given
> bucket (e.g., there may be only one user in Turkmenistan who
> successfully requested an FTE bridge over HTTPS on 2019-04-02), we
> should bin the statistics by rounding them up to the next multiple of,
> say, 10.  We should further export statistics infrequently -- maybe once
> a day.
> 
> Here's an example of a simple CSV format that takes into account the
> above:
> 
>  timestamp,mechanism,transport,country|domain,success,count,origin
>  1555977600,https,vanilla,it,successful,40,user
>  1555977600,https,obfs4,ca,unsuccessful,10,user
>  1555977600,email,vanilla,yahoo.com,successful,50,user
>  ...
> 
> What are your thoughts?

Over the next few months, Nick and I are going to work on
PrivCount for statistics generated by tor relays and bridges.
(I'll be on leave from today until late May.)

We haven't done the detailed design of PrivCount's API yet.

For Tor relay/bridge statistics, we'll have some Rust code
embedded in the tor binary (Data Collectors), which will
add noise, bin, and blind the statistics.

Then we'll have some aggregation servers (Tally Reporters)
which will aggregate and un-blind the results.

If we design the interfaces correctly, we should be able to
re-use the noise and bin code for BridgeDB. (The blinding is
redundant, until we have more than one BridgeDB.)

I imagine we could pass results to a command-line tool for
noise and binning. This tool would also be useful for tests.
(Tests are *so* much simpler when there's no network in the
middle.)

That way, all of Tor's relay, bridge, and BridgeDB statistics
will noised, binned, and reported in the same way.

I'm not sure if the timeframes will work out though: I'll be
doing the noise and binning when I get back at the end of May.

So we might need to do something quick and dirty until then.

T



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposing sbws changes

[teor]

Hi all,

We finished our first working version of sbws in March, and deployed
it to a directory authority. We're now working on deploying it to a
few more directory authorities:
https://trac.torproject.org/projects/tor/ticket/29290

We're also working on archiving and analysing the bandwidth files
produced by sbws and Torflow:
https://trac.torproject.org/projects/tor/ticket/21378

During this work, we've discovered some missing sbws features:
https://trac.torproject.org/projects/tor/ticket/30255

We need a better process for proposing and reviewing sbws changes.

At the moment, I am spending a lot of time reviewing and designing
sbws changes. And that's not sustainable. We need a process that works
when I go on leave, or get busy with other tasks.

I suggest that we use the tor proposals process:
https://gitweb.torproject.org/torspec.git/tree/proposals/001-process.txt

We can submit small changes as diffs to the bandwidth file spec:
https://gitweb.torproject.org/torspec.git/tree/bandwidth-file-spec.txt

But large changes, controversial changes, and changes with multiple
competing options should have their own proposal. Then, once we decide
what to do, we can integrate those changes into the spec.

T

--
teor
--



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Next Network Team Meeting Time

[teor]

Hi,

> On 19 Apr 2019, at 04:23, Ali Mmo  wrote:
> 
> How should i participate???
> 
> On Apr 18, 2019 9:35 PM, "Zachary Crownover"  
> wrote:
> How does one attend the meetings?
> 
> On Apr 18, 2019, at 9:58 AM, Ali Mmo  wrote:
> 
>> Hi excuse me I'm in the Iran and I can not attend the meeting

The network team meetings are online in the #tor-meeting channel on
the irc.oftc.net IRC network. irc.oftc.net supports IRC over Tor.

You'll need an IRC application to participate.

We rotate times, so that most people can participate at least once a
month.

>> On Apr 18, 2019 4:48 AM, "teor"  wrote:
>> Hi,
>> 
>> The next network team meeting is at 2300 UTC on Tuesday 23 April.
>> (Our "patch party" and "first meeting of the month" time.)
>> 
>> Here is our schedule for the next few months:
>> 
>> April schedule:
>> * Tuesday 23 April at 2300 UTC (at the "patch party" time)
>> * Monday 29 April at 1700 UTC
>> 
>> May schedule:
>> * Monday 6 May at 1700 UTC (first May meeting, but at the normal time)
>> * Monday 13 May at 1700 UTC
>> * Monday 20 May at 1700 UTC
>> * Monday 27 May at 1700 UTC
>> 
>> June schedule:
>> * Tuesday 4 June at 2300 UTC (first meeting of the month)
>> * Monday 10 June at 1700 UTC
>> 
>> T



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Next Network Team Meeting Time

[teor]

Hi,

The next network team meeting is at 2300 UTC on Tuesday 23 April.
(Our "patch party" and "first meeting of the month" time.)

Here is our schedule for the next few months:

April schedule:
* Tuesday 23 April at 2300 UTC (at the "patch party" time)
* Monday 29 April at 1700 UTC

May schedule:
* Monday 6 May at 1700 UTC (first May meeting, but at the normal time)
* Monday 13 May at 1700 UTC
* Monday 20 May at 1700 UTC
* Monday 27 May at 1700 UTC

June schedule:
* Tuesday 4 June at 2300 UTC (first meeting of the month)
* Monday 10 June at 1700 UTC

T

--
teor
--




signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Merger and Mainline Handovers

[teor]

Hi Nick, George, David,

(I'm sending this email to tor-dev so everyone knows how Core Tor
merges are going.)


Mainline Mergers

David is back from leave, so I'm going to stop doing mainline merges.
But please let me know if there's a merge I can help with.
(Email or Signal is best, IRC has a lot of backlog.)

Do we need to do a handover some time?
The next team meeting might be a good time.


Mainline Merge Ready Tickets

I moved my mainline merge trac wiki queries to this page:
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/MainlineMerges

That page should show all of the mainline merge_ready tickets, sorted
by owner and reviewer. Your name is in bold, so you can work out which
tickets you should merge. (We want 3 people to look at every ticket
before it merges, except for trivial changes.)

Here is our full list of task tracking wiki pages:
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam#TaskTracking


When does 0.4.0 stop being mainline?

It looks like people aren't merging backports to 0.4.0 any more.
That's probably a good idea: we should minimise release candidate changes.

When should I start doing 0.4.0 merges as part of the backports?


Backport Status

We released 0.4.0.4-rc last week, so I'm going to backport some
low-risk changes to 0.2.9 and later. Most of these changes have been
tested in 0.4.0.3-alpha.

I should be able to do the backports tomorrow or Tuesday.

Here are the backports for the next few days:
https://trac.torproject.org/projects/tor/wiki/user/teor#Backports:0.5daysperweek

Here are the backports I will do after I get back from my leave in May:
https://trac.torproject.org/projects/tor/wiki/user/teor/HiddenBackports

T

--
teor
--



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Next Network Team Meeting Time

[teor]

Hi,

The next network team meeting is at 2300 UTC on Tuesday 2 April.

Here is our full April schedule:
* Tuesday 2 April at 2300 UTC
* Monday  8 April at 1700 UTC
* Monday 15 April at 1700 UTC
* Tuesday 23 April at 1700 UTC (Monday 22 April is the Easter Monday public 
holiday)
* Monday 29 April at 1700 UTC

Daylight saving changed recently, so at least one of these meetings
will be at a different local time for you.

Our 1700 UTC meeting tracks north american daylight saving time.
Our 2300 UTC meeting does not change for daylight saving time.

T

--
teor
--



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Docker images for help people to run Tor (teor)

[teor]

Hi,

> On 1 Apr 2019, at 07:55, Alessandro Fiori  wrote:
> 
> Hi,
> I would like to share with the community some tests I've made for setting up 
> a "swarm" of Tor relays.
> 
> I setted up 3 exit relays and 3 middle relays on the same IPv4 address, to 
> make some testing, and the entire swarm works like a charm.
> 
> Due to Tor limitation, I have decided to build a template for preparing the 
> cluster, and upload it to GitHub.
> 
> You can find the example mini-cluster here:
> https://github.com/randomtable/ChimeraOS/tree/master/TorSwarm 
> 
> I have tested the exit capabilities on another network with different public 
> IP address.
> 
> All the relays have published their own descriptors, I have prepared the 
> containers as a single, separated machines.
> 
> The process is different from preparing a single relay, because of Docker 
> build process.
> 
> I have disabled caching during the cluster process, so i have been able to 
> modify torrc without issues.
> 
> Because  its very interesting, attached to this e-mail there is the console 
> Logs from the 6 relays.
> 
> All containers are read-only, so the entire relay will destroy at the end of 
> session.
> 
> I hope this can be useful.

Tor clients will only see 2 of your 6 relays, because the Tor directory 
authorities only publish 2 relays per IPv4 address.

Also, please use the deb.torproject.org repositories to get the latest version 
of tor:
Mar 31 19:57:08.000 [warn] Please upgrade! This version of Tor (0.3.3.9) is not 
recommended, according to the directory authorities. Recommended versions are: 
0.2.9.15,0.2.9.16,0.2.9.17,0.3.4.10,0.3.4.11,0.3.5.7,0.3.5.8,0.4.0.1-alpha,0.4.0.2-alpha,0.4.0.3-alpha

https://trac.torproject.org/projects/tor/wiki/TorRelayGuide/DebianUbuntu

T___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Docker images for help people to run Tor

[teor]

Hi,

(Please try to reply to the thread, not the digest!)

> On 30 Mar 2019, at 11:51, Alessandro Fiori  wrote:
> 
> Because the images are built separately, and Containers doesn't have 
> persistent data by default, i'm testing the launch of a "relay swarm", which 
> isn't the "Docker Swarm".
> The concept of a "Relay swarm" is to have multiple running relay containers 
> on the same user machine, without any conflict.
> 
> The idea (i will testing it very soon) is possible, by pulling an image and 
> running it multiple times, creating intermetiate containers.
> 
> A function like the "relay swarm" can be very useful, i don't know if this 
> approach can be risky, so i'm preparing a script for launching only one type 
> of relay swarm (middle or exit relays).
> 
> If there is no security concerns, this function will be a very key function 
> for grow up the Network, because a user can run multiple relays at a time.

Tor only allows 2 relays per IPv4 address.

T


signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Docker images for help people to run Tor

[teor]

Hi,

> On 29 Mar 2019, at 08:57, Alessandro Fiori  wrote:
> 
> I've decided to release two scripts (I'm releasing for Ubuntu too), for run 
> an Exit Relay or Hidden Service then, with Docker image.
> 
> If needed, let me know if there would be useful  releasing an image of a 
> Non-Exit-Relay.

Yes, many relay operators will want to run a non-exit relay.

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Tor Friendliness Scanner

[teor]

Hi,

> On 17 Mar 2019, at 08:04, Ryan Duff  wrote:
> 
>> The way we intended to do this was to send our FF "ground-truth"
>> collection through Tor, and specifically through the same exit node as
>> TTB uses. This way we can isolate the variable to the differences in the
>> browsers, rather than any network or other concerns. In addition, we are
>> working on developing a method for determining if content is dynamically
>> generated (and therefore different every time), or broken.
> 
> Won't this severely taint your "ground-truth" though? If you're ultimately 
> using the exit-node for both, then you'll really only end up measuring what 
> is friendly to the Tor Browser itself instead of Tor overall. Is that the 
> goal?
> 
> If not, with direct access to an exit node, you could run the test direct 
> from that node (not using Tor) for the "ground truth" and then use it as the 
> exit node of choice for the Tor Browser to have a solid comparison between 
> Tor vs Not-Tor.

Most sites block by IP (or IP range), so a direct connection using the
exit node's IP should give you very similar results to a Tor circuit
using the exit node's IP.

T



signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 301: Don't include package fingerprints in consensus documents

[teor]

> On 14 Mar 2019, at 03:50, Iain Learmonth  wrote:
> 
> Signed PGP part
> Hi,
> 
>>> On 25/02/2019 23:30, teor wrote:
>>>> Looks good to me, let's merge it as an "accepted" proposal?
> 
> This is now proposal 301.
> 
> What is the process by which this becomes "accepted"? Is this just a
> matter of someone making that commit?

Here's what "Accepted" means:

   Accepted: The proposal is complete, and we intend to implement it.
  After this point, substantive changes to the proposal should be
  avoided, and regarded as a sign of the process having failed
  somewhere.

https://gitweb.torproject.org/torspec.git/tree/proposals/001-process.txt#n152

I can't imagine us making any substantive changes to the proposal.

The reasons and actions are clear, and there are clearly-defined sub-tickets
for each of the tasks. We have successfully implemented similar tasks before.

Unless anyone objects in the next week, let's make this change:
https://trac.torproject.org/projects/tor/ticket/29776

T


signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Updating Proposal 299 (Preferring IPv4 or IPv6 based on IP Version Failure Count)

[teor]

> On 13 Mar 2019, at 08:44, Neel Chauhan  wrote:
> 
> Hi teor,
> 
>> Thanks, I opened a ticket to review and merge it here:
>> https://trac.torproject.org/projects/tor/ticket/29687
>> It should get reviewed within a week or two.
> 
> First off, thank you for doing this! As of now, it seems to have been merged.
> 
>>> I have added the "TrackFailures" flag to my GitHub PR and also mentioned 
>>> other flags will come in the future.
>> I like what you did here: I think having one option with extra flags is
>> better than two options.
> 
> Again, Thank you!
> 
> Also, can we mark Prop299 as "Accepted" or is there any updates needed to 
> this proposal?

I haven't reviewed the latest version, and I'm not sure if anyone else has 
either.
(I was hoping that someone else would review your latest version before it
got merged.)

Reading through it, I've noticed some things that I don't understand. I'll do a
review on your pull request. I'll also get a second team member to review the
proposal.

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Updating Proposal 299 (Preferring IPv4 or IPv6 based on IP Version Failure Count)

[teor]

> On 7 Mar 2019, at 09:07, Neel Chauhan  wrote:
> 
> Hi teor,
> 
> On 2019-03-05 21:12, teor wrote:
>> Sure. We usually do torspec changes using GitHub pull requests.
>> Can you open a pull request on https://github.com/torproject/torspec ?
>> Then link to the pull request from #27491 or in an email reply.
> 
> The GitHub PR is here: https://github.com/torproject/torspec/pull/61

Thanks, I opened a ticket to review and merge it here:
https://trac.torproject.org/projects/tor/ticket/29687

It should get reviewed within a week or two.

>> Here's one way we could do that:
>> Add a new option ClientAutoIPv6ORPortStrategy, which takes some flags.
>> When no flags are set (the default), we use the original
>> ClientAutoIPv6ORPort.
>> Then we add a flag for this proposal:
>>  - TrackFailures
>> Then other tickets can use other flags:
>>  - CheckLocalAddresses for
>> #27492 Try IPv4 or IPv6 more often based on public or private IP
>> addresses
>>  - CheckConsensus for
>>#27647 When randomly choosing IPv4 or IPv6, set IPv6 probability
>> based on IPv6 weight
>> What do you think?
>> Would you like to make these changes to this proposal?
> 
> These additions sound good.
> 
> I have added the "TrackFailures" flag to my GitHub PR and also mentioned 
> other flags will come in the future.

I like what you did here: I think having one option with extra flags is
better than two options.

T




signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] download consensus from mirrors based on frequency options in torrc

[teor]

Hi,

> On 7 Mar 2019, at 22:00, Pouya Miralaee  wrote:
> 
> according to these frequency options below, how should i specify that tor 
> does not or if it has to, only in necessary times downloads consensus from 
> directory servers and in all other cases, download them from mirrors or 
> fallbacks.
> 
> "ClientBootstrapConsensusAuthorityDownloadSchedule": "6, 11, 3600, 10800, 
> 25200, 54000, 111600, 262800",
> "ClientBootstrapConsensusFallbackDownloadSchedule": "0, 1, 4, 11, 3600, 
> 10800, 25200, 54000, 111600,262800",
> "ClientBootstrapConsensusAuthorityOnlyDownloadSchedule": "0, 3, 7, 3600, 
> 10800, 25200, 54000, 111600,262800",
> 
> I just intend to complete the decentralized situation in here. but i can not 
> figure out how these frequency options work together.

The defaults are fine for most clients.
And if you change them, you may be less anonymous.

All supported versions of Tor only use the first value in the list:
  o Minor features (configuration):
- The "DownloadSchedule" options have been renamed to end with
  "DownloadInitialDelay". The old names are still allowed, but will
  produce a warning. Comma-separated lists are still permitted for
  these options, but all values after the first are ignored (as they
  have been since 0.2.9). Closes ticket 23354.
https://gitweb.torproject.org/tor.git/tree/ReleaseNotes#n1891

Here's how you can use the authorities less:

1. Make sure you are using the default set of authorities.
(No DirAuthority or AlternateDirAuthority lines in your config.)

2. Avoid using the directory authorities when you're looking for a fallback:
  DirAuthorityFallbackRate 0.01
Don't set this value to zero, unless you are willing to fail if the
fallbacks you try are not available.

3. Delay authority fetches until you've checked a few fallbacks:
On a client:
  ClientBootstrapConsensusAuthorityDownloadSchedule "30"
Or however many seconds you're willing to wait if the fallbacks you try are
not available. (You can't make relays stop connecting to the authorities.)

And if you're using 0.3.4.8 or later, then you can spell the option like this:
  ClientBootstrapConsensusAuthorityDownloadInitialDelay 30
The old spellings work on all supported Tor versions.

Have fun!

T


signature.asc
Description: Message signed with OpenPGP
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev