Re: [tor-dev] Fact-checking a claim about relay/bridge fingerprint authentication

[David Fifield]

On Thu, Feb 15, 2024 at 10:54:22AM -0500, Roger Dingledine wrote:
> > If possible, we'd still like confirmation of (1) whether this is a good
> > characterization of the constraints involved when using a Tor bridge,
> > and (2) if 4.2 is the right part of tor-spec to cite for clients
> > disconnecting on an unexpected relay fingerprint.
> > 
> > https://github.com/turfed/snowflake-paper/blob/7181a5cdfe1e07cfb4ea6bc15c07d69cd793e908/snowflake.tex#L2396
> > A Tor bridge is identified by a long-term identity public key.
> > If, on connecting to a bridge, the client finds that the
> > bridge's identity is not the expected one, the client will
> > terminate the connection \cite[\S 4.2]{tor-spec}. The Tor client
> > can configure at most one identity per bridge; there is no way
> > to indicate (with a certificate, for example) that multiple
> > identities should be considered equivalent. This constraint
> > leaves two options: either all Snowflake bridges must share the
> > same cryptographic identity, or else it must be the client that
> > makes the choice of what bridge to use. While the former option
> > is possible to do (by synchronizing identity keys across
> > servers), every added bridge would increase the risk of
> > compromising the all-important identity keys. Our vision was
> > that different bridge sites would run in different locations
> > with their own management teams, and that any compromise of a
> > bridge site should affect that site only
> 
> This all sounds reasonable to me.
> 
> The phrase "all-important identity keys" made me pause, because in the
> obfs4 bridgedb case, it doesn't matter so much that the client checks
> the identity key, (a) because it's only one hop in the circuit, and
> the client will still check the identities for the other relays in the
> circuit, and (b) because it isn't all that meaningful to verify that it
> really is the stranger we picked for you at random -- so long as whoever
> it is extends your circuit to the relay you picked, and you verify that,
> you're getting most of what you can get.
> 
> Whereas for the Snowflake case, checking the identity key on the bridge
> is much more meaningful, first because we do actually know the bridge
> operator and we want to make sure you reached the real one, but more
> importantly because the Snowflake architecture puts the 'random stranger'
> in exactly the position to send you somewhere else if it wants.
> 
> So yes, sounds good.

Thanks for checking.

> As for which part of tor-spec to cite, note that Nick and others did a
> big reorg of tor-spec some months ago. It looks like 4.2 is from what
> used to be path-spec.txt, which is about how we choose paths. If I were
> to pick a piece of tor-spec to show that Tor clients decline to continue
> if the first hop they've picked can't prove it knows its identity key,
> I would pick section 2.3.1,
> https://spec.torproject.org/tor-spec/negotiating-channels.html#negotiating
> and the action is in the CERTS and AUTHENTICATE cells.
> 
> (For extra fun, I don't think anything promises that "2.3.1" will still
> be the number of this section in the future.)

Okay, that makes sense. What's now section 2.3.1 is the same
"Negotiating and initializing connections/channels" as what was
section 4 in the txt version we were referring to initially. So it looks
like we did have the right part of the spec.
https://gitlab.torproject.org/tpo/core/torspec/-/blob/29e445bd6e9efe82367b8a2b09a6c6aa0bc92b7b/spec/tor-spec/negotiating-channels.md
https://gitlab.torproject.org/tpo/core/torspec/-/blob/33308845cec54bfc0096b8ea0339a8ff183aa1b1/tor-spec.txt#L622

The new mdbook style makes it a little harder to refer to a specific
section. Since this is the only reference to tor-spec we have, I guess
what we'll do is change the bib entry to refer to the .md file of just
this section, with a GitLab permalink.

Is `author = {Roger Dingledine and Nick Mathewson}` still appropriate?
That was the authorship on tor-spec.txt, but the new .md doesn't have
it. Would `author = {{The Tor Project}}` be better?
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Fact-checking a claim about relay/bridge fingerprint authentication

[David Fifield]

The Snowflake paper has been conditionally accepted to Usenix Security
and we are now working on final revisions. As before, no response is
necessary, but if you have any comments, we can try to take them into
account up until about 2024-02-26. This is a current snapshot:
https://www.bamsoftware.com/papers/snowflake/snowflake.20240210.7181a5cd.pdf

If possible, we'd still like confirmation of (1) whether this is a good
characterization of the constraints involved when using a Tor bridge,
and (2) if 4.2 is the right part of tor-spec to cite for clients
disconnecting on an unexpected relay fingerprint.

https://github.com/turfed/snowflake-paper/blob/7181a5cdfe1e07cfb4ea6bc15c07d69cd793e908/snowflake.tex#L2396
A Tor bridge is identified by a long-term identity public key.
If, on connecting to a bridge, the client finds that the
bridge's identity is not the expected one, the client will
terminate the connection \cite[\S 4.2]{tor-spec}. The Tor client
can configure at most one identity per bridge; there is no way
to indicate (with a certificate, for example) that multiple
identities should be considered equivalent. This constraint
leaves two options: either all Snowflake bridges must share the
same cryptographic identity, or else it must be the client that
makes the choice of what bridge to use. While the former option
is possible to do (by synchronizing identity keys across
servers), every added bridge would increase the risk of
compromising the all-important identity keys. Our vision was
that different bridge sites would run in different locations
with their own management teams, and that any compromise of a
bridge site should affect that site only

On Tue, Oct 03, 2023 at 08:44:39PM -0400, David Fifield wrote:
> Cecylia, Arlo, Serene, Shelikhoo, and I are writing a research paper
> about Snowflake. Here is a draft:
> https://www.bamsoftware.com/papers/snowflake/snowflake.20231003.e6e1c30d.pdf
> 
> We're writing to check a factual claim in the section about having
> multiple backend bridges. Basically, we wanted it to be possible for
> there to be multiple Snowflake bridge sites run by different groups of
> people, and we did not want to share the same relay identity keys across
> all bridge sites, because of the increased risk of the keys being
> exposed. Therefore every bridge site has its own relay identity, which
> requires the client to know the relay fingerprints in advance and that
> it be the client (and not, e.g., the broker) that decides which bridge
> to use.
> 
> 1. Is our general description (quoted below) of the design constraints
>as they bear on Tor correct?
> 2. Is §4.2 "CERTS cells" the right part of tor-spec to cite to make our
>point?
>
> https://gitlab.torproject.org/tpo/core/torspec/-/blob/b345ca044131b2eb18e6ae0d5f23643a92aeff34/tor-spec.txt#L707
> 
> https://github.com/turfed/snowflake-paper/blob/e6e1c30dde6716dc5e54a32f2134f19068a7f395/snowflake.tex#L2146
>   A Tor bridge is identified by a long-term identity public key.
>   If, on connecting to a bridge, the client finds that the
>   bridge's identity is not the expected one, the client will
>   terminate the connection \cite[\S 4.2]{tor-spec}. The Tor client
>   can configure at most one identity per bridge; there is no way
>   to indicate (with a certificate, for example) that multiple
>   identities should be considered equivalent. This constraint
>   leaves two options: either all Snowflake bridges must share the
>   same cryptographic identity, or else it must be the client that
>   makes the choice of what bridge to use. While the former option
>   is possible to do (by synchronizing identity keys across
>   servers), every added bridge would increase the risk of
>   compromising the all-important identity keys. Our vision was
>   that different bridge sites would run in different locations
>   with their own management teams, and that any compromise of a
>   bridge site should affect that site only.
> 
> In my own experiments, providing an incorrect relay fingerprint leads to
> errors in connection_or_client_learned_peer_id:
> https://gitlab.torproject.org/tpo/core/tor/-/blob/tor-0.4.7.13/src/core/or/connection_or.c#L1897
> [warn] Tried connecting to router at 192.0.2.3:80 ID= 
> RSA_ID=2B280B23E1107BB62ABFC40DDCC8824814F80A71, but RSA + ed25519 identity 
> keys were not as expected: wanted  + 
> no ed25519 key but got 2B280B23E1107BB62ABFC40DDCC8824814F80A72 + 
> 1zOHpg+FxqQfi/6jDLtCpHHqBTH8gjYmCKXkus1D5Ko.
> [warn] Problem bootstrapping. Stuck at 14% (handshake): Handshaking with a 
> r

[tor-dev] Fact-checking a claim about relay/bridge fingerprint authentication

[David Fifield]

Cecylia, Arlo, Serene, Shelikhoo, and I are writing a research paper
about Snowflake. Here is a draft:
https://www.bamsoftware.com/papers/snowflake/snowflake.20231003.e6e1c30d.pdf

We're writing to check a factual claim in the section about having
multiple backend bridges. Basically, we wanted it to be possible for
there to be multiple Snowflake bridge sites run by different groups of
people, and we did not want to share the same relay identity keys across
all bridge sites, because of the increased risk of the keys being
exposed. Therefore every bridge site has its own relay identity, which
requires the client to know the relay fingerprints in advance and that
it be the client (and not, e.g., the broker) that decides which bridge
to use.

1. Is our general description (quoted below) of the design constraints
   as they bear on Tor correct?
2. Is §4.2 "CERTS cells" the right part of tor-spec to cite to make our
   point?
   
https://gitlab.torproject.org/tpo/core/torspec/-/blob/b345ca044131b2eb18e6ae0d5f23643a92aeff34/tor-spec.txt#L707

https://github.com/turfed/snowflake-paper/blob/e6e1c30dde6716dc5e54a32f2134f19068a7f395/snowflake.tex#L2146
A Tor bridge is identified by a long-term identity public key.
If, on connecting to a bridge, the client finds that the
bridge's identity is not the expected one, the client will
terminate the connection \cite[\S 4.2]{tor-spec}. The Tor client
can configure at most one identity per bridge; there is no way
to indicate (with a certificate, for example) that multiple
identities should be considered equivalent. This constraint
leaves two options: either all Snowflake bridges must share the
same cryptographic identity, or else it must be the client that
makes the choice of what bridge to use. While the former option
is possible to do (by synchronizing identity keys across
servers), every added bridge would increase the risk of
compromising the all-important identity keys. Our vision was
that different bridge sites would run in different locations
with their own management teams, and that any compromise of a
bridge site should affect that site only.

In my own experiments, providing an incorrect relay fingerprint leads to
errors in connection_or_client_learned_peer_id:
https://gitlab.torproject.org/tpo/core/tor/-/blob/tor-0.4.7.13/src/core/or/connection_or.c#L1897
[warn] Tried connecting to router at 192.0.2.3:80 ID= 
RSA_ID=2B280B23E1107BB62ABFC40DDCC8824814F80A71, but RSA + ed25519 identity 
keys were not as expected: wanted  + no 
ed25519 key but got 2B280B23E1107BB62ABFC40DDCC8824814F80A72 + 
1zOHpg+FxqQfi/6jDLtCpHHqBTH8gjYmCKXkus1D5Ko.
[warn] Problem bootstrapping. Stuck at 14% (handshake): Handshaking with a 
relay. (Unexpected identity in router certificate; IDENTITY; count 1; 
recommendation warn; host  at 
192.0.2.3:80)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] PT bridge reporting significant numbers of IP addresses after upgrade to 0.4.8.6; Conflux related?

[David Fifield]

The Snowflake bridges do not expose their plain ORPort (they have
`ORPort 127.0.0.1:auto` in torrc), and consequently they have always
reported ≈0  IP addresses in the bridge-ip-transports line of
bridge-extra-info descriptors, with virtually all connecting IP
addresses being instead attributed to the snowflake transport. (The 
count is low but nonzero, which I have rationalized as tor making some
small number of internal connections, or something.)

I upgraded one of the two Snowflake bridges from 0.4.7.13 to 0.4.8.6 on
2023-09-24. Since then, the number of  IP addresses has been roughly
equal to the number of snowflake IP addresses. The ORPort is still not
exposed; these are not external vanilla bridge users. Did something
change between these versions that might cause PT connections to be
double-counted, once for the transport and once for ?

Here are excerpted bridge-extra-info descriptors from before and after
the version upgrade. Note the bridge-ip-transports lines.

```
@type bridge-extra-info 1.3
extra-info crusty5 91DA221A149007D0FD9E5515F5786C3DD07E4BB0
master-key-ed25519 1y9CAtinlbrhDuYBSOBNiCU9Ck1lcY7LErxnzhtxVks
published 2023-09-19 10:46:08
transport snowflake
bridge-ip-versions v4=13528,v6=1384
bridge-ip-transports =8,snowflake=14912
```

```
@type bridge-extra-info 1.3
extra-info crusty5 91DA221A149007D0FD9E5515F5786C3DD07E4BB0
master-key-ed25519 1y9CAtinlbrhDuYBSOBNiCU9Ck1lcY7LErxnzhtxVks
published 2023-09-29 17:33:20
transport snowflake
bridge-ip-versions v4=2880,v6=336
bridge-ip-transports =1632,snowflake=1592
```

Below is a time series for a whole month.
The thing to notice is not the overall decrease in users on 2023-09-20;
that's due to
https://forum.torproject.org/t/problems-with-snowflake-since-2023-09-20-broker-failure-unexpected-error-no-answer/9346.
Rather, the thing to notice is that before 2023-09-24,  ≈ 0; and
after,  ≈ snowflake.

date snowflake
2023-08-30 4 45240  ← running 0.4.7.13 here
2023-08-3121193412
2023-09-0130195647
2023-09-0240200405
2023-09-0340201730
2023-09-0442195097
2023-09-0535190842
2023-09-0630189694
2023-09-0726191524
2023-09-0835195833
2023-09-0932196244
2023-09-1027146972
2023-09-11 6 55491
2023-09-1218182519
2023-09-1317173536
2023-09-1421161396
2023-09-1521166379
2023-09-1616174350
2023-09-1717165063
2023-09-1823164734
2023-09-1932164075
2023-09-2023121623
2023-09-21 2 25268
2023-09-22 0  4314
2023-09-23 0   746
2023-09-24   239   606  ← upgrade to 0.4.8.6 on this day
2023-09-25  1919  1900
2023-09-26  3533  3470
2023-09-27 11919 11687
2023-09-28 20242 19890
2023-09-29 26583 26100
2023-09-30 25929 25400
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Two features that would help load-balanced bridges

[David Fifield]

Linus Nordberg and I will have a paper at FOCI this summer on the
special way we run tor on the Snowflake bridges to permit better
scaling. It discusses the two workarounds from the post below, namely a
shim for predictable ExtORPort auth, and disabling onion key rotation.
This setup has been in place on Snowflake bridges since January 2022.
About 2.5% of Tor users (all users, not just bridge users) access Tor
using Snowflake, so it's not a niche use case even if it's just us.

One of the reviewers asked if there was a chance changes might be made
in tor that make our workarounds unnecessary. Is there anything to say
to this question? Might tor get a feature to control ExtORPort
authentication or onion key rotation, or is something that's planned to
stay as it is in favor of Arti? (Arti will probably remove the need for
the load-balanced multi-tor configuration, which will also remove the
need to disable onion key rotation, though better control over ExtORPort
auth could still be useful for running server PTs that are not child
processes of arti.)

Here is a draft of the paper, the relevant Sections are 3.1 and 3.2.
https://www.bamsoftware.com/papers/pt-bridge-hiperf/pt-bridge-hiperf.20230307.pdf
https://www.bamsoftware.com/papers/pt-bridge-hiperf/pt-bridge-hiperf.20230307.tex


On Mon, Feb 07, 2022 at 07:26:37PM -0700, David Fifield wrote:
> After the blocking of Tor in Russia in December 2022, the number of
> Snowflake users rapidly increased. Eventually the tor process became the
> limiting factor for performance, using all of one CPU core.
> 
> In a thread on tor-relays, we worked out a design where we run multiple
> instances of tor on the same host, all with the same identity keys, in
> order to effectively use all the server's CPU resources. It's running on
> the live bridge now, and as a result the bridge's bandwidth use has
> roughly doubled.
> 
> Design thread
>   
> https://forum.torproject.net/t/tor-relays-how-to-reduce-tor-cpu-load-on-a-single-bridge/1483
> Installation instructions
>   
> https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-Installation-Guide?version_id=6de6facbb0fd047de978a561213c59224511445f
> 
> Two details came up that are awkward to deal with. We have workaround
> for them, but they could benefit from support from core tor. They are:
> 
> 1. Provide a way to disable onion key rotation, or configure a custom
>onion key.
> 2. Provide a way to set a specific authentication cookie for ExtORPort
>SAFE_COOKIE authentication, or a new authentication type that doesn't
>require credentials that change whenever tor is restarted.
> 
> I should mention that, apart from the load-balancing design we settled
> on, we have brainstormed some other options for scaling the Snowflake
> bridge or bridges. At this point, none of these ideas can immediately be
> put into practice, because there's no way to tell tor "connect to one of
> these bridges at random, but only one," or "connect to this bridge, but
> accept any of these fingerprints."
> https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/28651
> 
> 
> # Disable onion key rotation
> 
> Multiple tor instances with the same identity keys will work fine for
> the first 5 weeks (onion-key-rotation-days + onion-key-grace-period-days),
> but after that time the instances will have independently rotated their
> onion keys, and clients will have connection failures unless the load
> balancer happens to connect them to the instance whose descriptor they
> have cached. This post investigates what the failure looks like:
> https://lists.torproject.org/pipermail/tor-relays/2022-January/020238.html
> 
> Examples of what could work here are a torrc option to set
> onion-key-rotation-days to a large value, an option to disable onion key
> rotation, an option to set a certain named file as the onion key.
> 
> What we are doing now is a bit of a nasty hack: we create a directory
> named secret_onion_key.old, so that a failed replace_file causes an
> early exit from rotate_onion_key.
> https://gitweb.torproject.org/tor.git/tree/src/feature/relay/router.c?h=tor-0.4.6.9#n494
> There are a few apparently benign side effects, like tor trying to
> rebuild its descriptor every hour, but it's effective at stopping onion
> key rotation.
> https://lists.torproject.org/pipermail/tor-relays/2022-January/020277.html
> 
> 
> # Stable ExtORPort authentication
> 
> ExtORPort (extended ORPort) is a protocol that lets a pluggable
> transport attach transport and client IP metadata to a connection, for
> metrics purposes. In order to connect to the ExtORPort, the pluggable
> transport needs to authenticate using a scheme like ControlPort
> authentication.
> https://

[tor-dev] goptlib moved to gitlab.torproject.org

[David Fifield]

I have moved the goptlib repository from git.torproject.org to
gitlab.torproject.org.

If you want to use it at the new location, change this import:
import "git.torproject.org/pluggable-transports/goptlib.git"
to this:
import 
"gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/goptlib"
and then run `go get` and `go mod tidy`.

Here is the issue about moving anti-censorship team projects from
git.torproject.org to gitlab.torproject.org generally:
https://bugs.torproject.org/tpo/anti-censorship/team/86#note_2823122

The old git.torproject.org URL should continue to work for the time
being.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Cross-user TLS traffic mixing in snowflake-server until 2023-03-13

[David Fifield]

Between 2022-10-01 and 2023-03-13, there was a bug in the software
deployed at Snowflake bridges that could cause parts of a user's stream
to be overwritten by parts of other users' streams. Though the Snowflake
team believes the privacy risks of the bug are minor, we are treating it
as a security bug and therefore making this announcement. The fix is
already deployed on the server, and no action is required by users.

## The cause

The source of the bug was issue 
https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40199
and specifically commit 
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/commit/839d2218837dfbd1682ff39b375f45660b3974b5,
which was intended as a performance optimization to reduce memory
allocation during the large influx of Snowflake users in late
September 2022.

The analysis surrounding the performance improvement incorrectly
concluded that certain memory buffers were not reused and therefore did
not need to be copied. But in fact, the buffers were reused, with the
effect that data meant for one user could be unpredictably overwritten
by data for another user.

## The effect

There is analysis at issue 
https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40260.
If our understanding of the bug is correct, it did not reveal plaintext
data or IP addresses to other users. The Snowflake bridge is a Tor
bridge, not an exit relay, so even a catastrophic failure could not leak
users' plaintext data or what destinations they were accessing.
A Snowflake connection consists of a layered set of protocols, and the
bug was in a layer that does not have access to IP addresses or any
other persistent identifiers.

* IP
* TCP
* HTTPS (WebSocket)
* session/retransmission ← The bug was here
* Tor TLS
* user data

At most, users and proxies would see fragments of other users' Tor TLS
streams. We believe the degree of information disclosure is less than
what an eavesdropper listening at the Snowflake bridge's HTTPS port
would see, which we already believe to be safe. The practical effect,
more than a loss of privacy, was excessive retransmissions at the
session/retransmission layer, and corrupted TLS data resulting in a
terminated connection.

## Remediation

We reverted the erroneous commit and added a test to check for the
erroneous buffer reuse. We deployed the update to both Snowflake bridges
on 2023-03-13.

https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/140
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/commit/b63d2272bfd262f5166c44f8f88330ed7a732009
https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40262

## Required action

No action is required by users. The bug was in code that was only used
by the Snowflake server, not proxies or clients.

## Thanks

The problem was discovered and first investigated by Haz Æ 41
(https://gitlab.torproject.org/hazae41) and reported on HackerOne:
https://hackerone.com/reports/1880610.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Is Arti expected to have better multi-CPU support than C-tor?

[David Fifield]

On Wed, Mar 08, 2023 at 06:30:42AM -0500, Nick Mathewson wrote:
> On Tue, Mar 7, 2023 at 4:07 PM David Fifield <[1]da...@bamsoftware.com> wrote:
> 
> Linus Nordberg and I are preparing a submission for FOCI about the
> special way we run tor on the Snowflake bridge. We run many tor
> processes with the same identity and onion keys, because otherwise tor
> being limited to one CPU would be the main bottleneck.
> 
> I'm writing to fact-check a claim about Arti and how we hope the current
> complicated procedure will not be needed in the future:
> 
>         The first and most important bottleneck to overcome is the
>         single-threaded nature of the Tor implementation.² A single Tor
>         process is limited to one CPU core: once Tor hits 100% CPU, the
>         performance of the bridge is capped, no matter the speed of the
>         network connection or the number of CPU cores
> 
>         ²We expect that Arti, the in-progress reimplementation of Tor,
>         will be natively multi-threaded, and remove this primary
>         complication.
> 
> Is this correct? Is a relay that uses a future version of Arti expected
> to be able to use all its CPU resources?
> 
> 
> 
> Yes, that's right.  There is no "main thread" in Arti; it's written in an
> asynchronous task-oriented style, and we use a runtime written in Rust (Tokio
> by default, but we abstract them so you can swap them out) to schedule tasks
> across multiple threads.
> 
> That said, we have spent approximately zero time so far tuning this
> multithreading, and I'd be surprised if it scales perfectly the first time. 
> Our first opportunity to show off here will be when we get onion service
> support later in this year.


Thank you, Nick.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Is Arti expected to have better multi-CPU support than C-tor?

[David Fifield]

Linus Nordberg and I are preparing a submission for FOCI about the
special way we run tor on the Snowflake bridge. We run many tor
processes with the same identity and onion keys, because otherwise tor
being limited to one CPU would be the main bottleneck.

I'm writing to fact-check a claim about Arti and how we hope the current
complicated procedure will not be needed in the future:

The first and most important bottleneck to overcome is the
single-threaded nature of the Tor implementation.² A single Tor
process is limited to one CPU core: once Tor hits 100% CPU, the
performance of the bridge is capped, no matter the speed of the
network connection or the number of CPU cores

²We expect that Arti, the in-progress reimplementation of Tor,
will be natively multi-threaded, and remove this primary
complication.

Is this correct? Is a relay that uses a future version of Arti expected
to be able to use all its CPU resources?

Here is the a draft of the submission. If you have any comments, our
submission deadline is 2023-03-15.

https://www.bamsoftware.com/papers/pt-bridge-hiperf/pt-bridge-hiperf.20230307.pdf
https://www.bamsoftware.com/papers/pt-bridge-hiperf/pt-bridge-hiperf.20230307.tex
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Metrics: Estimating fraction of reported directory-request statistics

[David Fifield]

On Thu, Apr 21, 2022 at 05:47:12PM +0200, Silvia/Hiro wrote:
> On 17/4/22 2:16, David Fifield wrote:
> > I am trying to reproduce the "frac" computation from the Reproducible
> > Metrics instructions:
> > https://metrics.torproject.org/reproducible-metrics.html#relay-users
> > Which is also Section 3 in the tech report on counting bridge users:
> > https://research.torproject.org/techreports/counting-daily-bridge-users-2012-10-24.pdf#page=4
> > 
> > h(R^H) * n(H) + h(H) * n(R\H)
> > frac = -
> >  h(H) * n(N)
> > 
> > My minor goal is to reproduce the "frac" column from the Metrics web
> > site (which I assume is the same as the frac above, expressed as a
> > percentage):
> > 
> > https://metrics.torproject.org/userstats-relay-country.csv?start=2022-04-01=2022-04-08=all=off
> > date,country,users,lower,upper,frac
> > 2022-04-01,,2262557,,,92
> > 2022-04-02,,2181639,,,92
> > 2022-04-03,,2179544,,,93
> > 2022-04-04,,2350360,,,93
> > 2022-04-05,,2388772,,,93
> > 2022-04-06,,2356170,,,93
> > 2022-04-07,,2323184,,,93
> > 2022-04-08,,2310170,,,91
> > 
> > I'm having trouble with the computation of n(R\H) and h(R∧H). I
> > understand that R is the subset of relays that report directory request
> > counts (i.e. that have dirreq-stats-end in their extra-info descriptors)
> > and H is the subset of relays that report directory request byte counts
> > (i.e. that have dirreq-write-history in their extra-info descriptors).
> > R and H partially overlap: there are relays that are in R but not H,
> > others that are in H but not R, and others that are in both.
> > 
> > The computations depend on some values that are directly from
> > descriptors:
> > n(R) = sum of hours, for relays with directory request counts
> > n(H) = sum of hours, for relays with directory write histories
> > h(H) = sum of written bytes, for relays with directory write histories
> > 
> > > Compute n(R\H) as the number of hours for which responses have been
> > > reported but no written directory bytes. This fraction is determined
> > > by summing up all interval lengths and then subtracting the written
> > > directory bytes interval length from the directory response interval
> > > length. Negative results are discarded.
> > I interpret this to mean: add up all the dirrect-stats-end intervals
> > (this is n(R)), add up all the dirreq-write-history intervals
> > (this is n(H)), and compute n(R\H) as n(R) − n(H). This seems wrong: it
> > would only be true when H is a subset of R.
> > 
> > > Compute h(R∧H) as the number of written directory bytes for the
> > > fraction of time when a server was reporting both written directory
> > > bytes and directory responses. As above, this fraction is determined
> > > by first summing up all interval lengths and then computing the
> > > minimum of both sums divided by the sum of reported written directory
> > > bytes.
> > This seems to be saying to compute h(R∧H) (a count of bytes) as
> > min(n(R), n(H)) / h(H). This is dimensionally wrong: the units are
> > hours / bytes. What would be more natural to me is
> > min(n(R), n(H)) / max(n(R), n(H)) × h(H); i.e., divide the smaller of
> > n(R) and n(R) by the larger, then multiply this ratio by the observable
> > byte count. But this, too, only works when H is a subset of R.
> > 
> > Where is this computation done in the metrics code? I would like to
> > refer to it, but I could not find it.
> > 
> > Using the formulas and assumptions above, here's my attempt at computing
> > recent "frac" values:
> > 
> > date   `n(N)`  `n(H)`   `h(H)`  `n(R)` `n(R\H)` `h(R∧H)` frac
> > 2022-04-01 166584 177638.  2.24e13 125491.   0   1.59e13 0.753
> > 2022-04-02 166951 177466.  2.18e13 125686.   0   1.54e13 0.753
> > 2022-04-03 167100 177718.  2.27e13 127008.   0   1.62e13 0.760
> > 2022-04-04 166970 177559.  2.43e13 126412.   0   1.73e13 0.757
> > 2022-04-05 166729 177585.  2.44e13 125389.   0   1.72e13 0.752
> > 2022-04-06 166832 177470.  2.39e13 127077.   0   1.71e13 0.762
> > 2022-04-07 166532 177210.  2.48e13 127815.   0   1.79e13 0.768
> > 2022-04-08 167695 176879.  2.52e13 127697.   0   1.82e13 0.761
> > 
> > The "frac" column does not match the CSV. Also notice that n(N) < n(H),
> > which should be impossible because H is supposed to be a subset of N
> > (N is the set of all relays). But this is what I get when I estimate
> >

Re: [tor-dev] Metrics: Estimating fraction of reported directory-request statistics

[David Fifield]

On Mon, Apr 18, 2022 at 03:45:29PM -0600, David Fifield wrote:
> I was initially interested in this for the purpose of better estimating
> the number of Snowflake users. But now I've decided "frac" is not useful
> for that purpose: since there is only one bridge we care about, it does
> not make sense to adjust the numbers to account for other bridges that
> may not report the same set of statistics. I don't plan to take this
> investigation any further for the time being, but here is source code to
> reproduce the above tables. You will need:
> https://collector.torproject.org/archive/relay-descriptors/consensuses/consensuses-2022-04.tar.xz
> https://collector.torproject.org/archive/relay-descriptors/extra-infos/extra-infos-2022-04.tar.xz
> 
> ./relay_uptime.py consensuses-2022-04.tar.xz > relay_uptime.csv
> ./relay_dir.py extra-infos-2022-04.tar.xz > relay_dir.csv
> ./frac.py relay_uptime.csv relay_dir.csv

Missed one of the source files.
import datetime

NUM_PROCESSES = 4

# "If the contained statistics end time is more than 1 week older than the
# descriptor publication time in the "published" line, skip this line..."
END_THRESHOLD = datetime.timedelta(days = 7)

# "Also skip statistics with an interval length other than 1 day."
# We set the threshold higher, because some descriptors have an interval a few
# seconds larger than 86400.
INTERVAL_THRESHOLD = datetime.timedelta(seconds = 9)

def datetime_floor(d):
return d.replace(hour = 0, minute = 0, second = 0, microsecond = 0)

TIMEDELTA_1DAY = datetime.timedelta(seconds = 86400)
def segment_datetime_interval(begin, end):
cur = begin
while cur < end:
next = min(datetime_floor(cur + TIMEDELTA_1DAY), end)
delta = next - cur
yield (cur.date(), delta / (end - begin), delta / TIMEDELTA_1DAY)
cur = next
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Metrics: Estimating fraction of reported directory-request statistics

[David Fifield]

On Sat, Apr 16, 2022 at 06:16:23PM -0600, David Fifield wrote:
> I am trying to reproduce the "frac" computation from the Reproducible
> Metrics instructions:
> https://metrics.torproject.org/reproducible-metrics.html#relay-users
> Which is also Section 3 in the tech report on counting bridge users:
> https://research.torproject.org/techreports/counting-daily-bridge-users-2012-10-24.pdf#page=4
> 
>h(R^H) * n(H) + h(H) * n(R\H)
> frac = -
> h(H) * n(N)
> 
> My minor goal is to reproduce the "frac" column from the Metrics web
> site (which I assume is the same as the frac above, expressed as a
> percentage):
> 
> https://metrics.torproject.org/userstats-relay-country.csv?start=2022-04-01=2022-04-08=all=off
> date,country,users,lower,upper,frac
> 2022-04-01,,2262557,,,92
> 2022-04-02,,2181639,,,92
> 2022-04-03,,2179544,,,93
> 2022-04-04,,2350360,,,93
> 2022-04-05,,2388772,,,93
> 2022-04-06,,2356170,,,93
> 2022-04-07,,2323184,,,93
> 2022-04-08,,2310170,,,91
> 
> I'm having trouble with the computation of n(R\H) and h(R∧H). I
> understand that R is the subset of relays that report directory request
> counts (i.e. that have dirreq-stats-end in their extra-info descriptors)
> and H is the subset of relays that report directory request byte counts
> (i.e. that have dirreq-write-history in their extra-info descriptors).
> R and H partially overlap: there are relays that are in R but not H,
> others that are in H but not R, and others that are in both.
>
> The computations depend on some values that are directly from
> descriptors:
> n(R) = sum of hours, for relays with directory request counts
> n(H) = sum of hours, for relays with directory write histories
> h(H) = sum of written bytes, for relays with directory write histories
>
> ...
> 
> Using the formulas and assumptions above, here's my attempt at computing
> recent "frac" values:
> 
> date   `n(N)`  `n(H)`   `h(H)`  `n(R)` `n(R\H)` `h(R∧H)` frac
> 2022-04-01 166584 177638.  2.24e13 125491.   0   1.59e13 0.753
> 2022-04-02 166951 177466.  2.18e13 125686.   0   1.54e13 0.753
> 2022-04-03 167100 177718.  2.27e13 127008.   0   1.62e13 0.760
> 2022-04-04 166970 177559.  2.43e13 126412.   0   1.73e13 0.757
> 2022-04-05 166729 177585.  2.44e13 125389.   0   1.72e13 0.752
> 2022-04-06 166832 177470.  2.39e13 127077.   0   1.71e13 0.762
> 2022-04-07 166532 177210.  2.48e13 127815.   0   1.79e13 0.768
> 2022-04-08 167695 176879.  2.52e13 127697.   0   1.82e13 0.761

I tried computing n(R\H) and h(R∧H) from the definitions, rather than by
using the formulas in the Reproducible Metrics guide. This achieves an
almost matching "frac" column, though it is still about 1% too high.

date   `n(N)`  `n(H)`   `h(H)`  `n(R)` `n(R\H)` `h(R∧H)` frac
2022-04-01 166584 177638.  2.24e13 125491. 90.9  1.96e13 0.930
2022-04-02 166951 177466.  2.18e13 125686.181.   1.92e13 0.937
2022-04-03 167100 177718.  2.27e13 127008.154.   2.00e13 0.942
2022-04-04 166970 177559.  2.43e13 126412.134.   2.14e13 0.936
2022-04-05 166729 177585.  2.44e13 125389. 94.6  2.15e13 0.938
2022-04-06 166832 177470.  2.39e13 127077.162.   2.11e13 0.940
2022-04-07 166532 177210.  2.48e13 127815.102.   2.18e13 0.938
2022-04-08 167695 176879.  2.52e13 127697.158.   2.21e13 0.926

I got this by taking an explicit set intersection between the R and H
time intervals. So, for example, if the intervals making up n(R) and
n(H) are (with their lengths):

n(R)[---10---]  [12]  [---9---]
n(H) [12][--16--]  [--7--]

Then the intersection n(R∧H) is:

n(R∧H)   [-5-]  [-5-][3]  [3]

h(R∧H) comes pro-rating the n(H) intervals, each of which is associated
with an h(H) byte count). Suppose the [12] interval represents
1000 bytes. Then each of the [-5-] intervals that result from it in the
intersection are worth 5/12 × 1000 = 417 bytes.

We get n(R\H) from n(R) − n(R∧H):

n(R\H)  [-5-][4-][-6--]

This seems overall more correct, though it required a more elaborate
computation than the Reproducible Metrics guide prescribes. I'm still
not sure why it does not match exactly, and I would still appreciate a
pointer to where Tor Metrics does the "frac" computation.

I was initially interested in this for the purpose of better estimating
the number of Snowflake users. But now I've decided "frac" is not useful
for that purpose: since there is only one bridge we care about, it does
not make sense to adjust the numbers to account for other bridges that
may not report the same set of statistics. I don't plan to take this
investigation any further for the time being, but here is source code to
reproduce

[tor-dev] Metrics: Estimating fraction of reported directory-request statistics

[David Fifield]

I am trying to reproduce the "frac" computation from the Reproducible
Metrics instructions:
https://metrics.torproject.org/reproducible-metrics.html#relay-users
Which is also Section 3 in the tech report on counting bridge users:
https://research.torproject.org/techreports/counting-daily-bridge-users-2012-10-24.pdf#page=4

   h(R^H) * n(H) + h(H) * n(R\H)
frac = -
h(H) * n(N)

My minor goal is to reproduce the "frac" column from the Metrics web
site (which I assume is the same as the frac above, expressed as a
percentage):

https://metrics.torproject.org/userstats-relay-country.csv?start=2022-04-01=2022-04-08=all=off
date,country,users,lower,upper,frac
2022-04-01,,2262557,,,92
2022-04-02,,2181639,,,92
2022-04-03,,2179544,,,93
2022-04-04,,2350360,,,93
2022-04-05,,2388772,,,93
2022-04-06,,2356170,,,93
2022-04-07,,2323184,,,93
2022-04-08,,2310170,,,91

I'm having trouble with the computation of n(R\H) and h(R∧H). I
understand that R is the subset of relays that report directory request
counts (i.e. that have dirreq-stats-end in their extra-info descriptors)
and H is the subset of relays that report directory request byte counts
(i.e. that have dirreq-write-history in their extra-info descriptors).
R and H partially overlap: there are relays that are in R but not H,
others that are in H but not R, and others that are in both.

The computations depend on some values that are directly from
descriptors:
n(R) = sum of hours, for relays with directory request counts
n(H) = sum of hours, for relays with directory write histories
h(H) = sum of written bytes, for relays with directory write histories

> Compute n(R\H) as the number of hours for which responses have been
> reported but no written directory bytes. This fraction is determined
> by summing up all interval lengths and then subtracting the written
> directory bytes interval length from the directory response interval
> length. Negative results are discarded.

I interpret this to mean: add up all the dirrect-stats-end intervals
(this is n(R)), add up all the dirreq-write-history intervals
(this is n(H)), and compute n(R\H) as n(R) − n(H). This seems wrong: it
would only be true when H is a subset of R.

> Compute h(R∧H) as the number of written directory bytes for the
> fraction of time when a server was reporting both written directory
> bytes and directory responses. As above, this fraction is determined
> by first summing up all interval lengths and then computing the
> minimum of both sums divided by the sum of reported written directory
> bytes.

This seems to be saying to compute h(R∧H) (a count of bytes) as
min(n(R), n(H)) / h(H). This is dimensionally wrong: the units are
hours / bytes. What would be more natural to me is
min(n(R), n(H)) / max(n(R), n(H)) × h(H); i.e., divide the smaller of
n(R) and n(R) by the larger, then multiply this ratio by the observable
byte count. But this, too, only works when H is a subset of R.

Where is this computation done in the metrics code? I would like to
refer to it, but I could not find it.

Using the formulas and assumptions above, here's my attempt at computing
recent "frac" values:

date   `n(N)`  `n(H)`   `h(H)`  `n(R)` `n(R\H)` `h(R∧H)` frac
2022-04-01 166584 177638.  2.24e13 125491.   0   1.59e13 0.753
2022-04-02 166951 177466.  2.18e13 125686.   0   1.54e13 0.753
2022-04-03 167100 177718.  2.27e13 127008.   0   1.62e13 0.760
2022-04-04 166970 177559.  2.43e13 126412.   0   1.73e13 0.757
2022-04-05 166729 177585.  2.44e13 125389.   0   1.72e13 0.752
2022-04-06 166832 177470.  2.39e13 127077.   0   1.71e13 0.762
2022-04-07 166532 177210.  2.48e13 127815.   0   1.79e13 0.768
2022-04-08 167695 176879.  2.52e13 127697.   0   1.82e13 0.761

The "frac" column does not match the CSV. Also notice that n(N) < n(H),
which should be impossible because H is supposed to be a subset of N
(N is the set of all relays). But this is what I get when I estimate
n(N) from a network-status-consensus-3 and n(H) from extra-info
documents. Also notice that n(R) < n(H), which means that H cannot be a
subset of R, contrary to the observations above.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Two features that would help load-balanced bridges

[David Fifield]

After the blocking of Tor in Russia in December 2022, the number of
Snowflake users rapidly increased. Eventually the tor process became the
limiting factor for performance, using all of one CPU core.

In a thread on tor-relays, we worked out a design where we run multiple
instances of tor on the same host, all with the same identity keys, in
order to effectively use all the server's CPU resources. It's running on
the live bridge now, and as a result the bridge's bandwidth use has
roughly doubled.

Design thread

https://forum.torproject.net/t/tor-relays-how-to-reduce-tor-cpu-load-on-a-single-bridge/1483
Installation instructions

https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-Installation-Guide?version_id=6de6facbb0fd047de978a561213c59224511445f

Two details came up that are awkward to deal with. We have workaround
for them, but they could benefit from support from core tor. They are:

1. Provide a way to disable onion key rotation, or configure a custom
   onion key.
2. Provide a way to set a specific authentication cookie for ExtORPort
   SAFE_COOKIE authentication, or a new authentication type that doesn't
   require credentials that change whenever tor is restarted.

I should mention that, apart from the load-balancing design we settled
on, we have brainstormed some other options for scaling the Snowflake
bridge or bridges. At this point, none of these ideas can immediately be
put into practice, because there's no way to tell tor "connect to one of
these bridges at random, but only one," or "connect to this bridge, but
accept any of these fingerprints."
https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/28651


# Disable onion key rotation

Multiple tor instances with the same identity keys will work fine for
the first 5 weeks (onion-key-rotation-days + onion-key-grace-period-days),
but after that time the instances will have independently rotated their
onion keys, and clients will have connection failures unless the load
balancer happens to connect them to the instance whose descriptor they
have cached. This post investigates what the failure looks like:
https://lists.torproject.org/pipermail/tor-relays/2022-January/020238.html

Examples of what could work here are a torrc option to set
onion-key-rotation-days to a large value, an option to disable onion key
rotation, an option to set a certain named file as the onion key.

What we are doing now is a bit of a nasty hack: we create a directory
named secret_onion_key.old, so that a failed replace_file causes an
early exit from rotate_onion_key.
https://gitweb.torproject.org/tor.git/tree/src/feature/relay/router.c?h=tor-0.4.6.9#n494
There are a few apparently benign side effects, like tor trying to
rebuild its descriptor every hour, but it's effective at stopping onion
key rotation.
https://lists.torproject.org/pipermail/tor-relays/2022-January/020277.html


# Stable ExtORPort authentication

ExtORPort (extended ORPort) is a protocol that lets a pluggable
transport attach transport and client IP metadata to a connection, for
metrics purposes. In order to connect to the ExtORPort, the pluggable
transport needs to authenticate using a scheme like ControlPort
authentication.
https://gitweb.torproject.org/torspec.git/tree/proposals/217-ext-orport-auth.txt?id=554d63ad3a60b705c3a5cbe2e3e9b33094a049dd#n75
tor generates a secret auth cookie and stores it in a file. When the
pluggable transport process is managed by tor, tor tells the pluggable
transport where to find the file by setting the TOR_PT_AUTH_COOKIE_FILE
environment variable.

In the load-balanced configuration, the pluggable transport server
(snowflake-server) is not run and managed by tor. It is an independent
daemon, so it doesn't have access to TOR_PT_AUTH_COOKIE_FILE (which
anyway would be a different path for every tor instance). The bigger
problem is that tor regenerates the auth cookie and rewrites the file on
every restart. All the tor instances have different cookies, and
snowflake-server does not know which it will get through the load
balancer, so it doesn't know what cookie to use.

Examples of what would work here are an option to use a certain file as
the auth cookie, an option to leave the auth cookie file alone if it
already exists, or a new ExtORPort authentication type that can use the
same credentials across multiple instances.

What we're doing now is using a shim program, extor-static-cookie, which
presents an ExtORPort interface with a static auth cookie for
snowflake-server to authenticate with, then re-authenticates to the
ExtORPort of its respective instance of tor, using that instance's auth
cookie.
https://lists.torproject.org/pipermail/tor-relays/2022-January/020183.html
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] GSoC 2021 - Alexa Top Sites Captcha and Tor Block Monitoring #Update

[David Fifield]

On Mon, Jul 12, 2021 at 05:01:35PM +0530, Apratim Ranjan Chakrabarty wrote:
> ** Looking forward for suggestions and comments as to how to improve on it.
> Also materials like research paper in this domain would be helpful **

Section IV-C of the ICLab paper has discussion of block page detection.
The first pass is regex for known block pages, but there is also
clustering by similar HTML structure and text.
https://censorbib.nymity.ch/#Niaki2020a
https://github.com/net4people/bbs/issues/52

The 2016 "Do You See What I See?" study seems to be in line with your
project. "The second-class treatment of anonymous users ranges from
outright rejection to ... imposing hurdles such as CAPTCHA-solving
Our study draws upon ... scans of the home pages of top-1,000 Alexa
websites through every Tor exit..." Section V-A has to do with scans of
top-ranked sites.
https://www.ndss-symposium.org/wp-content/uploads/2017/09/do-you-see-what-i-see-differential-treatment-anonymous-users.pdf
https://archive.org/details/ndss16doyousee
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Uptime stats for "Tor user can access an otherwise-functional hidden service"?

[David Fifield]

On Wed, May 05, 2021 at 03:27:23PM -0400, Holmes Wilson wrote:
> 3. Is there some incident log somewhere of problems that affected
> onion services network wide that includes how long these problems
> persisted for? (I don't see any onion service outage notes in this
> document, though I seem to remember there was an issue a few months
> back? https://metrics.torproject.org/news.html)

There's a small number of onion-related events at
https://gitlab.torproject.org/tpo/metrics/timeline
(search for "onion".)

Metrics news.html takes its input from the above timeline, but news.html
is currently out of sync, since before the January incident:
https://gitlab.torproject.org/tpo/metrics/timeline/-/issues/4
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Tails vs the capacity of the Meek bridges

[David Fifield]

On Fri, Mar 20, 2020 at 11:51:41AM +0100, anonym wrote:
> tl;dr: if Tails makes it too easy to use Meek bridges, could it overload the 
> current set of Meek bridges?

The default meek bridge is already overloaded, unfortunately. Users
complain that even though it works, it is too slow. Reports of 20 KB/s
are typical. See for example this recent comment from China:
https://bugs.torproject.org/33219#comment:9
> ...with the pre integrated meek bridges I just had 20 kb/s, at most
> 30, sometimes even lower than 20. So it took me over one hour to
> download the browser.

Here's the brandwidth chart for the default meek bridge btw. I would
guess that the bridge is capable of going faster, but it may have a
BandwidthRate set to keep costs from getting out of control.
https://metrics.torproject.org/rs.html#details/8F4541EEE3F2306B7B9FEF1795EC302F6B84DAE8

It's possible to set up a special bridge just for Tails users. It
requires setting up a bridge with meek-server (this is the cheap part)
and configuring a CDN to point to it (this is the expensive part). But
then you can let it run as fast as your budget allows.
https://trac.torproject.org/projects/tor/wiki/doc/meek#MicrosoftAzure
https://trac.torproject.org/projects/tor/wiki/doc/meek#Howtorunameek-serverbridge
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Shortcomings of the pluggable transports specification?

[David Fifield]

On Wed, Jun 12, 2019 at 04:41:34PM -0700, Philipp Winter wrote:
> We are working on improving Tor's pluggable transports specification:
> 
> 
> The goal is to make the spec useful to more people and fix issues that
> have accumulated over the years.  For more context, have a look at the
> following ticket, which we use to coordinate this effort:
> 
> 
> Before changing the spec, we need to understand its shortcomings and
> what issues implementers have run into.  For those of you who have
> experience with the spec -- either Tor's version 1.0 or version 2.1
> maintained by pluggabletransports.info -- please let us know:
> 
> * What version of the PT specification and what library implementation
>   (if any) are you using?
> 
> * What has your experience been with the PT specification?
> 
> * How would you improve the specification?

There are a couple of threads from 2017 relating to the development of
the 2.0 spec, which have some detailed comments.

Pluggable Transports 2.0 Specification, Draft 2
https://groups.google.com/d/topic/traffic-obf/sfDgcZk8s3s/discussion

Pluggable Transports 2.0 Specification, Draft 3
https://groups.google.com/d/topic/traffic-obf/bUo-OKnXSEI/discussion
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Release: obfs4proxy-0.0.10

[David Fifield]

On Sat, May 04, 2019 at 03:27:53PM +, Yawning Angel wrote:
> On 5/3/19 1:48 PM, Steve Snyder wrote:
> > FYI, obfs4proxy no longer recognizes address:port in this form:
> > 
> > ServerTransportListenAddr obfs4 [000.000.000.000]:443
> > 
> > Note the square brackets. Tor 0.3.5.8 / 0.4.0.5 still parses this
> > syntax, and obfs4proxy used to too. As of 0.0.10 it no longer does.
> 
> Odd.  None of that code, both in obfs4proxy and goptlib, has changed for
> years.  I'll look at it when I have a moment.

Might be this?

tor_addr_parse is overly permissive
https://bugs.torproject.org/23082
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] ICLab testing of default bridges

[David Fifield]

At the anti-censorship meeting today you mentioned talking to ICLab
about testing the default bridges. I believe that ICLab is already
testing at least a portion of the default bridges; they may have data
that you just have to ask for.

OONI and ICLab test the default bridges as a result of my, Lynn's, and
Qi's asking them to while we were studying default bridge blocking. The
data exist but we never carefully analyzed them. You can see what little
analysis we did at https://arxiv.org/abs/1709.08718 Figure 1 and
Section 5.9.

The data we used is in data/iclab/tor_jsons.tar.bz2 in
https://repo.eecs.berkeley.edu/git-anon/users/fifield/proxy-probe-paper.git.
It's a lot of JSON files that have TCP connect, TLS certificate fetch,
HTTP fetch, and TCP traceroute for each destination. We sent them the
list of bridges in September 2016 and they started measuring shortly
after that. The most recent update to the bridge list we sent them was
in August 2017 for https://bugs.torproject.org/23166.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] using obfs4 to tunnel to a SOCKS proxy server

[David Fifield]

On Fri, Jan 25, 2019 at 12:03:19AM +0100, Hans-Christoph Steiner wrote:
> Is this the same with other PT 1.1 daemons?  Or would Snowflake be
> different?  Seems like with obfs4, the load balancer using SNI would
> probably be the easiest for the wikipedia use case.

It will be the same with any other transport. They all have a uniform
interface through TOR_PT_ORPORT or TOR_PT_EXTENDED_SERVER_PORT.

Moat uses something basically similar with meek-server, feeding the
ORport into the local BridgeDB web server.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] RFC: Using `utls` in meek_lite.

[David Fifield]

On Thu, Jan 24, 2019 at 07:44:48AM +, Yawning Angel wrote:
> On 1/24/19 7:38 AM, David Fifield wrote:
> > I see, you're right. It has to do with the reuse of the initConn.
> 
> A proper "general" solution that solves that problem and the ALPN issue
> is to have a `initConn` and `http.RoundTripper` instance per destination
> host, and some additional locking.
> 
> With more implementation cleverness this could be brought down to two
> `http.RoundTripper` instances, and a host -> pointer + net.Conn map, and
> some locking.
> 
> But for something like meek_lite where the number of destination hosts
> is not large, the existing wrapper works fine and I don't see much
> reason to over engineer it.

I don't disagree, it's fine for this use case.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] RFC: Using `utls` in meek_lite.

[David Fifield]

On Thu, Jan 24, 2019 at 07:33:39AM +, Yawning Angel wrote:
> On 1/24/19 6:47 AM, David Fifield wrote:
> > // This also assumes that req.URL.Host will remain constant for the
> > // lifetime of the roundTripper, which is a valid assumption for 
> > meeklite.
> > 
> > Am I wrong, or is the actual restriction less strict? You can reuse the
> > roundTripper for different hosts--the ServerName is taken from the addr
> > argument to dialTLS--but only if those different hosts negotiate the
> > same ALPN, because the choice of http.Transport or http2.Transport is
> > made only once and persists for the lifetime of the roundTripper.
>
> The lock protecting `roundTripper.initConn` is only held in `dialTLS`,
> and the `roundTripper.transport` is not protected by a lock at all.
> 
> If the target host changes and there is simultaneous access (two threads
> call into `roundTripper.RoundTrip` right after initialization
> simultaneously), there is no guarantee that the connection used to
> create the inner `http.RoundTripper` instance will be passed to the
> correct thread.

I see, you're right. It has to do with the reuse of the initConn.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] RFC: Using `utls` in meek_lite.

[David Fifield]

On Mon, Jan 21, 2019 at 05:12:41AM +, Yawning Angel wrote:
> I just pushed a change to obfs4proxy master to use `utls` to mask the
> ClientHello signature (currently Chrome 70.x).
> 
> https://gitlab.com/yawning/obfs4/commit/4d453dab2120082b00bf6e63ab4aaeeda6b8d8a3

// This also assumes that req.URL.Host will remain constant for the
// lifetime of the roundTripper, which is a valid assumption for 
meeklite.

Am I wrong, or is the actual restriction less strict? You can reuse the
roundTripper for different hosts--the ServerName is taken from the addr
argument to dialTLS--but only if those different hosts negotiate the
same ALPN, because the choice of http.Transport or http2.Transport is
made only once and persists for the lifetime of the roundTripper.

For the same reason, I don't think we'll be able to use HelloRandomized,
only HelloRandomizedALPN or HelloRandomizedNoALPN. Otherwise we may
negotiate different ALPN even against the same server during the
lifetime of roundTripper. I tried adding a
conn.SetReadDeadline(time.Now().Add(10*time.Second))
inside dialTLS to force it to re-dial frequently, and with
HelloRandomized it does indeed eventually trip the "horrifically wrong"
branch with an error like:
net/http: HTTP/1.x transport connection broken: malformed HTTP response 
"\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x05\x00\x10\x00\x00\x00\x03\x00\x00\x00\xfa\x00\x06\x00\x10\x01@"
Despite the error, the client recovers quickly, redialing until it gets
a compatible ALPN.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] using obfs4 to tunnel to a SOCKS proxy server

[David Fifield]

On Wed, Jan 23, 2019 at 11:41:42AM +, Yawning Angel wrote:
> > For example, could the obfs4 server side provide a generic SOCKS proxy?
> 
> There is no functionality for doing such a thing in mainline obfs4proxy.
> 
> What currently will work is any one of:
> 
>  * Stick a proxy server of your choice behind the obfs4proxy server.
> From the application end it will essentially be connecting to a (for
> example) SOCKS5 proxy over another SOCKS5 proxy.
> 
>  * Connect the obfs4proxy server to a load-balancer or reverse-proxy
> that re-dispatches requests to the correct location based on the SNI
> block or `Host` header (depending on how you want to treat TLS).

This is the right answer. Fundamentally you need two layers of proxying:
one at the PT layer (obfs4proxy PT interface) and one at your
application layer (where you implement problem-specific logic like
domain whitelists).

On the server, you will point TOR_PT_ORPORT at a SOCKS server or load
balancer, rather than directly at the target web server. The
obfs4_server.sh script will work fine for that; you could also try
https://github.com/twisteroidambassador/ptadapter. The SOCKS server will
have to support a destination whitelist--or you could just put it on a
host with an appropriate outgoing firewall. Instead of a SOCKS server,
you could use load balancer/reverse proxy like Yawning says. Here are a
few that have SNI proxying (I've personally only used sslh):
https://www.haproxy.com/blog/enhanced-ssl-load-balancing-with-server-name-indication-sni-tls-extension/
https://github.com/yrutschle/sslh
https://github.com/dlundquist/sniproxy

But you're going to encounter an undesirable feature of this setup:
there's a 1:1 relationship between application-layer connections and
obfuscation-layer tunnels. That is, if the app makes 2 HTTPS connections
to 2 different Wikimedia domains, there will be 2 obfs4 tunnels
happening. It will work, but it's more conspicuous and will notionally
make website fingerprinting easier. What you may want is a multiplexing
protocol that collapses multiple streams into one on the client side (to
feed into the obfs4 tunnel) and splits them back apart again on the
server side. (In the usual Tor setup, it's the Tor protocol that serves
this multiplexing function--you only have one long-lived connection to
your guard, not a separate connection for every application-layer
stream.) Unfortunately I don't know of any out-of-the-box that does
this. You might try https://github.com/xtaci/smux; also lately I've been
thinking a lot about applying https://github.com/lucas-clemente/quic-go
to this problem.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] RFC: Using `utls` in meek_lite.

[David Fifield]

On Mon, Jan 21, 2019 at 05:12:41AM +, Yawning Angel wrote:
> I just pushed a change to obfs4proxy master to use `utls` to mask the
> ClientHello signature (currently Chrome 70.x).
> 
> https://gitlab.com/yawning/obfs4/commit/4d453dab2120082b00bf6e63ab4aaeeda6b8d8a3
> 
> I understand that this is being worked on for the original meek (see:
> https://bugs.torproject.org/29077), but I felt inspired and it was
> relatively easy to get something working.

Thanks, you found a clever approach that I hadn't thought of. I tried
just setting DialTLS on the main http.Transport--that doesn't work
because net/http doesn't know that utls has negotiated HTTP/2, and
starts sending HTTP/1.1 on an HTTP/2 connection. Setting DialTLS on an
http2.Transport works, but only with HTTP/2 servers.

If I may interpret, your code builds an http.RoundTripper wrapper around
http.Transport and http2.Transport. When the caller makes its first
request, the wrapper initiates the utls connection, then inspects what
protocol was negotiated with ALPN, and creates its own internal
http.Transport or http2.Transport as appropriate. Then, it simply
forwards all requests to its internal transport--also setting DialTLS on
the internal transport so that future connections will also use utls,
but re-using the same transport instead of making a new one each time.

This looks better than what I was trying to do. I will probably start
working on doing the meek-client implementation in this style.

As for the TODO, my plan was was to expose a "utls" SOCKS arg to make it
configurable per bridge, and just reuse the utls Client Hello ID names:
utls=HelloChrome_Auto
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Dormant Mode and pluggable transports

[David Fifield]

On Fri, Dec 14, 2018 at 04:28:26AM +0100, Alexander Færøy wrote:
> On Thu, Dec 13, 2018 at 04:15:56PM -0700, David Fifield wrote:
> > If transports need to become dormant too, then there needs to be some
> > way for tor to tell them to be. Now that https://bugs.torproject.org/28179
> > (Handle output from PT processes with the event loop) is almost
> > finished, perhaps the stdin/stdout channel would work for it. A request
> > to become dormant really does apply to the entire PT process (not just a
> > single transport or connection), so it's a good match for a
> > process-global channel like stdin. The PT process could respond with a
> > "SIGNAL DORMANT" message on its stdout, which would inform tor that the
> > PT has understood the request and will try to become dormant.
> 
> I've just opened bug #28849 for us to try to figure out how this should
> work both for the new process module itself, but especially for the only
> consumer of the process module right now: the pluggable transports.
> 
> One part of this that especially affects PT's running on Windows is that
> we would like to disable the Process I/O timer on Windows (which
> currently ticks once a second) when we are in the dormant mode. This
> would probably mean that once the stdout or stderr pipe's buffer, in the
> PT process, is full writing to stdout/stderr will block which would
> effectively be the same result as described in ticket #26360 (which
> #28179 as a side-effect also happens to fix).
> 
> The easy way out here would of course be to "just" terminate the PT's
> when we enter the dormant mode and re-spawn them when we leave the
> dormant mode.  If we decide to extend the PT protocol to handle `SIGNAL
> DORMANT` would we also need to have a method to inform the PT that it
> can start interacting with the rest of the world again?

As I think about it, I'm thinking that terminating the subprocess is
better from a KISS perspective. It's forward-compatible too, in the
sense that you can decide to start sending a "SIGNAL DORMANT" stdin
message in the future. And yes, if there's a "SIGNAL DORMANT" there
should also be a "SIGNAL ACTIVE" or something.

> Would it be bad if `SIGNAL DORMANT` also means that the PT should not
> write to stdout/stderr until it's been informed that we are no longer
> dormant?  :-)

That could be tricky. It may not be worth it.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Dormant Mode and pluggable transports

[David Fifield]

On Thu, Dec 13, 2018 at 03:56:50PM -0500, Nick Mathewson wrote:
> == Compatibility issues
> 
> I see two issues here: one minor, and one major.
> 
> Minor issue: some applications periodically make requests to the tor
> network on their own -- for example, Tor Browser's update requests.
> These requests prevent Tor from becoming dormant.  If this is
> undesired, we can add some way around this.
> 
> Major issue: some applications expect that Tor will always bootstrap
> when it starts, and delay presenting their own UI until Tor is ready.
> But if Tor starts as dormant, then it will not bootstrap until it
> receives a request from the client or a "SIGNAL ACTIVE"  command from
> the controller. This could lead to breakage as the application waits
> for Tor to tell it that it's ready, and Tor waits for somebody to tell
> it that it's needed.
> 
> Are all application developers okay with the issues above, and okay
> with working around them?  If not, we may need changes in Tor before
> 0.4.0.x is released.  Let's talk!

I'm thinking about how Dormant Mode will interact with pluggable
transports. Our current transports will (I think) become dormant when
tor does, because they only send something when tor does. At most, they
may chop up and pad some of tor's packets (like obfs4 iat-mode does),
but they don't send traffic of their own while tor is quiet. There's
nothing requiring that though: a transport is in general free to send
dummy traffic whenever it wants, perhaps as a form of traffic flow
obfuscation.

If transports need to become dormant too, then there needs to be some
way for tor to tell them to be. Now that https://bugs.torproject.org/28179
(Handle output from PT processes with the event loop) is almost
finished, perhaps the stdin/stdout channel would work for it. A request
to become dormant really does apply to the entire PT process (not just a
single transport or connection), so it's a good match for a
process-global channel like stdin. The PT process could respond with a
"SIGNAL DORMANT" message on its stdout, which would inform tor that the
PT has understood the request and will try to become dormant.

Or simpler but more drastic, tor could terminate its PT subprocesses
when it goes dormant (cleanly, by closing their stdin).
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Failure to connect to a 0.3.4.9 bridge--downgrading to 0.2.9.17 fixes it

[David Fifield]

On Thu, Dec 06, 2018 at 10:19:09PM -0700, David Fifield wrote:
> Here, a user reported a failure to connect to their own bridge, stopping
> at 25% bootstrapped. teor and I went through the basic troubleshooting
> steps of checking the bridge syntax, system time, and firewall settings.
> With or without obfs4 didn't make a difference. The user eventually
> tried downgrading the bridge from 0.3.4.9 to 0.2.9.17--and that fixed
> it. Their client version was 0.3.4.9.
> https://bugs.torproject.org/28717#comment:18
> 
> They also tried with two bridges from BridgeDB. The one running 0.3.4.9
> didn't work, and the one running 0.2.9.17 did.
> 
> A possible lead are these client log messages:
> [info] handle_response_fetch_desc(): Received server info (body size 0) from 
> server 'X.X.X.X:'
> [info] handle_response_fetch_desc(): Received http status code 404 ("Servers 
> unavailable") from server 'X.X.X.X:' while fetching 
> "/tor/server/authority.z". I'll try again soon.

The problem started in 0.3.4.1-alpha, according to the user's tests of
multiple versions on the bridge.

https://bugs.torproject.org/28717#comment:20
working 0.2.9.17
working 0.3.2.10 (git-31cc63deb69db819)
working 0.3.3.10 (git-2e94df92caee0fca)
working 0.3.3.6 (git-7dd0813e783ae16e)
working 0.3.3.7 (git-035a35178c92da94)
working 0.3.3.9 (git-45028085ea188baf)
not working 0.3.4.1-alpha (git-deb8970a29ef7427)
not working 0.3.4.2-alpha (git-bc951e83aac770d1)
not working 0.3.4.6-rc (git-6045c26d8442913e)
not working 0.3.4.7-rc (git-8465a8d84647c349)
not working 0.3.4.8 (git-da95b91355248ad8)
not working 0.3.4.9 (git-4ac3ccf2863b86e7)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Failure to connect to a 0.3.4.9 bridge--downgrading to 0.2.9.17 fixes it

[David Fifield]

Here, a user reported a failure to connect to their own bridge, stopping
at 25% bootstrapped. teor and I went through the basic troubleshooting
steps of checking the bridge syntax, system time, and firewall settings.
With or without obfs4 didn't make a difference. The user eventually
tried downgrading the bridge from 0.3.4.9 to 0.2.9.17--and that fixed
it. Their client version was 0.3.4.9.
https://bugs.torproject.org/28717#comment:18

They also tried with two bridges from BridgeDB. The one running 0.3.4.9
didn't work, and the one running 0.2.9.17 did.

A possible lead are these client log messages:
[info] handle_response_fetch_desc(): Received server info (body size 0) from 
server 'X.X.X.X:'
[info] handle_response_fetch_desc(): Received http status code 404 ("Servers 
unavailable") from server 'X.X.X.X:' while fetching 
"/tor/server/authority.z". I'll try again soon.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] obfs4, meek, active probing and the timeline of pluggable transports

[David Fifield]

On Sat, Oct 27, 2018 at 05:20:06PM +0530, Piyush Kumar Sharma wrote:
> 3.) I searched a lot but could not find the timeline in which pluggable
> transports were built. As in what was developed and deployed first, obfs4 or
> meek?

For questions like this, see our metrics timeline page:
https://trac.torproject.org/projects/tor/wiki/doc/MetricsTimeline

The three transports meek, ScrambleSuit, and obfs4  were deployed in Tor
Browser roughly at the same time. ScrambleSuit was a predecessor of
obfs4 that also deals with active probing. meek and ScrambleSuit were
first in 4.0-alpha-1 (Aug 2014) and 4.0 (Oct 2014). obfs4 was first in
4.5-alpha1 (Nov 2014) and 4.5 (Apr 2015).
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Idea which may or may not of been discussed

[David Fifield]

On Sat, Oct 13, 2018 at 12:21:49PM -0400, Matt Traudt wrote:
> Why wouldn't it be just as easy for censors to identify the small set of
> registered domains that Tor relays use and block TLS connections that
> involve them?

And in general, IMO pluggable transports are the right layer to address
this, not the Tor TLS layer. The way Tor uses TLS is already way more
complicated than it needs to be, partly because of past attempts to
build obfuscation into the core protocol rather than handling it as a
separate layer.
https://trac.torproject.org/projects/tor/wiki/org/projects/Tor/TLSHistory

The certificate server name is a pretty easy distinguishing feature--but
it's not the only one. But there are other ways in which the Tor TLS
handshake stands out, even if you use real server names with legit
certs. It's not easy to hack OpenSSL into perfectly imitating e.g., a
Firefox TLS fingerprint. That's why meek uses an instance of Firefox to
do its TLS, and why https://github.com/refraction-networking/utls
exists.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Domain Fronting, Meek, Cloudflare, and Encrypted SNI...

[David Fifield]

On Thu, Oct 04, 2018 at 09:37:18AM +0200, Andreas Krey wrote:
> A quick search indicates that aws and azure are already
> supporting it, although I'm unable to interpret whether that is
> actually the respective product you are/were using.

That's exactly it. Of course you can spin up a random EC2 machine or
Azure VM and connect to it with WebSocket. But a random AWS or Azure IP
address doesn't have the properties we want--it is easily blockable.
Why? Because there are no other valuable services running on the same IP
address, once it is discovered, the censor can block it with virtually
zero cost. (Not that it's an unworkable model--that's basically how
obfs4 works--it means you have to have to be careful about distributing
bridges, and keep replenishing the pool as they are inevitably blocked.)
Encrypted SNI does not help you here, not when there is only one service
running on the IP address.

We specifically use the CDN service, because the high degree of
co-hosting behind a CDN edge server makes it more expensive to block. In
this case, encrypted SNI (or formerly domain fronting) really does help,
because there are many and valuable sites hosted there, and connections
to different sites look the same from the censor's point of view. We're
not connecting via a cloud service just for fun, but because we depend
critically on that co-hosting to make blocking difficult. It's possible
that services other than the CDN services also have that property--it
would require some investigation to uncover--but then you would have to
ensure that the other service, too, supports WebSocket.

I'm not saying a WebSocket-in-TLS transport is a bad idea. It's just
that there's more to it than the protocol you use. You also have to go
through a proxy server that's expensive to block. As a proxy server,
most CDNs will support an HTTP/2 tunnel, but not WebSocket. And if you
don't have a CDN in the middle, you may as well use a plain CONNECT
proxy rather than WebSocket.

Related reading:
https://tools.ietf.org/html/draft-ietf-tls-sni-encryption-03#section-2.2
Many deployments still allocate different IP addresses to
different services, so that different services can be identified
by their IP addresses. However, content distribution networks
(CDN) commonly serve a large number of services through a small
number of addresses.
https://tools.ietf.org/html/draft-ietf-tls-esni-01#section-3.1
In principle, the provider might not be the origin for any
domains, but as a practical matter, it is probably the origin
for a large set of innocuous domains, but is also providing
protection for some private domains.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Domain Fronting, Meek, Cloudflare, and Encrypted SNI...

[David Fifield]

On Wed, Oct 03, 2018 at 07:01:21PM -0600, David Fifield wrote:
> And for that matter, why not a plain old HTTP CONNECT proxy? That would
> be even more efficient.

I should add that--leaving out domain fronting/encrypted SNI--there's an
implementation of exactly this, a pluggable transport built on an HTTP
proxy, by Sergey Frolov et al. He has been trying to get some attention
or buy-in to get it integrated into Tor Browser, but hasn't had much
luck so far. In my opinion, it will make a great alternative to obfs4
and be effective in many situations.

There's a bit more to it than I've described above. It can work with any
HTTP proxy (with HTTPS encryption to hide the destination from the
censor, of course)--but they've also implemented a proxy plugin for the
Caddy web server, which supports authentication. The authentication is
to resist active probing like the GFW does: a genuine client who got the
password through BridgeDB will be able to use the proxy, while a censor
probing IP address will just get the web server's normal pages. Check
the links for more info.

https://bugs.torproject.org/26923
https://github.com/sergeyfrolov/httpsproxy
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Domain Fronting, Meek, Cloudflare, and Encrypted SNI...

[David Fifield]

On Mon, Oct 01, 2018 at 07:55:31PM +0200, Andreas Krey wrote:
> On Mon, 24 Sep 2018 20:23:58 +0000, David Fifield wrote:
> ...
> > "encrypted SNI" part. But it's possible to do better: if you're willing
> > to abandon HTTP/1.1 compatibility and require HTTP/2, you can use the
> > "server push" feature to implement a serialization that's much more
> > efficient than the current one in meek.
> 
> How about websockets instead of trying to cram this into HTTP/2?

And for that matter, why not a plain old HTTP CONNECT proxy? That would
be even more efficient. But we're limited to what the CDN supports. Most
CDNs only support basic methods like GET and POST, not CONNECT or the
special headers needed by WebSocket.

Cloudflare does support WebSocket, though:
https://www.cloudflare.com/website-optimization/web-sockets/
https://blog.cloudflare.com/cloudflare-now-supports-websockets/
So this, combined with encrypted SNI, could be a viable technique when
tunneling through Cloudflare--it just wouldn't be portable to other
services. We even already have an existing WebSocket-based pluggable
transport implementation--it would need changes to the client to support
encrypted SNI.
https://gitweb.torproject.org/pluggable-transports/websocket.git/
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Information on the handling of relays churn

[David Fifield]

On Thu, Sep 27, 2018 at 08:21:06PM +0200, Adrien Luxey wrote:
>   • To which extent would you say that Tor is resilient to churn? What would 
> be
> the effects of a massive churn of relays? Where would be the bottleneck?

About churn specifically, the Sybil research of Winter, Ensafi, Loesing,
and Feamster has past measurements of churn rates, as well as source
code.

https://nymity.ch/sybilhunting/#Data
https://nymity.ch/sybilhunting/churn-values/
https://lists.torproject.org/pipermail/metrics-team/2016-February/64.html
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Domain Fronting, Meek, Cloudflare, and Encrypted SNI...

[David Fifield]

On Mon, Sep 24, 2018 at 01:46:10PM -0400, Nathaniel Suchy wrote:
> What this means:
> Effectively domain fronting works by sending a different SNI and host header.
> CDN providers like Cloudflare started double checking to make governments
> happy, scratch that line, I mean to protect their customers from fraud and
> abuse. They seem to of backtracked now. Encrypted SNI means that a firewall or
> coffee shop owner won’t be able to use SNI to see the real origin of TLS
> traffic.

What we would need in order for meek to used encrypted SNI would be
either:
 1) support for encrypted SNI in Go's crypto/tls package; or
 2) support for encrypted SNI in the Firefox that ships with Tor
Browser, which meek-client could use through its TLS camouflage
helper support.

IMO (2) is less desirable because I'd like to get rid of the TLS
camouflage helper support and replace it with a Go-level TLS camouflage
library: https://github.com/refraction-networking/utls. The TLS helper
works, but its complexity is a pain to deal with and leads to problems
like https://bugs.torproject.org/12774 https://bugs.torproject.org/25405.

Note, however, that a transport based on encrypted SNI doesn't have to
be meek per se. I use "meek" to refer to a specific combination of
domain fronting and a certain (fairly stupid and inefficient) HTTP/1.1
serialization of a bidirectional byte stream. It's conceptually
straightforward to simply replace the "domain fronting" part with an
"encrypted SNI" part. But it's possible to do better: if you're willing
to abandon HTTP/1.1 compatibility and require HTTP/2, you can use the
"server push" feature to implement a serialization that's much more
efficient than the current one in meek. It may even be better to start
over with a new codebase, it's not like meek's code is that large.

But tjr's point stands: I think it's best not to push anything like this
out until after encrypted SNI has seen some level of adoption by
browsers.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Ready to Integrate/Review New Marionette Version into Tor

[David Fifield]

On Tue, Jul 24, 2018 at 01:57:36PM -0400, John Helmsen wrote:
> Okay, I have generated a VM using VirtualBox of Ubuntu version 16.  I've had 
> to
> restart the build process a couple of times, since the hard drive was 10GB,
> then 20GB.  Now I am using a 50GB box, so it may work this time.
> 
> Should we create the branch using the -b command? ('git checkout -b
> tbb-8.0a9-build3'?) Otherwise, it complains about being headless.

It doesn't matter, just for the sake of doing a testbuild and priming
the cache of dependencies (you'll notice the git_clones directory get
filled in among others). But yes, you'll want a branch for integration,
something like
git checkout -b marionette-integration tbb-8.0a9-build3
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Ready to Integrate/Review New Marionette Version into Tor

[David Fifield]

On Tue, Jul 24, 2018 at 11:42:08AM -0400, John Helmsen wrote:
> Thank you, I have created the ticket as #26920. https://trac.torproject.org/
> projects/tor/ticket/26920#ticket.  Having downloaded the git project, it seems
> that this work cannot be performed on a Mac, since it doesn't run 'runc'.  Is
> that right?

Right, the README says "To build Tor Browser, you need a Linux
distribution that has support for runc (such as Debian jessie, Ubuntu
16.04, Fedora 20, etc ...)."

The tag I suggested, tbb-8.0a9-build3, is the tag of the most recent
alpha release (I think). I haven't tried building it myself--if it
doesn't work, you may just need to try a different tag. One of the Tor
Browser devs could suggest an alternative if it doesn't work on the
first try.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Ready to Integrate/Review New Marionette Version into Tor

[David Fifield]

On Fri, Jul 20, 2018 at 04:12:21PM -0400, John Helmsen wrote:
> We are in the process of writing the documentation for Marionette, but the
> documentation on the web page should be sufficient for at least getting a full
> evaluation started.  We'd like to have the evaluation complete by the end of
> next month, hopefully the middle of next month, and stand ready to make any 
> and
> all changes necessary.
> 
> A full set of documentation will also be written for designing your own
> protocols.  This is in process.
> 
> Please let us know what you need.

The Tor Browser developers may have more specific requests, but I can
suggest some steps to get started.

Open a ticket at https://trac.torproject.org/ for discussion and to
track progress.
Type: project
Component: Applications/Tor Browser
Keywords: marionette
The old ticket for FTE is a good reference: https://bugs.torproject.org/10362

And then it would help if you port your build process to the Tor Browser
build system. General information:
https://trac.torproject.org/projects/tor/wiki/doc/TorBrowser/Hacking
First, just build
git clone https://git.torproject.org/builders/tor-browser-build.git
cd tor-browser-build
git checkout tbb-8.0a9-build3
make testbuild # or, e.g., testbuild-linux-x86_64
Then you'll have to add a new project (consisting of a "build" and
"config" file) for Marionette and each of its dependencies. You can copy
from existing projects as templates. Here is the meek project, for
example:
https://gitweb.torproject.org/builders/tor-browser-build.git/tree/projects/meek
You'll also need to add bridge lines to:
https://gitweb.torproject.org/builders/tor-browser-build.git/tree/projects/tor-browser/Bundle-Data/PTConfigs/bridge_prefs.js
To build just one project, not an entire release, do e.g.:
rbm/rbm build gmp --target testbuild --target torbrowser-linux-x86_64
rbm/rbm build marionette --target testbuild --target 
torbrowser-linux-x86_64
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] man: "IPv6 addresses should be wrapped in square brackets"

[David Fifield]

On Sun, Jul 01, 2018 at 10:03:50AM +1000, teor wrote:
> When an option only takes an IP address, it does not make a difference.
> (As long as the underlying code uses tor_addr_parse().)

BTW there's currently a bug relating to this: if an address starts with
'[', tor_addr_parse strips the final byte whether or not it is ']'.
https://bugs.torproject.org/23082
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] permission denied when running snowflake-client with debian-tor user

[David Fifield]

On Mon, Jun 11, 2018 at 07:30:31PM +, Yawning Angel wrote:
> On Mon, 11 Jun 2018 13:24:19 -0400
> Arlo Breault  wrote: 
> > When you launch the client binary without providing a broker url
> > it tries to create a named pipe (mkfifo) to do signalling.
> > 
> > https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/client/rendezvous.go#n161
> 
> The PT spec explicitly forbids this behavior, to avoid this problem.

It's just a vestige of some early debugging code, don't worry about it.
Before we had the broker and everything, you had to manually copy and
paste rendezvous messages.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Connections failed to default obfs4 bridges

[David Fifield]

On Wed, Mar 28, 2018 at 10:57:13AM -0400, Rob Jansen wrote:
> In a recent connectivity test to the default obfs4 bridges [0], we found that 
> we are unable to connect to 10 or so of them (from open networks, i.e., no 
> local filtering).
> 
> Is this a feature, like some of them only respond to users in certain parts 
> of the world? Or is this a bug, like the default list of bridges refers to 
> old bridges that are no longer available? Or am I misunderstanding 
> functionality here?

Do you mean 10 distinct IP addresses, or 10 ports on a few IP addresses?
Not all the IP addresses in the list are distinct.

Even while Lynn Tsai, Qi Zhong, and I were closely monitoring default
bridge reachability, a lot of the default bridges were often offline,
because of reboots, iptables problems, etc. See for example the "Orbot
bridges" strip of Figure 5.2 here; the gray and red areas that precede
blocking are where the bridge was simply offline:
https://www.bamsoftware.com/papers/thesis/fifield-thesis.pdf#page=43

We have a lot of past measurements of default bridges. The rows with
site="eecs-login" are from the U.S.
https://www.bamsoftware.com/proxy-probe/ (download the repo, not
probe.csv.gz, which isn't as recent)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Pluggable transports research

[David Fifield]

On Wed, Jan 24, 2018 at 04:42:52PM -0800, Jodi Spacek wrote:
> I'm a master's student at the University of British Columbia (Vancouver,
> Canada) where I'm primarily researching anonymous systems and censorship. I
> would be delighted to contribute to pluggable transports. 
> 
> Of particular interest is image and audio data stenography - is anything is in
> the works for this or is it outdated? My aim is to add this functionality 
> while
> fully testing and evaluating it as part of my thesis project. I refer to the
> list of idea suggestions here: 
> https://trac.torproject.org/projects/tor/wiki/doc/PluggableTransports/ideas 

Circumvention research can probably learn a lot from steganography
research. Most of the "mainstream" research on circumvention (read: the
work I'm familiar with :D) is in CensorBib:
https://censorbib.nymity.ch/
However I've been meaning to see what else we can learn by bringing
related research into its scope. There's a thread of research by
Sebastian Zander et al. on covert channels that hardly intersects with
circumvention research; it would be a good contribution if you could
determine to what extent the two worlds can be joined. For example
"Reliable Transmission Over Covert Channels in First Person Shooter
Multiplayer Games" predates Rook and Castle. They developed an
evaluation framework that to my knowledge hasn't been applied to
circumvention protocols.
http://caia.swin.edu.au/cv/szander/cc/index.html
http://caia.swin.edu.au/cv/szander/cc/cchef/
"Provably Secure Steganography" by Hopper et al. could be relevant to
certain kinds of circumvention protocols.
https://www-users.cs.umn.edu/~hoppernj/tc-stego.pdf

The traffic-obf list is a group of circumvention researchers. They are
scheduling biweekly meetings on IRC. You could discuss some ideas there.
https://groups.google.com/d/msg/traffic-obf/VtsKZA2Akmk/-v3Ajct-AwAJ
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Dir auths using 2x bandwidth in last week

[David Fifield]

On Sun, Sep 17, 2017 at 07:32:13PM -0400, Roger Dingledine wrote:
> On Wed, Aug 09, 2017 at 11:36:27PM -0400, Roger Dingledine wrote:
> > https://atlas.torproject.org/#details/9695DFC35FFEB861329B9F1AB04C46397020CE31
> > https://atlas.torproject.org/#details/F2044413DAC2E02E3D6BCF4735A19BCA1DE97281
> > https://atlas.torproject.org/#details/BD6A829255CB08E66FBE7D3748363586E46B3810
> > https://atlas.torproject.org/#details/74A910646BCEEFBCD2E874FC1DC997430F968145
> > https://atlas.torproject.org/#details/7EA6EAD6FD83083C538F44038BBFA077587DD755
> > all show a big increase in sent bytes starting at the end of July.
> > 
> > It isn't growth in Tor users, since those have stayed relatively flat
> > in the last two weeks.
> > 
> > And the new rate seems to be the new normal -- it's showing no signs of
> > going back to the old rate.
> > 
> > I would assume it's outgoing directory stuff, since that's most of what
> > dir auths do.
> > 
> > Any guesses?
> 
> Well, whatever it was, it settled out -- back to normal -- once August
> ended.
> 
> https://atlas.torproject.org/#details/9695DFC35FFEB861329B9F1AB04C46397020CE31

The start and end of the doubling of dirauth bandwidth coincide with a
temporary doubling or tripling of reported users from the Netherlands:

https://people.torproject.org/~dcf/metrics-country.html?start=2017-07-01=2017-09-30=nl

There is a somewhat similar pattern in the Seychelles, Lithuania, and
Romania, all of which had unexplained increases at the same time as the
Netherlands.

https://lists.torproject.org/pipermail/metrics-team/2017-August/000428.html
https://people.torproject.org/~dcf/metrics-country.html?start=2017-07-01=2017-09-30=sc
https://people.torproject.org/~dcf/metrics-country.html?start=2017-07-01=2017-09-30=lt
https://people.torproject.org/~dcf/metrics-country.html?start=2017-07-01=2017-09-30=ro
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Dir auths using 2x bandwidth in last week

[David Fifield]

On Mon, Aug 21, 2017 at 01:56:16PM +1000, teor wrote:
> 
> > On 10 Aug 2017, at 13:36, Roger Dingledine  wrote:
> > 
> > https://atlas.torproject.org/#details/9695DFC35FFEB861329B9F1AB04C46397020CE31
> > https://atlas.torproject.org/#details/F2044413DAC2E02E3D6BCF4735A19BCA1DE97281
> > https://atlas.torproject.org/#details/BD6A829255CB08E66FBE7D3748363586E46B3810
> > https://atlas.torproject.org/#details/74A910646BCEEFBCD2E874FC1DC997430F968145
> > https://atlas.torproject.org/#details/7EA6EAD6FD83083C538F44038BBFA077587DD755
> > all show a big increase in sent bytes starting at the end of July.
> > 
> > It isn't growth in Tor users, since those have stayed relatively flat
> > in the last two weeks.
> > 
> > And the new rate seems to be the new normal -- it's showing no signs of
> > going back to the old rate.
> > 
> > I would assume it's outgoing directory stuff, since that's most of what
> > dir auths do.
> > 
> > Any guesses?
> 
> In July, Tor 0.3.0 became the most common relay version in the network,
> growing at quite a rapid rate:
> 
> https://metrics.torproject.org/versions.html
> 
> There doesn't seem to be any corresponding Tor Browser release in that
> timeframe:
> 
> 3 July: https://blog.torproject.org/blog/tor-browser-702-released
> 8 August: https://blog.torproject.org/blog/tor-browser-704-released

2 July is when deb.torproject.org switched to 0.3.0.

nusenu noted it here:
https://twitter.com/nusenu_/status/884128686764687361

I added a line already to
https://trac.torproject.org/projects/tor/wiki/doc/MetricsTimeline
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Tor and IP2Location LITE

[David Fifield]

On Sun, Aug 20, 2017 at 10:02:20PM +0200, Karsten Loesing wrote:
> Okay. Maybe we could do something with archive.org in that case. It's
> not that we do have a complete history for MaxMind's files, except that
> we could probably create our own history from Tor's Git repository which
> contains files based on MaxMind's files.

I have a script that walks through the history of tor's git geoip files.
#!/usr/bin/env python

import datetime
import getopt
import os.path
import socket
import subprocess
import sys

# Counts the size of per-country geoip allocations in the tor source code.
#
# Usage: ./scrape-geoip.py ~/src/tor > tor-geoip.csv
#
# ~/src/tor (or whatever the path is) must be a tor source repo; i.e. a clone of
# https://git.torproject.org/tor.git.

def usage(f=sys.stdout):
print >> f, """\
Usage: %s /path/to/tor
""" % sys.argv[0]

def history(dirname, filename):
proc = subprocess.Popen(["git", "log", "--reverse", "--date=short", "--pretty=%H %ad", filename],
cwd=dirname, stdout=subprocess.PIPE)
return proc.stdout

def git_show(dirname, filename, commithash):
proc = subprocess.Popen(["git", "show", commithash+":"+filename],
cwd=dirname, stdout=subprocess.PIPE)
return proc.stdout

def parse_geoip(f):
ccs = {}
for line in f:
if line.startswith("#"):
continue
parts = line.strip().split(",")
start = int(parts[0])
end = int(parts[1])
cc = parts[2].lower()
ccs.setdefault(cc, 0)
ccs[cc] += end - start + 1
return ccs

def ipv6_to_int(ipstr):
return long("0x" + socket.inet_pton(socket.AF_INET6, ipstr).encode("hex"), 16)

def parse_geoip6(f):
ccs = {}
for line in f:
if line.startswith("#"):
continue
parts = line.strip().split(",")
start = ipv6_to_int(parts[0])
end = ipv6_to_int(parts[1])
cc = parts[2].lower()
ccs.setdefault(cc, 0)
ccs[cc] += end - start + 1
return ccs


opts, args = getopt.gnu_getopt(sys.argv[1:], "h", ["help"])
for o, a in opts:
if o == "-h" or o == "--help":
usage()
sys.exit()

try:
TOR_PATH, = args
except ValueError:
usage(sys.stderr)
sys.exit(1)

print "date,ipv,country,count"

for line in history(TOR_PATH, "src/config/geoip"):
parts = line.strip().split()
commithash = parts[0]
date = datetime.datetime.strptime(parts[1], "%Y-%m-%d")

try:
ccs = parse_geoip(git_show(TOR_PATH, "src/config/geoip", commithash))
except Exception, e:
print >> sys.stderr, "Skipping %s %s: %s" % ("src/config/geoip", commithash, e)
continue
for cc, count in sorted(ccs.items()):
print ",".join([date.strftime("%Y-%m-%d"), "4", cc, str(count)])

for line in history(TOR_PATH, "src/config/geoip6"):
parts = line.strip().split()
commithash = parts[0]
date = datetime.datetime.strptime(parts[1], "%Y-%m-%d")

try:
ccs = parse_geoip6(git_show(TOR_PATH, "src/config/geoip6", commithash))
except Exception, e:
print >> sys.stderr, "Skipping %s %s: %s" % ("src/config/geoip6", commithash, e)
continue
for cc, count in sorted(ccs.items()):
print ",".join([date.strftime("%Y-%m-%d"), "6", cc, str(count)])
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Feature Request: please consider ship default Tor bridges

[David Fifield]

On Thu, Aug 17, 2017 at 05:19:44PM +, iry wrote:
> A set of Tor bridges are shipped with Tor browser bundle[0], helping
> users in Tor-censored area to connection to the Tor network. Since
> system Tor users may also face the censorship problem, shall we
> ship some Tor bridges along with the tor package?
> 
> The request is firstly reported[0] to Debian BTS and I got the
> following reply by Peter:
> 
> > If upstream starts shipping bridges with their Tor releases, that
> > would naturally result in the Tor package shipping bridges as
> > well.
> > 
> > I do not know whether that's a good idea or not, but I don't think 
> > deviating from upstream would be particularly worthwhile.

To get an idea of how frequently the list of default bridges has
changed, see the tbb-bridges keyword in the bug tracker:
https://trac.torproject.org/projects/tor/query?keywords=~tbb-bridges=time=id=summary=keywords=status=1=time

> The default bridge shipped with tor package should be exactly the same
> bridges contained in bridge_prefs.js[0] shipped with the latest stable
> TBB. This is because:
> 1. The servers hosting default bridges are set up for huge amount of
> traffic;
> 2. The servers hosting default bridges are probably audited by TPO for
> better security;
> 3. Using a different set of bridges will distinguish the
> anon-connection-wizard bridge users from the TBB bridge users, which
> compromises their anonymity.

There is an argument for using a different set of default bridges: when
one of the Tor Browser ones gets blocked, it won't affect the Debian
ones. For example, for a while, Orbot had some additional bridges that
Tor Browser did not have. When the firewall of China blocked the Tor
Browser bridges, the Orbot ones continued working for another nine
months (until they got blocked for a different reason). We know that at
least China and Kazakhstan pay attention to the default Tor Browser
bridges (and China blocks them as soon as they enter the source code,
even before a release).

So having a few bridges that are not shared with Tor Browser has that
advantage, at least. Of course, it's basically security by obscurity,
because a censor that can discover the Tor Browser bridges can (in
theory) also discover some other static list of bridges. But in practice
it will take censors time to build automation to read from a new list,
default bridges are security by obscurity anyway, though surprisingly
effective for that.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Pluggable Transports 2.0 Specification, Draft 2

[David Fifield]

On Wed, Jun 21, 2017 at 06:20:44AM +, Yawning Angel wrote:
> On Tue, 20 Jun 2017 21:27:35 -0700
> David Fifield <da...@bamsoftware.com> wrote:
> > Even closely affiliated projects like Orbot haven't been able to use
> > pluggable transports strictly according to the spec, because of the
> > various constraints on mobile platforms.
> 
> This is basically totally and utterly wrong.

I'm sorry. I guess this bit of folk knowledge I had picked up was wrong.
I thought it was somehow more complicated. I may have been confusing
Android and iOS issues.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Pluggable Transports 2.0 Specification, Draft 2

[David Fifield]

On Wed, Jun 21, 2017 at 01:16:20PM +1000, teor wrote:
> In general, is there a separate document or proposal that describes
> how Tor will implement the relevant interfaces? There doesn't seem
> to be much on Tor-specific issues in this spec.
> 
> There is one "Tor" note in the spec, maybe it should be in that
> separate document? Or maybe there should be more Tor notes in the
> spec?

As I understand it, one of the goals of the PT 2.0 spec is to make it
easier for projects *other* than Tor to use pluggable transports. The
current spec (the 1.0 spec) basically doesn't work for anyone other than
desktop Tor--this is something we've heard over and over from other
projects that use circumvention, and Brandon is well plugged in to those
conversations. Even closely affiliated projects like Orbot haven't been
able to use pluggable transports strictly according to the spec, because
of the various constraints on mobile platforms.

What various circumvention developers have said they prefer, is a
Go-based API. They would rather link against a library, than do the
whole subprocess-and-stdout thing that the PT 1.0 spec requires. That's
the reason for introducing the new Go stuff in the 2.0 spec. Inside and
outside Tor, a lot of projects have converged on implementing their
circumvention code in Go. The API of the 2.0 spec is based on the
internal architecture of obfs4proxy, which is de facto the main
implementation of most of Tor's pluggable transports.

If I understand correctly, Tor wouldn't have to implement glue code to
interface with the Go API. It could continue spawning subprocesses,
similar to what it does now. The executables it invokes, if they are
written in Go, will likely internally use a PT library that uses the 2.0
spec's API, but Tor wouldn't have to know those details.

PT 1.0 succeeded in being "pluggable" in one sense: it's easy to hotswap
a lot of circumvention technologies within Tor. But it failed in being
"pluggable" in another sense: it's not easy to share common transport
modules beyond Tor (in either direction). It would be great if the new
spec can realize that second sense of pluggability.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] The limits of timing obfuscation in obfs4

[David Fifield]

ich
proves that the client knows the bridge's out-of-band secret (this is
how obfs4 resists active probing). The server handshake reply is
similar, with the addition of an authentication tag:
k | a | p | m   (different values than in the client handshake)
This means that no matter how you schedule packet sending, the traffic
will always have this form, with a .. gap where the client is
waiting for the server's handshake to come back:
client  kpm..ddd
server  ...kapppmddd

There isn't a way to completely remove the client gap in obfs4 and still
follow the protocol. A future protocol could perhaps remove it (I say
perhaps because I haven't thought about the crypto implications) by
changing the client's handshake to have a second round of padding, which
it could send while waiting for the server to reply:
k | p | m1 | p | m2

[spec] 
https://gitweb.torproject.org/pluggable-transports/obfs4.git/tree/doc/obfs4-spec.txt?id=obfs4proxy-0.0.7
From 3699bbda1633b17eb5fae9ced6158df42fe1384b Mon Sep 17 00:00:00 2001
From: David Fifield <da...@bamsoftware.com>
Date: Sat, 10 Jun 2017 17:26:13 -0700
Subject: [PATCH] Queue writes through an independent write scheduler.

Send padding packets where there is no application data waiting.

The one place this doesn't work is in the client, after sending the
client handshake and before receiving the Y' and AUTH portions of the
server reply. During this time the client doesn't yet know the session
key and so cannot send anything. This version waits even longer, until
the entire server handshake has been received (including P_S, M_S, and
MAC_S).

This is set up to do sends in a fixed size of 500 bytes at a fixed rate
of 10 sends per second. As a hack, this rounds the client handshake size
to a multiple of 500 bytes, so that it doesn't stall waiting for the
final full chunk to be available to send.
---
 transports/obfs4/handshake_ntor.go |   8 ++-
 transports/obfs4/obfs4.go  | 122 -
 2 files changed, 114 insertions(+), 16 deletions(-)

diff --git a/transports/obfs4/handshake_ntor.go b/transports/obfs4/handshake_ntor.go
index ee1bca8..fb5935a 100644
--- a/transports/obfs4/handshake_ntor.go
+++ b/transports/obfs4/handshake_ntor.go
@@ -127,7 +127,13 @@ func newClientHandshake(nodeID *ntor.NodeID, serverIdentity *ntor.PublicKey, ses
 	hs.keypair = sessionKey
 	hs.nodeID = nodeID
 	hs.serverIdentity = serverIdentity
-	hs.padLen = csrand.IntRange(clientMinPadLength, clientMaxPadLength)
+	padLen := csrand.IntRange(clientMinPadLength, clientMaxPadLength)
+	// Hack: round total handshake size to a multiple of 500 for CBR mode.
+	padLen = padLen - (padLen + clientMinHandshakeLength) % 500
+	if padLen < 0 {
+		padLen += 500
+	}
+	hs.padLen = padLen
 	hs.mac = hmac.New(sha256.New, append(hs.serverIdentity.Bytes()[:], hs.nodeID.Bytes()[:]...))
 
 	return hs
diff --git a/transports/obfs4/obfs4.go b/transports/obfs4/obfs4.go
index 304097e..7210210 100644
--- a/transports/obfs4/obfs4.go
+++ b/transports/obfs4/obfs4.go
@@ -34,6 +34,7 @@ import (
 	"crypto/sha256"
 	"flag"
 	"fmt"
+	"io"
 	"math/rand"
 	"net"
 	"strconv"
@@ -265,7 +266,13 @@ func (sf *obfs4ServerFactory) WrapConn(conn net.Conn) (net.Conn, error) {
 		iatDist = probdist.New(sf.iatSeed, 0, maxIATDelay, biasedDist)
 	}
 
-	c := {conn, true, lenDist, iatDist, sf.iatMode, bytes.NewBuffer(nil), bytes.NewBuffer(nil), make([]byte, consumeReadSize), nil, nil}
+	c := {conn, true, lenDist, iatDist, sf.iatMode, bytes.NewBuffer(nil), bytes.NewBuffer(nil), make([]byte, consumeReadSize), make(chan []byte), nil, nil, make(chan bool)}
+
+	ws := newWriteScheduler(c)
+	go func() {
+		ws.run()
+		c.Close()
+	}()
 
 	startTime := time.Now()
 
@@ -290,8 +297,11 @@ type obfs4Conn struct {
 	receiveDecodedBuffer *bytes.Buffer
 	readBuffer   []byte
 
-	encoder *framing.Encoder
-	decoder *framing.Decoder
+	writeQueue chan []byte
+
+	encoder *framing.Encoder
+	decoder *framing.Decoder
+	codersReadyChan chan bool
 }
 
 func newObfs4ClientConn(conn net.Conn, args *obfs4ClientArgs) (c *obfs4Conn, err error) {
@@ -312,7 +322,13 @@ func newObfs4ClientConn(conn net.Conn, args *obfs4ClientArgs) (c *obfs4Conn, err
 	}
 
 	// Allocate the client structure.
-	c = {conn, false, lenDist, iatDist, args.iatMode, bytes.NewBuffer(nil), bytes.NewBuffer(nil), make([]byte, consumeReadSize), nil, nil}
+	c = {conn, false, lenDist, iatDist, args.iatMode, bytes.NewBuffer(nil), bytes.NewBuffer(nil), make([]byte, consumeReadSize), make(chan []byte), nil, nil, make(chan bool)}
+
+	ws := newWriteScheduler(c)
+	go func() {
+		ws.run()
+		c.Close()
+	}()
 
 	// Start the handshake timeout.
 	deadline := time.Now().Add(clientHandshakeTimeout)
@@ -343,9 +359,7 @@ func (conn *obfs4Conn) clientHandshake(nodeID *ntor.NodeID, peerIdentityKey *nto
 

Re: [tor-dev] Default bridges that are not publishing statistics

[David Fifield]

On Mon, Jun 05, 2017 at 11:51:04PM +1000, teor wrote:
> 
> Can you get logs (and torrcs) from those bridges to confirm whether
> they think they are producing extra info descriptors?

I've asked the operator but not gotten a reply yet.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Default bridges that are not publishing statistics

[David Fifield]

On Mon, Jun 05, 2017 at 03:15:08PM +1000, teor wrote:
> 
> > On 5 Jun 2017, at 15:06, David Fifield <da...@bamsoftware.com> wrote:
> > 
> > Calling get_extrainfo_descriptors from stem.descriptor.remote returns an
> > empty list. (499D92E08769BFC0B7941C74031335B9EC9E9BAE is the new
> > extra-info-digest I got from running just now.)
> > import stem.descriptor.remote
> > print list(stem.descriptor.remote.get_authorities())
> > print 
> > list(stem.descriptor.remote.get_server_descriptors("C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4"))
> > print 
> > list(stem.descriptor.remote.get_extrainfo_descriptors("499D92E08769BFC0B7941C74031335B9EC9E9BAE"))
> > This is the output. Bifroest is in the get_authorities list, so I assume
> > it's querying the bridge authority somewhere in there.
> > ['maatuska', 'tor26', 'Bifroest', 'longclaw', 'dizum', 'gabelmoo', 
> > 'moria1', 'dannenberg', 'Faravahar']
> > []
> > []
> 
> I think the documentation at:
> https://stem.torproject.org/api/descriptor/remote.html#stem.descriptor.remote.Query
> is out of date: as far as I know, newer stem versions try fallback
> directories rather than authorities.
> 
> Even if that's not the case, stem will choose a random authority for
> you, which will (8/9 times) tell you that it knows nothing about any
> bridge extra infos. (I don't think authorities cache bridge extra
> infos.)
> 
> Try passing Bifroest's address and DirPort to the endpoints= argument to
> get_extrainfo_descriptors():
> https://stem.torproject.org/api/descriptor/remote.html#stem.descriptor.remote.Query

Thanks again for your suggestions. Passing endpoints= doesn't seem to do
anything.
>>> import stem.descriptor.remote
>>> BIFROEST = ("37.218.247.217", 80)
>>> print 
list(stem.descriptor.remote.get_server_descriptors("C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4",
 endpoints=(BIFROEST,)))
[]
It also doesn't work when trying the fingerprint of another default
bridge, or of one I just got from bridge.torproject.org.
>>> print 
list(stem.descriptor.remote.get_server_descriptors("A09D536DD1752D542E1FBB3C9CE4449D51298239",
 endpoints=(BIFROEST,)))
[]
>>> print 
list(stem.descriptor.remote.get_server_descriptors("924E408BAE68C6C47A06BCDF6A44A53930708092",
 endpoints=(BIFROEST,)))
[]
I get the same output with and without the endpoints= argument, and
whether or not I provide the SHA-1 hash of the fingerprint.

Strangely, this call seems to just hang:
>>> print 
list(stem.descriptor.remote.get_extrainfo_descriptors("499D92E08769BFC0B7941C74031335B9EC9E9BAE",
 endpoints=(BIFROEST,)))


I tried some other things in Stem but none of them worked.
>>> from stem.control import Controller
>>> with Controller.from_port(port = 9051) as controller:
>>> controller.authenticate()
>>> print 
controller.get_server_descriptor("C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4")
router cymrubridge31 38.229.1.78 443 0 0...
>>> print 
controller.get_info("desc/id/C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4")
router cymrubridge31 38.229.1.78 443 0 0...
>>> print 
controller.get_info("extra-info/digest/499D92E08769BFC0B7941C74031335B9EC9E9BAE")
stem.InvalidArguments: GETINFO request contained unrecognized keywords: 
extra-info/digest/499D92E08769BFC0B7941C74031335B9EC9E9BAE

>>> import stem.connection
>>> import stem.socket
>>> control_socket = stem.socket.ControlPort(port = 9051)
>>> stem.connection.authenticate(control_socket)
>>> control_socket.send("GETINFO 
desc/id/C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4")
>>> print control_socket.recv()
desc/id/C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4=
router cymrubridge31 38.229.1.78 443 0 0...
OK
>>> control_socket.send("GETINFO 
extra-info/digest/499D92E08769BFC0B7941C74031335B9EC9E9BAE")
>>> print control_socket.recv()
Unrecognized key 
"extra-info/digest/499D92E08769BFC0B7941C74031335B9EC9E9BAE"
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Default bridges that are not publishing statistics

[David Fifield]

Thanks for your informative reply.

On Mon, Jun 05, 2017 at 02:37:00PM +1000, teor wrote:
> 
> > On 2 Jun 2017, at 08:20, David Fifield <da...@bamsoftware.com> wrote:
> > 
> > ...
> > And this Stem script:
> > from stem.control import Controller
> > with Controller.from_port(port = 9051) as controller:
> > controller.authenticate()
> > print 
> > controller.get_server_descriptor("C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4")
> ...
> > I can get the bridge-server-descriptor, but not the bridge-extra-info.
> > Here are some lines from bridge-server-descriptor:
> > platform Tor 0.2.9.10 on Linux
> > proto Cons=1-2 Desc=1-2 DirCache=1 HSDir=1 HSIntro=3 HSRend=1-2 
> > Link=1-4 LinkAuth=1 Microdesc=1-2 Relay=1-2
> > published 2017-06-01 21:19:10
> > fingerprint C8CB DB24 64FC 9804 A695 3143 7BCF 2BE3 1FDD 2EE4
> > uptime 613949
> > bandwidth 1073741824 1073741824 5628404
> > extra-info-digest 8B5F0BD647B3C4AF2C57F148FF6A1FB8B695B0AE 
> > 398ZHFBxUpTRfLxv+pSMY3BGMUYlzogXMG40dhjPgnA
> 
> You asked your bridge for its server descriptor and it gave it to you.
> 
> Did you try:
>  print 
> controller.get_extrainfo_descriptors("8B5F0BD647B3C4AF2C57F148FF6A1FB8B695B0AE")
> 
> https://stem.torproject.org/api/descriptor/remote.html#stem.descriptor.remote.DescriptorDownloader.get_extrainfo_descriptors
> 
> Since your bridge knows its own extra info descriptor, it should serve it to 
> you.
> 
> I don't know how to find a server that caches *all* bridge extra infos.
> Maybe you should try running these queries against the bridge authority?

get_extrainfo_descriptors doesn't work as a method of
stem.control.Controller:
AttributeError: 'Controller' object has no attribute 
'get_extrainfo_descriptors'

Calling get_extrainfo_descriptors from stem.descriptor.remote returns an
empty list. (499D92E08769BFC0B7941C74031335B9EC9E9BAE is the new
extra-info-digest I got from running just now.)
import stem.descriptor.remote
print list(stem.descriptor.remote.get_authorities())
print 
list(stem.descriptor.remote.get_server_descriptors("C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4"))
print 
list(stem.descriptor.remote.get_extrainfo_descriptors("499D92E08769BFC0B7941C74031335B9EC9E9BAE"))
This is the output. Bifroest is in the get_authorities list, so I assume
it's querying the bridge authority somewhere in there.
['maatuska', 'tor26', 'Bifroest', 'longclaw', 'dizum', 'gabelmoo', 
'moria1', 'dannenberg', 'Faravahar']
[]
[]

> (How does OnionOO do it?)

I thought that Onionoo was getting extrainfos from Collector. I assumed
that was the reason why Onionoo doesn't work for the two bridges in
question, because they aren't in Collector.

> > (Furthermore, with the torrc shown above, tor doesn't save *any*
> > extra-info descriptors, despite the presence of "DownloadExtraInfo 1"
> > and "FetchUselessDescriptors 1".
> 
> This is because your bridge is your directory guard.
> 
> If you want to fetch relay extra infos through your bridge, you need to
> configure it to download and serve relay extra infos. I'm not sure how to do
> that (maybe DirCache 1?), so I opened this ticket:
> https://trac.torproject.org/projects/tor/ticket/22492

I'm not really interested in fetching extrainfos through the bridge. I
was only trying to do that for the bridge itself, in order to see how
much traffic the bridge is getting. I thought if Stem couldn't do it, I
might be able to pull the answer out of tor's state directory. Actually
I suspected that the bridge would not cache extrainfos for other
bridges, but I thought it might at least return its own extrainfo.

What I'm trying to understand is why these two bridges are behaving
differently than other default bridges, even though as far as I know
they are configured the same way (obfs4 is the only open port, ORport is
firewalled shut). I'm concerned that we are undercounting the number of
obfs4 users while these bridges are not reporting statistics.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Default bridges that are not publishing statistics

[David Fifield]

On Wed, May 24, 2017 at 09:31:45PM -0700, David Fifield wrote:
> On Sat, May 06, 2017 at 09:25:11AM -0700, David Fifield wrote:
> > Okay, thanks. It still doesn't fully make sense to me, because while
> > some of the default bridges are in Atlas, not all of them are (for
> > example the two from https://bugs.torproject.org/21917). I don't think
> > it's possible that they haven't gotten *any* client traffic.
> > 
> > I wonder if it has something to do with the tor version number?
> 
> I checked all the other default bridges and all of them but 2 are
> publishing statistics:
> C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4  cymrubridge31
> 0BAC39417268B96B9F514E7F63FA6FBA1A788955  cymrubridge33
> I will contact the operators and ask them to try setting
> "AssumeReachable 1".

The operator of these bridges set "AssumeReachable 1" and the bridges
still are not publishing statistics ("No Tor relays or bridges matched
your query :("):
https://atlas.torproject.org/#details/C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4
https://atlas.torproject.org/#details/0BAC39417268B96B9F514E7F63FA6FBA1A788955

According to the operator, the bridges are running version 0.2.9.10, so
my hypothesis about it having something to do with old version numbers
is wrong.

Roger suspects that the reason the bridges haven't reported any
statistics is because they haven't received any connections. That
possibility seems unlikely to me, because they are default bridges
shipping in Tor Browser, and additionally I have an hourly test running
that makes an obfs4 connection and builds a circuit.

I tried downloading one of the bridge's bridge-extra-info descriptors
(which would contain information about how many connections it has had),
but I can't quite make it work. Using this torrc file:
UseMicroDescriptors 0
DownloadExtraInfo 1
FetchUselessDescriptors 1
ClientTransportPlugin obfs4 exec /usr/bin/obfs4proxy
ControlPort 9051
CookieAuthentication 1
UseBridges 1
Bridge obfs4 38.229.1.78:80 C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4 
cert=Hmyfd2ev46gGY7NoVxA9ngrPF2zCZtzskRTzoWXbxNkzeVnGFPWmrTtILRyqCTjHR+s9dg 
iat-mode=1
And this Stem script:
from stem.control import Controller
with Controller.from_port(port = 9051) as controller:
controller.authenticate()
print 
controller.get_server_descriptor("C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4")
I can get the bridge-server-descriptor, but not the bridge-extra-info.
Here are some lines from bridge-server-descriptor:
platform Tor 0.2.9.10 on Linux
proto Cons=1-2 Desc=1-2 DirCache=1 HSDir=1 HSIntro=3 HSRend=1-2 
Link=1-4 LinkAuth=1 Microdesc=1-2 Relay=1-2
published 2017-06-01 21:19:10
fingerprint C8CB DB24 64FC 9804 A695 3143 7BCF 2BE3 1FDD 2EE4
uptime 613949
bandwidth 1073741824 1073741824 5628404
extra-info-digest 8B5F0BD647B3C4AF2C57F148FF6A1FB8B695B0AE 
398ZHFBxUpTRfLxv+pSMY3BGMUYlzogXMG40dhjPgnA

(Furthermore, with the torrc shown above, tor doesn't save *any*
extra-info descriptors, despite the presence of "DownloadExtraInfo 1"
and "FetchUselessDescriptors 1". The datadir contains cached-certs,
cached-consensus, and cached-descriptors, but no cached-extrainfo. Only
after I change "UseBridges 1" to "UseBridges 0" does the
cached-extrainfo file appear--but then of course it doesn't contain any
information on bridges.)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Default bridges that are not publishing statistics

[David Fifield]

On Sat, May 06, 2017 at 09:25:11AM -0700, David Fifield wrote:
> Okay, thanks. It still doesn't fully make sense to me, because while
> some of the default bridges are in Atlas, not all of them are (for
> example the two from https://bugs.torproject.org/21917). I don't think
> it's possible that they haven't gotten *any* client traffic.
> 
> I wonder if it has something to do with the tor version number?

I checked all the other default bridges and all of them but 2 are
publishing statistics:
C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4  cymrubridge31
0BAC39417268B96B9F514E7F63FA6FBA1A788955  cymrubridge33
I will contact the operators and ask them to try setting
"AssumeReachable 1".

I suspect it has something to do with the Tor version number. Here are
the other default bridges and their version numbers. The only old one is
the last one, which is the one I had to set AssumeReachable on to get it
to show up.

A09D536DD1752D542E1FBB3C9CE4449D51298239  LeifEricson  obfs3,obfs4,fte  Tor 
0.3.1.0-alpha-dev on Linux
1E05F577A0EC0213F971D81BF4D86A9E4E8229ED  ndnop0   obfs3  Tor 0.2.9.10 
on Linux
4C331FA9B3D1D6D8FB0D8FBBF0C259C360D97E6A  ndnop2   obfs3  Tor 0.2.9.10 
on Linux
AF9F66B7B04F8FF6F32D455F05135250A16543C9  Unnamed Tor 0.2.8.7 
on Linux
0E858AC201BF0F3FA3C462F64844CBFFC7297A42  pdxtorbridge01  Tor 0.2.9.10 
on Linux
1E326AAFB3FCB515015250D8FCCC8E37F91A153B  wisctorbridge01 Tor 0.2.8.8 
on Linux
FC562097E1951DCC41B7D7F324D88157119BB56D  wisctorbridge02 Tor 0.2.9.10 
on Linux
A17A40775FBD2CA1184BF80BFC330A77ECF9D0E9  wisctorbridge03 Tor 0.2.9.10 
on Linux
8DFCD8FB3285E855F5A55EDDA35696C743ABFC4E  ndnop3   obfs4  Tor 0.2.9.10 
on Linux
BBB28DF0F201E706BE564EFE690FE9577DD8386D  ndnop5   obfs4  Tor 0.2.9.10 
on Linux
752CF7825B3B9EA6A98C83AC41F7099D67007EA5  riemann  obfs4  Tor 0.2.8.9 
on Linux
7B126FAB960E5AC6A629C729434FF84FB5074EC2  noether  obfs4  Tor 0.2.8.7 
on Linux
A832D176ECD5C7C6B58825AE22FC4C90FA249637  MaBishomarim obfs4  Tor 0.2.9.10 
on Linux
8FB9F4319E89E5C6223052AA525A192AFBC85D55  Mosaddeghobfs4  Tor 0.2.9.10 
on Linux
00DC6C4FA49A65BD1472993CF6730D54F11E0DBB  JonbesheSabz obfs4  Tor 0.2.9.10 
on Linux
C73ADBAC8ADFDBF0FC0F3F4E8091C0107D093716  GreenBeltobfs4  Tor 0.2.9.8 
on Linux
FE7840FE1E21FE0A0639ED176EDA00A3ECA1E34D  Azadiobfs4  Tor 0.2.9.10 
on Linux
CDF2E852BF539B82BD10E27E9115A31734E378C2  Lisbeth  obfs4  Tor 0.2.9.10 
on Linux
FC259A04A328A07FED1413E9FC6526530D9FD87A  NX01 obfs4  Tor 0.3.0.6 
on Linux
B9E7141C594AF25699E0079C1F0146F409495296  TorLandMeek  meek   Tor 0.2.9.10 
on Linux
97700DFE9F483596DDA6264C4D7DF7641E1E39CE  cymrubridge02meek   Tor 0.2.9.9 
on Linux
2B280B23E1107BB62ABFC40DDCC8824814F80A72  Unnamed  snowflake  Tor 
0.2.5.12 on Linux


appendix: commands to generate this data:
grep -o -E '"\w+ [0-9.:]+ [0-9A-F]+' bridge_prefs.js | sed -e 's/.//' | while 
read transport addr fpr; do wget -c -O $fpr.json 
https://onionoo.torproject.org/details?fingerprint=$(echo $fpr | xxd -r -p | 
sha1sum | awk '{print $1}'); done
for a in *.json; do echo -n "$a "; jq -j 
'.bridges[]|(.hashed_fingerprint,"\t",.nickname,"\t",(.transports//[]|join(",")),"\t",.platform)'
 $a; echo; done
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Why is my bridge not publishing statistics?

[David Fifield]

On Sat, May 06, 2017 at 09:25:11AM -0700, David Fifield wrote:
> > You're right that this is a fragile situation. Maybe we should recommend
> > that if you firewall your ORPort, you also set "AssumeReachable 1"
> > in your torrc?
> 
> I've just set "AssumeReachable 1"; let's see if that helps anything.

Setting "AssumeReachable 1" seems to have worked.

https://atlas.torproject.org/#details/5481936581E23D2D178105D44DB6915AB06BFB7F
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Why is my bridge not publishing statistics?

[David Fifield]

On Sat, May 06, 2017 at 03:41:28AM -0400, Roger Dingledine wrote:
> On Fri, May 05, 2017 at 04:30:52PM -0700, David Fifield wrote:
> > But if it's the case that an unreachable ORPort causes descriptors not
> > to be uploaded, then why do the default obfs4 bridges appear in Atlas?
> 
> Tor relays (and bridges) test their reachability by making circuits
> that loop back to themselves, and they consider themselves reachable
> when an incoming connection sends a create cell (see the end of
> onionskin_answer()).
> 
> You might think that these two actions are more connected, i.e. that
> it needs to be one of the loop circuits that sends the create cell,
> but no, they're completely disconnected. So the relay (or bridge)
> can launch all the loop circuits it wants, and they can all fail, but
> if something causes an incoming connection that sends a create cell,
> it will happily conclude that it's reachable.
> 
> So it's likely that the reason the default bridges are publishing to
> the bridge authority is because somebody used them via obfs4, at which
> point they decided they were reachable, at which point they decided it
> was cool to publish.

Okay, thanks. It still doesn't fully make sense to me, because while
some of the default bridges are in Atlas, not all of them are (for
example the two from https://bugs.torproject.org/18050). I don't think
it's possible that they haven't gotten *any* client traffic.

I wonder if it has something to do with the tor version number?

> You're right that this is a fragile situation. Maybe we should recommend
> that if you firewall your ORPort, you also set "AssumeReachable 1"
> in your torrc?

I've just set "AssumeReachable 1"; let's see if that helps anything.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Why is my bridge not publishing statistics?

[David Fifield]

I searched for the Snowflake bridge in Atlas, and couldn't find it. Its
fingerprint is 2B280B23E1107BB62ABFC40DDCC8824814F80A72. Its torrc is
stock "Last updated 9 October 2013 for Tor 0.2.5.2-alpha" except for
these settings:
ContactInfo David Fifield <d...@torproject.org>
SOCKSPort 0
ORPort 9001
BridgeRelay 1
ExtORPort auto
ServerTransportPlugin snowflake exec /usr/local/bin/snowflake-server 
--acme-hostnames snowflake.bamsoftware.com --acme-email d...@torproject.org 
--log /var/log/tor/snowflake-server.log
ServerTransportListenAddr snowflake 0.0.0.0:443

Its ORPort 9001 is blocked by the local firewall, because it is meant to
be only a Snowflake bridge, and not a vanilla bridge. (Most of the
default Tor Browser obfs4 bridges are configured the same way, with
their ORPort blocked.) There are these messages in the log (which I
exxpected):
[warn] Your server (37.218.242.151:9001) has not managed to confirm 
that its ORPort is reachable. Please check your firewalls, ports, address, 
/etc/hosts file, etc.

Why is the bridge not appearing in Atlas? I initially suspected
https://bugs.torproject.org/18050, whose changelog entry is:
- Check that both the ORPort and DirPort (if present) are reachable
  before publishing a relay descriptor. Otherwise, relays publish a
  descriptor with DirPort 0 when the DirPort reachability test takes
  longer than the ORPort reachability test.
  Closes bug #18050. Reported by "starlight", patch by "teor".
  Bugfix on 0.1.0.1-rc, commit a1f1fa6ab on 27 Feb 2005.
But if it's the case that an unreachable ORPort causes descriptors not
to be uploaded, then why do the default obfs4 bridges appear in Atlas?
For example:

https://atlas.torproject.org/#details/D9C805C955CB124D188C0D44F271E9BE57DE2109

https://atlas.torproject.org/#details/D3D4A456FCB5F301F092F6A49ED671B84B432FB8

https://atlas.torproject.org/#details/11A3982C417AF37230F576006405BE5338F41C09
Actually, now that I look at it, I notice some other default bridges are
not present in Atlas, for example the two from
https://bugs.torproject.org/21917, which went out in Tor Browser 6.5.2:
C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4
0BAC39417268B96B9F514E7F63FA6FBA1A788955

What's going on and how can we fix it? You can find a list of default
bridge fingerprints here:
https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/Bundle-Data/PTConfigs/bridge_prefs.js
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] "firefox --app" for meek-http-helper

[David Fifield]

On Sun, Mar 26, 2017 at 02:28:00PM +, anonym wrote:
> Tails uses the Tor Launcher shipped in Tor Browser, but it's run as a
> stand-alone XUL application (`firefox --app ...`), so the *web*
> browser isn't started as part of it.

Sorry to change the subject, but should we be running meek-http-helper
using "firefox --app"? I didn't know about that before. It sounds like
it could solve some of the problems associated with having multiple
Firefox profiles.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Flashproxy has been Deactivated by Stanford? Why?

[David Fifield]

On Mon, Dec 19, 2016 at 09:53:25AM -0800, David Fifield wrote:
> The badge was deactivated by Stanford (without my knowledge, but I found
> out a while ago). I arranged with them to move it to alternate hosting
> and have them install a redirect, but that has been a low priority
> behind other work on Snowflake.
> 
> I'm sorry about the confusion. If I get some time I'll add a notice to
> the flash proxy main page saying that it's been retired.

I've set up separate hosting for the flash proxy badge files at
https://flashproxy.bamsoftware.com/, and added redirects from the
https://crypto.stanford.edu/flashproxy/ URLs to there. Already deployed
badges should continue to work, given the redirects. I added a note to
the top of the home page saying that flash proxy is deprecated.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Flag blocked websites

[David Fifield]

On Fri, Mar 10, 2017 at 03:46:03PM -0500, Boter42 wrote:
> I'm also trying to implement an automatic scan of specific lists of websites 
> to
> check their behaviour towards Tor. I'm using ooniprobe but I lack some
> technical skills (mainly to filter out false positives), I'll see if I can set
> up a team.

You might be able to use/adapt some code we wrote to mine OONI reports
for cases of Tor blocking:

http://sec.cs.ucl.ac.uk/users/smurdoch/papers/ndss16doyousee.pdf
https://archive.org/details/ndss16doyousee

The actual OONI-processing code is in a Git repo at:
https://www.bamsoftware.com/git/ooni-tor-blocks.git

One catch, though, is that you'll have to adapt the ooni.py file to
handle OONI's web_connectivity tests. The code was originally written
before web_connectivity existed, so it only works with the http_requests
test.

If you want to run your own active tests, we have some patches on top of
exitmap:
git clone -b l7_tor_limits https://www.bamsoftware.com/git/exitmap.git
But you will need to do some work to bring them up to date.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Flag blocked websites

[David Fifield]

On Fri, Mar 10, 2017 at 08:27:01AM -0500, Boter42 wrote:
> I think it would be important to have a way to flag/report those websites that
> can't be access by the users while they're using the tor browser. 
> 
> Is there already a solution to do this? Do you think it would be a good tool? 
> 
> It would be great to have an updated records of this kind of websites so that
> we can push website owners to make the Tor user-experience as smooth as
> possible. 

There's an informal yet quite large list here:
https://trac.torproject.org/projects/tor/wiki/org/doc/ListOfServicesBlockingTor
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] OnionGatherer: evaluating status of hidden services

[David Fifield]

On Fri, Mar 10, 2017 at 12:58:55PM +0100, Massimo La Morgia wrote:
> we are a research group at Sapienza University, Rome, Italy. We do research on
> distributed systems, Tor, and the Dark Web. As part of our work, we have
> developed OnionGatherer, a service that gives up-to-date information about 
> Dark
> Web hidden services to Tor users.

...and presumably helps you build a crowdsourced list of onion services
that you plan to use for some other research purpose?

If you're planning a research project on Tor users, you should write to
the research safety board and get ideas about how ot do it in a way that
minimizes risk.
https://research.torproject.org/safetyboard.html

This idea seems, to me, to have a lot of privacy problems. You're asking
people to use Chrome instead of Tor Browser, which means they will be
vulnerable to a lot of fingerprinting and trivial deanonymization
attacks. Your extension reports not only the onion domains that it
finds, but also the URL of the page you were browsing at the time:
var onionsJson = JSON.stringify({onions:onions, website: 
window.location.href});
You need to at least inform your research subjects/users what of their
private data you are storing and what you are doing with it.

You're using two different regexes for onion URLs that aren't the same.
The one used during replacement doesn't match "https", so I guess it
will fail on URLs like https://facebookcorewwwi.onion/.
/^(http(s)?:\/\/)?.{16}(\.onion)\/?.*$/
/(http:\/\/)?\b[\w\d]{16}\.onion(\/[\S]*|)/
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] GAEuploader

[David Fifield]

On Sun, Jan 22, 2017 at 03:53:16PM -0800, Katherine Li wrote:
> I would really appreciate user testing on GAEuploader. You can download it 
> at: 
> https://github.com/katherinelitor/GAEuploader/releases
> README: https://github.com/katherinelitor/GAEuploader
> Tor wiki page, containing step-by-step screenshots of how to use:
> https://trac.torproject.org/projects/tor/wiki/doc/GAEuploader

I would like to echo this call for testing. GAEuploader automates the
process of uploading the meek code to App Engine. If you already have a
Google account, it's pretty easy: the program will open a browser so you
can authenticate and accept the App Engine terms of service, then upload
the code under a domain name you choose. At the end, GAEuploader prints
out a bridge line to paste into Tor Browser.

The difference between this and the old meek-google is that the App
Engine code you upload is just for you, not shared with anyone else.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Flashproxy has been Deactivated by Stanford? Why?

[David Fifield]

On Tue, Dec 20, 2016 at 01:21:04AM +0800, to...@riseup.net wrote:
> It turned out that the entire code has been commented out and apparently
> Flashproxy became
> out of service. Why? Has the project discontinued, or just down for
> maintenance?

Flash proxy is basically retired now. It was removed from Tor Browser a
year ago (https://bugs.torproject.org/17428) after it had been
supplanted by more effective transports. I don't know why there was a
blog post on December 16 promoting flash proxy, because it's no longer
used.

Even when flash proxy was part of Tor Browser, it had very few users
(less than 100; see 
https://metrics.torproject.org/userstats-bridge-transport.html?start=2013-01-01=websocket),
probably because of the difficulty of running it as a client
(https://trac.torproject.org/projects/tor/wiki/doc/PluggableTransports/FlashProxy/Howto).
Compare those user numbers to meek (current about 10K) or obfs4 (30K).

The reason I haven't asked people to stop running the flash proxy badge
is we're working on a new pluggable transport along the same lines but
without the usability challenges:
https://trac.torproject.org/projects/tor/wiki/doc/Snowflake. I was
thinking about adapting existing flash proxy badges to provide capacity
to Snowflake instead. This would go for Cupcake as well. The need to get
the badge running again hasn't been pressing because Snowflake isn't
deployed yet, but we're getting close.

The badge was deactivated by Stanford (without my knowledge, but I found
out a while ago). I arranged with them to move it to alternate hosting
and have them install a redirect, but that has been a low priority
behind other work on Snowflake.

I'm sorry about the confusion. If I get some time I'll add a notice to
the flash proxy main page saying that it's been retired.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] automatically detect many new identical/similar bridges

[David Fifield]

On Wed, Dec 14, 2016 at 10:09:00AM +, nusenu wrote:
> in the context of [1] I'm wondering if it makes sense to add bridge
> support to ornetradar.
> 
> If there is any value to automatically detect multiple new bridges:
> 
> - Do bridges publish ContactInfo in their descriptor? If not: Why not?
> (it shouldn't disclose their bridge location)

From my understanding, bridge descriptors may include ContactInfo, but
Collector removes it before publishing. So you can distinguish
descriptors that originally had ContactInfo set from those that did not,
but you can't tell what the ContactInfo was.

https://collector.torproject.org/#bridge-descriptors
> 5. Replace contact information: If there is contact information in a
> descriptor, the contact line is changed to somebody.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Using fingerprint of cached relay bypasses bridge?

[David Fifield]

Someone on #tor-project IRC reported that you can bypass your pluggable
transport if you use the fingerprint of an ordinary relay already known
to Tor in your bridge line. I would file a ticket but I haven't been
able to reproduce it.

The example the IRC user gave was this, meant to be pasted into Tor
Browser as a custom bridge:
Bridge meek 0.0.2.0:2 3C3A6134E4B5B7D1C18AD4E86EE23FAC63866554 
url=https://d2zfqthxsdq309.cloudfront.net/ front=a0.awsstatic.com
The fingerprint is the wrong one for this bridge. It should be
B9E7141C594AF25699E0079C1F0146F409495296 for the bridge nicknamed
TorLandMeek. It is instead for the relay nicknamed traffic70 at
188.138.1.166:9001.

The claim is that if tor has already cached a descriptor with
fingerprint 3C3A6134E4B5B7D1C18AD4E86EE23FAC63866554, then it will make
a direct connection for the purpose of making a one-hop circuit. "it's
about one hop tunnel when exit is entry" says the IRC user. They point
to these parts of the source code:

https://gitweb.torproject.org/tor.git/tree/src/or/circuituse.c?id=tor-0.2.8.9#n2010
r = node_get_by_nickname(conn->chosen_exit_name, 1);
https://gitweb.torproject.org/tor.git/tree/src/or/circuituse.c?id=tor-0.2.8.9#n2015
extend_info = extend_info_from_node(r, conn->want_onehop ? 1 : 0);

I wasn't able to reproduce it. I used this torrc file:

DataDirectory datadir
UseBridges 1
Bridge meek 0.0.2.0:2 3C3A6134E4B5B7D1C18AD4E86EE23FAC63866554 
url=https://d2zfqthxsdq309.cloudfront.net/ front=a0.awsstatic.com
ClientTransportPlugin meek exec ./meek-client --log meek-client.log

First I tried with a cold cache, and got a fingerprint mismatch:

Nov 01 18:00:53.000 [notice] Bootstrapped 10%: Finishing handshake with 
directory server
Nov 01 18:00:54.000 [warn] Tried connecting to router at 0.0.2.0:2, but 
identity key was not as expected: wanted 
3C3A6134E4B5B7D1C18AD4E86EE23FAC63866554 but got 
B9E7141C594AF25699E0079C1F0146F409495296.
Nov 01 18:00:54.000 [warn] Problem bootstrapping. Stuck at 10%: Finishing 
handshake with directory server. (Unexpected identity in router certificate; 
IDENTITY; count 1; recommendation warn; host 
3C3A6134E4B5B7D1C18AD4E86EE23FAC63866554 at 0.0.2.0:2)

Then I commented out "UseBridges 1", let the bootstrap finish, and
uncommented "Use Bridges 1" again. I got the same output:

Nov 01 18:05:06.000 [notice] Bootstrapped 10%: Finishing handshake with 
directory server
Nov 01 18:05:09.000 [warn] Tried connecting to router at 0.0.2.0:2, but 
identity key was not as expected: wanted 
3C3A6134E4B5B7D1C18AD4E86EE23FAC63866554 but got 
B9E7141C594AF25699E0079C1F0146F409495296.
Nov 01 18:05:09.000 [warn] Problem bootstrapping. Stuck at 10%: Finishing 
handshake with directory server. (Unexpected identity in router certificate; 
IDENTITY; count 1; recommendation warn; host 
3C3A6134E4B5B7D1C18AD4E86EE23FAC63866554 at 0.0.2.0:2)

I used tcpdump to check for connections to 188.138.1.166, and didn't see
any.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Call for help on testing core tor releases

[David Fifield]

On Thu, Oct 13, 2016 at 02:29:19PM -0400, isab...@riseup.net wrote:
> Hello Tor community!
> 
> The Core Tor Team would like to improve our release process by getting
> it more tested so bugs are found earlier, so stable releases can get out
> faster and without any big bugs.
> 
> During Tor's Meeting in Seattle a couple of weeks ago, we discussed this
> topic and decided to organize a QA team (with the help of the community)
> for core tor.

There are some people on the tor-qa who test Tor Browser before release;
maybe some of them will help? Or maybe you want to do core tor QA on
this list?
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-qa
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Tor Relays on Whonix Gateway

[David Fifield]

On Wed, Oct 19, 2016 at 10:35:16PM +0200, ban...@openmailbox.org wrote:
> On 2016-10-17 10:24, isis agora lovecruft wrote:
> > 
> > You're planning to enable "ServerTransportPlugin snowflake" on Whonix
> > Gateways
> > by default?  And then "ClientTransportPluging snowflake" on workstations
> > behind the gateway?
> 
> I was planning to enable the server by default (I thought WebRTC was P2P
> though) but after looking at it some more I don't think it's a good idea.

It doesn't make sense to run the Snowflake server on a lot of bridges
anyway. It's not like the obfs* model where you need lots of bridges in
order to get IP diversity. Snowflake gets IP diversity by routing
through web browsers. The bridge itself may even be blocked by the
censor; it doesn't matter.

The server component of Snowflake isn't even WebRTC. Snowflake is WebRTC
between the client and the browser proxy, then WebSocket (which is
easier to program) between the browser proxy and the bridge. The server
component is actually just a WebSocket server, borrowed from flash
proxy.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 274: A Name System API for Tor Onion Services

[David Fifield]

On Fri, Oct 07, 2016 at 04:06:51PM -0400, George Kadianakis wrote:
>In particular, onion addresses are currently composed of 16 random base32
>characters, and they look like this:
> 
>   3g2upl4pq6kufc4m.onion
>   vwakviie2ienjx7t.onion
>   idnxcnkne4qt76tg.onion
>   vwakviie2ienjx6t.onion

Is there any meaning intended by the near-collision in the 2nd and 4th
addresses?

>If all plugins fail to successfuly perform the name resolution, Tor SHOULD
>default to using the exit node for name resolution.
>XXX or not?  because of leaks?

It would be really nice if it were possible for the browser to show some
kind of UI indication that a name I type into the URL bar is going to be
handled specially. Even now, the URL bar is magical: if you typo a DNS
name, the browser does a web search and leaks whatever you just typed. I
think in the protocol as proposed, there's no way for the browser to
predict whether a name will be looked up in a special way using a
plugin, or even to learn what the mapping was after it has occurred.

How do you think this will interact with attacker-controlled names in
HTML? Let's say someone has a local petname database where
"bank.petname" points to the onion site of their bank. What happens when
HTML says
http://bank.petname/favicon.ico;>
Will it show the icon of the user's bank, whatever bank it happens to
be? I guess my question is if attackers are allowed to query the user's
local naming setup, even in an indirect way, or does the magic only
happen for names that are typed directly into the URL bar? (Because the
mapping happens below the SOCKS layer, I guess it could not apply only
to the URL bar.)

>Hence, the contribution of this proposal is a modular Name System API
>(NSA)

This acronym might cause confusion :)

>|
> $~~$
>|  1.$  4. GNS magic!! 
>  $
>|  User: SOCKS CONNECT to$ debian.zkey -> 
> sejnfjrq6szgca7v.onion$
>|http://debian.zkey/ 
> $~~$~~~$
>|   $
>  +-|-+ $
>  |+v-+ 2. +-+|   3.$
>  ||Tor   | debian.zkey|Tor  ||   debian.zkey 
> +-$---+
>  ||Networking->Naming   ->
>  |
>  ||Submodule ||Submodule||  Tor Name System API  |  
> GNS|
>  ||  <- <-  
> wrapper|
>  ||  | 6. | ||5. |
>  |
>  |+|-+ sejnfjrq6szgca7v.onion +-+|sejnfjrq6szgca7v.onion 
> +-+
>  +-|-+
>|  7.
>|  Tor: Connect to
>|   http://sejnfjrq6szgca7v.onion/
>v

So here, the browser thinks it is connecting to debian.zkey (the URL bar
says "debian.zkey"). But Tor is really connecting to sejnfjrq6szgca7v.onion
in the background. What name does the browser put in its Host header? It
can't be the onion name, because there's no feedback from the naming
module back to the user application layer. It must be "debian.zkey"
then. If that is a petname, then it just got leaked to the server. I can
imagine this might also cause a problem with some virtual hosting setups
(though I suppose those are not very common for onion services). If the
user uses HTTPS, e.g. https://facebook.zkey/, then they'll get a TLS
name mismatch error, even if https://facebookcorewwwi.onion/ exists and
has a valid certificate--so using the naming system is not a transparent
replacement for memorizing the onion address. Maybe non-HTTP protocols
will also have problems.

It would be nice if there were some standard external way to just do a
name→onion query. As specified, there's no easy way for e.g. the browser
to query how a name will be mapped, short of launching its own instance
of the GNS wrapper and communicating with the over the Name System API.
I wonder, perhaps there's a way to extend the Tor RESOLVE cell to
support this feature? So you could do, for example
$ tor-resolve debian.zkey
sejnfjrq6szgca7v.onion

>When launching name plugins, Tor sets various environment variables to pass
>data to the name plugin (e.g. NS API version, state directory, etc.). More
>information on the environment variables at [INITENVVARS].
> 
>After a name plugin initializes and parses all needed environment
>variables, it communicates with Tor using its stdin/stdout.

I worry that repeating the envvar and stdin/stdout design of the PT spec
will cause problems for 

[tor-dev] uProxy adds Tor support

[David Fifield]

https://blog.uproxy.org/2016/09/uproxy-adds-tor-support.html

This blog post says that uProxy gained support for proxying others'
traffic through Tor.

uProxy client → censor → uProxy server → Tor → destination

In the classic uProxy deployment scenario, the client and server are
people who know each other. (I say "classic" because there are other
deployment modalities now.) They each run a browser extension. The
client encapsulates its web traffic and sends it to the server over
WebRTC. The server then issues the web requests and sends the responses
back over the WebRTC channel.

A potential problem is that the uProxy server's own web browsing gets
mixed up with that of the client. The client could, for example, get the
server in trouble by accessing sketchy web sites. That's what the Tor
integration is meant to solve.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Tor Browser downloads and updates graphs

[David Fifield]

On Mon, Sep 12, 2016 at 11:12:15AM -0400, Mark Smith wrote:
> On 9/11/16 3:45 PM, David Fifield wrote:
> >> * We don't know what (8) or (9) is but it seems to us we are losing
> >> users over time and are only getting them back slowly if at all. A
> >> weekday/weekend pattern is visible there as well.
> > 
> > Does Tor Browser continue checking for further updates in the span of
> > time between when it downloads an update and when it is restarted? For
> > example, you are running 6.0, the browser downloads the 6.0.1 update and
> > stages it and asks you to restart; does the browser check for updates
> > until you actually restart? If not, then the decreases in update pings
> > might be people being tardy in restarting their browser.
> 
> That is a good theory, but I don't think update checks occur if there is
> a pending update. The code that checks and returns early is here:
> 
> https://gitweb.torproject.org/tor-browser.git/tree/toolkit/mozapps/update/nsUpdateService.js?h=tor-browser-45.4.0esr-6.0-1#n2388

Oh, thanks for finding that source code link. I looked for that code and
didn't find it.

But that's exactly what I'm saying: once someone has downloaded an
update, they stop sending update pings until their next restart, which
might explain the decreases in update pings at (8) and (9) in the graphs.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Pluggable transport idea: TLS session resumption

[David Fifield]

Here's an idea for a new pluggable transport. It's just a TLS tunnel,
but with a twist that allows the server's certificate to be omitted,
depriving the censor of many classification features, such as whether
the certificate is signed by a CA, the certificate's lifetime, and
whether the commonName matches the server's IP address. I got the idea
from ShadowsocksR:
https://github.com/breakwa11/shadowsocks-rss/blob/master/ssr.md#2%E6%B7%B7%E6%B7%86%E6%8F%92%E4%BB%B6

The trick that makes it work is RFC 5077 session tickets. How it's
supposed to work is, the client makes a TLS connection as usual, and the
server sends back a session ticket before beginning the flow of
application data. The session ticket is just a random blob from the
client's point of view; from the server's point of view it's an
authenticated ciphertext that encodes session parameters such as the
ciphersuite and master secret. The next time the client connects to the
same server, it sends the session ticket along with its ClientHello and
the server can skip sending its certificate.

Here's a simple way to adapt session tickets for circumvention. The
client *always* sends a session ticket when making a connection, even
the very first time. The ticket doesn't come from a previous
communication with the server; the client generates the ticket itself.
The ticket could for instance be a random string--the censor wouldn't be
able to distinguish--but we can also use those bits for other purposes.
When the server receives the ClientHello, it *pretends* that the ticket
was a valid one, and finishes the handshake using hardcoded session
parameters, just statically configuring what it would have encoded into
a session ticket.

The protocol as just described would be vulnerable to active probing;
the censor could test for servers by sending them garbage session
tickets and seeing how they respond. But that's easy to fix. We can, for
example, let the client and server have a shared secret, and have the
server treat the session ticket as the client part of an obfs4
handshake--which conveniently resembles a random blob. If the session
ticket doesn't pass obfs4 authentication, then the TLS server can
respond as it naturally would if a client sent an invalid session
ticket; i.e., issue a new ticket and do a full handshake (then send
dummy data, I guess). The server can also honor its own legitimately
issued tickets, but still send dummy data in response. Only clients who
know the shared secret will be able to access the proxy functionality of
the server.

In order to block such a transport, the censor will have to look at
features other than the server certificate. It could, for example:
 * block all session tickets (dunno how costly that would be)
 * statefully track which tickets servers have issued, and block
   connections that use an unknown ticket.
 * track the fraction of connections that use session tickets on each
   TLS server.
 * active-probe the server in order to get a certificate, and then look
   at certificate features.
We'd have to do some research to find out the distribution of session
ticket sizes in the wild. (RFC 5077 recommends a specific format:
https://tools.ietf.org/html/rfc5077#section-4.)

I didn't think of this idea; it comes from ShadowsocksR and its
tls1.2_ticket_auth plugin.
https://github.com/breakwa11/shadowsocks-rss/blob/master/ssr.md#2%E6%B7%B7%E6%B7%86%E6%8F%92%E4%BB%B6
The documentation is in Chinese. Here is what Google Translate says in
English:
Analog TLS1.2 handshake at client has a session ticket situation
is connected. Currently complete analog implementation, software
testing through capture perfectly disguised as TLS1.2. Because
there is no ticket so send certificates and other complicated
steps, so the firewall can not make a judgment based on the
certificate. At the same time it comes with the ability of
certain anti-replay attacks. In case of a replay attack is
searched end log in to the service, you can grep "replay attack"
search, you can use this plug-in found in your area there is no
line of interference for TLS. TLS relatively powerless firewall,
anti-blocking ability than other plug-ins should be strong, but
may also encounter a lot of interference, but the agreement
itself will check out any interference, then disconnect
encounter interference, to avoid long waits, so that customers
end browser or reconnect themselves. This plug-in is compatible
with the original agreement (the server is configured to require
tls1.2_ticket_auth_compatible), one more than the original
protocol handshake causes the connection will take longer, use C
# client open automatically reconnect when performed better than
the other plug-ins. Support for custom parameters, parameters
SNI, it sends the host name field, this function is very similar
to the TOR 

Re: [tor-dev] HTTPS Everywhere

[David Fifield]

On Mon, Sep 05, 2016 at 10:28:26PM +0530, AKASH DAS wrote:
> Can I know the issues that are currently in https everywhere.

I don't know if this is what you're looking for, but here are some open
bug tracker tickets.

https://trac.torproject.org/projects/tor/query?status=!closed=HTTPS+Everywhere%2FEFF-HTTPS+Everywhere
https://trac.torproject.org/projects/tor/query?status=!closed=HTTPS+Everywhere%2FHTTPS+Everywhere%3A+Chrome
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Some information about Tor relays

[David Fifield]

On Fri, Aug 26, 2016 at 04:46:45AM +, Liu, Zhuotao wrote:
> Thanks for that info, David. That seems valuable to me. :) 
> 
> However, I am a bit confused about the definition 
> 
> "cell-circuits-per-decile": Mean number of circuits that are included in any 
> of the deciles, 
> rounded up to the next integer. 
> 
> What is the exact meaning of 'decile'? Is it one tenth of a hour? Or 
> something else? 

I don't know. My reading of dir-spec says it is probably the 0%–10%,
10%–20%, 20%–30%, etc. circuits counting by number of cells.

https://spec.torproject.org/dir-spec

"cell-processed-cells" num,...,num NL
Mean number of processed cells per circuit, subdivided into
deciles of circuits by the number of cells they have processed in
descending order from loudest to quietest circuits.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Some information about Tor relays

[David Fifield]

On Fri, Aug 26, 2016 at 01:42:38AM +, Liu, Zhuotao wrote:
> This is Sky from University of Illinois. Currently we are working on research
> project related with Tor. 
> 
> To help us to better design and evaluation our proposal, we need some
> information about the Tor relays that is currently unavailable from the Atlas.
> Thus, if someone who operates Tor relays could provide us such information,
> your help will be greatly appreciated. :) 
> 
> We hope to have an estimate about computation capacity of Tor relays. For
> instance, how many circuits a relay can maintain when its CPU is driven to
> about 100%? On average, how many circuits are maintained by a busy guard and
> what the CPU utilization is. These kinds of information would be really
> helpful. 

I don't know about CPU usage, but as for circuits, I think you can get
them from @extra-info and @bridge-extra-info descriptors:
https://collector.torproject.org/#type-extra-info
https://collector.torproject.org/#type-bridge-extra-info
For example, see some of the files at
https://collector.torproject.org/recent/relay-descriptors/extra-infos/

The "cell-circuits-per-decile" lines might be interesting to you.
https://spec.torproject.org/dir-spec:
"cell-circuits-per-decile" num NL
Mean number of circuits that are included in any of the deciles,
rounded up to the next integer.

For parsing the descriptor files, you can use Stem:

https://stem.torproject.org/_modules/stem/descriptor/extrainfo_descriptor.html
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] GreatFire Circumvention Central: tests of speed and stability of circumvention tools in China

[David Fifield]

https://en.greatfire.org/blog/2016/jul/greatfireorg-now-testing-vpn-speed-and-stability-china
https://cc.greatfire.org/en
"Our newest website, Circumvention Central (CC), aims to provide
real-time information and data about circumvention solutions
that work in China. Since 2011, we have been collecting data
about blocked websites in China and now we will add data about
the effectiveness of VPNs and other circumvention tools."

They list various VPNs and other tools. Tor is on the list. (This page
has JavaScript charts that aren't working for me in Tor Browser.)
https://cc.greatfire.org/en/provider/Tor

At the moment, these are the figures I see:
Speed (kbps)Stability
1. HideMyAss504 (+39%)  98% (-1%)
2. VyprVPN  445 (−33%)  98% (+11%)
3. FreeBrowser  440 (+13%)  88% (+1%)
4. XX-Net   356 (New)   97%
5. Astrill  294 (−32%)  97% (+1%)
6. Lantern  262 (+15%)  95% (+3%)
7. VPNSecure259 (New)   87%
8. Psiphon  218 (+6%)   89% (+2%)
9. Ultrasurf206 (+460%) 86% (−4%)
10. Freegate188 (−29%)  91% (+1%)
11. PureVPN 151 (−25%)  94% (+1%)
12. Tor 112 (+237%) 96% (+1%)

The test details have interesting values under the "Settings" header:
https://cc.greatfire.org/en/provider/Tor#http-tests
"0", "socks5", "是", "lantern-proxy", "lantern-proxy,no bridge",
"lantern-proxy,meek-arzue", "proxy,no bridge", "lantern-proxy,obfs4",
"obfs4", "否", "meek-arzue", "obfs4,sock5 proxy".

There's a tool on the list I didn't know, XX-Net. Its GitHub page says
"It uses google app engine (GAE) as a proxy server through the
firewall" and "A Reborn GoAgent."
https://github.com/XX-net/XX-Net
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] meek-server performance improvements?

[David Fifield]

I saw you say on IRC that you had an idea for improving the efficiency
of meek-server. What's your idea? The server hosting meek-azure is
passing 90% CPU at times.

One idea I've seen is using one connection for upstream data
(data-carrying POSTs, emptry responses), and one connection for
downstream data (GET, one big chunked download). As I understand it,
this is what Lantern's enproxy does:
https://github.com/getlantern/enproxy
So far I've avoided this idea because of the possibly weird traffic
patterns.

I've also done some preliminary work on allowing the server to send data
in larger chunks, which should reduce polling:
https://trac.torproject.org/projects/tor/ticket/12857#comment:4
All that's really needed here is a decision on how to let the client
indicate that it supports such a server, as the change is backward
incompatible.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Using Let's Encrypt for meek bridges

[David Fifield]

On Fri, Apr 08, 2016 at 05:28:45PM -0700, George Tankersley wrote:
> > I'm looking for ideas of good ways to handle TLS certificates and their
> > renewal for meek bridges. I want to use Let's Encrypt for this process,
> > and I hope that someone who knows Let's Encrypt well can contribute some
> > ideas.
> 
> > ideally not requiring new complicated code in meek-server itself
> 
> If you're OK with some amount of new code, there are Go client libraries that
> might be sufficiently flexible:
> 
> - https://github.com/xenolf/lego
> - https://ericchiang.github.io/go/tls/lets/encrypt/letsencrypt/2015/11/13/
> a-letsencrypt-client-for-go.html
> 
> I'll give this a try soon if you haven't already.

Thanks. You are welcome to try. I haven't done anything yet.

My ideal situation would be that I can run an acme client *outside* of
meek-server, the same way I can do for Apache. Something like this:
letsencrypt-auto certonly [some other arguments, maybe --webroot] 
meek.bamsoftware.com

Adding acme code to meek-server seems less desirable to me. I am trying
to keep the feature count low to keep the bug count low. meek-server is
under 500 lines and I wouldn't want to add more than, say, 200 lines for
acme support. It might have to write new certificate files to the
filesystem, which means worrying about privileges. It's possible,
though, that I don't properly understand how an acme library should work
and these concerns aren't appropriate.

My current inclination is to add an --acme-webroot option to
meek-server, and install a restricted http.Handler that allows serving
files from that directory. It will only serve files whose names are in
the specific format that acme requires. Maybe it can only serve files
whose creation time is recent, as an additional safeguard against
accidentally serving non-acme files.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] iObfs: obfs4proxy on iOS

[David Fifield]

On Mon, Apr 04, 2016 at 12:04:45AM -0400, Mike Tigas wrote:
> [again, cross-posted to tor-dev and guardian-dev.]
> 
> A quick status report on this: it works! Hit a big epiphany, figured out
> how to get `gomobile` to emit the necessary bits, then went wild.
> 
> Some example stdout from Onion Browser connecting to Tor via obfs4,
> meek_lite (google), and scramblesuit:
> https://gist.github.com/mtigas/f1b9a3a8befa6f60d517eb2340f3cdd4
> 
> There are trivial forks of obfs4[1] and goptlib[2] that simply hard-code
> some options that are normally sent as environment variables because
> obfs4proxy runs in managed mode[3]. (It's the best I have right now
> until I can figure out a better way to communicate between obfs4proxy
> and the iOS bits.) I’ve tacked a few other quick thoughts at the bottom
> of the iObfs readme[4]. As a quick test I've started building it into
> Onion Browser (iobfs branch[5]), which is what got the output linked above.
> 
> [1]:
> https://github.com/mtigas/obfs4/compare/1df5c8ffe8f4aa2614323698e8008f1ab1fb7a18...mtigas:iObfs-201604-dev
> [2]:
> https://github.com/mtigas/goptlib/compare/f17a5f239f705d7e39a8bccbebdf9927cc99dbeb...mtigas:iObfs-201604-dev

This is radical. Maybe you don't need the fork of goptlib if you do
os.Setenv on the relevant variables before calling pt.ClientSetup in
obfs4?
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Using Let's Encrypt for meek bridges

[David Fifield]

I'm looking for ideas of good ways to handle TLS certificates and their
renewal for meek bridges. I want to use Let's Encrypt for this process,
and I hope that someone who knows Let's Encrypt well can contribute some
ideas.

All three of the meek bridges use HTTPS to receive connections from the
CDN, so they need TLS certificates. For example, when you use
meek-azure, your traffic is forwarded to the bridge at
https://meek.bamsoftware.com/. How it works now is I do the usual domain
validation procedure with a CA, receive an email to show that I control
the domain, install the cert and key, and then run the server like this:
meek-server --cert /etc/meek/cert.pem --key /etc/meek/key.pem

When I used Let's Encrypt in the past, using the --webroot option, it
wanted to write a file to the URL path "/.well-known/acme-challenge/{token}".
That won't work for meek-server as it exists today, because meek-server
never serves files from the filesystem. But it could. Perhaps we could
add an option like --acme-webroot that would allow serving files from a
single whitelisted directory.

I notice that there are other ways of proving domain ownership (HTTP,
TLS SNI, DNS). Maybe we could use one of those?
https://ietf-wg-acme.github.io/acme/#identifier-validation-challenges

I also note that there are third-party plugins:
https://github.com/letsencrypt/letsencrypt/wiki/Plugins
Maybe there could be a plugin for meek-server (ideally not requiring new
complicated code in meek-server itself).

Currently you have to restart meek-server in order to make it notice a
changed certificate and key file. It would be better if that were not
necessary--maybe we could periodically stat the files, and re-load them
if they have changed?

This is going to be an issue for Snowflake as well, because we will want
to use WebSocket over TLS for the server component.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Request for feedback/victims: cfc

[David Fifield]

During the OONI survey to find instances of server-side Tor blocking, we
found a few variations on CloudFlare captcha pages. They don't all say
"Attention Required!". Apparently there is an option to customize the
page, but few sites make use of it. Here are the regexes we used
(excerpted from https://www.bamsoftware.com/git/ooni-tor-blocks.git):
if status == 403:
if server == "cloudflare-nginx" and re.search("Attention 
Required! \\| CloudFlare|One more step to access", body):
return True, "403-CLOUDFLARE"
if server == "cloudflare-nginx" and re.search("|", body):
# A customized captcha page.
return True, "403-CLOUDFLARE"
if server == "cloudflare-nginx" and re.search("Access denied \\| 
[^ ]* used CloudFlare to restrict access", body):
# With this one you don't get a captcha. May be controlled by the
# site operator.
return True, "403-CLOUDFLARE"
if status == 503:
if re.search("", body):
return True, "503-CLOUDFLARE"
I now think the 'server == "cloudflare-nginx"' tests are unnecessary.
The last two patterns above don't even give you a captcha to solve, just
deny access. You might want to limit your detection to 403 and 503
responses (or maybe exempt 200-series and 300-series responses).

These are a couple of sites that used customized CloudFlare:
https://4chan.org/ ("Verification Required")
https://yelp.com/ ("You're not barking up the wrong tree...")
yelp.com only started using CloudFlare a little while ago. It's a funny
case, because they *also* implement a hard Tor blacklist. Once you get
through the CloudFlare captcha 403, you get a 503 from a different
system.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Summary of meek's costs moved to tor-project list

[David Fifield]

I decided to move the meek cost emails to the tor-project list, because
they are more project-y than dev-y. Here is the email for January 2016:

https://lists.torproject.org/pipermail/tor-project/2016-February/000101.html

There's a table of all previous summaries here:

https://trac.torproject.org/projects/tor/wiki/doc/meek#Costs
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Configuration of tor relay using setup files (use of API via Tor Expert Bundle)

[David Fifield]

On Sun, Feb 07, 2016 at 03:44:35PM +, Nathan Bliss wrote:
> Is there a way to configure a bridge in tor (e.g. meek) via the config files
> from the command line without having to use the GUI in the Tor browser? I've
> been searching for documentation on this, so if I've missed it I would be
> grateful for a pointer to where this is in the docs... I found something about
> the torrc but this was an option for configuring a tor server node as a bridge
> whereas I want to configure the bridge as a client (in client mode).

Here is an example torrc for configuring a client:

https://gitweb.torproject.org/pluggable-transports/meek.git/tree/meek-client/torrc
Just comment out the "Bridge" lines other than the one you need. Then to
run it, you can do
tor -f torrc
Depending on your setup, you might have to provide the full paths to tor
or torrc. For example,
C:\path\whatever\tor.exe -f C:\path\whatever\torrc

If you need the web browser TLS camouflage (see
https://gitweb.torproject.org/pluggable-transports/meek.git/tree/README),
then it'll be easiest to run tor manually from inside the Tor Browser
directory tree with its torrc-defaults file. That will look something
like this:
Browser\TorBrowser\Tor\tor.exe --defaults-torrc 
Browser\TorBrowser\Data\Tor\torrc-defaults -f your-own-torrc
where "your-own-torrc" is a file containing the bridge you are using
(like in the previous paragraph).

> I need to do this in order to have automation scripts working with search
> engines such as google. Google always seems to be able to detect http
> connection via a script or via the tor interface running in standard mode. 
> Only
> a client connection via google-meek bridge seems to work and I'm looking for a
> way to do this via the tor stem API or via the config files.

This doesn't make sense to me. Are you saying that you're making client
connections through the Tor network to Google, and Google behaves
differently depending on whether you connect using a pluggable transport
or not? That seems unlikely. Web sites have no way of knowing what
pluggable transport you're using. It's more likely that you just got
lucky with your exit nodes while you were using meek-google. I suggest
you run some more experiments to confirm your guess.

In any event, if you are using meek for purposes other than
circumventing censorship, please take the time to set up your own App
Engine instance (it's easy) and pay for it yourself, otherwise you are
taking away bandwidth and capacity from actual censored users. To set up
your own instance, you will have to follow these instructions. Download
the "appengine" files from the meek source code, edit the file
"app.yaml" to use your App Engine app name, and then upload it with
"goapp deploy".
https://gitweb.torproject.org/pluggable-transports/meek.git/tree/appengine
https://gitweb.torproject.org/pluggable-transports/meek.git/tree/appengine/README
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Introducing Snowflake (webrtc pt)

[David Fifield]

On Mon, Jan 25, 2016 at 02:34:42PM -0800, Serene wrote:
> Snowflake is a webrtc pluggable transport inspired by flashproxy.
> (https://gitweb.torproject.org/pluggable-transports/snowflake.git)
> Arlo, David, and I have made lots of progress on it lately, and it now
> appears to have reached minimum viability.

I started a wiki page.
https://trac.torproject.org/projects/tor/wiki/doc/Snowflake
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Does Orbot use default obfs4 bridges?

[David Fifield]

On Tue, Jan 19, 2016 at 03:29:38PM -0500, Nathan Freitas wrote:
> 
> On Tue, Jan 19, 2016, at 02:52 PM, David Fifield wrote:
> > Does Orbot have a list of default built-in obfs4 bridges? Or do users
> > fetch them dynamically? I looked in the source code and found default
> > meek bridges but not default obfs4.
> 
> No we have no default Obfs4 bridges built-in. It has been on my list to
> get done, because we really should, obviously to make the user
> experience better, and to alleviate meek traffic.
> 
> > I'm asking because we recently added a few new high-capacity default
> > obfs4 bridges.
> > https://bugs.torproject.org/18071 (1 bridge)
> > https://bugs.torproject.org/18091 (2 bridges)
> 
> I will get this built-in ASAP.

The list of Tor Browser built-in bridges is here:
https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/Bundle-Data/PTConfigs/bridge_prefs.js
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Does Orbot use default obfs4 bridges?

[David Fifield]

Does Orbot have a list of default built-in obfs4 bridges? Or do users
fetch them dynamically? I looked in the source code and found default
meek bridges but not default obfs4.

I'm asking because we recently added a few new high-capacity default
obfs4 bridges.
https://bugs.torproject.org/18071 (1 bridge)
https://bugs.torproject.org/18091 (2 bridges)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] How many exits exit from an IP address different than their OR address? (10.7%)

[David Fifield]

On Sun, Jan 17, 2016 at 10:24:47PM +, cacahuatl wrote:
> On Sun, Jan 17, 2016 at 01:01:03PM +0100, coderman wrote:
> > misguided because it won't work as you expect, the right way to check
> > is to build circuits and see where they exit from. you can do this
> > yourself!
>
> Tor Project already does it for you with archived data and everything:
> https://collector.torproject.org/#exit-lists
> "The exit list service TorDNSEL publishes exit lists containing the IP
> addresses of relays that it found when exiting through them."

Oh, wow, thanks. I did not know that TorDNSEL was already providing this
service. Combined with 
https://collector.torproject.org/#type-network-status-consensus-3
it should be possible to do what I did with my exitmap scan for any time
in the past.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Questions about censorship detection paper

[David Fifield]

On Tue, Jan 12, 2016 at 11:49:19PM +, John wrote:
> Hi David,
> 
> Thank you, these pointers were very helpful. Do you know if there is
> some kind of resource that lists known censorship events? I'd like to
> see how good the approach from the paper does at identifying them.

For Tor-specific censorship, a list that is reasonably complete and up
to date is: http://www.eecs.berkeley.edu/~sa499/tor_timeline.pdf. It was
composed by mining the Tor issue tracker and blog for events.

For general censorship, I'm not aware of any systematic or complete list
of censorship events. There are quite a lot of research papers that
study one particular place for a period of time, and from them we know
of some incidents.

Here's a list of some measurement papers I'm aware of that might discuss
some events. You might also look at OpenNet and Freedom House reports.


Internet censorship in Iran: A first look
https://www.usenix.org/conference/foci13/workshop-program/presentation/aryan

The Anatomy of Web Censorship in Pakistan
https://www.usenix.org/conference/foci13/workshop-program/presentation/Nabi

Dimming the Internet: Detecting throttling as a mechanism of censorship in Iran
http://arxiv.org/abs/1306.4361

Global Network Interference Detection Over the RIPE Atlas Network
https://www.usenix.org/system/files/conference/foci14/foci14-anderson.pdf

Analysis of Country-wide Internet Outages Caused by Censorship
http://conferences.sigcomm.org/imc/2011/docs/p1.pdf

Characterizing Web Censorship Worldwide: Another Look at the OpenNet Initiative 
Data
http://censorbib.nymity.ch/pdf/Gill2015a.pdf
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Summary of meek's costs, December 2015

[David Fifield]

On Mon, Jan 11, 2016 at 02:51:12PM -0900, Jesse V wrote:
> On 01/11/2016 02:42 PM, David Fifield wrote:
> > We still have support from
> > Google, so that $561.29 actually costs about $61.29.
> 
> Oh, I was not aware of this. When does the support expire, and how much
> would it cost (in terms of Tor's budget) to double the bandwidth to 16
> Mbits, for instance? I'm just thinking about the implications if there
> were donations for meek.

It expires pretty soon. March, I think?

I don't know how much usage will increase with a higher rate limit, but
I'm sure that we could saturate 24 Mbps, because that's about where we
were before instituting rate limiting. I think of it the other way:
rather than picking a speed and figuring out what it will cost, set a
cost and then adjust your speed to meet it.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Questions about censorship detection paper

[David Fifield]

On Tue, Jan 12, 2016 at 07:21:39AM +, John wrote:
> I ran into the technical report from George Danezis about an
> anomaly-based censorship-detection system for Tor. I have a few
> questions that I hope you can help me with.
> 
> Is there an implementation available of the approach described in the paper?

Go to
https://metrics.torproject.org/userstats-relay-country.html
and check "Show possible censorship events if available".

I think the source code is here:
https://gitweb.torproject.org/karsten/metrics-web.git/tree/detector

> The paper talks about finding anomalies in the time series of the number
> of requests to directory servers from certain jurisdictions. Where can I
> find this data about the requests to directory servers?

https://collector.torproject.org/

I think you want the "dirreq-v3-reqs" lines. For example, here is a
recently archived extra-info consensus:
https://collector.torproject.org/recent/relay-descriptors/extra-infos/2016-01-12-07-05-27-extra-infos
(The file will expire after a few days; after that go to
https://collector.torproject.org/archive/relay-descriptors/extra-infos/.)

Here's a line from the file showing the number of requests by country
for one of the relays:
dirreq-v3-reqs 
us=7808,ru=4400,de=4112,fr=2584,gb=1848,it=1352,es=1208,jp=1192,br=1184,nl=1024,ua=1008,ca=912,pl=728,au=696,md=680,in=584,se=464,ar=448,mx=448,id=384,be=336,at=320,tr=288,tw=288,ch=280,cz=264,il=264,kr=248,ir=240,th=232,kz=224,pt=224,gr=216,sg=216,fi=208,my=208,ro=192,dk=184,hu=176,ae=168,ve=168,by=160,cl=160,ie=160,vn=160,za=152,ph=144,no=136,co=128,hk=128,lv=128,nz=120,sk=120,bg=112,eg=112,pk=112,rs=104,bd=88,ma=80,ec=72,ng=72,dz=64,hr=64,pe=64,si=64,ee=56,lt=56,do=48,tn=48,az=40,ba=40,cr=40,is=40,jo=40,kw=40,qa=40,uy=40,af=32,am=32,bo=32,ci=32,cy=32,gh=32,gt=32,iq=32,ke=32,lu=32,pr=32,uz=32,et=24,ge=24,lb=24,mk=24,mn=24,sa=24,sd=24,sv=24,sy=24,tm=24,al=16,bh=16,bj=16,cu=16,gd=16,kg=16,kh=16,ly=16,ni=16,np=16,om=16,ps=16,re=16,sn=16,tt=16,??=8,ad=8,ao=8,as=8,aw=8,ax=8,bb=8,bf=8,bm=8,bn=8,bs=8,bw=8,cd=8,cg=8,cm=8,cn=8,cv=8,dm=8,gf=8,gm=8,gp=8,gq=8,gw=8,gy=8,hn=8,ht=8,im=8,jm=8,km=8,ky=8,li=8,lk=8,me=8,mg=8,mq=8,mr=8,mt=8,mu=8,mz=8,na=8,nc=8,ne=8,pa=8,pf=8,pg=8,pm=8,p
 y=8,rw=8,sc=8,sm=8,sr=8,tj=8,tz=8,ug=8,vi=8,vu=8,ye=8,yt=8,zm=8,zw=8
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] How many exits exit from an IP address different than their OR address? (10.7%)

[David Fifield]

I wanted to know how many exits exit from an address that is different
from their OR address. The answer is about 10.7%, 109/1018 exits. The
interesting part is that of those 109 mismatches, 87 have an exit
address that differs from the OR address in all four octets; i.e., the
IP addresses used by the exit are not even in the same /8.

Also, there are several groups of exits whose OR addresses *are* in a
related subnet, but which all exit through the same, unrelated, IP
address. See for example 109.236.82.* (all exiting through
185.108.128.7), 178.17.171.* (all exiting through 37.48.65.71), and
217.23.11.* (all exiting through 85.17.31.120) in the table below.

I wrote an exitmap module that hits an IP-checker website. The module
wrote a CSV file where each row included the exit's OR address and the
address as reported by the remote web site. I ignored IPv6 OR addresses.
I did not find any fingerprint that had more than one IPv4 address
(considering the .fingerprint and .or_addresses members from Stem).
Percentages are just raw counts, not weighted by exit probability.


Total number of exits: 1018 (100.0%)
Number of failures: 18 (1.8%)
Number of matches: 891 (87.5%)
Total number of mismatches: 109 (10.7%)
Number of 1-octet mismatches: 19 (1.9%)
Number of 2-octet mismatches: 3 (0.3%)
Number of 3-octet mismatches: 0 (0.0%)
Number of 4-octet mismatches: 87 (8.5%)

Table of mismatches
   * = mismatch 1 octet
  ** = mismatch 2 octets
 *** = mismatch 3 octets
 = mismatch 4 octets
  fingerprint  or_address  remote_address mark
 B060482C784788B8A564DECD904E14CB305C8B38  176.10.104.241  176.10.104.240*
 DC41244B158D1420C98C66F7B5E569C09DCE98FE  176.10.104.241  176.10.104.240*
 487092BA36F4675F2312AA09AC0393D85DAD6145  176.10.104.244  176.10.104.243*
 DE7DE889E0D1A5F397AE35642060B84999581203  176.10.104.244  176.10.104.243*
 7A65CC8F45134F3393A0295EFFD2980DD885E8E2   176.123.6.157   176.123.6.155*
 A7E78D9880BB0793409D386D15E83B3B1236B19F  178.175.128.54  178.175.128.50*
 8BF22BEA5F854F2BB2E09C71260ADF590D355E36  213.61.149.125  213.61.149.100*
 80D73E75A30BEEF627604B7014753304764E0723  213.61.149.126  213.61.149.100*
 5D263037FC175596B3A344132B0B755EB8FB1D1C   31.185.27.203 31.185.27.1*
 C974508A98446F36169FB248655BCD50DF17F14C  37.130.227.134  37.130.227.133*
 170EC06D58E9094A027F4169514ADD98D30983A6   46.235.226.27  46.235.226.226*
 D3BE73ECE50DA9243DC0DF9DA6ED25027B82D38550.7.178.10150.7.178.100*
 FEDEF82551BB49DB54BE6C77D27AD0E20A8D6FD650.7.178.10250.7.178.100*
 10E13E340651D0EF66B4DEBF610B3C0981168107  77.247.181.164  77.247.181.162*
 06E123865C590189B3181114F23F0F13A7BC0E69  77.247.181.166  77.247.181.162*
 664BDC0344771122FC6C4F577BF0AEC3F4ED5456   80.73.242.142   80.73.242.130*
 9E3E47D7B92144CBAB0D487CCE192531F6A55833   81.209.35.111   81.209.35.112*
 410C286061266C562049178C6BF1E04060F51F7C   94.103.175.85   94.103.175.86*
 F4B72EA6FD0EACF652B6C200611F37244F2B31F3  94.242.228.107  94.242.228.108*
 62BEE61EB88D4A81C3BA3931D6FA999D706AC4D562.149.13.57   62.149.12.153   **
 1BC17B72F90998F0BAEE25E36FFC140C9D7A8D7A 85.119.82.485.119.84.57   **
 AA60D088E8317BBA3D2CF96C706AB4452FB2F57D  90.182.128.100   90.182.235.46   **
 4E915A5CCDBBDD4BD4B7F09A5AFB581AB1AD7EF2 107.181.178.204  104.156.228.72 
 03B4386E579BEBCF7605D0FB17A688B35C342D5D   109.236.82.50   185.108.128.7 
 C3B1E12BCA2F307B64737AB84993E7B414FD3D09   109.236.82.50   185.108.128.7 
 B9DC592CBC10EC7428730BF658F9211B8773800F   109.236.82.51   185.108.128.7 
 DE89B3B6954FFC0F76F6B37A05E2AD6EF3B046F1   109.236.82.51   185.108.128.7 
 CF9F5B88C0F95EB6C5547CE5FB38503049DF784D   109.236.82.52   185.108.128.7 
 57275A9356CFB1D36E64B117D44AB9F67711E483   109.236.82.53   185.108.128.7 
 D1E52741A7DCB4D51C7EFC82A654E2527516E1B0   109.236.82.53   185.108.128.7 
 7837F5B740223D19D8A0D3127D2A8EB9FDA59F67  148.163.73.112 206.217.208.162 
 B6045286F21E7078203E1B03C8E00669A2154724  148.163.73.112 206.217.208.162 
 D225C2EC81E044E3947BC49929E46003ED62C89F  151.80.164.147  46.166.186.238 
 23A69F271ACBFACB20467559D1C04F13DEF74B4B  153.92.126.135  128.127.105.94 
 4236DF494DC91E1245234AA676F1F993ECB067F5  153.92.126.135  128.127.105.94 
 27CE487BABE6C2128FB2D1A801C7CB715B48EC6B   153.92.126.19  168.1.6.34 
 C369EB6BFDFB996864746944544B82C9684C46EA   153.92.126.19  168.1.6.34 
 5453CAF677B8E754C4D5EFA1FEDD8ABB16950FCB  153.92.127.143168.1.10.226 
 756F68024C6E55FF7142AD933C135F88B552A932  153.92.127.143168.1.10.226 
 B0E7C83FA0C728997BB60CB0C758A3A3FDC3C1DB 173.243.112.148  162.216.46.165 
 E6BE2BEAAF4A543F9D94CD20CB459ECFEA3AD1B1 173.243.112.148  162.216.46.165 
 7585DDCEB14783C69960E777BA258F5B3948F1C9  176.126.85.175 159.122.133.198 
 CBBCD92AD9479C8FBB8AA17FF22C7EC206FF5B1A  176.126.85.175 159.122.133.198 
 

Re: [tor-dev] Go version in Gitian descriptors

[David Fifield]

On Sun, Jan 03, 2016 at 11:01:25PM -0600, Jeremy Rand wrote:
> I noticed that it looks like Tor Project is using Go 1.4.2 to build
> the pluggable transports in Gitian.  I'm curious why a newer version
> of Go isn't used.  My understanding is that Go 1.4.2 (or earlier) is
> needed to build Go 1.5 because 1.5's source code is itself in Go.
> Would using Go 1.5 be as simple as building 1.4.2 in Gitian (as is
> done now), and then using 1.4.2 to build 1.5, and then placing 1.5 in
> PATH instead of 1.4.2 as is done now?  Have obstacles been identified
> in such a configuration, or is it just that no one tried it?

It's just that nobody has tried it. Here's the ticket for the most
recent update (to 1.4.2) if you want to make a new patch and file a new
ticket:
https://bugs.torproject.org/15448
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Bitcoin-paid hidden meek relays?

[David Fifield]

On Thu, Dec 10, 2015 at 12:09:36PM +0100, Jacek Wielemborek wrote:
> List,
> 
> I'm starting this thread in relation to the following discussion:
> 
> https://security.stackexchange.com/questions/107490/how-will-france-block-tor
> 
> I definitely don't have a perfect knowledge of Tor, so correct me if I'm
> wrong at any point. I heard that Tor entry nodes right now are
> distributed to users as IP addresses, which are scarce resources that
> could be blocked.
> 
> In order to make this more difficult, pluggable transports were
> introduced - those try to use other protocols to define communication
> with Tor network, lowering the cost of introducing new way to connect
> with it.
> 
> One that particularly caught my attention is Meek, which uses CDNs to
> evade censorship. CDNs share same IPs for many critical Internet
> serices, so blocking them is not an option and having them behind TLS
> makes it even more complicated.
> 
> The problem I noticed though is that the costs of Meek go up and if I
> read the reports from David Fifield (the maintainer of Meek), the
> bandwidth has to be limited to avoid abuse. This slows the transport
> down and I thought of another approach.
> 
> Instead of using a limited amount of Meek nodes, we could encourage
> users to run those on their own and distribute them in a way similar to
> regular Tor entry nodes. The catch would be that those nodes would
> require a small Bitcoin pre-payment that covers the cost of bandwidth
> used. The node would require the user to pass some proof of payment
> during the connection and not respond at all if there was no payment
> made. Here's an example way this could work:
> 
> 1. User sends a message to brid...@bridges.torproject.org with a request
> for a meek bridge and a GPG public key attached,
> 2. The bridge service replies with a Bitcoin address created just for
> the user,
> 3. As soon as the user sends any Bitcoins, it receives another e-mail
> with the hostname of the Meek relay and a token that one should use
> 
> Together with Amazon AWS, this could be an automated solution to the
> bandwidth and payment problem that would result in short-lived relays
> that are difficult to detect.

I like the idea of bridges that users pay for according to their use.
There are a lot of unanswered questions about how exactly the payments
and accounting work. It seems to me that this is a case where the
details are very important (I could be wrong).

I know of some research along these lines. "Proof-of-Work as Anonymous
Micropayment: Rewarding a Tor Relay" by Alex Biryukov and Ivan
Pustogarov (they worked out some of the protocol details):
In this paper we propose a new micropayments scheme which can be
used to reward Tor relay operators. Tor clients do not pay Tor
relays with electronic cash directly but submit proof of work
shares which the relays can resubmit to a crypto-currency mining
pool. Relays credit users who submit shares with tickets that
can later be used to purchase improved service.

One misunderstood thing about meek: it doesn't require a CDN to work.
You can fairly easily upload your own meek proxy (as a PHP or Python
script) to any HTTPS web host. The CDN and its collateral damage are
only required when you make the proxies widely known, as we do with
meek-google, meek-amazon, and meek-azure in Tor Browser. If you run your
own and do not share it widely, then you do not need a CDN.

For example, here is the PHP version:
https://gitweb.torproject.org/pluggable-transports/meek.git/tree/php
Upload the index.php file to a web host service that supports HTTPS.
Then configure its URL as a bridge line in Tor Browser:
meek 0.0.2.0:4 url=https://mysite.example.com/index.php

It's conceivable that we could even distribute such bridge lines in
BridgeDB. People could upload their PHP files and we could have a lot of
meek bridges. But BridgeDB is not currently set up for that. I don't
know exactly how it works, but BridgeDB assumes that the bridge is
running at the same place as a full Tor relay, which isn't the case for
a PHP forwarder like this. We also don't have a way for such PHP proxies
to automatically self-publish their URLs for others to use.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Better relay uptime visualisation

[David Fifield]

On Mon, Dec 07, 2015 at 02:51:23PM -0500, Philipp Winter wrote:
> I spent some time improving the existing relay uptime visualisation [0].
> Inspired by a research paper [1], the new algorithm uses single-linkage
> clustering with Pearson's correlation coefficient as distance function.
> The idea is that relays are grouped next to each other if their uptime
> (basically a binary sequence) is highly correlated.  Check out the
> following gallery.  It contains monthly relay uptime images, dating back
> to 2007:
> 

How about just taking the XOR of two sequences as the distance?

It would be interesting to know if there are any near-perfect
anticorrelations; i.e., one relay starts when another stops.

> Another practical problem is that it's cumbersome to learn the relay
> fingerprint of a given column.  I'm looking into JavaScript/HTML tricks
> that can show text when you hover over a region in the image.  Perhaps
> somebody knows more?

One way is to set an onmousemove handler that inserts text into a
preexisting element. For example (untested):




var OUTPUT_ELEM = document.getElementById("output");
/* Get an event's coordinates relative to a given element. */
function elem_coords(event, elem) {
var rect = elem.getBoundingClientRect();
/* http://stackoverflow.com/a/872537 */
if (typeof pageXOffset !== "undefined") {
scrollLeft = pageXOffset;
scrollTop = pageYOffset;
} else if (document.documentElement !== undefined && 
document.documentElement.clientHeight !== undefined) {
scrollLeft = document.documentElement.scrollLeft;
scrollTop = document.documentElement.scrollTop;
} else {
scrollLeft = document.body.scrollLeft;
scrollTop = document.body.scrollTop;
}
var x = event.pageX - (scrollLeft + rect.left);
var y = event.pageY - (scrollTop + rect.top);
return { x: x, y: y };
}
function onmousemove_callback(event) {
var c = elem_coords(event, img_element);
OUTPUT_ELEM.innerText = get_text_for_coordinates(c.x, c.y);
}
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Better relay uptime visualisation

[David Fifield]

On Tue, Dec 08, 2015 at 10:47:08AM +1100, Tim Wilson-Brown - teor wrote:
> 
> On 8 Dec 2015, at 10:43, Tom Ritter <[1]t...@ritter.vg> wrote:
> 
> On 7 December 2015 at 13:51, Philipp Winter <[2]p...@nymity.ch> wrote:
> 
> I spent some time improving the existing relay uptime visualisation
> [0].
> Inspired by a research paper [1], the new algorithm uses 
> single-linkage
> clustering with Pearson's correlation coefficient as distance 
> function.
> The idea is that relays are grouped next to each other if their uptime
> (basically a binary sequence) is highly correlated.  Check out the
> following gallery.  It contains monthly relay uptime images, dating
> back
> to 2007:
> <[3]https://nymity.ch/sybilhunting/uptime-visualisation/>
> 
> If you aren't familiar with this type of visualisation: Every image
> shows the uptime of all Tor relays that were online in a given month.
> Every row is a consensus and every column is a relay.  White pixels
> mean
> that a relay was offline and black pixels means that a relay was
> online.  Red pixels are used to highlight suspiciously similar
> clusters.
> 
> 
> That's really cool.  It seems to imply that the majority of the tor
> network stop operating halfway through the month though... Do the
> other tor graphs take into account hibernating relays?  For example, I
> would expect the time-to-download graph would be somewhat affected:
> [4]https://metrics.torproject.org/torperf.html?graph=torperf=
> 2015-10-01=2015-10-31=all=5mb
> 
> 
> Hibernating relays run from the start of their first period to gauge load.
> Then they start at a random time during the day/month, but early enough that
> they think they'll still use all their bandwidth.
> 
> I wonder if we're seeing another phenomenon? (daily / monthly server 
> restarts?)
> Or we could be seeing hibernation failing to work as intended.

Relays turn on or off all the time. Of all the descriptors seen in a
year, less than 10% are continuously running the whole time. The rest
either started at some time or stopped at some time or both. See an
example here for 2014:

https://people.torproject.org/~dcf/graphs/microdescs/microdescs-2014-short.png
All we're seeing is the distributions of the dates at which the subset
of relays that stopped during the month actually stopped, which seems
pretty uniform. I'll bet that if you look at those relays in the
previous month, they are running at the end of the month, not
hibernating.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Graphs - Estimated Traffic Capacity

[David Fifield]

On Fri, Nov 20, 2015 at 01:38:56PM -0500, David Goulet wrote:
> Anyway, if you think this algorithm could be improved, please respond. If you
> think this algorithm is wrong, please respond. If you can reproduce the result
> on your own with this algo, omg please respond! :) The above could be totally
> wrong but hopefully we did a fairly good job. Please remember, this is an
> estimate. :)

I would be interested to see the capacity by pluggable transport: how
much headroom each PT has and which ones need more bridges.

I tried it a few months ago but I didn't have the idea of looking at the
read/write bytes history. I tried using measured/advertised bandwidth
and using the speed of consensus downloads from the
dirreq-v3-tunneled-dl line.

[tor-dev] Per-transport bridge bandwidth and bridge counts
https://lists.torproject.org/pipermail/tor-dev/2015-July/009136.html
Using measured bandwidth:
https://people.torproject.org/~dcf/graphs/pt-bandwidth-2015-07-20/pt-bandwidth.png
Using median dirreq-v3-tunneled-dl rate:
https://people.torproject.org/~dcf/graphs/pt-bandwidth-2015-07-20/pt-dl.md.png
Source code:
https://people.torproject.org/~dcf/graphs/pt-bandwidth-2015-07-20.tar.gz
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Summary of meek's costs, October 2015

[David Fifield]

On Fri, Nov 20, 2015 at 05:50:51PM -0600, Tom Ritter wrote:
> On 18 November 2015 at 16:32, David Fifield <da...@bamsoftware.com> wrote:
> > There was an unfortunate outage of meek-amazon (not the result of
> > censorship, just operations failure). Between 30 September and 9 October
> > the bridge had an expired HTTPS certificate.
> > [tor-talk] Outage of meek-amazon
> > 
> > https://lists.torproject.org/pipermail/tor-talk/2015-October/039231.html
> > 
> > https://lists.torproject.org/pipermail/tor-talk/2015-October/039234.html
> > And then, as a side effect of installing a new certificate, the bridge's
> > fingerprint changed, which caused Tor Browser to refuse to connect. It
> > used to be that we didn't include fingerprints for the meek bridges, but
> > now we do, so we didn't anticipate this error and didn't notice it
> > quickly.
> > Update the meek-amazon fingerprint to 
> > B9E7141C594AF25699E0079C1F0146F409495296
> > https://trac.torproject.org/projects/tor/ticket/17473
> > [tor-talk] Changed fingerprint for meek-amazon bridge (attn support)
> > 
> > https://lists.torproject.org/pipermail/tor-talk/2015-November/039397.html
> > Interestingly, the meek-amazon bridge still had about 400 simultaneous
> > users (not as much as normal) during the time when the fingerprint
> > didn't match. I would have expected it to go almost to zero. Maybe it's
> > people using an old version of Tor Browser (from before March 2015) or
> > some non–Tor Browser installation.
> 
> It seems like it would be better to use the SPKI rather than the cert
> fingerprint, this would allow you to reissue the same key and keep
> things working for older clients.

The fingerprint I'm talking about is the relay fingerprint, not the
HTTPS/X.509 one. The HTTPS certificate and the relay identity
fingerprint are completely independent. It just happened that in this
case, the relay was so configured, that when it rebooted to start using
the new HTTPS cert, it also generated a new identity key.

We're not pinning the HTTPS cert and in fact we can't; it's just used
for confidentiality on the CDN↔meek-server link.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Summary of meek's costs, October 2015

[David Fifield]

Here's the summary of meek's CDN fees for October 2015.

App Engine +   Amazon +   Azure = total by month
February 2014$0.09 +   -- +  -- = $0.09
March 2014   $0.00 +   -- +  -- = $0.00
April 2014   $0.73 +   -- +  -- = $0.73
May 2014 $0.69 +   -- +  -- = $0.69
June 2014$0.65 +   -- +  -- = $0.65
July 2014$0.56 +$0.00 +  -- = $0.56
August 2014  $1.56 +$3.10 +  -- = $4.66
September 2014   $4.02 +$4.59 +   $0.00 = $8.61
October 2014$40.85 +  $130.29 +   $0.00 =   $171.14
November 2014  $224.67 +  $362.60 +   $0.00 =   $587.27
December 2014  $326.81 +  $417.31 +   $0.00 =   $744.12
January 2015   $464.37 +  $669.02 +   $0.00 =  $1133.39
February 2015  $650.53 +  $604.83 +   $0.00 =  $1255.36
March 2015 $690.29 +  $815.68 +   $0.00 =  $1505.97
April 2015 $886.43 +  $785.37 +   $0.00 =  $1671.80
May 2015   $871.64 +  $896.39 +   $0.00 =  $1768.03
June 2015  $601.83 +  $820.00 +   $0.00 =  $1421.83
July 2015  $732.01 +  $837.08 +   $0.00 =  $1569.09
August 2015$656.76 +  $819.59 + $154.89 =  $1631.24
September 2015 $617.08 +  $710.75 + $490.58 =  $1818.41
October 2015   $672.01 +  $110.72 + $300.64 =  $1083.37
--
total by CDN  $7443.58 + $7987.32 + $946.11 = $16377.01 grand total

https://metrics.torproject.org/userstats-bridge-transport.html?graph=userstats-bridge-transport=2015-08-01=2015-10-31=meek

In October we had about 4,000 simultaneous users all month.

There was an unfortunate outage of meek-amazon (not the result of
censorship, just operations failure). Between 30 September and 9 October
the bridge had an expired HTTPS certificate. 
[tor-talk] Outage of meek-amazon
https://lists.torproject.org/pipermail/tor-talk/2015-October/039231.html
https://lists.torproject.org/pipermail/tor-talk/2015-October/039234.html
And then, as a side effect of installing a new certificate, the bridge's
fingerprint changed, which caused Tor Browser to refuse to connect. It
used to be that we didn't include fingerprints for the meek bridges, but
now we do, so we didn't anticipate this error and didn't notice it
quickly.
Update the meek-amazon fingerprint to 
B9E7141C594AF25699E0079C1F0146F409495296
https://trac.torproject.org/projects/tor/ticket/17473
[tor-talk] Changed fingerprint for meek-amazon bridge (attn support)

https://lists.torproject.org/pipermail/tor-talk/2015-November/039397.html
Interestingly, the meek-amazon bridge still had about 400 simultaneous
users (not as much as normal) during the time when the fingerprint
didn't match. I would have expected it to go almost to zero. Maybe it's
people using an old version of Tor Browser (from before March 2015) or
some non–Tor Browser installation.

Our grant for meek-azure ran out and now it costs money. Accordingly
I've rate-limited it to limit costs. I set it to 1.1 MB/s on 2 October
and to 0.8 MB/s on 30 October.
[tor-talk] meek-azure now rate-limited
https://lists.torproject.org/pipermail/tor-talk/2015-October/039169.html

meek-google was also out for a few days around 30 October because I
messed up the app upload and pointed it to the wrong bridge.

If you want to set up your own bridge and CDN instance without rate
limiting, I can help you do it. Here are some docs to look at:
https://trac.torproject.org/projects/tor/wiki/doc/meek#Howtorunameek-serverbridge
https://trac.torproject.org/projects/tor/wiki/doc/meek#GoogleAppEngine
https://trac.torproject.org/projects/tor/wiki/doc/meek#AmazonCloudFront
https://trac.torproject.org/projects/tor/wiki/doc/meek#MicrosoftAzure


== App Engine a.k.a. meek-google ==

Here is how the Google costs broke down:
2842 GB $341.06
6619 instance hours $330.95
Compared to the previous month:
2871 GB $344.53
5451 instance hours $272.55

https://globe.torproject.org/#/bridge/88F745840F47CE0C6A4FE61D827950B06F9E4534


== Amazon a.k.a. meek-amazon ==

Usage of meek-amazon was quite low this month because of bridge outages
having to do with an expired HTTPS certificate and a changed relay
fingerprint.

Asia Pacific (Singapore)   6M requests  $8.24   36 GB  $5.05
Asia Pacific (Sydney)635K requests  $0.791 GB  $0.21
Asia Pacific (Tokyo)   1M requests  $1.715 GB  $0.71
EU (Ireland)  39M requests $46.81  130 GB  $9.97
South America (Sao Paulo)  2M requests  $4.985 GB  $1.23
US East (Northern Virginia)   24M requests $24.36   86 GB  $6.64
--
total 74M requests $86.89  266 GB $23.81


https://globe.torproject.org/#/bridge/F4AD82B2032EDEF6C02C5A529C42CFAFE516564D
(Note new fingerprint.)


== Azure a.k.a. meek-azure ==

Zone 1  1652 GB  $202.51
Zone 2   586 GB   $98.13
--
total   2238 GB  $300.64

https://globe.torproject.org/#/bridge/AA033EEB61601B2B7312D89B62AAA23DC3ED8A34