Re: [tor-relays] DDOS alerts from my provider
I haven't read it yet, but there's a short paper at FOCI this year analyzing a case study of a DDoS attack on relays operated by the authors. "A case study on DDoS attacks against Tor relays" Tobias Höller, René Mairhofer https://www.petsymposium.org/foci/2024/foci-2024-0014.php On Mon, Jul 08, 2024 at 07:34:51PM +0200, Rafo (r4fo.com) via tor-relays wrote: > I have been running a relay for a few months now without any problems. But > this > week I’ve received 2 DDoS alerts from my provider (Netcup), both are ~3 > gigabits. They seem to be coming from other Tor relays. > I’m running an Invidious like instance on my server (which uses around 600 > megabits) but I have a 2.5 gigabit port. So I configured my Tor relay to use > 300-400 megabits. > I’m not sure where that 3 gigabit of data comes from. > I have lowered my advertised bandwidth to 100 megabits, would that be enough > to > prevent these kind of issues? ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Running a high-performance pluggable transports Tor bridge (FOCI 2023 short paper)
On Mon, Dec 11, 2023 at 08:13:17PM +0100, Felix wrote: > Thank you for the paper and the presentation. > > Chapter 3 (Multiple Tor processes) shows the structure: > > > mypt - HAproxy = multiple tor services > > At the end of chapter 3.1 it is written > > the loss of country- and transport-specific metrics > > How will the metrics data be pulled out of the multiple tor services to > fetch *all* metrics data? Or will only one of them be looked at, without > full data representation? The key is that every instance of tor must have a different nickname. That way, even though they all have the same relay identity key, Tor Metrics knows to count all the descriptors separately. So, for instance, on one snowflake bridge (identity 2B280B23E1107BB62ABFC40DDCC8824814F80A72), we use nicknames: flakey1, flakey2, …, flakey12 and on another bridge (identity 8838024498816A039FCBBAB14E6F40A0843051FA) we use nicknames: crusty1, crusty2, …, crusty12 Instructions for setting up nicknames can be found at https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-Installation-Guide#tor It used to be the case that Tor Metrics did not understand the descriptors of this kind of multi-instance bridge. If you had N instances, it would count only 1 of them per time period. But Tor Metrics has now known about this kind of bridge (multiple descriptors per time period with the same identity key but different nicknames) for more than a year: https://gitlab.torproject.org/tpo/network-health/metrics/website/-/issues/40047 https://gitlab.torproject.org/tpo/network-health/metrics/website/-/merge_requests/42 Relay Search still does not know about multi-instance bridges, though. If you look up such a bridge, it will display one of the multiple instances more or less at random. In the case of the current snowflake bridges, you have to multiply the numbers on Relay Search pages by 12 to get the right numbers. https://metrics.torproject.org/rs.html#details/2B280B23E1107BB62ABFC40DDCC8824814F80A72 https://metrics.torproject.org/rs.html#details/8838024498816A039FCBBAB14E6F40A0843051FA There's a special repository for making graphs of snowflake users. This was necessary in the time before Tor Metrics natively understood multi-instance bridges, and I still use it because it offers some extra flexibility over what metrics.torproject.org provides. With some small changes, the same code could work for other pluggable transports, or even single bridges. https://gitlab.torproject.org/dcf/snowflake-graphs This is a sample of the graph output: https://forum.torproject.org/t/snowflake-daily-operations-november-2023-update/10575 ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Running a high-performance pluggable transports Tor bridge (FOCI 2023 short paper)
On Mon, Sep 04, 2023 at 02:09:50AM -0600, David Fifield wrote: > Linus Nordberg and I wrote a short paper that was presented at FOCI > 2023. The topic is how to use all the available CPU capacity of a server > running a Tor relay. > > This is how the Snowflake bridges are set up. It might also be useful > for anyone running a relay that is bottleneck on the CPU. If you have > ever run multiple relays on one IP address for better scaling (if you > are one of the relay operators affected by the recent > AuthDirMaxServersPerAddr change), you might want to experiment with this > setup. The difference is that all the instances of Tor have the same > relay fingerprint, so they operate like one big relay instead of many > small relays. > > https://www.bamsoftware.com/papers/pt-bridge-hiperf/ The workshop presentation video (22 minutes) of this paper has just become available on YouTube. The paper homepage has a copy of the video too. https://www.youtube.com/watch?v=UkUQsAJB-bg&list=PLWSQygNuIsPc8bOJ2szOblMK4i6T79S1m&index=5 The other FOCI 2023 issue 2 videos are online as well: https://www.youtube.com/playlist?list=PLWSQygNuIsPc8bOJ2szOblMK4i6T79S1m ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Bridge with iat-mode=2 very slow
On Tue, Sep 26, 2023 at 02:22:06PM +, Split via tor-relays wrote: > I run the obfs4 bridge, in the parameters I specify to use iat-modr=2. As a > result, the bridge is VERY, VERY SLOW. Connection speed is on average 100 kb/ > sec. When I remove the iat-mode=2 parameter, the speed becomes 8-10 times > higher. > > I checked the processor load, it is at 3%, the problem is unlikely to be with > the processor, the RAM load is at 20%. I can't figure out if this is expected > behavior or if the speed drop is too significant. Please tell me, is this the > normal maximum speed of the bridge with iat-mode=2 or can it be increased > somehow? I don't know what's a typical bandwidth range to expect, but it's expected that iat-mode=2 is much slower than normal. Use iat-mode=1 for less aggressive traffic shaping. Some old visualizations of iat-mode's effect on packet sizes and timing: https://www.bamsoftware.com/talks/pets-2017-menace/#figures ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Quick bugfix sharing regarding obfs4 malfunctioning
On Thu, Sep 07, 2023 at 02:12:36PM +0200, telekobold wrote: > I just want to share some quick bugfix with you (sorry if this is obvious to > you or has been written somewhere else). > > Suddenly, I got the following error messages on my two bridges running on > Debian 11 appearing in the logs (in /var/log/tor/notices.log and in the nyx > output) every second until a restart: > > [warn] Managed proxy "/usr/bin/obfs4proxy" process terminated > with status code 65280 > [warn] Server managed proxy encountered a method error. (obfs4 > listen tcp 0.0.0.0:443: bind: permission denied) > [warn] Managed proxy '/usr/bin/obfs4proxy' was spawned > successfully, but it didn't launch any pluggable transport listeners! > > When restarting the corresponding bridge, in the startup process the second > and the third of the above warning messages again appeared in the logs. So > obfs4 was suddenly not usable any more. Port 443 is not blocked in the > bridge's firewalls. > > A bit research reveled that apparently, an automatic update set the systemd > setting "NoNewPrivileges=no" in /lib/systemd/system/tor@default.service and > tor@.service [1] back to yes, which caused the above issue. After setting it > back and restarting, everything works fine now and instead of the warning > messages mentioned above, the following message appears in the log again: > > [notice] Registered server transport 'obfs4' at '[::]:443' There's a better way to set `NoNewPrivileges=no` that will not get overwritten in an upgrade. Use a systemd override: https://bugs.torproject.org/tpo/core/tor/18356#note_2439960 ``` systemctl edit tor@.service tor@default.service ``` Enter this text in both editors that appear: ``` [Service] NoNewPrivileges=no ``` Then run ``` service tor restart ``` This will create files /etc/systemd/system/tor@.service.d/override.conf and /etc/systemd/system/tor@default.service.d/override.conf that will not be overwritten in an upgrade. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
[tor-relays] Running a high-performance pluggable transports Tor bridge (FOCI 2023 short paper)
Linus Nordberg and I wrote a short paper that was presented at FOCI 2023. The topic is how to use all the available CPU capacity of a server running a Tor relay. This is how the Snowflake bridges are set up. It might also be useful for anyone running a relay that is bottleneck on the CPU. If you have ever run multiple relays on one IP address for better scaling (if you are one of the relay operators affected by the recent AuthDirMaxServersPerAddr change), you might want to experiment with this setup. The difference is that all the instances of Tor have the same relay fingerprint, so they operate like one big relay instead of many small relays. https://www.bamsoftware.com/papers/pt-bridge-hiperf/ > The pluggable transports model in Tor separates the concerns of > anonymity and circumvention by running circumvention code in a > separate process, which exchanges information with the main Tor > process over local interprocess communication. This model leads to > problems with scaling, especially for transports, like meek and > Snowflake, whose blocking resistance does not rely on there being > numerous, independently administered bridges, but which rather forward > all traffic to one or a few centralized bridges. We identify what > bottlenecks arise as a bridge scales from 500 to 10,000 simultaneous > users, and then from 10,000 to 50,000, and show ways of overcoming > them, based on our experience running a Snowflake bridge. The key idea > is running multiple Tor processes in parallel on the bridge host, with > externally synchronized identity keys. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] (Announcement) WebTunnel, a new pluggable transport for bridges, now available for deployment
On Thu, Jun 29, 2023 at 03:38:13PM +0100, Shelikhoo wrote: > How to test and report issues > - > > You can test the WebTunnel bridge by using the most recent version of Tor > Browser Alpha (https://www.torproject.org/download/alpha/). Currently, > WebTunnel is only distributed over the HTTPS distributor (torrc > setting:'BridgeDistribution https'). > > You can report issues on the Tor Project GitLab Anti-censorship group: > https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/webtunnel. > > Given that this new PT is only available now on Tor Browser Alpha, relay > operators should not expect significant usage or a large number of users at > the > moment. > > Please let us know if you encountered any difficulty with WebTunnel setup. > Thanks for your contribution to the Tor ecosystem. I tried getting a bridge line from https://bridges.torproject.org/bridges/?transport=webtunnel but it gave me an error: "It seems there was an error getting your QRCode." Are there just not enough bridges for the distributor to distribute yet? ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Security implications of disabling onion key rotation?
On Thu, Jun 01, 2023 at 01:21:30PM -0400, Roger Dingledine wrote: > Thanks Nick! I endorse Nick's response, with two additions: > > On Thu, Jun 01, 2023 at 09:07:17AM -0400, Nick Mathewson wrote: > > Onion key rotation limits the time range in which this kind of attack > > is useful: it will only work for as long as the onion key is listed in > > a live directory. > > For bridges it is a little bit different, because bridges don't have > an onion key listed in any public (consensus style) directory document > that clients get. Rather, the client connects to the bridge directly and > fetches a full timestamped descriptor from the bridge, which is signed > by the bridge's identity key, and which includes the onion key that the > client should use. Thanks, that was a subtlety I had missed. Since we are writing about bridges, I mostly want to give the bridge perspective. We had formerly written this: A relay's current onion keys appear in the Tor network consensus; when clients make circuits through it, they expect it to use certain onion keys. We've now changed it to: Tor clients cache a bridge's onion public keys when they connect; subsequent connections only work if the cached keys are among the bridge's two most recently used sets of onion keys. Here's my old post when I tested what would happen if a client cached one onion key on the first attempt and then the onion key was not the same on the second attempt: https://lists.torproject.org/pipermail/tor-relays/2022-January/020238.html > So if you have broken an old (rotated) onion key for a bridge, the > proper attack involves MITMing the connection to the bridge, breaking > or stealing the bridge's identity key too, and crafting a new descriptor > that lists the old onion key. > > Whereas if the bridge never rotates the onion key, then you would be > able to successfully attack the CREATE cell that the client sends to > the bridge -- but only if you could see it, which would involve MITMing > the connection to the bridge and also being able to convince the client > that you are the bridge, which I think implies having or breaking the > identity key too. Doesn't seem so bad. So it sounds like compromise of an onion key is no worse than compromise of an identity key, because with an identity key an attacker could cook up and sign a new onion key. The exception is that if an attacker somehow got an identity key but not current onion keys, and it's a bridge that's affected rather than a relay, then the attacker would not be able to fool clients that had previously connected and cached the past genuine onion keys. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Security implications of disabling onion key rotation?
On Thu, Jun 01, 2023 at 09:07:17AM -0400, Nick Mathewson wrote: > On Wed, May 24, 2023 at 8:54 PM David Fifield wrote: > [...] > > > > What are the risks of not rotating onion keys? My understanding is that > > rotation is meant to enhance forward security; i.e., limit how far back > > in time past recorded connections can be attacked in the case of key > > compromise. https://spec.torproject.org/tor-design Section 4 says: > > Short-term keys are rotated periodically and independently, to > > limit the impact of key compromise. > > This is an interesting question! > > So, compromising an onion key shouldn't be enough on its own to break > forward secrecy. The circuit extension handshakes use an additional > set of ephemeral keys as part of the negotiation process, which are > discarded immediately after the handshake. (This is the > diffie-hellman keys in TAP, and the x/X y/Y keypairs in ntor.) > Assuming that this is done properly, and all the cryptographic > assumptions hold, these keys alone should make it impossible to > decrypt anything after the session keys are discarded. > > The purpose of the onion key is, rather, to make it impossible for > somebody else to impersonate the target relay. If somebody steals > your onion key, and they have their own relay R, then they can use > your onion key to impersonate you whenever somebody tries to extend a > circuit from R to you. > > Onion key rotation limits the time range in which this kind of attack > is useful: it will only work for as long as the onion key is listed in > a live directory. > > (Now, any attacker who can steal your onion key can probably also > steal your identity key too, if you don't keep that offline, and use > it to impersonate you for even longer. The advantage of using a stolen > onion key is that it's much harder to detect; all the attacks I can > think of that use a stolen identity key involve, whereas the > onion-key-theft attack occurs when you are already in a perfect > position to be a MITM.) Thanks, that helps. If I understand correctly, compromise of an onion key allows an attacker to impersonate the relay because it is effectively the relay's "identity" as far as CREATE cells and circuit_send_first_onion_skin are concerned; i.e., the public onion key is the "B" in the ntor handshake, in which the relay's actual long-term identity key doesn't play a role. The only way the identity keys figure into it is that they (via the signing keys) sign the consensus documents that inform clients what onion keys to expect. The way I'm planning to summarize this is that, with onion key rotation disabled, you need to treat the now long-term onion keys as if they were long-term identity keys. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
[tor-relays] Security implications of disabling onion key rotation?
Linus Nordberg and I have had a paper accepted to FOCI 2023 on the special pluggable transports configuration used on the Snowflake bridges. That design was first hashed out on this mailing list last year. https://forum.torproject.net/t/tor-relays-how-to-reduce-tor-cpu-load-on-a-single-bridge/1483/11 https://github.com/net4people/bbs/issues/103 https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival%20Guides/Snowflake%20Bridge%20Installation%20Guide There is a draft of the paper here: https://www.bamsoftware.com/papers/pt-bridge-hiperf/pt-bridge-hiperf.20230307.pdf https://www.bamsoftware.com/papers/pt-bridge-hiperf/pt-bridge-hiperf.20230307.tex A question that more than one reviewer asked is, what are the security implications of disabling onion key rotation as we do? (Section 3.2 in the draft.) It's a good question and one we'd like to address in the final draft. What are the risks of not rotating onion keys? My understanding is that rotation is meant to enhance forward security; i.e., limit how far back in time past recorded connections can be attacked in the case of key compromise. https://spec.torproject.org/tor-design Section 4 says: Short-term keys are rotated periodically and independently, to limit the impact of key compromise. Do the considerations differ when using ntor keys versus TAP keys? ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] inet_csk_bind_conflict
On Mon, Dec 12, 2022 at 10:18:53PM +0100, Anders Trier Olesen wrote: > > It is surprising, isn't it? It certainly feels like calling connect > > without first binding to an address should have the same effect as > > manually binding to an address and then calling connect, especially if > > the address you bind to is the same as the kernel would have chosen > > automatically. It seems like it might be a bug, but I'm not qualified to > > judge that. > > Yes, I'm starting to think so too. And strange that Cloudflare doesn't mention > stumbling upon this problem in their blogpost on running out of ephemeral > ports. [1] > [1]https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections > If I find the time, I'll make an attempt at understanding exactly what is > going > on in the kernel. Cloudflare has another blog post today that gets into this topic. https://blog.cloudflare.com/the-quantum-state-of-a-tcp-port/ It investigates the difference in behavior between inet_csk_bind_conflict and __inet_hash_connect that I commented on at https://forum.torproject.net/t/tor-relays-inet-csk-bind-conflict/5757/13 and https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40201#note_2839472. Setting the IP_BIND_ADDRESS_NO_PORT option leads to __inet_hash_connect; not setting it leads to inet_csk_bind_conflict. The author attributes the difference in behavior to the fastreuse field in the bind hash bucket: > The bucket might already exist or we might have to create it first. > But once it exists, its fastreuse field is in one of three possible > states: -1, 0, or +1. > > … > > …inet_csk_get_port() skips conflict check for fastreuse == 1 buckets. > …__inet_hash_connect() skips buckets with fastreuse != -1. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
[tor-relays] Draft workshop submission "Running a high-performance pluggable transports Tor bridge"
Linus Nordberg and I have been working together to run the main Snowflake bridge since April 2022. We are preparing a short paper (4 pages) for the FOCI workshop (https://foci.community/) on the special procedures required to operate a bridge that gets the large volume of traffic that a Snowflake bridge does. This is a draft of the submission: https://www.bamsoftware.com/papers/pt-bridge-hiperf/pt-bridge-hiperf.20230307.pdf https://www.bamsoftware.com/papers/pt-bridge-hiperf/pt-bridge-hiperf.20230307.tex I am posting the draft here to see if anyone has feedback or comments. If you do, our submission deadline is 2023-03-15. The main ideas were worked out in a thread on this very list: https://forum.torproject.net/t/tor-relays-how-to-reduce-tor-cpu-load-on-a-single-bridge/1483 The core problem is that a single tor process is limited to a single CPU core. The main fix is to run multiple tor processes, and synchronize their identity and onion keys externally. It has been essential to enable scaling in the Snowflake bridge; but the same trick may be useful (minus the pluggable transport part) for other relays that process a much larger amount of traffic than average, such as large exit relays. (Though it's not clear that one exit with 2× the capacity is really any different than 2 exits in the same family.) This is also somewhat related to the current discussion about increasing the limit on the number of relays per IP address, which is a concession to the limited scalability of one tor process: https://bugs.torproject.org/tpo/core/tor/40744 > While this is great in general, it also adds significant complexity > for Tor operators since Tor does not scale well on multi-core CPU's. > In order to effectively use a modern server with a large amount of > threads, you would need to run many Tor relays. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
On Fri, Dec 16, 2022 at 04:27:06AM +, Gary C. New via tor-relays wrote: > On Tuesday, December 13, 2022, 07:35:23 PM MST, David Fifield > wrote: > > On Tue, Dec 13, 2022 at 07:29:45PM +, Gary C. New via tor-relays wrote: > >> On Tuesday, December 13, 2022, 10:11:41 AM PST, David Fifield > >> wrote: > >> > >> Am I correct in assuming extor-static-cookie is only useful within the > >> context > >> of bridging connections between snowflake-server and tor (not as a > >> pluggable > >> transport similar to obfs4proxy)? > > > That's correct. extor-static-cookie is a workaround for a technical > > problem with tor's Extended ORPort. It serves a narrow and specialized > > purpose. It happens to use the normal pluggable transports machinery, > > but it is not a circumvention transport on its own. It's strictly for > > interprocess communication and is not exposed to the Internet. You don't > > need it to run a Snowflake proxy. > > Created a Makefile for extra-static-cookie for OpenWRT and Entware: > > https://forum.openwrt.org/t/extor-static-cookie-makefile/145694 I appreciate the enthusiasm, but I should reiterate: there is no reason to ever use this tool on OpenWRT. Packaging it is a mistake. If you think you need it, you misunderstand what it is for. > > I am not sure what your plans are with running multiple obfs4proxy, but > > if you just want multiple obfs4 listeners, with different keys, running > > on different ports on the same host, you don't need a load balancer, > > extor-static-cookie, or any of that. Just run multiple instances of tor, > > each with its corresponding instance of obfs4proxy. The separate > > instances don't need any coordination or communication. > > The goal of running multiple obfs4proxy listeners is to offer numerous, unique > bridges distributed across several servers maximizing resources and > availability. If the purpose is running on several different servers, you don't need a load balancer and you don't need extor-static-cookie. Those tools are meant for running *one* instance of a pluggable transport on *one* server. If you want to distribute bridges over multiple servers, just run one instance each of tor and obfs4proxy on multiple servers, in the normal way. You don't need anything fancy. > > You could, in principle, use the same load-balanced setup with > > obfs4proxy, but I expect that a normal bridge will not get enough users > > to justify it. It only makes sense when the tor process hits 100% CPU > > and becomes a bottleneck, which for the Snowflake bridge only started > > to happen at around 6,000 simultaneous users. > > Hmm... If normal bridges will not see enough users to justify the deployment > of numerous, unique bridges distributed over several servers--this may be a > deciding factor. I don't have enough experience with normal bridges to know. Some pluggable transports, like obfs4, need there to be many bridges, because they are vulnerable to being blocked by IP address. Each individual bridge does not get much traffic, because there are so many of them. With obfs4, it's not about load, it's about address diversity. Just run multiple independent bridges if you want to increase your contribution. Snowflake is unlike obfs4 in that it does not depends on there being multiple bridges for its blocking resistance. Snowflake gets its address diversity at a different layer—the Snowflake proxies. There are many proxies, but there only needs to be one bridge. However, that one bridge, because it receives the concentrated traffic of many users, needs special scaling techniques. > >> What about a connection flow of haproxy/nginx => (snowflake-server => > >> extor-static-cookie => tor) on separate servers? > > > You have the order wrong (it's snowflake-server → haproxy → > > extor-static-cookie → tor), but yes, you could divide the chain at any > > of the arrows and run things on different hosts. You could also run half > > the extor-static-cookie + tor on one host and half on another, etc. > > I've installed and started configuring snowflake-server and have some > questions > after reading the README: > > In short, I'm trying to get a sense of whether it makes sense to run a > Snowflake Bridge and Normal Bridge on the same public addresses? There is no reason at all to run a Snowflake bridge. No user will ever connect to it, because Snowflake bridges are not distributed through BridgeDB like obfs4 bridges are; they are shipping in configuration files with Tor Browser or Orbot. There is no need for volunteers to run Snowflake bridges, and no benefit to them doing so. If you w
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
On Tue, Dec 13, 2022 at 07:29:45PM +, Gary C. New via tor-relays wrote: > On Tuesday, December 13, 2022, 10:11:41 AM PST, David Fifield > wrote: > > > The Snowflake proxy is not a pluggable transport. You just > run it as a > > normal command-line program. There is no torrc involved, and the proxy > > does not interact with a tor process at all. > > Thank you for the clarification. It seems I incorrectly assumed that > extor-static-cookie was a wrapper for snowflake-proxy. > > "To work around this problem, there is a shim called [1]extor-static-cookie > that presents an ExtORPort with a fixed, unchanging authentication key on a > static port, and forwards the connections (again as ExtORPort) to tor, using > that instance of tor's authentication key on an ephemeral port. One > extor-static-cookie process is run per instance of tor, using > ServerTransportPlugin and ServerTransportListenAddr." > > Am I correct in assuming extor-static-cookie is only useful within the context > of bridging connections between snowflake-server and tor (not as a pluggable > transport similar to obfs4proxy)? That's correct. extor-static-cookie is a workaround for a technical problem with tor's Extended ORPort. It serves a narrow and specialized purpose. It happens to use the normal pluggable transports machinery, but it is not a circumvention transport on its own. It's strictly for interprocess communication and is not exposed to the Internet. You don't need it to run a Snowflake proxy. I am not sure what your plans are with running multiple obfs4proxy, but if you just want multiple obfs4 listeners, with different keys, running on different ports on the same host, you don't need a load balancer, extor-static-cookie, or any of that. Just run multiple instances of tor, each with its corresponding instance of obfs4proxy. The separate instances don't need any coordination or communication. The reason for all the complexity in the Snowflake is that there is *one* instance of the pluggable transport (snowflake-server) that needs to communicate with N instances of tor. And the only reason for doing that is that tor (C-tor) doesn't scale beyond one CPU. If tor could use more than one CPU (as we hope Arti will), we would not need or use any of these workarounds. You could, in principle, use the same load-balanced setup with obfs4proxy, but I expect that a normal bridge will not get enough users to justify it. It only makes sense when the tor process hits 100% CPU and becomes a bottleneck, which for the Snowflake bridge only started to happen at around 6,000 simultaneous users. > What about a connection flow of haproxy/nginx => (snowflake-server => > extor-static-cookie => tor) on separate servers? You have the order wrong (it's snowflake-server → haproxy → extor-static-cookie → tor), but yes, you could divide the chain at any of the arrows and run things on different hosts. You could also run half the extor-static-cookie + tor on one host and half on another, etc. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
On Mon, Dec 12, 2022 at 08:19:53PM +, Gary C. New via tor-relays wrote: > I am having some issues or misunderstandings with implementing Snowflake Proxy > within Tor. I assumed that implementing Snowflake Proxy within Tor would be > similar to OBFS4Bridge in that Tor would initialize Snowflake Proxy as a > managed Pluggable Transport listening on the assigned > ServerTransportListenAddr. I can see Snowflake Proxy initiate outbound > requests, but I don't see it listen on the specified ServerTransportListenAddr > and Port. The Snowflake proxy is not a pluggable transport. You just run it as a normal command-line program. There is no torrc involved, and the proxy does not interact with a tor process at all. Unlike, say, obfs4, in Snowflake the bridges are centralized and the proxies are decentralized. If you run a proxy you don't also run a bridge. If it helps the mental model: the standalone proxy program in Snowflake does exactly the same thing as the browser extension proxy (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake-webext). Those browser proxies don't have an attached tor process; neither does the command-line proxy. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
On Sun, Dec 11, 2022 at 04:25:06AM +, Gary C. New via tor-relays wrote: > I was successfully able to get Snowflake cross-compiled and installed for > OpenWRT and Entware as a package. Thanks, nice work. > # opkg files snowflake > Package snowflake (2.4.1-1) is installed on root and has the following files: > /opt/bin/proxy > /opt/bin/client > /opt/bin/probetest > /opt/bin/broker > /opt/bin/server > /opt/bin/distinctcounter I don't think it makes sense to package the server or broker for OpenWRT. The client and proxy, sure. But the server and broker do not even run on the same host in an actual deployment. distinctcounter is just a metrics utility for the broker: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/95 ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] inet_csk_bind_conflict
On Mon, Dec 12, 2022 at 12:39:50AM +0100, Anders Trier Olesen wrote:
> I wrote some tests[1] which showed behaviour I did not expect.
> IP_BIND_ADDRESS_NO_PORT seems to work as it should, but calling bind without
> it
> enabled turns out to be even worse than I thought.
> This is what I think is happening: A successful bind() on a socket without
> IP_BIND_ADDRESS_NO_PORT enabled, with or without an explicit port configured,
> makes the assigned (or supplied) port unavailable for new connect()s (on
> different sockets), no matter the destination. I.e if you exhaust the entire
> net.ipv4.ip_local_port_range with bind() (no matter what IP you bind to!),
> connect() will stop working - no matter what IP you attempt to connect to. You
> can work around this by manually doing a bind() (with or without an explicit
> port, but without IP_BIND_ADDRESS_NO_PORT) on the socket before connect().
>
> What blows my mind is that after running test2, you cannot connect to anything
> without manually doing a bind() beforehand (as shown by test1 and test3
> above)!
> This also means that after running test2, software like ssh stops working:
>
> When using IP_BIND_ADDRESS_NO_PORT, we don't have this problem (1 5 6 can be
> run in any order):
Thank you for preparing that experiment. It's really valuable, and it
looks a lot like what I was seeing on the Snowflake bridge: calls to
connect would fail with EADDRNOTAVAIL unless first bound concretely to a
port number. IP_BIND_ADDRESS_NO_PORT causes bind not to set a concrete
port number, so in that respect it's the same as calling connect without
calling bind first.
It is surprising, isn't it? It certainly feels like calling connect
without first binding to an address should have the same effect as
manually binding to an address and then calling connect, especially if
the address you bind to is the same as the kernel would have chosen
automatically. It seems like it might be a bug, but I'm not qualified to
judge that.
If I am interpreting your results correctly, it means that either of the
two extremes is safe: either everything that needs to bind to a source
address should call bind with IP_BIND_ADDRESS_NO_PORT, or else
everything (whether it needs a specific source address or not) should
call bind *without* IP_BIND_ADDRESS_NO_PORT. (The latter situation is
what we've arrived at on the Snowflake bridge.) The middle ground, where
some connections use IP_BIND_ADDRESS_NO_PORT and some do not, is what
causes trouble, because connections that do not use
IP_BIND_ADDRESS_NO_PORT somehow "poison" the ephemeral port pool for
connections that do use IP_BIND_ADDRESS_NO_PORT (and for connections
that do not bind at all). It would explain why causing HAProxy not to
use IP_BIND_ADDRESS_NO_PORT resolved errors in my case.
> > Removing the IP_BIND_ADDRESS_NO_PORT option from Haproxy and
> > *doing nothing else* is sufficient to resolve the problem.
>
> Maybe there are other processes on the same host which calls bind() without
> IP_BIND_ADDRESS_NO_PORT, and blocks the ports? E.g OutboundBindAddress or
> similar in torrc?
OutboundBindAddress is a likely culprit. We did end up setting
OutboundBindAddress on the bridge during the period of intense
performance debugging at the end of September.
One thing doesn't quite add up, though. The earliest EADDRNOTAVAIL log
messages started at 2022-09-28 10:57:26:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40198
Whereas according to the change history of /etc on the bridge,
OutboundBindAddress was first set some time between 2022-09-29 21:38:37
and 2022-09-29 22:37:06, over 30 hours later. I would be tempted to say
this is a case of what you initially suspected, simple tuple exhaustion
between two static IP addresses, if not for the fact that pre-binding an
address resolved the problem in that case as well ("I get EADDRNOTAVAIL
sometimes even with netcat, making a connection to the haproxy port—but
not if I specify a source address in netcat"). But I only ran that
netcat test after OutboundBindAddress had been set, so there may have
been many factors being conflated.
Anyway, thank your for the insight. I apologize if I was inconsiderate
in my prior reply.
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] inet_csk_bind_conflict
On Sat, Dec 10, 2022 at 09:59:14AM +0100, Anders Trier Olesen wrote: > IP_BIND_ADDRESS_NO_PORT did not fix your somewhat similar problem in your > Haproxy setup, because all the connections are to the same dst tuple port> > (i.e 127.0.0.1:ExtORPort). > The connect() system call is looking for a unique 5-tuple srcport, dstip, dstport>. In the Haproxy setup, the only free variable is > srcport , so toggling > IP_BIND_ADDRESS_NO_PORT makes no difference. No—that is what I thought too, at first, but experimentally it is not the case. Removing the IP_BIND_ADDRESS_NO_PORT option from Haproxy and *doing nothing else* is sufficient to resolve the problem. Haproxy ends up binding to the same address it would have bound to with IP_BIND_ADDRESS_NO_PORT, and there are the same number of 5-tuples to the same endpoints, but the EADDRNOTAVAIL errors stop. It is counterintuitive and unexpected, which why I took the trouble to write it up. As I wrote at #40201, there are divergent code paths for connect in the kernel when the port is already bound versus when it is not bound. It's not as simple as filling in blanks in a 5-tuple in otherwise identical code paths. Anyway, it is not true that all connections go to the same (IP, port). (There would be no need to use a load balancer if that were the case.) At the time, we were running 12 tor processes with 12 different ExtORPorts (each ExtORPort on a different IP address, even: 127.0.3.1, 127.0.3.2, etc.). We started to have EADDRNOTAVAIL problems at around 3000 connections per ExtORPort, which is far too few to have exhausted the 5-tuple space. Please check the discussion at #40201 again, because I documented this detail there. I urge you to run an experient yourself, if these observations are not what you expect. I was surprised, as well. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
On Sat, Dec 10, 2022 at 05:19:43AM +, Gary C. New via tor-relays wrote: > I'm in the process of trying to cross-compile snowflake for OpenWRT and > Entware. Are there any other dependencies to compile snowflake other than Go? The README should list dependencies. Setting GOOS and GOARCH should be sufficient. > Do you know if it's possible to configure multiple pluggable transports with > different listeners within a single torrc? Yes. You cannot configure multiple listeners for the same transport, but you can use multiple different transports at once. Use use different sets of ServerTransportPlugin / ServerTransportListenAddr / ServerTransportOptions, or ClientTransportPlugin / Bridge for the client side. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
On Fri, Dec 09, 2022 at 08:43:26AM +, Gary C. New wrote: > In my implementation of the loadbalanced OBFS4 configuration, it appears that > BridgeDB still tests the ORPort for availability and without it marks the > OBFS4 bridge as being down. I see. Then yes, I suppose it is still necessary to expose the ORPort. > I gather that default bridges don't require a DistributionMethod as your > loadbalanced Snowflake configuration is set to "none?" That's correct. Default bridges are not distributed by rdsys, they are distributed in the configuration of Tor Browser itself. See extensions.torlauncher.default_bridge.* in about:config. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
On Fri, Dec 09, 2022 at 10:16:47AM +0100, Toralf Förster wrote: > On 12/9/22 07:02, David Fifield wrote: > > But now there is rdsys and bridgestrap, which may have the ability to > > test the obfs4 port rather than the ORPort. I cannot say whether that > > removes the requirement to expose the ORPort. > > Would be a step toward to make scanning for bridges harder IMO, if the > ORPort is no longer needed to be exposed. You are entirely correct. It's been noted as a discoverability vulnerability for over 10 years now. But so far attempts to resolve https://bugs.torproject.org/tpo/core/tor/7349 have fallen short. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] inet_csk_bind_conflict
On Fri, Dec 09, 2022 at 09:47:07AM +, Alexander Færøy wrote: > On 2022/12/01 20:35, Christopher Sheats wrote: > > Does anyone have experience troubleshooting and/or fixing this problem? > > Like I wrote in [1], I think it would be interesting to hear if the > patch from pseudonymisaTor in ticket #26646[2] would be of any help in > the given situation. The patch allows an exit operator to specify a > range of IP addresses for binding purposes for outbound connections. I > would think this could split the load wasted on trying to resolve port > conflicts in the kernel amongst the set of IP's you have available for > outbound connections. This sounds similar to a problem we faced with the main Snowflake bridge. After usage passed a certain threshold, we started getting constant EADDRNOTAVAIL, not on the outgoing connections to middle nodes, but on the many localhost TCP connections used by the pluggable transports model. https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40198 https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40201 Long story short, the only mitigation that worked for us was to bind sockets to an address (with port number unspecified, and with IP_BIND_ADDRESS_NO_PORT *unset*) before connecting them, and use different 127.0.0.0/8 addresses or ranges of addresses in different segments of the communication chain. https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/120 https://gitlab.torproject.org/dcf/extor-static-cookie/-/commit/a5c7a038a71aec1ff78d1b15888f1c75b66639cd IP_BIND_ADDRESS_NO_PORT was mentioned in another part of the thread (https://lists.torproject.org/pipermail/tor-relays/2022-December/020895.html). For us, this bind option *did not help* and in fact we had to apply a workaround for Haproxy, which has IP_BIND_ADDRESS_NO_PORT hardcoded. *Why* that should be the case is a mystery to me, as is why it is true that bind-before-connect avoids EADDRNOTAVAIL even when the address manually bound to is the very same address the kernel would have automatically assigned. I even spent some time reading the Linux 5.10 source code trying to make sense of it. In the source code I found, or at least think I found, code paths for the behvior I observed; but the behavior seems to go against how bind and IP_BIND_ADDRESS_NO_PORT are documented to work. https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40201#note_2839472 > Although my understanding of what Linux is doing is very imperfect, my > understanding is that both of these questions have the same answer: > port number assignment in `connect` when called on a socket not yet > bound to a port works differently than in `bind` when called with a > port number of 0. In case (1), the socket is not bound to a port > because you haven't even called `bind`. In case (2), the socket is not > bound to a port because haproxy sets the `IP_BIND_ADDRESS_NO_PORT` > sockopt before calling `bind`. When you call `bind` *without* > `IP_BIND_ADDRESS_NO_PORT`, it causes the port number to be bound > before calling `connect`, which avoids the code path in `connect` that > results in `EADDRNOTAVAIL`. > > I am confused by these results, which are contrary to my understanding > of what `IP_BIND_ADDRESS_NO_PORT` is supposed to do, which is > precisely to avoid the problem of source address port exhaustion by > deferring the port number assignment until the time of `connect`, when > additional information about the destination address is available. But > it's demonstrable that binding to a source port before calling > `connect` avoids `EADDRNOTAVAIL` errors in our use cases, whatever the > cause may be. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
On Fri, Dec 09, 2022 at 01:09:05AM +, Gary C. New wrote: > Is it truly necessary to expose the ORPort to the World in a pluggable > transport configuration? I don't know if it is necessary for ordinary bridges to expose the ORPort. For a long time, it was necessary, because BridgeDB used the ORPort to check that a bridge was running, before distributing it to users. See: https://bugs.torproject.org/tpo/core/tor/7349 But now there is rdsys and bridgestrap, which may have the ability to test the obfs4 port rather than the ORPort. I cannot say whether that removes the requirement to expose the ORPort. https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/merge_requests/36 For the special case of the default bridges shipped with Tor Browser, it is not necessary to export the ORPort, because those bridges are not distributed by rdsys. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] security update for obfs4proxy
On Fri, Oct 14, 2022 at 06:08:38PM +0200, Toralf Förster wrote: > On 10/14/22 11:28, meskio wrote: > > The latest version of obfs4proxy (0.0.14) comes with an important security > > fix. > > Is there a Changelog available ? The below issue, which is currently confidential, has details of what was fixed. The issue is scheduled to become public by 2022-11-15. https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/obfs4/40007 ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
On Fri, Mar 04, 2022 at 09:40:01PM +, Gary C. New wrote: > I see that the metrics change has been reverted. > > If/When the metrics change is implemented, will loadbalanced Tor Relay Nodes > need to be uniquely named or will they all be able to use the same nickname? When I made my own combined graphs, I relied on different instances having different nicknames. I don't know an easy way to distinguish the descriptors of different instances otherwise. You could conceivably do it by analyzing the periodicity of different instances' publishing schedules. (Start one instance on the hour, another at :10, another at :20, etc.) But that seems fragile, not to mention annoying to deal with. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
On Thu, Mar 03, 2022 at 08:13:34PM +, Gary C. New wrote: > Has Tor Metrics implemented your RFC related to Written Bytes per Second and > Read Bytes per Second on Onionoo? > > As of the 27th of February, I've noticed a change in reporting that accurately > reflects the aggregate of my Tor Relay Nodes opposed to the previously > reported > Single Tor Node. Are you seeing a similar change for snowflake.torproject.org? You're right. I see a change since 2022-02-27, but in the case of the snowflake bridge the numbers look wrong, about 8× too high. I posted an update on the issue. Thanks for noticing. https://gitlab.torproject.org/tpo/network-health/metrics/onionoo/-/issues/40022#note_2783524 > Additionally, other than the hourly stacktrace errors in the syslog, the > secure_onion_key workaround seems to be working well without any ill > side-effects. I've been able to operate with the same secure_onion_key for > close to 5 weeks, now. Have you run into any issues? Yes, it's still working well here. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
The load-balanced Snowflake bridge is running in production since 2022-01-31. Thanks Roger, Gary, Roman for your input. Hopefully reproducible installation instructions: https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-Installation-Guide?version_id=6de6facbb0fd047de978a561213c59224511445f Observations since: https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40095#note_2774428 Metrics graphs are currently confused by multiple instances of tor uploading descriptors under the same fingerprint. Particularly in the interval between 2022-01-25 and 2022-02-03, when a production bridge and staging bridge were running in parallel, with four instances being used and another four being mostly unused. https://metrics.torproject.org/rs.html#details/5481936581E23D2D178105D44DB6915AB06BFB7F https://metrics.torproject.org/userstats-bridge-transport.html?start=2021-11-10&end=2022-02-08&transport=snowflake Since 2022-02-03, it appears that Metrics is showing only one of the four running instances per day. Because all four instances are about equally used (as if load balanced, go figure), the values on the graph are 1/4 what they should be. The reported bandwidth of 5 MB/s is actually 20 MB/s, and the 2500 clients are actually 1. All the necessary data are present in Collector, it's just a question of data processing. I opened an issue for the Metrics graphs, where you can also see some manually made graphs that are closer to the true values. https://bugs.torproject.org/tpo/network-health/metrics/onionoo/40022 I started a thread on tor-dev about the issues of onion key rotation and ExtORPort authentication. https://lists.torproject.org/pipermail/tor-dev/2022-February/thread.html ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
On Sat, Jan 29, 2022 at 02:54:40AM +, Gary C. New via tor-relays wrote: > > > From your documentation, it sounds like you're running everything on the > > > same machine? When expanding to additional machines, similar to the file > > > limit issue, you'll have to expand the usable ports as well. > > > I don't think I understand your point. At 64K simultaneous connections, you > > run out of source ports for making connection 4-tuple unique, but I don't > > see how the same or different hosts makes a difference, in that respect. > > On many Linux distros, the default ip_local_port_range is between 32768 - > 61000. > > The Tor Project recommends increasing it. > > ``` > # echo 15000 64000 > /proc/sys/net/ipv4/ip_local_port_range > ``` Thanks, that's a good tip. I added it to the installation guide. > > I'm not using nyx. I'm just looking at the bandwidth on the network > > interface. > > If you have time, would you mind installing nyx to validate observed > similarities/differences between our loadbalanced configurations? I don't have plans to do that. > > > I'm glad to hear you feel the IPv6 reporting appears to be a > > > false-negative. Does this mean there's something wrong with IPv6 > > > Heartbeat reporting? > > > > I don't know if it's wrong, exactly. It's reporting something different > > than what ExtORPort is providing. The proximate connections to tor are > > indeed all IPv4. > > I see. Perhaps IPv6 connections are less prolific and require more time to > ramp? No, it's not that. The bridge has plenty of connections from clients that use an IPv6 address, as the bridge-stats file shows: ```plain bridge-ip-versions v4=15352,v6=1160 ``` It's just that, unlike a direct TCP connection as the the case with a guard relay, the client connections pass through a chain of proxies and processes on the way to the tor: client → Snowflake proxy → snowflake-server WebSocket server → extor-static-cookie adapter → tor. The last link in the chain is IPv4, and evidently that is what the heartbeat log reports. The client's actual IP address is tunnelled, for metrics purposes, through this chain of proxies and processes, to tor using a special protocol called ExtORPort (see USERADDR at https://gitweb.torproject.org/torspec.git/tree/proposals/196-transport-control-ports.txt). It looks like the bridge-stats descriptor pays attention to the USERADDR information and the heartbeat log does not, that's all. > After expanding my reading of your related "issues," I see that your VPS > provider only offers up to 8 cores. Is it possible to spin-up another VPS > environment, with the same provider, on a separate VLAN, allowing route/ > firewall access between the two VPS environments? This way you could test > loadbalancing a Tor Bridge over a local network using multiple virtual > environments. Yes, there are many other potential ways to further expand the deployment, but I do not have much interest in that topic right now. I started the thread for help with a non-obvious point, namely getting past the bottleneck of a single-core tor process. I think that we have collectively found a satisfactory solution for that. The steps after that for further scaling are relatively straightforward, I think. Running one instance of snowflake-server on one host and all the instances of tor on a nearby host is a logical next step. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
> On the matter of onion key rotation, I had the idea of making the onion key
> files read-only. Roger did some source code investigation and said that it
> might work to prevent onion key rotation, with some minor side effects. I
> plan to give the idea a try on a different bridge. The possible side effects
> are that tor will continue trying and failing to rotate the onion key every
> hour, and "force a router descriptor rebuild, so it will try to publish a new
> descriptor each hour."
Making secret_onion_key and secret_onion_key_ntor read-only does not quite
work, because tor first renames them to secret_onion_key.old and
secret_onion_key_ntor.old before writing new files. (Making the *.old files
read-only does not work either, because the `tor_rename` function first unlinks
the destination.)
https://gitweb.torproject.org/tor.git/tree/src/feature/relay/router.c?h=tor-0.4.6.9#n497
But a slight variation does work: make secret_onion_key.old and
secret_onion_key_ntor.old *directories*, so that tor_rename cannot rename a
file over them. It does result in an hourly `BUG` stack trace, but otherwise it
seems effective.
I did a test with two tor instances. The rot1 instance had the directory hack
to prevent onion key rotation. The rot2 had nothing to prevent onion key
rotation.
```plain
# tor-instance-create rot1
# tor-instance-create rot2
```
/etc/tor/instances/rot1/torrc:
```plain
Log info file /var/lib/tor-instances/rot1/onionrotate.info.log
BridgeRelay 1
AssumeReachable 1
BridgeDistribution none
ORPort 127.0.0.1:auto
ExtORPort auto
SocksPort 0
Nickname onionrotate1
```
/etc/tor/instances/rot2/torrc:
```plain
Log info file /var/lib/tor-instances/rot2/onionrotate.info.log
BridgeRelay 1
AssumeReachable 1
BridgeDistribution none
ORPort 127.0.0.1:auto
ExtORPort auto
SocksPort 0
Nickname onionrotate2
```
Start rot1, copy its keys to rot2, then start rot2:
```plain
# service tor@rot1 start
# cp -r /var/lib/tor-instances/rot1/keys /var/lib/tor-instances/rot2/
# chown -R _tor-rot2:_tor-rot2 /var/lib/tor-instances/rot2/keys
# service tor@rot2 start
```
Stop the two instances, check that the onion keys are the same, and that
`LastRotatedOnionKey` is set in both state files:
```plain
# service tor@rot1 stop
# service tor@rot2 stop
# ls -l /var/lib/tor-instances/rot*/keys/secret_onion_key*
-rw--- 1 _tor-rot1 _tor-rot1 888 Jan 28 22:57
/var/lib/tor-instances/rot1/keys/secret_onion_key
-rw--- 1 _tor-rot1 _tor-rot1 96 Jan 28 22:57
/var/lib/tor-instances/rot1/keys/secret_onion_key_ntor
-rw--- 1 _tor-rot2 _tor-rot2 888 Jan 28 23:05
/var/lib/tor-instances/rot2/keys/secret_onion_key
-rw--- 1 _tor-rot2 _tor-rot2 96 Jan 28 23:05
/var/lib/tor-instances/rot2/keys/secret_onion_key_ntor
# md5sum /var/lib/tor-instances/rot*/keys/secret_onion_key*
fb2a8a8f9de56f061eccbb3fedd700c4
/var/lib/tor-instances/rot1/keys/secret_onion_key
2066ab7e01595adf42fc791ad36e1fc5
/var/lib/tor-instances/rot1/keys/secret_onion_key_ntor
fb2a8a8f9de56f061eccbb3fedd700c4
/var/lib/tor-instances/rot2/keys/secret_onion_key
2066ab7e01595adf42fc791ad36e1fc5
/var/lib/tor-instances/rot2/keys/secret_onion_key_ntor
# grep LastRotatedOnionKey /var/lib/tor-instances/rot*/state
/var/lib/tor-instances/rot1/state:LastRotatedOnionKey 2022-01-28 22:57:14
/var/lib/tor-instances/rot2/state:LastRotatedOnionKey 2022-01-28 23:11:04
```
Set `LastRotatedOnionKey` 6 weeks into the past to force an attempt to rotate
the keys the next time tor is restarted:
```plain
# sed -i -e 's/^LastRotatedOnionKey .*/LastRotatedOnionKey 2021-12-15
00:00:00/' /var/lib/tor-instances/rot*/state
# grep LastRotatedOnionKey /var/lib/tor-instances/rot*/state
/var/lib/tor-instances/rot1/state:LastRotatedOnionKey 2021-12-15 00:00:00
/var/lib/tor-instances/rot2/state:LastRotatedOnionKey 2021-12-15 00:00:00
```
Create the secret_onion_key.old and secret_onion_key_ntor.old directories in
the rot1 instance.
```plain
# mkdir -m 700 /var/lib/tor-instances/rot1/keys/secret_onion_key{,_ntor}.old
```
Check the identity of keys before starting:
```plain
# md5sum /var/lib/tor-instances/rot*/keys/secret_onion_key*
fb2a8a8f9de56f061eccbb3fedd700c4
/var/lib/tor-instances/rot1/keys/secret_onion_key
2066ab7e01595adf42fc791ad36e1fc5
/var/lib/tor-instances/rot1/keys/secret_onion_key_ntor
md5sum: /var/lib/tor-instances/rot1/keys/secret_onion_key_ntor.old: Is a
directory
md5sum: /var/lib/tor-instances/rot1/keys/secret_onion_key.old: Is a directory
fb2a8a8f9de56f061eccbb3fedd700c4
/var/lib/tor-instances/rot2/keys/secret_onion_key
2066ab7e01595adf42fc791ad36e1fc5
/var/lib/tor-instances/rot2/keys/secret_onion_key_ntor
```
Start both instances:
```plain
# service tor@rot1 start
# service tor@rot2 start
```
Verify that the rot1 instance is still using the same onion keys, while rot2
has rotated them:
```plain
# ls -ld /var/lib/tor-instances/rot*/keys/secret_onion_key*
-rw--- 1 _tor-rot1 _tor-rot1 888 Jan 28 23:45
/var/lib/tor-instances/rot1/keys/
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
> With regard to loadbalanced Snowflake sessions, I'm curious to know what > connections (i.e., inbound, outbound, directory, control, etc) are being > displayed within nyx? I'm not using nyx. I'm just looking at the bandwidth on the network interface. > Your Heartbeat logs continue to appear to be in good health. When keys are > rotated, We're trying to avoid rotating keys at all. If the read-only files do not work, we'll instead probably periodically rewrite the state file to push the rotation into the future. > > I worried a bit about the "0 with IPv6" in a previous comment. Looking at > > the bridge-stats files, I don't think there's a problem. > > I'm glad to hear you feel the IPv6 reporting appears to be a false-negative. > Does this mean there's something wrong with IPv6 Heartbeat reporting? I don't know if it's wrong, exactly. It's reporting something different than what ExtORPort is providing. The proximate connections to tor are indeed all IPv4. > Are your existing 8 cpu's only single cores? Is it too difficult to upgrade > with your VPS provider? Sure, there are plenty of ways to increase resources of the bridge, but I feel that's a different topic. Thanks for your comments. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
On Tue, Jan 25, 2022 at 11:21:10PM +, Gary C. New via tor-relays wrote: > It's nice to see that the Snowflake daemon offers a native configuration > option for LimitNOFile. I ran into a similar issue with my initial > loadbalanced Tor Relay Nodes that was solved at the O/S level using ulimit. > It would be nice if torrc had a similar option. LimitNOFile is actually not a Snowflake thing, it's a systemd thing. It's the same as `ulimit -n`. See: https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Process%20Properties > From your documentation, it sounds like you're running everything on the same > machine? When expanding to additional machines, similar to the file limit > issue, you'll have to expand the usable ports as well. I don't think I understand your point. At 64K simultaneous connections, you run out of source ports for making connection 4-tuple unique, but I don't see how the same or different hosts makes a difference, in that respect. > I found your HAProxy configuration in your “Draft installation guide.” It > seems you’re using regular TCP streaming mode with the Snowflake instances vs > transparent TCP streaming mode, which is a notable difference with the > directly loadbalanced Tor Relay configuration. I admit I did not understand your point about transparent proxying. If it's about retaining the client's source IP address for source IP address pinning, I don't think that helps us. This is a bridge, not a relay, and the source IP address that haproxy sees is several steps removed from the client's actual IP address. haproxy receives connections from a localhost web server (the server pluggable transport that receives WebSocket connections); the web server receives connections from Snowflake proxies (which can and do have different IP addresses during the lifetime of a client session); only the Snowflake proxies themselves receive direct traffic from the client's own source IP address. The client's IP address is tunnelled all the way through to tor, for metrics purposes, but that uses the ExtORPort protocol and the load balancer isn't going to understand that. I think that transparent proxying would only transparently proxy the localhost IP addresses from the web server, which doesn't have any benefit, I don't think. What's written in the draft installation guide is not the whole file. There's additionally the default settings as follows: ``` global log /dev/loglocal0 log /dev/loglocal1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners stats timeout 30s user haproxy group haproxy daemon # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets defaults log global modehttp option httplog option dontlognull timeout connect 5000 timeout client 5 timeout server 5 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http ``` > You might test using a timeout value of 0s (to disable the timeout at the > loadbalancer) and allow the Snowflake instances to preform state checking to > ensure HAProxy isn’t throttling your bridge. Thanks for that hint. So far, 10-minute timeouts seem not to be causing a problem. I don't know this software too well, but I think it's an idle timeout, not an absolute limit on connection lifetime. > Currently, as I only use IPv4, I can't offer much insight as to the lack of > IPv6 connections being reported (that's what my logs report, too). On further reflection, I don't think there's a problem here. The instances' bridge-stats and end-stats show a mix of countries and v4/v6. https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40095#note_2772684 > Regarding metrics.torproject.org... I expect you'll see that written-bytes > and read-bytes only reflect that of a single Snowflake instance. However, > your consensus weight will reflect the aggregate of all Snowflake instances. Indeed, the first few data poin
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
The DNS record for the Snowflake bridge was switched to a temporary staging
server, running the load balancing setup, at 2022-01-25 17:41:00. We were
debugging some initial problems until 2022-01-25 18:47:00. You can read about
it here:
https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40095#note_2772325
Snowflake sessions are now using the staging bridge, except for those that
started before the change happened and haven't finished yet, and perhaps some
proxies that still have the IP address of the production bridge in their DNS
cache. I am not sure yet what will happen with metrics, but we'll see after a
few days.
On the matter of onion key rotation, I had the idea of making the onion key
files read-only. Roger did some source code investigation and said that it
might work to prevent onion key rotation, with some minor side effects. I plan
to give the idea a try on a different bridge. The possible side effects are
that tor will continue trying and failing to rotate the onion key every hour,
and "force a router descriptor rebuild, so it will try to publish a new
descriptor each hour."
https://gitweb.torproject.org/tor.git/tree/src/feature/relay/router.c?h=tor-0.4.6.9#n523
```
if (curve25519_keypair_write_to_file(&new_curve25519_keypair, fname,
"onion") < 0) {
log_err(LD_FS,"Couldn't write curve25519 onion key to \"%s\".",fname);
goto error;
}
// ...
error:
log_warn(LD_GENERAL, "Couldn't rotate onion key.");
if (prkey)
crypto_pk_free(prkey);
```
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
On Tue, Jan 04, 2022 at 11:57:36PM -0500, Roger Dingledine wrote: > Hm. It looks promising! But we might still have a Tor-side problem remaining. > I think it boils down to how long the KCP sessions last. > > The details on how exactly these bridge instances will diverge over time: > > The keys directory will start out the same, but after four weeks > (DEFAULT_ONION_KEY_LIFETIME_DAYS, used to be one week but in Tor > 0.3.1.1-alpha, proposal 274, we bumped it up to four weeks) each bridge will > rotate its onion key (the one clients use for circuit-level crypto). That is, > each instance will generate its own fresh onion key. > > The two bridge instances actually haven't diverged completely at that point, > since Tor remembers the previous onion key (i.e. the onion key from the > previous period) and is willing to receive create cells that use it for one > further week (DEFAULT_ONION_KEY_GRACE_PERIOD_DAYS). So it is after 5 weeks > that the original (shared) onion key will no longer work. > > Where this matters is (after this 5 weeks have passed) if the client connects > to the bridge, fetches and caches the bridge descriptor of instance A, and > then later it connects to the bridge again and gets passed to instance B. In > this case, the create cell that the client generates will use the onion key > for instance A, and instance B won't know how to decrypt it so it will send a > destroy cell back. I've done an experiment with a second snowflake bridge that has the same identity keys but different onion keys. A client can bootstrap with either one starting from a clean state, but it fails if you bootstrap with one and then try to bootstrap with the other using the same DataDirectory. The error you get is ```plain onion_skin_ntor_client_handshake(): Invalid result from curve25519 handshake: 4 ``` The first bridge is the existing "prod" snowflake bridge with nickname: * flakey The other "staging" bridge is the load-balanced configuration with four instances. All four instances currently have the same onion keys; which however are different from the "prod"'s onion keys. (The onion keys actually come from a backup I made.) * flakey1 * flakey2 * flakey3 * flakey4 Bootstrapping "prod" with a fresh DataDirectory "datadir.prod" works. Here is torrc.prod: ```plain UseBridges 1 SocksPort auto DataDirectory datadir.prod ClientTransportPlugin snowflake exec ./client -keep-local-addresses -log snowflake.log Bridge snowflake 192.0.2.3:1 2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.torproject.net/ max=1 ice=stun:stun.voip.blackberry.com:3478,stun:stun.altar.com.pl:3478,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.sonetel.net:3478,stun:stun.stunprotocol.org:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478 ``` Notice `new bridge descriptor 'flakey' (fresh)`: ```plain snowflake/client$ tor -f torrc.prod [notice] Tor 0.3.5.16 running on Linux with Libevent 2.1.8-stable, OpenSSL 1.1.1d, Zlib 1.2.11, Liblzma 5.2.4, and Libzstd 1.3.8. [notice] Bootstrapped 0%: Starting [notice] Starting with guard context "bridges" [notice] Delaying directory fetches: No running bridges [notice] Bootstrapped 5%: Connecting to directory server [notice] Bootstrapped 10%: Finishing handshake with directory server [notice] Bootstrapped 15%: Establishing an encrypted directory connection [notice] Bootstrapped 20%: Asking for networkstatus consensus [notice] new bridge descriptor 'flakey' (fresh): $2B280B23E1107BB62ABFC40DDCC8824814F80A72~flakey at 192.0.2.3 [notice] Bootstrapped 25%: Loading networkstatus consensus [notice] I learned some more directory information, but not enough to build a circuit: We have no usable consensus. [notice] Bootstrapped 40%: Loading authority key certs [notice] The current consensus has no exit nodes. Tor can only build internal paths, such as paths to onion services. [notice] Bootstrapped 45%: Asking for relay descriptors for internal paths [notice] I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 0/6673, and can only build 0% of likely paths. (We have 100% of guards bw, 0% of midpoint bw, and 0% of end bw (no exits in consensus, using mid) = 0% of path bw.) [notice] Bootstrapped 50%: Loading relay descriptors for internal paths [notice] The current consensus contains exit nodes. Tor can build exit and internal paths. [notice] Bootstrapped 57%: Loading relay descriptors [notice] Bootstrapped 64%: Loading relay descriptors [notice] Bootstrapped 73%: Loading relay descriptors [notice] Bootstrapped 78%: Loading relay descriptors [notice] Bootstrapped 80%: Connecting to the Tor network [notice] Bootstrapped 85%: Finishing handshake with first hop [notice] Bootstrapped 90%: Establishing a Tor circuit [notice] Bootstrapped 100%: Done ``` Bootstrapping "staging" with
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
On Thu, Dec 30, 2021 at 10:42:51PM -0700, David Fifield wrote: > One complication we'll have to work out is that ptadapter doesn't have a > setting for ExtORPort forwarding. ptadapter absorbs any ExtORPort > information and forwards an unadorned connection onward. The idea I had > to to work around this limitation is to have ptadapter, rather than > execute snowflake-server directly, execute a shell script that sets > TOR_PT_EXTENDED_SERVER_PORT to a hardcoded address (i.e., to haproxy) > before running snowflake-server. Though, I am not sure what to do about > the extended_orport_auth_cookie file, which will be different for > different tor instances. There are a number of potential ways to deal with the complication of ExtORPort authentication, from alternative ExtORPort authentication types, to ExtORPort-aware load balancing. With a view towards deploying something in the near future, I wrote this program that enables an external pluggable transport to talk to tor's ExtORPort and authenticate as if it had an unchanging authentication cookie. https://gitlab.torproject.org/dcf/extor-static-cookie The difficulty with load-balancing multiple tor instances, with respect to ExtORPort, is that to authenticate with the ExtORPort you need to read a cookie from a file on disk, which tor overwrites randomly every time it starts. If you do not know which instance of tor will receive your forwarded traffic, you do not know which ExtORPort cookie to use. The extor-static-cookie program presents an ExtORPort interface, however it reads its authentication cookie that is independent of any instance of tor, which you can write once and then leave alone. The external server pluggable transport can read from the shared authentication cookie file as well. Every instance of tor runs a copy of extor-static-cookie, all using the same authentication cookie file. The extor-static-cookie instances receive ExtORPort authentication from the external server pluggable transport, along with the USERADDR and TRANSPORT metadata, then re-authenticate and echo that information to their respective tor's ExtORPort. So we change from this: ___ .->|tor| ___ | --- ->|snowflake-server|->|haproxy|-+->|tor| --- | --- '->|tor| --- to this: ______ .->|extor-static-cookie|->|tor| ___ | ------ ->|snowflake-server|->|haproxy|-+->|extor-static-cookie|->|tor| --- | ------ '->|extor-static-cookie|->|tor| ------ I have a similar setup running now on a test bridge, with one instance of obfs4proxy load-balancing to two instances of tor. ## Setup notes Install extor-static-cookie: # apt install golang # git clone https://gitlab.torproject.org/dcf/extor-static-cookie # (cd extor-static-cookie && go build) # install -o root -g root extor-static-cookie/extor-static-cookie /usr/local/bin/ Generate a shared authentication cookie file: # mkdir -m 755 /var/lib/extor-static-cookie # extor-static-cookie/gen-auth-cookie > /var/lib/extor-static-cookie/static_extended_orport_auth_cookie Install a first instance of tor and configure it as a bridge: # apt install tor # tor-instance-create o1 /etc/tor/instances/o1/torrc: BridgeRelay 1 PublishServerDescriptor 0 AssumeReachable 1 SocksPort 0 ORPort 127.0.0.1:auto ExtORPort auto ServerTransportPlugin extor_static_cookie exec /usr/local/bin/extor-static-cookie /var/lib/extor-static-cookie/static_extended_orport_auth_cookie ServerTransportListenAddr extor_static_cookie 127.0.0.1:10001 Notice we set `ExtORPort auto` (this is tor's own ExtORPort), and also pass `127.0.0.1:10001` to extor-static-cookie, which is the ExtORPort that the external server pluggable transport will talk to. Start the first instance, which will generate keys: systemctl start tor@o1 Install a second instance of tor and configure it as a bridge (with a different ServerTransportListenAddr port): # tor-instance-create o2 /etc/tor/instances/o2/torrc: BridgeRelay 1 PublishServerDescriptor 0 AssumeReachable 1 SocksPort 0 ORPort 127.0.0.1:auto ExtORPort auto ServerTransportPlugin extor_static_cookie exec /usr/local/bin/extor-static-cookie /var/lib/extor-static-coo
Re: [tor-relays] How to reduce tor CPU load on a single bridge?
On Mon, Dec 27, 2021 at 04:00:34PM -0500, Roger Dingledine wrote: > On Mon, Dec 27, 2021 at 12:05:26PM -0700, David Fifield wrote: > > I have the impression that tor cannot use more than one CPU core???is that > > correct? If so, what can be done to permit a bridge to scale beyond > > 1×100% CPU? We can fairly easily scale the Snowflake-specific components > > around the tor process, but ultimately, a tor client process expects to > > connect to a bridge having a certain fingerprint, and that is the part I > > don't know how to easily scale. > > > > * Surely it's not possible to run multiple instances of tor with the > > same fingerprint? Or is it? Does the answer change if all instances > > are on the same IP address? If the OR ports are never used? > > Good timing -- Cecylia pointed out the higher load on Flakey a few days > ago, and I've been meaning to post a suggestion somewhere. You actually > *can* run more than one bridge with the same fingerprint. Just set it > up in two places, with the same identity key, and then whichever one the > client connects to, the client will be satisfied that it's reaching the > right bridge. Thanks for this information. I've done a test with one instance of obfs4proxy forwarding through a load balancer to two instances of tor that have the same keys, and it seems to work. It seems like this could work for Snowflake. > (A) Even though the bridges will have the same identity key, they won't > have the same circuit-level onion key, so it will be smart to "pin" > each client to a single bridge instance -- so when they fetch the bridge > descriptor, which specifies the onion key, they will continue to use > that bridge instance with that onion key. Snowflake in particular might > also want to pin clients to specific bridges because of the KCP state. > > (Another option, instead of pinning clients to specific instances, > would be to try to share state among all the bridges on the backend, > e.g. so they use the same onion key, can resume the same KCP sessions, > etc. This option seems hard.) Let's make a distinction between the "frontend" snowflake-server pluggable transport process, and the "backend" tor process. These don't necessarily have to be 1:1; either one could be run in multiple instances. Currently, the "backend" tor is the limiting factor, because it uses only 1 CPU core. The "frontend" snowflake-server can scale to multiple cores in a single process and is comparatively unrestrained. So I propose to keep snowflake-server as a single process, and to run multiple tor processes. That eliminates the dimension of KCP state coordination, and should last us until snowflake-server outgrows the resources of a single host. The snowflake-server program is a managed proxy; i.e., it expects to run with certain environment variables set by a managing process, normally tor. We'll need to instead run snowflake-server apart from any single tor instance. Probably the easiest way to do that in the short term is with ptadapter (https://github.com/twisteroidambassador/ptadapter), which converts a pluggable transport into a TCP proxy, forwarding to an address you specify. Then we can have ptadapter forward to a load balancer like haproxy. The load balancer will then round-robin over the ORPorts of the available tor instances. The tor instances can all be on the same host (run as many instances as you have CPU cores), which may or may not be the same host on which snowflake-server is running. Currently we have this: ___ -->|snowflake-server|-->|tor| --- (run by tor) The design I'm proposing is this: ___ .->|tor| ___ | --- -->|snowflake-server|-->|haproxy|-+->|tor| --- | --- (run by ptadapter) '->|tor| --- I believe that the "pinning" of a client session to particular tor instance will work automatically by the fact that snowflake-server keeps an outgoing connection alive (i.e., through the load balancer) as long as a KCP session exists. One complication we'll have to work out is that ptadapter doesn't have a setting for ExtORPort forwarding. ptadapter absorbs any ExtORPort information and forwards an unadorned connection onward. The idea I had to to work around this limitation is to have ptadapter, rather than execute snowflake-server directly, execute a shell script that sets TOR_PT_EXTENDED_SERVER_PORT to a hardcoded address (i.e., to haproxy) before running snowflake-server. Th
[tor-relays] How to reduce tor CPU load on a single bridge?
The main Snowflake bridge (https://metrics.torproject.org/rs.html#details/5481936581E23D2D178105D44DB6915AB06BFB7F) is starting to become overloaded, because of a recent substantial increase in users. I think the host has sufficient CPU and memory headroom, and pluggable transport process (that receies WebSocket connections and forwards them to tor) is scaling across multiple cores. But the tor process is constantly using 100% of one CPU core, and I suspect that the tor process has become a bottleneck. Here are issues about a recent CPU upgrade on the bridge, and observations about the proportion of CPU used by different processes: https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40085 https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40086#note_2767961 I have the impression that tor cannot use more than one CPU core—is that correct? If so, what can be done to permit a bridge to scale beyond 1×100% CPU? We can fairly easily scale the Snowflake-specific components around the tor process, but ultimately, a tor client process expects to connect to a bridge having a certain fingerprint, and that is the part I don't know how to easily scale. * Surely it's not possible to run multiple instances of tor with the same fingerprint? Or is it? Does the answer change if all instances are on the same IP address? If the OR ports are never used? * OnionBalance does not help with this, correct? * Are there configuration options we could set to increase parallelism? * Is migrating to a host with better single-core performance the only immediate option for scaling the tor process? Separate from the topic of scaling a single bridge, here is a past issue with thoughts on scaling beyond one bridge. it looks as though there are not ways to do it that do not require changes to the way tor handles its Bridge lines. https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/28651 * Using multiple snowflake Bridge lines does not work well, despite that we could arrange to have the Snowflake proxy connect the user to the expected bridge, because tor will try to connect to all of them, not choose one at random. * Removing the fingerprint from the snowflake Bridge line in Tor Browser would permit the Snowflake proxies to round-robin clients over several bridges, but then the first hop would be unauthenticated (at the Tor layer). It would be nice if it were possible to specify a small set of permitted bridge fingerprints. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Dropping packets with TCP MSS=1400 to foil GFW active probing
On Mon, Aug 20, 2018 at 02:25:40PM -0400, Nathaniel Suchy wrote: > Interesting. Is there any reason to not use an obfuscated bridge? No, not really. obfs4 resists active probing without any special additional steps. But I can think of one reason why the MSS trick is worth trying, anyway. Due to a longstanding bug (really more of a design issue that's hard to repair), you can't run an obfs4 bridge without also running a vanilla (unobfuscated) bridge on a different port on the same IP address. So if anyone ever connects to that vanilla port, the bridge will get probed and the entire IP address blocked, including the obfs4 port. https://bugs.torproject.org/7349 ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Dropping packets with TCP MSS=1400 to foil GFW active probing
On Sun, Aug 19, 2018 at 07:41:26PM -0400, Nathaniel Suchy wrote: > Is China successfully probing OBFS4 bridges? Or does this apply more to non > obfs bridges? China doesn't dynamically probe obfs4 bridges. (More precisely: they may try to probe, but the probes don't result in blocks for obfs4.) They only block obfs4 bridges whose addresses they learn in other ways. The MSS=1400 trick works, the authors report, even for non-obfuscated bridges. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
[tor-relays] Dropping packets with TCP MSS=1400 to foil GFW active probing
A paper from FOCI 2018 by Arun Dunna, Ciarán O'Brien, and Phillipa Gill on the subject of Tor bridge blocking in China has this interesting suggestion (Section 5.2): https://www.usenix.org/conference/foci18/presentation/dunna To do this, we write a series specific rules using iptables in order to drop packets from Chinese scanners. ... We use a rule to drop incoming Tor packets with an MSS of 1400. Further investigation would be needed to analyze potential false positives... We note that this method of dropping scan traffic successfully keeps our bridge relays from being blocked and allows our client in China to maintain access to the bridge. Like https://github.com/NullHypothesis/brdgrd, surely this trick won't work forever, but if you're setting up a new bridge, it's worth a try? This is completely untested, but I think the iptables rule would look something like this: iptables -A INPUT --protocol tcp --dport [your-bridge-port] -m tcpmss --mss 1400 -j DROP Then, after a while, check /var/lib/tor/stats/bridge-stats and see if you have any connections from "cn". ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Call for a big fast bridge (to be the meek backend)
On Mon, Sep 15, 2014 at 07:12:23PM -0700, David Fifield wrote: > The meek pluggable transport is currently running on the bridge I run, > which also happens to be the backend bridge for flash proxy. I'd like to > move it to a fast relay run by an experienced operator. I want to do > this both to diffuse trust, so that I don't run all the infrastructure, > and because my bridge is not especially fast and I'm not especially > adept at performance tuning. > > All you will need to do is run the meek-server program, add some lines > to your torrc, and update the software when I ask you to. The more CPU, > memory, and bandwidth you have, the better, though at this point usage > is low enough that you won't even notice it if you are already running a > fast relay. I think it will help if your bridge is located in the U.S., > because that reduces latency from Google App Engine. > > The meek-server plugin is basically just a little web server: > https://gitweb.torproject.org/pluggable-transports/meek.git/tree/HEAD:/meek-server A couple of other requirements have shown themselves in helping set up a meek-server relay. The first is that it has to be a 0.2.5.x version of tor. This is so that the ExtORPort option will be supported. The ExtORPort option is needed in order to collect statistics on pluggable transports (see #4773). One of the lines you will have to add to torrc is: ExtORPort auto If your tor does not support the option, you will see in the log: [warn] Failed to parse/validate config: Unknown option 'ExtORPort'. Failing. The second requirement is that the relay needs to have either BridgeRelay or DirPort set. If neither of these options is set, the relay will not be a directory cache, and clients will not be able to bootstrap past "20%: Asking for networkstatus consensus." As I understand it, #12538 will make it so that every relay is a directory cache, but it is not committed yet. I'll check back privately with the people who offered to run a relay and check if setting these options is okay. David Fifield ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
[tor-relays] Bridges included in Tor Browser -- should they regen keys because of Heartbleed?
I wondered whether operators of bridges that are included in the browser bundle should generate new identity keys after upgrading their OpenSSL. The argument for generating new keys is that old keys may have been compromised by Heartbleed. The argument against is that a new fingerprint will prevent existing browser bundle users from using the default bridges, because the fingerprint is built into the browser: https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/Bundle-Data/PTConfigs/bridge_prefs.js What I heard from some developers is that it would be good to set up new bridges with new keys (could be on the same IP address), and give the new information to the browser devs so they can put it in the bundles. Leave the old ones running for a while until usage drops off. A question is how to actually do this, running two copies of tor on the same IP. Offhand I would say that using a separate DataDirectory will be enough, but I don't know for sure. David Fifield ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
