[tor-dev] Damian's Status Report - May 2015
Squirrels! Five newborn squirrels are in our tree! And they... oh right, code. Talk about code, Damian, like a professional. ... awww, but damn they're cute. Stem Release 1.4.1 Though not as adorable as baby squirrels the new release of Stem is still pretty sweet. Ephemeral hidden services, substantially faster descriptor parsing, and a host of other improvements can all be yours with a simple upgrade! https://blog.torproject.org/blog/stem-release-14 Nyx Log Panel Rewrite No, Nyx isn't done but its log panel is! This is the third major curses component to be rewritten, and I'm delighted to say it now sucks less! How did it suck, you ask? Though casual users may not have noticed arm had a laughably inefficient O(n^3) approach to log deduplication. This was in fact so terrible arm simply turned off deduplication when it started bogging down the system too much. This is just one of they myriad of improvements, but in Nyx deduplication is O(n^2) and can easily be made linear (... though fuzzy matching makes this grosser than I'd like so first seeing if further improvement is warranted). Few other noteworthy things were... * Thanks to DonnchaC Stem's tor message parsing is now a dramatic 300x faster when using python3! https://github.com/DonnchaC/stem/pull/1 * Wrote a simple bandwidth scanner to exemplify manual circuit construction for our tutorials... https://stem.torproject.org/tutorials/to_russia_with_love.html#custom-path-selection * Sambuddha both made our tutorial examples downloadable (#10411) and more easily testable (#16191)! Cheers! -Damian ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
[tor-dev] Visualising similarities between relay descriptors
Visualising the similarity between two Tor relay descriptors helps with finding Sybil attacks. I added code to sybilhunter [0] that takes as input relay descriptors, determines all (n^2)/2 pairwise similarities, and outputs DOT code (part of Graphviz) that illustrates relay clusters and what makes them similar. For now, this functionality is just a set of hard-coded rules that determine, e.g.: - Do the relays have the same, non-default exit policy? - Do the relays have a similar uptime? - Do the relays run on the same platform? To give you an idea of what this looks like, I took all relay descriptors archived by CollecTor [1] for 2015-05-30 and calculated similarities by running: $ sybilhunter -data 2015-05-30/ -cumulative -matrix -threshold 6 -visualise sim.dot $ dot -o sim.svg -Tsvg sim.dot The resulting graph is online [2]. Vertices are relays (nickname in the first line, followed by the first eight hex digits), and the edge labels show the similarities. Unsurprisingly, there are several relay clusters that probably should be in a family, but aren't. For example, the startor*, torpids*, manningsnowden*, and Montharkan* relays. There are, however, also several relay clusters, often named default, that share the first two hex digits of their fingerprint. This is unlikely to be a coincidence, so they might have wanted to position themselves in the DHT. Please let me know if you have any suggestions on how to improve the tool or its visualisation. [0] https://gitweb.torproject.org/user/phw/sybilhunter.git/ [1] https://collector.torproject.org/recent/relay-descriptors/server-descriptors/ [2] https://www.nymity.ch/sybilhunting/svg/2015-05-30_similarities.svg Cheers, Philipp ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] valid MyFamily syntax variations ('$FP=nick' ?)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi, That would be invalid. I know there are some families which do it. You are free to enter a nickname at MyFamily parameter in your torrc, but it won't actually work for clients. Tor will still start on relay side and go on with it. It used to work back when we were using Named / Unnamed flags. If the relay had the Named flag, you could just enter its nickname at Tor would automatically map that nickname to its proper fingerprint. It did not work for relays with Unnamed flag or relays without both Named and Unnamed flags. I have opened a ticket for this on trac, but we fixed the FAQ. https://trac.torproject.org/projects/tor/ticket/12092 On 5/31/2015 10:48 PM, nusenu wrote: Hi, according to the spec [1] relay ops can choose between fingerprint (with and without $ prefix) and nicknames when constructing their MyFamily configs. The man page recommends fingerprints. Now I'm faced with [2][3] families that use a combination: $fp=nick Are they actually valid or is this a bug or can one actually specify arbitrary strings there? (dir auths and onionoo apparently accept them) thanks [1] https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n504 [2] https://atlas.torproject.org/#details/2B76359D3AAA94CA107F447D2D14E481 A99EB207 (search for KosakaKirino) [3] https://collector.torproject.org/recent/relay-descriptors/server-descr iptors/2015-05-30-14-05-48-server-descriptors (this url has a lifetime) ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) iQEcBAEBCAAGBQJVa2o+AAoJEIN/pSyBJlsR6qgH/jOO//s8CGKkrGYRxWMHrSWK JUhH4xj8YPm7fTiDDH2MmKerGbC3N1l2ENKs03f1zDWfzxuxuWch2kdA+mN7GUaB xPAeAyksk4Rp6eMs9DhnZLUTLC1vHbta7Yx9qlVs5sc5aeR+HbKMl7jfIHStGQxF swhY2Er4bvMj4VbtMnWbcqTP1037UyEIriTgdLxXgeb1B58BW/i3sjtqDpWuZSrc 7a1SCYOGfIzu96WJi542LGVft2VJmlwFlZwV8Nq3ph2cq6AhBfxVuTKauoUcdG43 JbdWmiYzTd9UpoIoc8X6Z9SJIz6FXMKKWIBN/xvNCJpueveDw9sz9IaFEtdvooo= =JQWJ -END PGP SIGNATURE- ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] valid MyFamily syntax variations ('$FP=nick' ?)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi, based on your answer and linked trac entry, I assume my point was not entirely clear. I was specifically wondering about a combination of *both* ($FP=nickname) something I haven't seen till now: Example as provided in the URLs of my last email: $2B76359D3AAA94CA107F447D2D14E481A99EB207=Konata,$827BF0F1E52B82D69B3CE3DFEE1F1FE3A11042FD=KosakaKirino (onionoo excerpt) have a look at this relay's family: https://atlas.torproject.org/#details/2B76359D3AAA94CA107F447D2D14E481 A99EB207 Normally you see, fingerprints only: $FP1, $FP2, ... or Nicknames only: PPTOR0042,PPTOR0043,PPTOR0044, ... or nicknames next to fingerprints: $FP1, oilsrv1, ... -BEGIN PGP SIGNATURE- iQIcBAEBCgAGBQJVa23KAAoJEFv7XvVCELh02vQP/R/d0kNfQM3r30CZj4jXExbT XIuas66C/tqxtF8BPSQ3+xCUSStknbCEuhoMnjDJr4J5G5M5RaFk7XSOl4DbZ56/ GoVVKvG6HYEUHdnkAwk7dUDJQELZ+4UB4s24wPV/mFkzjx/503dA7TWeVWE9O2/E rnUvPPo8rMZ8vh6rksgBId3XvqFl6ShHduToNe83leomwl1V9qZY5F28dUrBkcFp ZIPhPsnonZYXRqlsW6WJlFWwAfTou5do8H7yJSAZd6+0+w0PLZKoF0pzH7vE5Ht8 T9dKQSw/9ZFbL8iJMdf0lVWpkhDGKHkDEOx7X8oRAVBrKdYj7xvnGhcj8XkwLGp3 h0L3pTqH3ApqlohCyuSA3ePYw4Cwy1ZJt0ix0t8PDRmm+j/r2uxELZHyWF7tV8CS LCpKLnWIecEOtA9fnn4eCTLqQQ/PZgSqJ3pJdhK8wFQXSHOD0vr0xjssPDohIfE3 xtBSQocainlCLtaWc4ickrDpZgldIVSuKaSRxGs67Pxd4qxxaQEDPG3F+UrCbUJU Xoy2bD9LXhD56H8mKp35L0wOBK/rl81HHlVztNqUVvNN/kyHvFFb1KRhtQaXKTYg NLefBY9/a+P3cHZ/Yv6PlugyNWQmc5QiqrLn3er11SOWde6Rg3si2jEEAz9MYiAT A8J3fn94VkaYR1f4uIlz =1VTB -END PGP SIGNATURE- ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
[tor-dev] valid MyFamily syntax variations ('$FP=nick' ?)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi, according to the spec [1] relay ops can choose between fingerprint (with and without $ prefix) and nicknames when constructing their MyFamily configs. The man page recommends fingerprints. Now I'm faced with [2][3] families that use a combination: $fp=nick Are they actually valid or is this a bug or can one actually specify arbitrary strings there? (dir auths and onionoo apparently accept them) thanks [1] https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n504 [2] https://atlas.torproject.org/#details/2B76359D3AAA94CA107F447D2D14E481A99EB207 (search for KosakaKirino) [3] https://collector.torproject.org/recent/relay-descriptors/server-descriptors/2015-05-30-14-05-48-server-descriptors (this url has a lifetime) -BEGIN PGP SIGNATURE- iQIcBAEBCgAGBQJVa2WkAAoJEFv7XvVCELh0jEwQAJqdeZaFpRWPfI9/17o7aEKI Jx8Lp811GGwr4STO41ixwOIQ69hjNuav47apcmR4AsUVAq8zkSnzljKyYtVnp7qH 4EHx0kqIop92s9de1AFKg2XDnZ3u4eSN6z53VXeO2+/jFl2CgMVarzLYPHNBBF/x ev3zMk8gYsGA7K5TowZ+HnMalXhHLpTvO+iPPzNF6eY3/pG31hImFfHvhkJ5nrzB gex9wmCTXigM6I1mcT8mK8OH/2+MkNFra2h4tcEmqzmzRdWx1UWDTgl97Jp2injL UxuHL1RkoGLqlZ/MDiMiZPa4JurNRKgBXdZNsASOvcX8ZiPiCzqbbCm4hN4Rsidi ROWR8AcadA/9a5Ovi0z8IhxUiF8Ks2Wadeovp4w+E2NuYvFNmUpvkH0jVSViSE8n v+XClj0DX8oHirRqp80M0o5GpOB2hsRzT0+QhRFKYbikW2CRziM1ECWknw3aj3/z VPohoo6bkkLLBfrrXdgqclvgtoG/zPl8Ef203pmyTjPCFEdx4mJTLzq+hmjoaPsI h+RweF+HvDc0qM+6fiYeHExQl7YMNuT6GbG58lK6V8c+r++FIrPwfyOepAddoE0p dTAHfqYMYhDL952z916tmNNPnHXsUu8hVaIVuB9RvYjAU9FpPAz5Q5CDaIu+F4TY kb9dxHK1c8IHOGamYMOD =tbZE -END PGP SIGNATURE- ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] valid MyFamily syntax variations ('$FP=nick' ?)
Hi nusenu, The spec isn't done :P Seriously though, no it's not a bug. If you check nodelist [0] you'll see that this type of hex-encoded nickname is normal for generating a descriptor. If you check CollecTor history for the node your mention [1] you'll see the result of building a descriptor. Metrics-lib parses this out in ServerDescriptorImpl [2]. Which then shows up in Atlas via Onionoo in DetailsStatus [3] during NodeDetailsStatusUpdater [4]. I hope that clears things up. (I know I skipped a full stack trace but its C and Java--cut me some slack) --leeroy [0] https://gitweb.torproject.org/tor.git/tree/src/or/nodelist.c#n499 [1] https://collector.torproject.org/recent/relay-descriptors/server-descriptors/2015-05-31-20-05-49-server-descriptors [2] https://gitweb.torproject.org/metrics-lib.git/tree/src/org/torproject/descriptor/impl/ServerDescriptorImpl.java#n341 [3] https://gitweb.torproject.org/onionoo.git/tree/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java [4] https://gitweb.torproject.org/onionoo.git/tree/src/main/java/org/torproject/onionoo/updater/NodeDetailsStatusUpdater.java ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Sanitizing bridge descriptors containing ed25519 fields
On Sat, May 30, 2015 at 5:16 PM, Karsten Loesing kars...@torproject.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 30/05/15 16:44, Nick Mathewson wrote: On Fri, May 29, 2015 at 4:23 PM, Karsten Loesing kars...@torproject.org wrote: router euler [scrubbed] 8000 0 0 identity-ed25519 -BEGIN ED25519 CERT- [scrubbed] -END ED25519 CERT- Base64-decode that block, throw it into SHA256(), base64-encode the result, format as block. But wouldn't the result be much shorter? There's no new fingerprint equivalent, like fingerprint-ed25519, is there? Oh dear. That blob is a certificate, not a key. It changes over time, because the key that it certifies varies over time. The format is documented in section 2.1 of proposal 220; the actual identity key is in an extension labeled with type 04 (see section 2.2.1). Maybe we should add a fingerprint-ed25519 line? It would be redundant, but maybe valuable. What do you think? I took a quick look at proposal 220, but not to the point that makes me entirely certain about this ed25519 stuff. If anything below remains unclear, that might be because I'm not clear about it myself. Please question everything I'm saying, even more than usual. Let's also include Damian on this thread. He usually has good ideas about descriptors, and he knows about sanitizing bridge descriptors. And let's add Isis who is working a lot with bridge descriptors, too. (There may be more people we should ask, but hey, it's tor-dev@.) So, I think a fingerprint-ed25519 line would be useful. It would make the bridge descriptor sanitizing process much easier. It would also facilitate debugging network problems, because people can just grep descriptors rather than using specialized tools that know how to decode the cert. And with microdescriptors in place it should be okay to add this line even if it's redundant information, because clients would never download it. Hm. Okay, that sounds solid enough. I'll try to hack it in tonight or Monday, and add it to prop220. If the server descriptor in #16227 were a bridge descriptor, I'd do the following steps for sanitizing it: - Remove identity-ed25519 line and subsequent crypto block. +1 - Keep yet-to-be-added fingerprint-ed25519 line, but SHA256 its value. - Keep extra-info digest line, but SHA1 and SHA256 its values. Suggestion: Don't use raw SHA256(x) but instead use SHA256(sanitize ID for bridge descriptor | x) or SHA256(sanitize extra-info digest for bridge descriptor | x). That way we don't need to worry about colliding with something else that decides to SHA256 these. - Remove onion-key line and subsequent crypto block. - Remove signing-key line and subsequent crypto block. - Remove onion-key-crosscert line and subsequent crypto block. - Remove ntor-onion-key-crosscert line and subsequent crypto block. - Keep ntor-onion-key line, mostly because we didn't remove it so far; unless it should be removed? - Remove router-sig-ed25519 line. - Remove router-signature line and subsequent crypto block. - Add router-digest line with SHA1 of SHA1 of descriptor content and SHA256 of SHA256 of descriptor content; the rationale is that we don't have to write entirely new digests into the network status in order to match r lines with descriptors. That all sounds fine. I also added the extra-info descriptor that corresponds to the server descriptor to #16227: https://trac.torproject.org/projects/tor/ticket/16227#comment:5 I wonder, what's the best way for including the ed25519 identity in the extra-info descriptor? How about extending the first line to: extra-info Truie SHA1-of-RSA SHA256-of-ed25519 Possible downsides are that this additional value might break existing code and that it might be problematic to get rid of the SHA1-of-RSA part later. But the same issues would come up with the extra-info-digest line in server descriptors, and maybe there are good solutions. Otherwise, a separate fingerprint-ed25519 line might work here, too. Plausible. In order to sanitize such an extra-info descriptor, I'd do these steps: - Keep extra-info line, but SHA1 (and possibly SHA256) its value(s). See above. - Or, keep yet-to-be-added fingerprint-ed25519 line, but SHA256 its value. See above. - Remove router-sig-ed25519 line. - Remove router-signature line and subsequent crypto block. - Add router-digest line with SHA1 of SHA1 of descriptor content and SHA256 of SHA256 of descriptor content; same rationale as above, but with the extra-info-digest line in server descriptors. What do you think? Sounds fine. ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
[tor-dev] Tor Browser IsolateSOCKSAuth behavior questions.
Hello, I've been working on a dumb hack that lets me do things like this: https://imgur.com/3mah244 (Yes, that's a single Tor Browser instance, separate windows used for illustrative purposes.) It's still very raw and doesn't do everything I want it to do, so I'm not really releasing the code yet, but I have some questions regarding how Tor Browser behaves when setting the SOCKS username for isolation purposes. Ideally I want my shim to enforce isolation between the various upstreams (Tor, I2P, whatever) correctly to avoid cross-protocol probing (and to shield the I2P administration interface from eeevil websites). This appears to be straight forward if the application is Tor Browser because IsolateSOCKSAuth is always used at first glance (I will assume for now that if users decide to use things like torsocks that do not use isolation this way that they know what they are doing). My question is, what causes Tor Browser to set the SOCKS username to --unknown-- and what the behavior should be in that case if: * The destination is a .onion address. * The destination is a .i2p address. * The destination is the I2P management console. I'm fairly sure this should be deny. * The destination is any other address (will be dispatched over Tor if running, I don't think I will attempt to support I2P outproxies because they suck). (I think allow because things break otherwise?) For destinations that are .onion/.i2p, I plan to be fairly strict about making sure the SOCKS5 target and the username matches (I need to be more relaxed for sites on the regular intertubes since cross-site resources are loaded (I may make this behavior configurable...). Is this dumb? Is it common for foo.onion to load resources off bar.onion? How about in I2P land? The final form of my shim will support running with any combination of nothing (Tor Browser just for the privacy benefits, probably unsafe, I may reconsider this), I2P, and Tor (Though the most useful configuration is probably I2P + Tor). Thanks in advance, -- Yawning Angel pgp4CFhRjXQuC.pgp Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
