[tor-dev] Globe without search term
21:43 #tor-dev: rndm karsten, looks like this now if it gets the maximum number of results http://globe.rndm.de/#/search/query= You're not online now, so replying via email: Looks really good! Minor tweaks to the phrasing: To avoid too many requests, - the better motivation is that a) we don't want to overload the poor Onionoo server and b) reduce delays and bandwidth requirements for the client. I think users will understand b) better and not care much about a). How about To reduce waiting time and limit bandwidth usage, ? try to use a search word - this message also shows up if the user tried a search term that is too general. How about try to refine your search ? Thanks! Nice work! Best, Karsten ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Using Stem's descriptor fetching module to replace the Java consensus-health checker
On 8/12/13 2:02 AM, Damian Johnson wrote: Hi Karsten, just finished throwing together a script that does seven of the eighteen DocTor checks. The rest shouldn't be hard, just take a little elbow grease... https://gitweb.torproject.org/atagar/tor-utils.git/commitdiff/1e49c33 Cool! I quickly looked at the commit, and this is probably due to the early state of the script, but I thought I'd mention it anyway: I wondered how the single try block around get_consensuses, get_votes, and run_checks would respond to single directory authorities being unavailable, closing connections in the middle of the download, taking longer than 60 seconds, etc. I think the 60 seconds thing might be handled fine, but would the other I/O errors make the script not run any checks? As for the website, why is that part of the same codebase as the monitors? The site doesn't look to make use of the derived warnings. Is this simply a kludge since they both make use of the same descriptor data? The kludge is that checks and website don't share code, not that there's a website. The idea of the typical use case is that people receive a warning via email or IRC and then go to the website to learn more details. Here's how I could imagine integrating checks and website more closely: for each type of warning, there's a separate class that knows how to do the following things: - look at previously downloaded consensuses and/or votes to decide if there's something to warn about, - print out a warning message if something's not okay, - decide on a severity, - define rate limiting of this warning message, and - produce the HTML for the website. Note that the large table at the end of the current consensus-health page is probably different, because it contains much more information than what's required to further investigate a warning message. We should not include that table in your Python rewrite. I just took it out from the metrics website to see if anybody cares. For reference, here's the archived latest consensus-health.html that contains that table: https://people.torproject.org/~karsten/volatile/consensus-health-2013-08-12-07-00-00.html#relayflags The website might be a good use case for Hyde (http://ringce.com/hyde). Plausible, yes. Can't say much about tools, but something to generate static HTML sounds like a fine choice. That said, this feels like it should belong in the metrics-web repository... No, we should rather move the website output to its own subdomain, e.g., doctor.tpo. It's a kludge that it's on the metrics website. It doesn't belong there, as much as ExoneraTor and relay search don't belong there. All the best, Karsten ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Using Stem's descriptor fetching module to replace the Java consensus-health checker
On 8/12/13 10:51 AM, Damian Johnson wrote: The kludge is that checks and website don't share code, not that there's a website. The idea of the typical use case is that people receive a warning via email or IRC and then go to the website to learn more details. I disagree. This repository contains two very distinct applications: * monitors for issues with the votes * a website that renders the present content of the votes If someone's only interested in a presentation of vote contents, then DocTor shouldn't be their tool. If vote contents are interesting to anyone, we should add them to Onionoo and have some Onionoo client present them. This is not what I have in mind for DocTor. I'm only interested in providing directory authority operators with the information they need to fix problems with the voting process. (Also note that you only mention votes above. But DocTor also looks at problems with serving consensuses, e.g., connection problems when downloading the consensus, or serving outdated consensuses.) The use cases for each are associated, but bundling them together makes about as much sense as lumping vidalia and tor within the same repository. That's not really true. What you don't see right now is that problems with the consensus or votes would be highlighted in the website output. For example, if either gabelmoo or mori1 or tor26 is missing a certain recommended version, that line will be printed in red. So, there's a close relation between status notifications and the website, just not in the code. Personally I'm a big fan of these monitors, but less so the website. I don't think it's especially useful (precious few people have cause to find a side-by-side comparison of vote attributes to be interesting, and fewer still would opt for this over reading the documents). But that said, it's not overly much code. I might toy with Hyde to generate the site after finishing the monitors but no promises. That part is not something I would want to own for the long term, though. Well, maybe let's step back then and find a solution that you're happy to own for the long term. Once your tool is online, I'm planning to shut down the current consensus-health checker including the website output, so your tool should contain all the information that people need to fix problems in the consensus. In my view, writing the results of a DocTor run to a website and highlighting problems in red was the easiest way to provide directory authority operators with all information they need. I could also imagine adding additional information about warnings to the bottom of status notification emails. So, warnings on top and then one paragraph for each warning requiring additional information. Or maybe there are other ways to provide this additional information. We should also include Sebastian and Peter in this discussion, because they cared about consensus-health output in the past and may have more suggestions. Cc'ed them. Thanks, Karsten ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Status report - Stream-RTT
On Saturday 10 August 2013 23:52:44 Damian Johnson wrote: If I understand this correctly you're thinking that multiple calls to extend_circuit() cause parallel EXTENDCIRCUIT requests, and the first response would be used for both callers. Is that right? Yes. If so then I would be very interested if you actually see that behaviour. Stem provides thread safe controller communication. See the msg() method of the BaseController - though the Controller's methods are called in parallel the actual socket requests are done in serial to prevent that exact issue that you describe. That looks fine to me. I obviously drew the wrong conclusion from the issues I have encountered. My fault, sorry. Best, Robert signature.asc Description: This is a digitally signed message part. ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] [GSoC 2013] Status report - Searchable metrics archive
On 8/10/13 9:28 PM, Kostas Jakeliunas wrote: * I don't think we can avoid using certain postgresql extensions (if only one) - which means that deploying will always take more than apt-get pip install, but I believe it is needed; Can you give an example of a query that won't be executed efficiently without this extension and just fine with it? Maybe we can tweak that query somehow so it works fine on a vanilla PostgreSQL. Happy to give that some thoughts. I'd really want to avoid using stuff that is not in Debian. Or rather, if we really need to add non-standard extensions, we need more than thinking and believing that it's unavoidable. :) Overall, I'm spending a bit too much time on a specific problem, but at least I have a more intimate lower-level knowledge of PostgreSQL, which turns out to be very relevant to this project. I hope to be able to soon move to extending Onionoo support and providing a clean API for getting lists of consensuses in which a particular relay was present. And maybe start with the frontend. :) Sounds like a plan, except for the frontend part. Scratch that, I'd say, and focus on the API. Making it shiny can come once everything works as expected. Best, Karsten ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] [GSoC 2013] Status report - Searchable metrics archive
Karsten, this won't be a very short email, but I honestly swear I did revise it a couple of times. :) This is not urgent by any measure, so whenever you find time to reply will be fine. ctrl+f to observe: for some precise data / support for my plan re: using the pg_prewarm extension. On Mon, Aug 12, 2013 at 2:16 PM, Karsten Loesing kars...@torproject.orgwrote: On 8/10/13 9:28 PM, Kostas Jakeliunas wrote: * I don't think we can avoid using certain postgresql extensions (if only one) - which means that deploying will always take more than apt-get pip install, but I believe it is needed; Can you give an example of a query that won't be executed efficiently without this extension and just fine with it? Maybe we can tweak that query somehow so it works fine on a vanilla PostgreSQL. Happy to give that some thoughts. I'd really want to avoid using stuff that is not in Debian. Or rather, if we really need to add non-standard extensions, we need more than thinking and believing that it's unavoidable. :) First off, the general idea. I know this might not sound convincing (see below re: this), but any query that uses an index will take significantly longer to execute if it needs to load parts of the index from disk. More precisely, query time deviation and max(query_time) inversely correlates with the percentage of the index in question in memory. The larger the index, the more difficult it is to 'prep' it into cache, the more unpredictable query exec time gets. Take a look at the query used to join descriptors and network statuses given some nickname (could be any other criterion, e.g. fingerprint or IP address): https://github.com/wfn/torsearch/blob/master/misc/nested_join.sql We use the following indexes while executing that query: * lower(nickname) on descriptor * (substr(fingerprint, 0, 12), substr(lower(digest), 0, 12)) on statusentry (this one is used to efficiently join descriptor table with statusentry: (fingerprint, descriptor) pair is completely unique in the descriptor table, and it is fairly unique in the statusentry table (whereas a particular fingerprint usually has lots and lots of rows in statusentry)); this index uses only substrings because otherwise, it will hog memory on my remote development machine (not EC2), leaving not much for other indexes; this composite substring index still takes ~2.5GB for status entries (only) in the range between [2010-01; 2013-05] as of now * validafter on statusentry (the latter *must* stay in memory, as we use it elsewhere as well; for example, when not given a particular search criterion, we want to return a list of status entries (with distinct fingerprints) sorted by consensus validafter in descending order) We also want to keep a fingerprint index on the descriptor table because we want to be able to search / look up by fingerprint. I'm thinking of a way to demonstrate the efficiency of having the whole index in memory. For now, let me summarize what I have observed, intersect with what is relevant now: running the aforementioned query on some nickname that we haven't queried for since the last restart of postgresql, it might take, on average, about 1.5 to 3 seconds to execute on EC2, and considerably longer on my development db if it is a truly popular nickname (otherwise, more or less the same amount of time); sometimes a bit longer - up to ~4s (ideally it should be rather uniform since the indexes are *balanced* trees, but.. and autovacuum is enabled.) Running that same query later on (after we've run other queries after that first one), it will take = 160ms to execute and return results (this is a conservative number, usually it's much faster (see below)). Running EXPLAIN (ANALYZE, BUFFERS) shows that what happened was that there was no [disk] read next to index operations - only buffer hit. This means that there was no need to read from disk during all the sorting - only when we knew which rows to return did we need to actually read them from disk. (There are some nuances, but at least this will be true for PostgreSQL = 9.2 [1], which I haven't tried yet - there might be some pleasant surprises re: query time. Last I checked, Debian 9.0 repository contains postgresql 9.1.9.) Observe: 1a. Run that query looking for 'moria2' for the first time since postgresql restart - relay is an old one, only one distinct fingerprint, relatively few status entries: http://sprunge.us/cEGh 1b. Run that same query later on: http://sprunge.us/jiPg (notice: no reads, only hits; notice query time) 2a. Run query on 'gabelmoo' (a ton of status entries) for the first time (development machine, query time is rather insane indeed): http://sprunge.us/fQEK 2b. Run that same query on 'gablemoo' later on: http://sprunge.us/fDDV PostgresSQL is rather clever: it will keep the parts of indexes more often used in cache. What pg_prewarm simply does is: * load all (or critical for us) indexes to memory (and load them whole), which is possible
Re: [tor-dev] Using Stem's descriptor fetching module to replace the Java consensus-health checker
I'm only interested in providing directory authority operators with the information they need to fix problems with the voting process. Hi Karsten. I'm going to focus on the monitors for now and come back to discussion of a website after Sebastian and Peter have a chance to respond. That said, we totally agree on this point - the goal of this project is solely to help detect and resolve issues with authorities and consensus generation. If the website is vital to that then great (I still think it should be its own repository, but it's then definitely worth keeping). However, if we can do just as well by including additional information in the warnings then that would be even more maintainable in the long term. My thoughts on this are probably best explained by a tangent. For years I've run a sybil checker called consensusTracker.py [1]. This checker did a bit more than just watch for sybil attacks. It also generated a pretty html report of the wax and wane in relay counts over the week among other things. While these bells and whistles were pretty, no one cared about them and rightfully so. The sole purpose of a sybil checker is to provide us a notification saying Oi! Potential problem here, recent relay additions are The html report, while pretty, was both useless and made the code far, far more complicated than it needed to be. Finally I replaced that 601 line script with a far simpler 116 line counterpart that does just what we want and nothing more [2]. The monitor is now more reliable, maintainable, and can easily be updated in the future when necessary (something I couldn't do with the mess that was the previous script). The lesson that I learned from this was Start with the ending goal of a project and code toward that. Anything else will just result in feature creep. This isn't to necessarily say the DocTor website is unnecessary (only the site's intended audience like Peter can tell us that), but I definitely think we should figure out what he needs before resolving to keep it. Cheers! -Damian [1] https://gitweb.torproject.org/atagar/tor-utils.git/blob/e537044:/consensusTracker.py [2] https://gitweb.torproject.org/atagar/tor-utils.git/commitdiff/9b7de30 ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Globe without search term
On 12.08.2013 08:23, Karsten Loesing wrote: 21:43 #tor-dev: rndm karsten, looks like this now if it gets the maximum number of results http://globe.rndm.de/#/search/query= You're not online now, so replying via email: Looks really good! Minor tweaks to the phrasing: To avoid too many requests, - the better motivation is that a) we don't want to overload the poor Onionoo server and b) reduce delays and bandwidth requirements for the client. I think users will understand b) better and not care much about a). How about To reduce waiting time and limit bandwidth usage, ? try to use a search word - this message also shows up if the user tried a search term that is too general. How about try to refine your search ? Thanks! Nice work! Best, Karsten Thanks for your correction, I'm going to add it soon. I've seen that you've included the field parameter to limit the fields that onionoo returns. Do you think it would be better to use the field parameter and show a limited amount of data or try to get everything and only display the fields that have a value? Btw I had an idea to create a small js snippet that allows users to show data of an relay or bridge. It would use the onionoo api to send a request with a given fingerprint and renders the result in a small piece of html. This way someone won't need to link to a specified globe/atlas page and can directly show visitors information about an relay/bridge. I don't know if it would be useful for anyone. What's your opinion on that? Cheers, Christian ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
[tor-dev] Proposal 221: Stop using CREATE_FAST
Filename: 221-stop-using-create-fast.txt Title: Stop using CREATE_FAST Authors: Nick Mathewson Created: 12 August 2013 Target: 0.2.5.x Status: Open 0. Summary I propose that in 0.2.5.x, Tor clients stop sending CREATE_FAST cells, and use CREATE or CREATE2 cells instead as appropriate. 1. Introduction The CREATE_FAST cell was created to avoid the performance hit of using the TAP handshake on a TLS session that already provided what TAP provided: authentication with RSA1024 and forward secrecy with DH1024. But thanks to the introduction of the ntor onionskin handshake in Tor 0.2.4.x, for nodes with older versions of OpenSSL, the TLS handshake strength lags behind with the strength of the onion handshake, and the arguments against CREATE no longer apply. Similarly, it's good to have an argument for circuit security that survives possible breakdowns in TLS. But when CREATE_FAST is in use, this is impossible: we can only argue forward-secrecy at the first hop of each circuit by assuming that TLS has succeeded. So let's simply stop sending CREATE_FAST cells. 2. Proposed design Currently, only clients will send CREATE_FAST, and only when they have FastFirstHopPK set to its default value, 1. I propose that we change FastFirstHopPK from a boolean to also allow a new default auto value that tells Tor to take a value from the consensus. I propose a new consensus parameter, usecreatefast, default value taken to be 1. Once enough versions of Tor support this proposal, the authorities should set the value for usecreatefast to be 0. In the series after that (0.2.6.x?), the default value for FastFirstHopPK should be 0. (Note that CREATE_FAST must still be used in the case where a client has connected to a guard node or bridge without knowing any onion keys for it, and wants to fetch directory information from it.) 3. Alternative designs We might make some choices to preserve CREATE_FAST under some circumstances. For example, we could say that CREATE_FAST is okay if we have a TLS connection with a cipher, public key, and ephemeral key algorithm of a given strength. We might try to trust the TLS handshake for authentication but not forward secrecy, and come up with a first-hop handshake that did a simple curve25519 diffie-hellman. We might use CREATE_FAST only whenever ntor is not available. I'm rejecting all of the above for complexity reasons. We might just change the default for FastFirstHopPK to 1 in 0.2.5.x-alpha. It would make early users of that alpha easy for their guards to distinguish. 4. Performance considerations This will increase the CPU requirements on guard nodes; their cpuworkers would be more heavily loaded as 0.2.5.x is more adopted. I believe that, if guards upgrade to 0.2.4.x as 0.2.5.x is under development, the commensurate benefits of ntor will outweigh the problems here. This holds even more if we wind up with a better ntor implementation or replacement. 5. Considerations on client detection Right now, in a few places, Tor nodes assume that any connection on which they have received a CREATE_FAST cell is probably from a non-relay node, since relays never do that. Implementing this proposal would make that signal unreliable. We should do this proposal anyway. CREATE_FAST has never been a reliable signal, since FastFirstHopPK 0 is easy enough to type, and the source code is easy enough to edit. Proposal 163 and its successors have better ideas here anyway. ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
[tor-dev] [draft] Proposal 220: Migrate server identity keys to Ed25519
Here's a proposal I wrote about node key migration. I hope it meshes well with the authority identity migration proposal that Jake and Linus are doing. There are probably holes and mistakes here: let's fix them. Filename: 220-ecc-id-keys.txt Title: Migrate server identity keys to Ed25519 Authors: Nick Mathewson Created: 12 August 2013 Target: 0.2.5.x Status: Draft [Note: This is a draft proposal; I've probably made some important mistakes, and there are parts that need more thinking. I'm publishing it now so that we can do the thinking together.] 0. Introduction In current Tor designs, identity keys are limited to 1024-bit RSA keys. Clearly, that should change, because RSA doesn't represent a good performance-security tradeoff nowadays, and because 1024-bit RSA is just plain too short. We've already got an improved circuit extension handshake protocol that uses curve25519 in place of RSA1024, and we're using (where supported) P256 ECDHE in our TLS handshakes, but there are more uses of RSA1024 to replace, including: * Router identity keys * TLS link keys * Hidden service keys This proposal describes how we'll migrate away from using 1024-bit RSA in the first two, since they're tightly coupled. Hidden service crypto changes will be complex, and will merit their own proposal. In this proposal, we'll also (incidentally) be extirpating a number of SHA1 usages. 1. Overview When this proposal is implemented, every router will have an Ed25519 identity key in addition to its current RSA1024 public key. Ed25519 (specifically, Ed25519-SHA-512 as described and specified at http://ed25519.cr.yp.to/) is a desirable choice here: it's secure, fast, has small keys and small signatures, is bulletproof in several important ways, and supports fast batch verification. (It isn't quite as fast as RSA1024 when it comes to public key operations, since RSA gets to take advantage of small exponents when generating public keys.) (For reference: In Ed25519 public keys are 32 bytes long, private keys are 64 bytes long, and signatures are 64 bytes long.) To mirror the way that authority identity keys work, we'll fully support keeping Ed25519 identity keys offline; they'll be used to sign long-ish term signing keys, which in turn will do all of the heavy lifting. A signing key will get used to sign the things that RSA1024 identity keys currently sign. 1.1. 'Personalized' signatures Each of the keys introduced here is used to sign more than one kind of document. While these documents should be unambiguous, I'd going to forward-proof the signatures by specifying each signature to be generated, not on the document itself, but on the document prefixed with some distinguishing string. 2. Certificates and Router descriptors. 2.1. Certificates When generating a signing key, we also generate a certificate for it. Unlike the certificates for authorities' signing keys, these certificates need to be sent around frequently, in significant numbers. So we'll choose a compact representation. VERSION [1 Byte] TYPE[1 Byte] CERTIFIED_KEY [32 Bytes] EXPIRATION_DATE [4 Bytes] EXTRA_STUFF [variable length] SIGNATURE [64 Bytes] The VERSION field holds the value [01]. The TYPE field holds the value [01]. The CERTIFIED_KEY field is an Ed25519 public key. The expiration date is a day, given in DAYS since the epoch, after which this certificate isn't valid. The EXTRA_STUFF field is left for a future version of this format. [ Is EXTRA_STUFF a good idea? -NM] Before processing any certificate, parties MUST know which identity key it is supposed to be signed by, and then check the signature. The signature is formed by signing the first N-64 bytes of the certificate prefixed with the string Tor node signing key certificate v1. We also specify a revocation document for revoking a signing key or an identity key. Its format is: FIXED_PREFIX[8 Bytes] VERSION [1 Byte] KEYTYPE [1 Byte] IDENTITY_KEY[32 Bytes] REVOKED_KEY [32 Bytes] PUBLISHED [8 Bytes] EXTRA_STUFF [variable length] SIGNATURE [64 Bytes] FIXED_PREFIX is REVOKEID or REVOKESK. VERSION is [01]. KEYTYPE is [01] for revoking a signing key or [02] or revoking an identity key. REVOKED_KEY is the key being revoked; IDENTITY_KEY is the node's Ed25519 identity key. PUBLISHED is the time that the document was generated, in seconds since the epoch. EXTRA_STUFF is left for a future version of this document. The SIGNATURE is generated with the same key as in IDENTITY_KEY, and covers the entire revocation, prefixed with Tor key revocation v1. Using these revocation documents is
