Re: [tor-dev] Starting with contributing to Anonymous Local Count Statistics.

[George Kadianakis]

Jaskaran Singh  writes:

> [ text/plain ]
> Hi,
>
> I thought the project idea had already been depreciated in favor of
> counting unique users by directory fetches. No?
>

Yes, we do count unique users by directory fetches for the "active Tor
users" metric: https://metrics.torproject.org/userstats-relay-country.html

But we also use in-memory data structures tracking IP addresses to count
unique users per-country: 
https://metrics.torproject.org/userstats-bridge-combined.html?start=2017-11-04=2018-02-02=dz

I was not aware that we are planning to deprecate the latter in favor of
counting directory fetches. Did you get that from somewhere? Perhaps it
could make sense, not sure.

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Starting with contributing to Anonymous Local Count Statistics.

[George Kadianakis]

Aruna Maurya  writes:

> [ text/plain ]
> Hey!
>
> I was going through the Tor Volunteer page and came across the Anonymous
> local count statistics project. As a student it would be a great starting
> point and an even bigger opportunity to get a chance to collaborate and
> learn in the process.
>
> I would like to contribute to it, and would love to start as soon as
> possible. It would be great if someone could guide me through.
>

Hello Aruna,

thanks for reaching out.

I also find this project interesting. I'd like to help you but my time
is quite limited lately.

What would you like guidance with?

With regards to design, I suggest you take a look at the last comments
of this trac ticket:  
https://trac.torproject.org/projects/tor/ticket/7532#comment:22
Particularly it seems like the PCSA algorithm might be a reasonable way
forward.

With regards to coding, I suggest you familiarize yourself with the Tor
codebase. Some specific places to look at would be the way that Tor
currently counts users. For example, see geoip_note_client_seen() and
its callers, for when bridges register new clients to their stats
subsystem. Also check geoip_format_bridge_stats() for when bridges
finally report those stats.

Let us know if you have any specific questions!

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Bandwidth Authority Progress

[George Kadianakis]

teor  writes:

> Hi asn,
>
> Original Subject: Re: [tor-project] Meeting notes, network team meeting, 18 
> Dec
>
>> On 19 Dec 2017, at 07:28, Nick Mathewson  wrote:
>> 
>>  - Met with David Stainton, Moritz and others. Talked about relay load
>>balancing and bandwidth dirauths. People are sad about the state of the
>>network: some relays are overloaded while others idle, many overloaded
>>relays cant even establish circuits to each other. Need to do something
>>about it: deploy bwscanner and start thinking about peerflow. What about
>>isis' bridge bandwidth scanner?
>
> Can you tell us a bit more about this meeting?
> What can people do if they want to be involved?
> When is the next meeting, or how can people find out about it?
>

Hey teor,

thanks for following through.

The "meeting" was impromptu and IRL because we all happened to be at the
same place. There is no next meeting and it's up to us (Tor/network
team) to figure out what are the next steps here.

This week I did some digging to explore the various possible ways
forward also based on your email here: 
https://www.mail-archive.com/tor-dev@lists.torproject.org/msg09912.html
Here are my findings:

Possibility a) Develop peerflow and deploy it in place of torflow

   Peerflow is an exciting and secure bandwidth measurement system
   published in PETS 2017: 
https://ohmygodel.com/publications/peerflow-popets2017.pdf

   Unfortunately, it seems quite complicated to develop from scratch and
   will probably require _significant_ engineering time to actually make
   it a deployed reality (understand, develop, test, deploy). This is
   probably the solution we would like to pursue if we had a grant and a
   dedicated developer.

Possibility b) Finalize bwscanner and deploy in place of torflow

   bwscanner is a project by Aaron/David/Donncha: and can be found here:
   https://github.com/TheTorProject/bwscanner

   It seems to implement the torflow design (2-hop circs && buckets) but
   in a cleaner and better codebase. From what I understand, the main
   part of the project is done, but there has been minimal testing on
   the real network (there are unittests tho) and also the final output
   file with the bandwidth weights has not been completely finalized.

   This project is not quite there yet, and will require some
   non-trivial engineering time, but it's probably a much easier task
   compared to peerflow due to the design being more understood and
   already coded. I think 2-3 weeks of developer time could be quite
   fruitful here. I also heard that some bw auth operators are eager to
   run bwscanner instead of torflow on their setup in January.

Possibility c) Adapt the bridge bw scanner that is currently being developed

   Apparently isis and another developer are currently writing a bridge
   bandwidth scanner for bridgedb, that could in theory be extended to
   scan the whole network. They are currently writing some sort of Rust
   library that will be used by the scanner, and the project is ETA
   around March 2018. The whole development process is pretty opaque so
   I have no idea what's going on. Also, there probably needs to be
   considerable work to extend it from a simple bridge scanner to a real
   relay scanner, and the final result will probably look like bwscanner
   above.

Currently my intuition is to work on (b) above, while also preparing the
ground for (a) which seems to be The Right Thing.

I'm not sure what's the right way forward here in terms of project
management, since the network team seems overloaded and I haven't heard
of anyone willing to take this on...

Ideally we would probably apply for some sort of grant on this work so
that some actual developer time is allocated. I think this is definitely
fundable work since it deeply impacts the *performance* and security of
the Tor network, and basically the network has no chance of surviving in
greater loads if the status quo persists.

I'll try to think more about this problem in the future, these are just
my thoughts from a few hours of digging.  

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] UX improvement proposal: Onion auto-redirects using Alt-Svc HTTP header

[George Kadianakis]

As discussed in this mailing list and in IRC, I'm posting a subsequent
version of this proposal. Basic improvements:
- Uses a new custom HTTP header, instead of Alt-Svc or Location.
- Does not do auto-redirect; it instead suggests the onion based on
  antonella's mockup: 
https://trac.torproject.org/projects/tor/attachment/ticket/21952/21952.png




UX improvement proposal: Onion redirects using Onion-Location HTTP header


1. Motivation:

   Lots of high-profile websites have onion addresses these days (e.g. Tor ,
   NYT, blockchain, ProPublica).  All those websites seem confused on what's
   the right way to inform their users about their onion addresses. Here are
   some confusion examples:
 a) torproject.org does not even advertise their onion address to Tor users 
(!!!)
 b) blockchain.info throws an ugly ASCII page to Tor users mentioning their 
onion
address and completely wrecking the UX (loses URL params, etc.)
 c) ProPublica has a "Browse via Tor" section which redirects to the onion 
site.

   Ideally there would be a consistent way for websites to inform their users
   about their onion counterpart. This would provide the following positives:
 + Tor users would use onions more often. That's important for user
   education and user perception, and also to partially dispell the darkweb 
myth.
 + Website operators wouldn't have to come up with ad-hoc ways to advertise
   their onion services, which sometimes results in complete breakage of
   the user experience (particularly with blockchain)

   This proposal specifies a simple way forward here that's far from perfect,
   but can still provide benefits and also improve user-education around onions
   so that in the future we could employ more advanced techniques.

   Also see Tor ticket #21952 for more discussion on this:
  https://trac.torproject.org/projects/tor/ticket/21952

2. Proposal

   We introduce a new HTTP header called "Onion-Location" with the exact same
   restrictions and semantics as the Location HTTP header. Websites can use the
   Onion-Location HTTP header to specify their onion counterpart, in the same
   way that they would use the Location header.

   The Tor Browser intercepts the Onion-Location header (if any) and informs
   the user of the existense of the onion site, giving them the option to visit
   it. Tor Browser only does so if the header is served over HTTPS.

   Browsers that don't support Tor SHOULD ignore the Onion-Location header.

3. Improvements

4. Drawbacks

4.1. No security/performance benefits

   While we could come up with onion redirection proposals that provide
   security and performance benefits, this proposal does not actually provide
   any of those.

   As a matter of fact, the security remains the same as connecting to normal
   websites (since we trust its HTTP headers), and the performance gets worse
   since we first need to connect to the website, get its headers, and then
   also connect to the onion.

   Still _all_ the website approaches mentioned in the "Motivation" section
   suffer from the above drawbacks, and sysadmins still come up with ad-hoc
   ways to inform users abou their onions. So this simple proposal will still
   help those websites and also pave the way forward for future auto-redirect
   techniques.

4.2. Defining new HTTP headers is not the best idea

   This proposal defines a new non-standard HTTP header. This is not great
   because it makes Tor into a "special" thing that needs to be supported with
   special headers. However, the fact that it's a new HTTP header that only
   works for Tor is a positive thing since it means that non-Tor browsers will
   just ignore it.

   Furthermore, another drawback is that this HTTP header will increase the
   bandwidth needlessly if it's also served to non-Tor clients. Hence websites
   with lots of client traffic are encouraged to use tools that detect Tor
   users and only serve the header to them (e.g. tordnsel).

5. The future

   As previously discussed, this is just a simple proposal to introduce the
   redirection concept to people, and also to help some sysadmins who are
   currently coming up with weird ways to inform people about their
   onions. It's not the best way to do this, but it's definitely one of the
   simplest ways.

   In the future we could implement with more advanced auto-redirect proposals 
like:

   a) Have a "domains to onions" map into HTTPS-everywhere and have it do the
  autoredirects for us (performance benefits, and security benefits under 
many
  threat models).

   b) Bake onion addresses into SSL certificates and Let's Encrypt as suggested
  by comment:42 in #21952.

   But both of the designs above require non-trivial engineering/policy work
   and would still confuse people. So I think 

Re: [tor-dev] Rebooting work on proposal #247 (guard discovery)

[George Kadianakis]

Mike Perry <mikepe...@torproject.org> writes:

> George Kadianakis:
>> Hello Mike,
>> 
>> I'm finally getting out of the prop224/microdescriptor bug pile, and
>> getting more time to start working on guard stuff like prop247 again.
>> 
>> I'm planning to spend a few days next week to regain knowledge on
>> prop247. I'll check out the notes from the Wilmington hackfest, re-read
>> my old simulator's code, etc.
>

Hey, and thanks for the reply!

> I was not involved in with Prop271, so I am not deeply familiar with it.
> However, it has several things we do not need. In particular, the plan
> for prop247 still is to treat consensus information as the official
> notion of vanguard reachability, so there is no need to try to determine
> censorship, firewall, or local network reachability information. If a
> node is in the consensus, it stays in our vanguard set and does not get
> replaced until it actually leaves the consensus. This is consistent with
> how the consensus is currently used for interior hops, and mitigates
> path bias attacks.
>
> If it is dead-simple to use only the consensus uptime portions of
> prop271 without the reachability code, I could be convinced of that. But
> as it is, the rotation times do not need to be as long as guards, and
> the implementation simplification here is attractive. Plus, nodes that
> fall completely out of the consensus periodically like this are probably
> bad choices anyway..
>
> What do you think?
>  

Sounds plausible.

I'm slightly worried that not tracking transient reachability status
might cause situations where all the L2 guards are temporarily down and
hence brings complete the service/client to complete halt. I'm not sure
how likely this is to happen, and it surely depends on L2 size and L2
node selection parameters.

I have not thought about the engineering aspects of this. I think
bending the prop271 code to do this might be a PITA. But I'm also not
sure if not using the guard layer is gonna be easier either. I imagine
there will be engineering complexities either way here. Will try to
figure this out in the coming weeks.

>> I know you have thought more about prop247 the past months, and it would
>> be great if you could brief me up on any updates that I should know
>> about. Specifically I'm wondering if you have any new insights on how
>> the proposed prop247 changes interact with Tor's guard algorithm (prop271)?
>> 
>> Also any other things I should know about from your work on the
>> performance simulator? Perhaps ideas about performance, topology or path
>> restrictions?
>
> Yes. I have decided to simplify everything as much as possible. I am
> going with a mesh topology for the prop247 performance tests (via
> https://bugs.torproject.org/13837,  https://bugs.torproject.org/23101
> and https://bugs.torproject.org/24487). That is the simplest option to
> implement and test for performance, and intuitively seems to have
> almost as good security properties as the bin version (unless your
> security simulator tells us otherwise).
>

Sounds reasonable!

> I am also aiming for these high-level design goals, most important
> first:
>
> 0. All service-side circuits use 3 hops of vanguards.
> 1. Hidden services should avoid trivially disclosing their third
>vanguard to a non-network adversary (ie one that is not running nodes
>but that is watching either HSDESCS or connecting to the service).
>This means their paths look like this:
>S - G - L2 - L3 - HSDIR
>S - G - L2 - L3 - Intro
>S - G - L2 - L3 - M - Rend
> 2. Clients should avoid revealing their third vanguard hop to services
>and to nodes that have information about which service they are
>accessing. This means that their paths look like this:
>C - G - L2 - L3 - M - HSDIR
>C - G - L2 - L3 - M - Intro
>C - G - L2 - L3 - Rend
> 3. Clients use 3 hops of vanguards for all hidden service circuits.
>
> If we do all of these, it will mean that we will have long path lengths
> (8 hop rends), but it also means that it is easy to reason about
> linkability and information disclosure. My thinking is that we should do
> the performance tests with the safest option first (ie: all of these
> goals), and see exactly how bad it is, and then make compromises if it
> turns out to be much worse performance than status quo.
>
> In the event of bad performance, I would alter property #3 before
> messing with property #2, and alter #2 before property #1, but I could
> be talked into a different strategy, or driven to one based on data.
>

Sounds reasonable.

In general, I imagine that this feature is gonna be opt-in initially
which makes me worry less about performance in the beginning.

Also, I'm cur

Re: [tor-dev] UX improvement proposal: Onion auto-redirects using Alt-Svc HTTP header

[George Kadianakis]

teor  writes:

>> 
>> On 16 Nov 2017, at 00:38, Alec Muffett  wrote:
>> 
>>> I think it's important to point out that a Tor client is never
>>> guaranteed to hold a *definitive* consensus.
>>> 
>> That's why I say "(mostly) definitive" in my text - my feeling is that a 
>> locally-held copy of the consensus to be queried is going to be on average 
>> of far higher quality, completeness, and non-stagnancy than something that 
>> one tries to scrape out of Onionoo every 15 minutes.
>
> Please don't use a consensus or a tor client to check for exits for
> this purpose. It produces significant numbers of false negatives,
> because some exits use other IP addresses for their exit traffic.
>

I'm actually not a fan of Alec's idea, and I agree with you that there
will be a significant number of false negatives, but it might be worth
pointing out that IIUC false negatives are probably not so damaging in
this use case, because it would result in users getting thrown to the
normal website instead of the onion site, because the website didn't
realize they are Tor users. So not much damage done there.

False positives are a bit more damaging for reachability because it
means that the website would throw normal users to the onion website
which would fail, but that's not so likely (except if exit node
operators surf from their exit node, or if an exit node IP is shared to
other people).
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] UX improvement proposal #2: Encrypted bookmarks for onions

[George Kadianakis]

Hello,

here is another onion-related UX improvement proposal. We still don't
have a plan for how to concretely fix the onion naming issue, and we
recently released next gen onions so names just got bigger. Ideally we
should start experimenting with solutions sooner than later (also see
https://blog.torproject.org/cooking-onions-names-your-onions).

I think a local-solution akin to bookmarks makes sense to start with, so
I opened a trac ticket today about encrypted bookmarks on Tor Browser:
  https://trac.torproject.org/projects/tor/ticket/24310

Please let me know if you are aware of firefox addon projects that do
encrypted bookmarks that we could use or start basing our work on.

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] UX improvement proposal: Onion auto-redirects using Alt-Svc HTTP header

[George Kadianakis]

Alec Muffett  writes:

> On 15 Nov 2017 12:18, "Iain R. Learmonth"  wrote:
>
> Is this not what TorDNSEL does?
> https://www.torproject.org/projects/tordnsel.html.en
>
>
> Hi Iain!
>

Hey Alec,

thanks for the feedback.

> That certainly sounds like it will give you the answer! But although it
> would give the right kind of answer, it is not what I am asking for.
>
> At the scale of websites like Facebook or the New York Times, a timely
> response is required for the purposes of rendering a page. The benefits of
> solving the problem at "enterprise" scale then trickle down to
> implementations of all sizes.
>
> Speaking as a programmer, it would be delightfully easy to make a DNS query
> and wait for a response to give you an answer... but then you have to send
> the query, wait for propagation, wait for a result, trust the result, debug
> cached versions of the results, leak the fact that all these lookups are
> going on, and so forth.
>
> This all adds adds up to latency and cost, as well as leaking metadata of
> your lookups; plus your local DNS administrator will hate you (cf: doing
> name resolution for every webpage fetch for writing Apache logs, is frowned
> upon.  Better to log the raw IP address and resolve it later if you need.
>
> On the other hand: if you are running a local Tor daemon, a copy of the
> entire consensus is held locally and is (basically) definitive.  You query
> it with near zero lookup latency, you get an instant response with no
> practical lag behind "real time", plus there are no men in the middle, and
> there is no unwanted metadata leakage.
>

I think it's important to point out that a Tor client is never
guaranteed to hold a *definitive* consensus. Currently Tor clients can
stay perfectly happy with a consensus that is up to 3 hours old, even if
they don't fetch the latest one (which gets made every hour).

In general, the Tor network does not have a definitive state at any
point, and different clients/relays can have different states at the
same time.

If we were to create "the definitive exit node oracle" we would need a
Tor client that polls the dirauths the second a new consensus comes out,
and maybe even then there could be desynchs. Perhaps it's worthwhile
doing such a thing, and maybe that's exactly what tordnsel is doing, but
it's something that can bring extra load to the dirauths and should not
be done in many instances.

Furthermore, you said that enterprises might be spooked out by
tor-specific "special" HTTP headers, but now we are discussing weird tor
modules that communicate with the Tor daemon to decide whether to
redirect clients, so it seems to me like an equally "special" Tor setup
for sysadmins.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Understanding the guard/md issue (#21969)

[George Kadianakis]

George Kadianakis <desnac...@riseup.net> writes:

> Hey Tim,
>

OK updates here.

We merged #23895 and #23862 to 032 and master.

#23817 is now in needs_review and hopefully will get in the next 032 alpha.
I think this next alpha should be much better in terms of mds.

Next tickets in terms of importance should probably be #23863 and #24113.
I have questions/feedback in both of them, and I'm ready to move in.

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] UX improvement proposal: Onion auto-redirects using Alt-Svc HTTP header

[George Kadianakis]

Tom Ritter  writes:

> I am a big proponent of websites advertising .onions in their Alt-Srv.
>
>> 4.2. No security/performance benefits
>>
>>While we could come up with auto-redirect proposals that provide security
>>and performance benefits, this proposal does not actually provide any of
>>those.
>>
>>As a matter of fact, the security remains the same as connecting to normal
>>websites (since we trust its HTTP headers), and the performance gets worse
>>since we first need to connect to the website, get its headers, and then
>>also connect to the onion.
>>
>>Still _all_ the website approaches mentioned in the "Motivation" section
>>suffer from the above drawbacks, and sysadmins still come up with ad-hoc
>>ways to inform users abou their onions. So this simple proposal will still
>>help those websites and also pave the way forward for future auto-redirect
>>techniques.
>
> I envision a future Onion Everywhere extension like HTTPS Everywhere
> that works similar to the HSTS preload list. Crawlers validate a
> websites intention to be in the Onion Everywhere extension, and we
> cache the Alt-Srv information so it is used on first load.
>

Yep, that's yet another cool way to do this. 

>
>> 4.3. Alt-Svc does not do exactly what we want
>>
>>I read in a blog post [*] that using Alt-Svc header "doesn’t change the 
>> URL
>>in the location bar, document.location or any other indication of where 
>> the
>>resource is; this is a “layer below” the URL.". IIUC, this is not exactly
>>what we want because users will not notice the onion address, they will 
>> not
>>get the user education part of the proposal and their connection will 
>> still
>>be slowed down.
>>
>>I think we could perhaps change this in Tor Browser so that it rewrites 
>> the
>>onion address to make it clear to people that they are now surfing the
>>onionspace.
>>
>>[*]: https://www.mnot.net/blog/2016/03/09/alt-svc
>
>
> I am a big opponent of changing the semantics of Alt-Srv.
>
> We'd have to change the semantics to only do redirection for onion
> domains. We'd also have to figure out how to handle cases where the
> onion lives alongside non-onion (which takes precedence?) We'd also
> have to maintain and carry this patch ourselves because it's pretty
> antithetical to the very intent of the header and I doubt the
> networking team at Mozilla would be interested in maintaining it.
>
> Besides those issues, it also eliminates Alt-Srv as a working option
> to something *else* websites may want: to silently redirect users to
> their .onion _without_ the possibility of confusion for the user by
> changing the address bar. I think Alt-Srv is an option for partial
> petname support in TB.
>
> There is a perfectly functioning mechanism for redirecting users: the
> Location header. It does a lot of what you want: including temporary
> or persistent redirects, updating the addess bar. Obviously it doesn't
> work for all users, most don't know what .onion is, so Facebook isn't
> going to deploy a .onion Location redirect even if they attempted to
> detect TB users.
>
> But they could send a new Onion-Redirect header that is recognized and
> processed (with a notification bar) by any UA that supports Tor and
> wants to implement it. This header would have a viable path to uplift,
> support in extensions, and even standardization. Onion Everywhere can
> preload these headers too.
>

Agreed, the semantics of Alt-Svc are not what we want, and it's probably
not a good idea to change it from an engineering/policy perspective.

Establishing our own header, with the same semantics as Location, seems
to be a cleaner way to approach this.

When I find some time (hopefully this or next week) I will fix up the
proposal based on received feedback.

>
> On 14 November 2017 at 11:25, teor  wrote:
>>> 4. Drawbacks
>>
>> You missed the biggest one:
>>
>> If the onion site is down, the user will be redirected to the downed site.
>> (I've used onion site redirects with this issue, it's really annoying.)
>> Similarly, if a feature is broken on the onion site, the user will be
>> redirected to a site they can't use.
>>
>> Or if the user simply wants to use the non-onion site for some reason
>> (maybe they want a link they can share with their non-onion friends,
>> or maybe they don't want to reveal they're using Tor Browser).
>>
>> Users *must* have a way to disable the redirect on every redirect.
>
> Right now, I don't agree with this. (I reserve the right to change my
> mind.) Websites can already redirect users to broken links through
> mistakes. Why is "my onion site is not running" a scenario we want to
> code around but "my subdomain is not running" is not? If a website
> wants to redirect users they should be responsible enough to monitor
> the uptime of their onion domain and keep it running.  Maybe we need
> better tooling for that, but that's 

[tor-dev] UX improvement proposal: Onion auto-redirects using Alt-Svc HTTP header

[George Kadianakis]

UX improvement proposal: Onion auto-redirects using Alt-Svc HTTP header


1. Motivation:

   Lots of high-profile websites have onion addresses these days (e.g. Tor ,
   NYT, blockchain, ProPublica).  All those websites seem confused on what's
   the right way to inform their users about their onion addresses. Here are
   some confusion examples:
 a) torproject.org does not even advertise their onion address to Tor users 
(!!!)
 b) blockchain.info throws an ugly ASCII page to Tor users mentioning their 
onion
address and completely wrecking the UX (loses URL params, etc.)
 c) ProPublica has a "Browse via Tor" section which redirects to the onion 
site.

   Ideally there would be a consistent way for websites to inform their users
   about their onion counterpart. This would provide the following positives:
 + Tor users would use onions more often. That's important for user
   education and user perception, and also to partially dispell the darkweb 
myth.
 + Website operators wouldn't have to come up with ad-hoc ways to advertise
   their onion services, which sometimes results in complete breakage of
   the user experience (particularly with blockchain)

   This proposal specifies a simple way forward here that's far from perfect,
   but can still provide benefits and also improve user-education around onions
   so that in the future we could employ more advanced techniques.

   Also see Tor ticket #21952 for more discussion on this:
  https://trac.torproject.org/projects/tor/ticket/21952

2. Proposal

   Websites use the Alt-Svc HTTP header to specify their onion counterpart:
 http://httpwg.org/http-extensions/alt-svc.html

   The Tor Browser intercepts that header (if any) and auto-redirects the user
   to the onion site. Tor Browser only does so if the header is served over 
HTTPS.

   (That's it.)

3. Improvements

3.1. User education through notifications

   To minimize the probability of users freaking out about auto-redirects Tor
   Browser could inform the user that the auto-redirect is happening. This
   could happen with a small notification bar [*] below the URL bar informing
   users that "Tor Browser is auto-redirecting you to the secure onion site".

   The notification bar could also have a question mark button that provides
   the user with additional information on the merits of onion sites and why
   they should like them.

   [*]: like this one: 
http://www.topdreamweaverextensions.com/UserFiles/Image/firefox-bar.jpg

3.2. Auto-redirects too intrusive? Make them optional.

   If we think that auto redirects are too intrusive, we should consider making
   them optional, or letting the website specify the desired behavior.

   If a website wants to specify an onion address but doesn't like
   auto-redirects, it could specify that as part of Alt-Svc and we could still
   inform the user that an onion is available using a notification bar again.

4. Drawbacks

   This proposal is far from the perfect solution of course. Here are a few
   reasons why:

4.1. Auto-redirects are scary

   Novice users or paranoid users might freak out with auto-redirects since
   it's unexpected. They might think they are under attack or they got hacked.

   An argument against that is that we are already using HTTPS-everywhere which
   is basically a redirection layer, and while its redirection technique is
   simpler, sometimes the derivative URL is not similar to the original URL,
   and we still don't see people freaking out. I think that's because people
   trust Tor Browser to do the right thing. Combining this knowledge with
   section 3.1 might solve this problem.

4.2. No security/performance benefits

   While we could come up with auto-redirect proposals that provide security
   and performance benefits, this proposal does not actually provide any of
   those.

   As a matter of fact, the security remains the same as connecting to normal
   websites (since we trust its HTTP headers), and the performance gets worse
   since we first need to connect to the website, get its headers, and then
   also connect to the onion.

   Still _all_ the website approaches mentioned in the "Motivation" section
   suffer from the above drawbacks, and sysadmins still come up with ad-hoc
   ways to inform users abou their onions. So this simple proposal will still
   help those websites and also pave the way forward for future auto-redirect
   techniques.

4.3. Alt-Svc does not do exactly what we want

   I read in a blog post [*] that using Alt-Svc header "doesn’t change the URL
   in the location bar, document.location or any other indication of where the
   resource is; this is a “layer below” the URL.". IIUC, this is not exactly
   what we want because users will not notice the onion address, they will not
   get the user education part of the proposal and their connection will still
   be slowed 

Re: [tor-dev] Understanding the guard/md issue (#21969)

[George Kadianakis]

teor <teor2...@gmail.com> writes:

>> On 29 Oct 2017, at 01:19, George Kadianakis <desnac...@riseup.net> wrote:
>> 
>> 
>> 
>> Let me know what you think! Perhaps you have other ideas here of how we
>> should approach this issue.
>
> Fix #23817 by implementing a failure cache and going to a fallback if all
> primary guards fail. I think that would be a solution for #23863 as well.
>
> And if a few fallbacks don't have the guard's microdesc, mark the guard as
> down. It's likely it's microdesc is just not on the network for some reason.
>

Hey Tim!

Please see
https://trac.torproject.org/projects/tor/ticket/23817#comment:6 for an
implementation plan of the failure cache concept. If it makes sense to
you, I will try to implement it this week.

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Understanding the guard/md issue (#21969)

[George Kadianakis]

teor <teor2...@gmail.com> writes:

>> On 29 Oct 2017, at 01:19, George Kadianakis <desnac...@riseup.net> wrote:
>> 
>> Hey Tim,
>> 
>> just wanted to ask a clarifying question wrt #21969.
>> 
>> First of all there are various forms of #21969 (aka the "missing
>> descriptors for some of our primary entry guards" issue). Sometimes it
>> occurs for 10 mins and then goes away, whereas for other people it
>> disables their service permanently (until restart). I call this the
>> hardcore case of #21969. It has happened to me and disabled my service
>> for days, and I've also seen it happen to other people (e.g. dgoulet).
>> 
>> So. We have found various md-related bugs and put them as children of
>> #21969. Do you think we have found the bugs that can cause the hardcore
>> case of #21969? That is, is any of these bugs (or a bug combo) capable
>> of permanently disabling an onion service?
>
> Yes, this bug is disabling:
>

Thanks for the reply, Tim.

> #23862, where we don't update guard state unless we have enough
> directory info.
>
> When tor gets in a state where it doesn't have enough directory info
> due to another bug, this makes sure it will never get out of that state.
> Because it will never mark its directory guards as up when it gets a
> new consensus, and therefore it will never fetch microdescs, find out
> it has enough directory info, and build circuits.
>

Hmm, just want to make sure I get this.

My understanding with #23862 is that Tor would never mark its directory
guards as up like you say, but it _would still_ fetch microdescs using
fallback directories because of the way
directory_pick_generic_dirserver() works. Isn't that the case?

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] v3 hidden services: inconsistencies between spec and implementation

[George Kadianakis]

inkylatenoth  writes:

> Whilst implementing v3 hidden services myself I found some
> inconsistencies between the specs and the current implementation. I
> wanted to share these in case someone from the Tor organization wants to
> update the specs and/or the implementation.
>

Hello inkylatenoth!

That's a great post and thanks for catching all these issues and
innacuracies! We are definitely interested in consistency and fixing the
spec (and implementation if needed).

> # rend-spec-v3.txt
>
> ## 2.4
>
> * after decrypting the `superencrypted' object from a descriptor, the
>   resulting document does not end with the NL character. This means that
>   it does not strictly conform to the document meta-format described in
>   section 1.2 of dir-spec.txt.
>

Hmm... This might be worth fixing on the implementation if possible (and
if it won't break things). Otherwise, let's patch the spec.

> ## A.2
>
> * the blinded key param is defined as H(BLIND_STRING | A | s | B | N).
>   In practice I found that I had to add a null byte after BLIND_STRING
>   in order to reach the same value as the C implementation:
>
> param = H(BLIND_STRING | INT_1(\x00) | A | s | B | N)
>
>   In all other cases where a string constant is used like this (e.g.
>   computing the nonce N above), I found that the trailing null byte is
>   not required.
>

Ouch. This might be an artifact of the way strings are implemented in C.

I guess a spec patch might be the right thing to do here, otherwise too
much stuff will break. 

> * when clamping the blinding factor, the second bitwise operation is
>   `param[31] &= 127' in the spec but `param[31] &= 63' in the C
>   implementation. These are equivalent in practice when followed by the
>   third operation (`param[31] |= 64'), but it might be nice to use a
>   consistent representation for the benefit of human readers.
>

Hmm... Yeah there are various ways to do the clamping for ed25519 keys.

I think we should edit the spec to reflect the clamping we do on the code.

> # 220-ecc-ids-keys.txt
>
> # 2.1
>
> * 'The signature is formed by signing the first N-64 bytes of the
>   certificate prefixed with the string "Tor node signing key certificate
>   v1".' I found this to be false; the signatures only validate without
>   the string prefix.
>

Ouch... I think we should edit the spec and consider if there are any
security risks here.

> ## A.1
>
> * I realized that the certificate types here are outdated. The
>   signing-key extension is listed as type [04], when in rend-spec-v3.txt
>   and the C implementation it is type [08].

Let's fix the spec here too...

---

Inkylatenoth, let me know if you are interested in drafting a spec/code
patch for the issues you found!!! If you are not interested, I can try
to do them myself at some point in the next weeks (been pretty busy with
stuff lately).

Also, let us know if your independent implementation is a public thing
we should know about. Seems interesting :)

Thanks again!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Understanding the guard/md issue (#21969)

[George Kadianakis]

Hey Tim,

just wanted to ask a clarifying question wrt #21969.

First of all there are various forms of #21969 (aka the "missing
descriptors for some of our primary entry guards" issue). Sometimes it
occurs for 10 mins and then goes away, whereas for other people it
disables their service permanently (until restart). I call this the
hardcore case of #21969. It has happened to me and disabled my service
for days, and I've also seen it happen to other people (e.g. dgoulet).

So. We have found various md-related bugs and put them as children of
#21969. Do you think we have found the bugs that can cause the hardcore
case of #21969? That is, is any of these bugs (or a bug combo) capable
of permanently disabling an onion service?

It seems to me that all the bugs identified so far can only cause #21969
to occur for a few hours before it self-heals itself. IIUC, even the
most fundamental bugs like #23862 and #23863 are only temporarily, since
eventually one of the dirguards will fetch the missing mds and give them
to the client. Do you think that's the case?

I'm asking you because I plan to spend some serious time next week on
#21969-related issues, and I'd like to prioritize between bug hunting
and bug fixing. That is, if the root cause of the hardcore case of
#21969 is still out there, I'd like to continue bug hunting until I find
it.

Let me know what you think! Perhaps you have other ideas here of how we
should approach this issue.

Cheers!! :)

PS: Sending this as an email since our timezones are making it kind hard
to synch up on IRC.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Padding prop224 REND1 cells to blend them with legacy cells

[George Kadianakis]

teor <teor2...@gmail.com> writes:

>> On 20 Sep 2017, at 00:44, George Kadianakis <desnac...@riseup.net> wrote:
>> 
>> Legacy RENDEZVOUS1 cells are bigger than the prop224 ones. The prop224
>> spec suggests we pad the new cells so that they look similar in size to
>> the legacy ones.
>> 
>> ...
>> 
>> The suggestion is to pad the prop224 cells to 168 bytes using random data.
>> 
>> Would that work? My main question is whether the g^y part of the legacy
>> cell has any distinguishers that could distinguish it from random data.
>> It's encoded using OpenSSL's BN_bn2bin() and it's a 1024 bit DH public
>> key. Are there any algebraic or openssl structure distinguishers we
>> should be worrying about, or is random data sufficient to mask it out?
>
> What's the threat model here?
>
> I ask because regardless of whether the RENDEZVOUS1 cell plaintext is
> distinguishable between v2 and v3, the rend point can distinguish v2 and
> v3 using this one neat trick:
> * if the service extends using TAP, the protocol is v2
> * if the service extends using ntor, the protocol is v3
>

Thanks for the discussion and research, Ian and teor! 

I summarized the findings here: 
https://trac.torproject.org/projects/tor/ticket/23420#comment:5

Not sure what's the right approach here.

Perhaps I'm fine with doing nothing at this point, and figuring this out
in the future if v4 ever comes.

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Padding prop224 REND1 cells to blend them with legacy cells

[George Kadianakis]

Hello Ian, (and other cryptographers on the list)

here is a quick question which you might be able to answer super fast:

Legacy RENDEZVOUS1 cells are bigger than the prop224 ones. The prop224
spec suggests we pad the new cells so that they look similar in size to
the legacy ones.

Here is how the legacy ones look like:

   RC   Rendezvous cookie  [20 octets]
   g^y  Diffie-Hellman [128 octets]
   KH   Handshake digest   [20 octets]

Here is how the prop224 ones look like:

   RENDEZVOUS_COOKIE  [20 bytes]
   HANDSHAKE_INFO [64 bytes]

The suggestion is to pad the prop224 cells to 168 bytes using random data.

Would that work? My main question is whether the g^y part of the legacy
cell has any distinguishers that could distinguish it from random data.
It's encoded using OpenSSL's BN_bn2bin() and it's a 1024 bit DH public
key. Are there any algebraic or openssl structure distinguishers we
should be worrying about, or is random data sufficient to mask it out?

Thanks!!! :)

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Open topics of prop247: Defending Against Guard Discovery Attacks using Vanguards

[George Kadianakis]

s7r <s...@sky-ip.org> writes:

> George Kadianakis wrote:
>> Hello,
>> 
>> here is some background information and summarizing of proposal 247
>> "Defending Against Guard Discovery Attacks using Vanguards" for people
>> who plan to work on this in the short-term future.
>> 
>
> Hello,
>
> I have discussed in Amsterdam briefly with David about this and want to
> further discuss it here where everyone can throw an eye on it. I have an
> idea to use a different technique that will replace the concept of
> vanguards and will only activate in case the hidden service might be
> under attack - I refer here to Hidden Service Guard Discovery Attack,
> which is currently fast, easy, (relatively, depending on adversary)
> cheap and effective. It will also address the load balancing questions
> we have in the context of using vanguards and make it much harder and
> more expensive for an adversary to mount a HS Guard Discovery attack.
>
> The main idea was discussed last year a little bit:
> https://lists.torproject.org/pipermail/tor-dev/2016-January/010291.html
>
> but its initial logic (banning a suspicious rendezvous point for a
> period of time) had some issues as pointed out by arma:
> https://lists.torproject.org/pipermail/tor-dev/2016-January/010292.html
>
> So, to mitigate those, we could do something different:
>
> Each hidden service server will keep in persistent memory the following
> information and always validate a rule before selecting the path for any
> rendezvous circuit:
> - total number of successfully established rendezvous circuits for the
> last 24 hours;
> - middle probability based on consensus weight fraction for each
> rendezvous relay in the list (if this value is less than 0.3, it
> defaults to 0.3);
> - hop 2 and 3 from the last circuit used to each rendezvous relay in the
> list;
> - number of successfully established rendezvous circuits per each
> rendezvous relay fingerprint in the last 24 hours.
>
> A table with the required columns would look like this:
> Fingerprint | Middle Prob | Last circ hop2,hop3 | num circs last 24 hrs
>
> A rendezvous relay is considered suspicious when the number of
> successfully established circuits in the last 24 hours per a certain
> rendezvous relay exceeds with more than x2 factor the number of expected
> circuits having that relay as rendezvous point.
>

Hello s7r,

and thanks for helping with this and approaching it from a different
direction.

Personally, I'd be really surprised if any solution that statistically
marks relays or paths as "suspicious" will ever work for this particular
problem. That's because the adversary does not need that many paths to
succeed, and also because the adversary has lots of time to carry out
the attack. I also dislike these solutions since the HS operator cannot
really distinguish between an actual attack that just happened, or their
HS getting slashdotted, or someone trolling them by connecting a
thousand times.

Furthermore, I don't understand why your proposal marks rendezvous
points as suspicious, and not the paths themselves. After all, the
attacker can choose a diffferent rendezvous point everytime, as long as
the HS makes fresh circuits for each one.

Also if you are suggesting the reuse of hops 2 and 3 for multiple
circuits, you are basically suggesting a layered guard approach which is
what prop247 also tries to do, and FWIW it's not simple at all
(especially from an engineering perspective).

Cheers!

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Clarifications on guard-spec.txt

[George Kadianakis]

Roger Dingledine  writes:

> Hi folks!
>
> I'm catching up on my proposals, and I really like guard-spec.txt.
> Nicely done!
>
> I made some fixes that I hope are straightforward:
> https://lists.torproject.org/pipermail/tor-commits/2017-May/122942.html
> https://lists.torproject.org/pipermail/tor-commits/2017-May/122948.html
> https://lists.torproject.org/pipermail/tor-commits/2017-May/122986.html
>
> And I have four remaining questions/topics that could use some feedback
> from the spec authors/implementors rather than just having me bust in
> and change things.
>

Second email:

Please check branch `guard_spec_roger` in my torspec repo for fixes to
all the issues below (I think), apart from issue (B) which probably
deserves its own ticket.

Feel free to merge, and let me know if I missed something.

> ---
>
> A) We don't really say what we mean by "adding" a guard to an ordered
> list. In particular:
>
>>  We add new members to {CONFIRMED_GUARDS} when we mark a circuit
>>  built through a guard as "for user traffic."
>
> When we say "add", do we mean "append" here?
>
>>  To compute primary guards, take the ordered intersection of
>>  {CONFIRMED_GUARDS} and {FILTERED_GUARDS}, and take the first
>>  {N_PRIMARY_GUARDS} elements.  If there are fewer than
>>  {N_PRIMARY_GUARDS} elements, add additional elements to
>>  PRIMARY_GUARDS chosen _uniformly_ at random from
>>  ({FILTERED_GUARDS} - {CONFIRMED_GUARDS}).
>
> Similarly, does "add" mean "append"?
>
>>  Once an element has been added to {PRIMARY_GUARDS}, we do not remove it
>>  until it is replaced by some element from {CONFIRMED_GUARDS}. Confirmed
>>  elements always proceed unconfirmed ones in the {PRIMARY_GUARDS} list.
>
> This one looks like a bug as currently stated: by "proceed", do we mean
> "precede"?
>
> ---
>
> B) In Sec 4.10, whenever we get a new consensus, we:
>
>>  For every guard in {SAMPLED_GUARDS}, we update {IS_LISTED} and
>>  {FIRST_UNLISTED_AT}.
>
> In the old design, we also believed the Running flag in the new
> consensus, that is, we marked the guards as reachable again. It looks
> from sampled_guards_update_from_consensus() like we no longer do that.
>
> Is that old behavior considered a bug and we intentionally stopped,
> or did we not consider it?
>
> I am ok with "that was stupid behavior, so we stopped", but if we didn't
> know that we used to do it, maybe we should decide whether it is stupid
> behavior. :)
>
> ---
>
> C) In Sec A.4, in the state file we have
>
>> "bridge_addr" -- If the guard is a bridge, its configured
>> address and OR port. Optional.
>
> How does this play with bridges that have pluggable transports? In my
> "bridge obfs2 128.31.0.34:51715" line, the ORPort of the bridge is
> not listed.
>
> It looks from
> https://trac.torproject.org/projects/tor/ticket/21027
> like we did some fixing here, but the spec didn't get an update?
>
> ---
>
> D) In Sec 4.8, when a circuit succeeds:
>
>>  * If this circuit was , it is now
>>.  You may not yet attach streams to it.
>>Then check whether the {last_time_on_internet} is more than
>>{INTERNET_LIKELY_DOWN_INTERVAL} seconds ago:
>>
>>   * If it is, then mark all {PRIMARY_GUARDS} as "maybe"
>> reachable.
>>
>>   * If it is not, update the list of waiting circuits. (See
>> [UPDATE_WAITING] below)
>
> [Where INTERNET_LIKELY_DOWN_INTERVAL has been picked as 10 minutes.]
>
> To make sure I understand this one, consider the following scenario:
>
> * We go offline, and our current circuits fail.
> * We try to make new circuits through each of our primary guards,
>   failing for each (because we're offline) and marking them down.
> * Then we move to the remaining confirmed+usable ones, and mark those
>   down too.
> * Then we work through the rest of USABLE_FILTERED_GUARDS, marking them
>   down too. As we mark them down, USABLE_FILTERED_GUARDS shrinks,
>   causing us to add new elements to SAMPLED_GUARDS to replenish
>   USABLE_FILTERED_GUARDS.
> * After we've brought SAMPLED GUARDS to 60, we stop adding new ones.
>   At this point, we are out of tries:
>
>>  * Otherwise, if USABLE_FILTERED_GUARDS is empty, we have exhausted
>>  all the sampled guards.  In this case we proceed by marking all guards
>>  as  reachable so that we can keep on sampling.
>
> Does this mean that we mark our *already-sampled* guards as reachable,
> thus making USABLE_FILTERED_GUARDS big again, and we loop through the
> above steps again, and we keep looping until we come back online?
>
> The above "so that we can keep on sampling" phrase confuses me, because it
> seems like there's no way that we would arrive at "USABLE_FILTERED_GUARDS
> is empty" without also 

Re: [tor-dev] Clarifications on guard-spec.txt

[George Kadianakis]

Roger Dingledine  writes:

> Hi folks!
>
> I'm catching up on my proposals, and I really like guard-spec.txt.
> Nicely done!
>
> I made some fixes that I hope are straightforward:
> https://lists.torproject.org/pipermail/tor-commits/2017-May/122942.html
> https://lists.torproject.org/pipermail/tor-commits/2017-May/122948.html
> https://lists.torproject.org/pipermail/tor-commits/2017-May/122986.html
>

Hello,

all three commits above LGTM. Thanks for cleaning up the proposal!

> And I have four remaining questions/topics that could use some feedback
> from the spec authors/implementors rather than just having me bust in
> and change things.
>
> ---
>
> A) We don't really say what we mean by "adding" a guard to an ordered
> list. In particular:
>
>>  We add new members to {CONFIRMED_GUARDS} when we mark a circuit
>>  built through a guard as "for user traffic."
>
> When we say "add", do we mean "append" here?
>
>>  To compute primary guards, take the ordered intersection of
>>  {CONFIRMED_GUARDS} and {FILTERED_GUARDS}, and take the first
>>  {N_PRIMARY_GUARDS} elements.  If there are fewer than
>>  {N_PRIMARY_GUARDS} elements, add additional elements to
>>  PRIMARY_GUARDS chosen _uniformly_ at random from
>>  ({FILTERED_GUARDS} - {CONFIRMED_GUARDS}).
>
> Similarly, does "add" mean "append"?
>
>>  Once an element has been added to {PRIMARY_GUARDS}, we do not remove it
>>  until it is replaced by some element from {CONFIRMED_GUARDS}. Confirmed
>>  elements always proceed unconfirmed ones in the {PRIMARY_GUARDS} list.
>
> This one looks like a bug as currently stated: by "proceed", do we mean
> "precede"?
>

All the suggestions above seem correct.
We should open a ticket and make a patch.

> ---
>
> B) In Sec 4.10, whenever we get a new consensus, we:
>
>>  For every guard in {SAMPLED_GUARDS}, we update {IS_LISTED} and
>>  {FIRST_UNLISTED_AT}.
>
> In the old design, we also believed the Running flag in the new
> consensus, that is, we marked the guards as reachable again. It looks
> from sampled_guards_update_from_consensus() like we no longer do that.
>
> Is that old behavior considered a bug and we intentionally stopped,
> or did we not consider it?
>
> I am ok with "that was stupid behavior, so we stopped", but if we didn't
> know that we used to do it, maybe we should decide whether it is stupid
> behavior. :)
>

Hmm, I also don't see us currently doing this, but it might be worth doing.

I imagine it's not going to do much in most cases since primary guards
have a pretty short retry period, but still it might be a good idea
since it will keep our guard list more up to date.

Perhaps we should consider adding this behavior to the spec and code.

> ---
>
> C) In Sec A.4, in the state file we have
>
>> "bridge_addr" -- If the guard is a bridge, its configured
>> address and OR port. Optional.
>
> How does this play with bridges that have pluggable transports? In my
> "bridge obfs2 128.31.0.34:51715" line, the ORPort of the bridge is
> not listed.
>
> It looks from
> https://trac.torproject.org/projects/tor/ticket/21027
> like we did some fixing here, but the spec didn't get an update?
>

Hmm, seems like that part of the spec is not really accurate indeed!

Maybe we can just update the spec and say that `bridge_addr` contains
the configured address and port of the bridge which can be either the
ORPort or PT port?

> ---
>
> D) In Sec 4.8, when a circuit succeeds:
>
>>  * If this circuit was , it is now
>>.  You may not yet attach streams to it.
>>Then check whether the {last_time_on_internet} is more than
>>{INTERNET_LIKELY_DOWN_INTERVAL} seconds ago:
>>
>>   * If it is, then mark all {PRIMARY_GUARDS} as "maybe"
>> reachable.
>>
>>   * If it is not, update the list of waiting circuits. (See
>> [UPDATE_WAITING] below)
>
> [Where INTERNET_LIKELY_DOWN_INTERVAL has been picked as 10 minutes.]
>
> To make sure I understand this one, consider the following scenario:
>
> * We go offline, and our current circuits fail.
> * We try to make new circuits through each of our primary guards,
>   failing for each (because we're offline) and marking them down.
> * Then we move to the remaining confirmed+usable ones, and mark those
>   down too.
> * Then we work through the rest of USABLE_FILTERED_GUARDS, marking them
>   down too. As we mark them down, USABLE_FILTERED_GUARDS shrinks,
>   causing us to add new elements to SAMPLED_GUARDS to replenish
>   USABLE_FILTERED_GUARDS.
> * After we've brought SAMPLED GUARDS to 60, we stop adding new ones.
>   At this point, we are out of tries:
>

Seems like a reasonable description. I haven't looked at this part of
the code in a while to be 100% sure 

[tor-dev] Open topics of prop247: Defending Against Guard Discovery Attacks using Vanguards

[George Kadianakis]

Hello,

here is some background information and summarizing of proposal 247
"Defending Against Guard Discovery Attacks using Vanguards" for people
who plan to work on this in the short-term future.

I include a list of open design topics (probably not exhaustive) and a list of
engineering topics. Some engineering stuff can be done parallel to the design 
stuff.

 Background info 

* Proposal: 
https://gitweb.torproject.org/torspec.git/tree/proposals/247-hs-guard-discovery.txt
* Discussion:
** Initial prop247 thread: 
https://lists.torproject.org/pipermail/tor-dev/2015-July/009066.html
** Recent prop247 thread: 
https://lists.torproject.org/pipermail/tor-dev/2015-September/009497.html
** Reading group notes of prop247: 
https://lists.torproject.org/pipermail/tor-dev/2016-January/010265.html

 Design topics 

* Optimize proposal parameters
** Optimize guardset sizes
** Optimize guardset lifetimes and prob distributions (minXX/maxXX/uniform?)
** To take informed decision, we might need a prop247 simulator, or an actual 
PoC with txtorcon

* HOW to choose second-layer and third-layer guards?
** Should they be Guards? middles? Vanguards? Serious security / load balancing 
implications!
** Can guardsets share guards between them or are they disjoint? Particularly 
third-layer sets
** background: 
https://lists.torproject.org/pipermail/tor-dev/2016-January/010265.html

* HOW to avoid side-channel guard discovery threats?
** Can IP/RP be the same as first-layer guard?
** Can first-layer guard be the same as third-layer guard?
** background: 
https://gitweb.torproject.org/user/mikeperry/torspec.git/commit/?h=guard_discovery_dev2

* Change path selection for IP circs to avoid third-layer guard linkability 
threats.
** Switch from [HS->G1->M->IP] to [HS->G1->G2->G3->IP] or even to 
[HS->G1->G2->G3->M->IP].
** Consider the latter option for HSDir circs as well?
** background: 
https://gitweb.torproject.org/user/mikeperry/torspec.git/commit/?h=guard_discovery_dev2

* Should prop247 be optional or default?
** Consider making it optional for a testing period?

* How does prop247 affects network performance and load balancing?
** especially if it's enabled by default?
** Update load balancing proposal?

* Correct behavior for multiple HSes on single host?

* Does prop247 influence guard fingerprinting (#10969) and should we care 
enough?

 Engineering topics 

* What's a good entrynodes API to implement prop247? 
* What's a good state file API to implement prop247?

* Write prop247 simulator to verify security goals and optimize proposal 
parameters (see above).

* Write PoC with txtorcon!
* Write PoC with little-t-tor!


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Action items wrt ed25519 onion address verification in prop224 (was Re: [tor-project] Network team meetings notes from 17 April 2017)

[George Kadianakis]

Ian Goldberg <i...@cs.uwaterloo.ca> writes:

> On Tue, Apr 25, 2017 at 03:38:37PM +0300, George Kadianakis wrote:
>> > It turns out the point whose packed representation is 32 bytes of 0x00
>> > is a torsion point; it is the point (-1,0).
>> >
>> > Indeed, these are the 7 pure torsion points in ed25519:
>> >
>> > 26e8958fc2b227b045c3f489f2ef98f0d5dfac05d3c63339b13802886d53fc05
>> >  =(-1,0)
>> > c7176a703d4dd84fba3c0b760d10670f2a2053fa2c39ccc64ec7fd7792ac037a
>> > ec7f =(0,-1)
>> > c7176a703d4dd84fba3c0b760d10670f2a2053fa2c39ccc64ec7fd7792ac03fa
>> > 0080 =(1,0)
>> > 26e8958fc2b227b045c3f489f2ef98f0d5dfac05d3c63339b13802886d53fc85
>> >
>> > So just take any of the above points, and add it to a valid pubkey to
>> > get an invalid pubkey.
>> >
>> > You should probably also check points not on the curve at all, such as:
>> >
>> > e19c65de75c68cf3b7643ea732ba9eb1a3d20d6d57ba223c2ece1df66feb5af0
>> >
>> > If you generate a 32-byte string at random, about 1/2 the time it won't
>> > be on the curve at all (that is, if P is the unpack of those 32 bytes,
>> > 8*l*P is *not* the identity), about 7/16 of the time it is on the curve,
>> > but has a torsion component (8*l*P is the identity, but l*P is not), and
>> > 1/16 of the time it's a valid pubkey (l*P is the identity, but P is
>> > not).
>> >
>> 
>> Good stuff Ian.
>> 
>> I pushed a new branch `bug22006` that:
>> - Checks that the pubkey is not the identity element itself.
>> - Adds tests based on the points you listed above.
>> 
>> Check it out here:
>>  https://gitweb.torproject.org/user/asn/tor.git/log/?h=bug22006_v2
>
> It looks to me as though you're only checking the pure torsion points
> above.  You should *add* one of those points to a valid pubkey in order
> to get a point to check.  For example, the points:
>
> 300ef2e64e588e1df55b48e4da0416ffb64cc85d5b00af6463d5cc6c2b1c185e
> f43e3a046db8749164c6e69b193f1e942c7452e7d888736f40b98093d814d5e7
> c9fff3af0471c28e33e98c2043e44f779d0427b1e37c521a6bddc011ed1869af
>
> would be good additional tests (all should fail; they have order 8l, 4l,
> 2l respectively).
>
> This one should pass:
>
> 4ba2e44760dff4c559ef3c38768c1c14a8a54740c782c8d70803e9d6e3ad8794
>

Oops yes, I was actually testing the pure torsion points, instead of
generating fresh points P = b*B + t*T as I think you are suggesting.

Doing ed25519 addition in the tor unittests is kind of a PITA, so I'll
just borrow the points you are suggesting above and test those directly.

Pushed a fixup commit here:
https://gitweb.torproject.org/user/asn/tor.git/commit/?h=bug22006_v2=eb3530fc8999b3a3028d60f86bd04445273f247f

>> Another thing:
>> 
>> My understanding is that this is a special-case validation and it's
>> expensive,
>
> It's a single scalar multiplication (and packing, I suppose).  I guess
> "expensive" is relative; you might time it to see if the cost matters to
> you.
>
>> so I opted to not perform it everytime we generate a new
>> ed25519 keypair or when we receive one from the internet. So for
>> example, I'm not doing it when we extract the ed25519 signing pubkey
>> that signs the HS descriptor, since we don't care if there are
>> equivalent forms of that key.
>
> You indeed don't need to do it when you generate a key yourself in a
> known-good way (unless you're paranoid about fault attacks, which I
> don't think you are).  I'm a little wary about ever not doing it on a
> pubkey you receive from the Internet, though.  I would want to know if
> someone were sending me a malformed crypto-relevant value.
>

Hmm, and that's just to ensure that there are no equivalent forms of
those pubkeys, right? And also to ensure that no one is sending random
garbage to us. Because IIUC using an ed25519 pubkey with a torsion
component does not really affect the security of signature verification
in other ways.

Anyhow, these two commits do the validation in a bunch of places we
receive ed25519 pubkeys from the net:

https://gitweb.torproject.org/user/asn/tor.git/commit/?h=bug22006_v2=e37c0c3fa5d572421e52d4cd144f0617ad4d10e0

https://gitweb.torproject.org/user/asn/tor.git/commit/?h=bug22006_v2=8965bf278e5efe3f17e6d99123659a26da02c589

Thanks for the script as well! :)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Action items wrt ed25519 onion address verification in prop224 (was Re: [tor-project] Network team meetings notes from 17 April 2017)

[George Kadianakis]

Ian Goldberg <i...@cs.uwaterloo.ca> writes:

> On Mon, Apr 24, 2017 at 04:47:44PM +0300, George Kadianakis wrote:
>> Ian Goldberg <t...@cypherpunks.ca> writes:
>> 
>> > On Thu, Apr 20, 2017 at 03:40:58PM +0300, George Kadianakis wrote:
>> >> Hey Ian,
>> >> 
>> >> so the current problem with ed25519-donna is that when I'm doing the
>> >> above, I'm getting the following 32-byte key after ge25519_pack(). Here
>> >> it is hexed up:
>> >> 
>> >>0x0100
>> >> 
>> >> I don't see any methods in ed25519-donna for checking a point or a
>> >> packed point to see if it's the point at infinity (identity element).
>> >
>> > There actually is such a function, but it's buried in a weird place.
>> > ge25519_is_neutral_vartime in ed25519-donna-batchverify.h does what you
>> > want, but it's static, so you would need to copy it (and remove the line
>> > involving batch_point_buffer).  Probably not worthwhile.
>> >
>> > I can easily believe the representation of the identity (neutral)
>> > element in ed25519 is the hex value above.  (And it is; see below.)
>> >
>> >> I've been assuming that the point at infinity would be an all-zeroes
>> >> 32-byte array, because that's what we are doing in curve25519, but I
>> >> actually have no idea how the array representation of ge25519_pack()
>> >> works. Here is our process for curve25519:
>> >>
>> >> https://gitweb.torproject.org/tor.git/tree/src/common/crypto_curve25519.c#n123
>> >
>> > Curve25519 only outputs the x coordinate, which is indeed 0, so you get
>> > the all-zero value.  ed25519 outputs the y coordinate (which is the
>> > 255-bit value 0x000...0001 in little-endian format) and the single-bit
>> > parity of the x coordinate (which is 0), so you do get the hex you give
>> > above.  (The identity element is the point (0,1) in Edwards curves, not
>> > actually the point at infinity.)
>> >
>> >> The above packed point is an all zeroes 32-byte string, apart from the
>> >> very first bit which is set. Could it be that the first bit is the
>> >> compressed sign of the 'x' coordinate or something, and that's actually
>> >> the point at infinity?
>> >> 
>> >> But then what's the right way to check for the point at infinity? Check
>> >> that everything is 0 apart from the first sign bit?
>> >
>> > Yes, pack the point, and compare it to the above 32-byte value.
>> >
>> >> And after I figure this out, I need to do the same for the reference
>> >> implementation of ed25519, because as it seems Tor uses two simultaneous
>> >> implementations of ed25519:
>> >> https://gitweb.torproject.org/tor.git/tree/src/ext/ed25519
>> >
>> > Yes, hopefully that implementation packs points in the same way!
>> >
>> >> Thanks for the help Ian :) Very much appreciated!
>> >
>> > No worries.
>> >
>> 
>> [CCing tor-dev since people might find it interesting]
>> 
>> Thanks for the advice Ian :)
>> 
>> With your help, I have now implemented the validation check. Check out
>> the ticket #22006 and the branch `bug22006` here:
>> 
>> https://gitweb.torproject.org/user/asn/tor.git/commit/?h=bug22006=628dd1adfdc4cbde449d6ec6808a9f6aa6f6f6c4
>
> You should also check that the point it not *itself* the identity
> element.
>
>> One last thing I'd like to do here is test our validation function
>> against a pubkey with torsion component to make sure that it gets
>> rejected. How would you go about generating such a pubkey?
>> 
>> I was thinking of using the CRT to find an integer a \in Z, s.t
>>a == 0 (mod l) && a == 1 (mod 8)
>> but then I didn't know how to go from that scalar with a torsion
>> component to an ed25519 point...
>
> It turns out the point whose packed representation is 32 bytes of 0x00
> is a torsion point; it is the point (-1,0).
>
> Indeed, these are the 7 pure torsion points in ed25519:
>
> 26e8958fc2b227b045c3f489f2ef98f0d5dfac05d3c63339b13802886d53fc05
>  =(-1,0)
> c7176a703d4dd84fba3c0b760d10670f2a2053fa2c39ccc64ec7fd7792ac037a
> ec7f =(0,-1)
> c7176a703d4dd84fba3c0b760d10670f2a2053fa2c39ccc64ec7fd7792ac03fa
> 0

Re: [tor-dev] Interest in collaborating on a standard Ed25519 key blinding scheme?

[George Kadianakis]

Tony Arcieri  writes:

> I'm trying to gauge interest on the IRTF's CFRG mailing list regarding
> collaborating on a draft for a standard Ed25519 hierarchical derivation /
> key blinding scheme:
>
> https://mailarchive.ietf.org/arch/msg/cfrg/lM1ix9R-0tVzhZorQhQlKvi4wpA
>
> The post makes several mentions of Tor's work in the space in regard to the
> next-generation hidden services design.
>
> I think it'd be great if Tor were to collaborate on the design of such a
> scheme and adopt it for the new hidden services design. I see a lot of
> convergent evolution in this space and think it would be great if there
> were a single standard everyone could implement.
>
> Even if you don't, I think there are some ideas from similar schemes Tor
> should fold back into its own design, particularly in regard to how certain
> bits of the private scalar are "clamped". Some discussion of that here:
>
> https://moderncrypto.org/mail-archive/curves/2017/000862.html
>
> tl;dr: clamp the third highest bit of the root scalar to zero (in addition
> to the bits normally clamped in the non-canonical Ed25519 private scalar),
> and use 224-bit child scalars.
>

An update:

After lots of discussions in the Amsterdam Tor meeting, the following
approach was suggested for cleansing keys of their torsion components
that is more friendly towards hierarchical key-derivation schemes:
https://moderncrypto.org/mail-archive/curves/2017/000866.html

However, my current intuition is to just not do this for hidden service
ed25519 blinded keys. Those keys are only used for signing descriptors
which should be safe to do, and we don't plan to use them for D-H any
time soon. If we or some crazy app EVER decides to use those ephemeral
keys for key exchange, we would need to use a special DH function that
kills the tensor component of keys before using them, as suggested by
Trevor here: https://moderncrypto.org/mail-archive/curves/2017/000874.html

Please let me know if you think this is not a good idea!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Action items wrt prop224 onion address encoding (was Re: Proposition: Applying an AONT to Prop224 addresses?)

[George Kadianakis]

Michael Rogers <mich...@briarproject.org> writes:

> On 11/04/17 11:45, George Kadianakis wrote:
>> We basically add the canonical onion address in the inner encrypted
>> layer of the descriptor, and expect the client to verify it. I made this
>> feature optional in case we ever decide it was a bad idea.
>
> Is the version number also included in the blinded key derivation? I
> haven't been keeping up with prop224 developments, so apologies if
> that's already been settled, but in your previous email it sounded like
> it was one of the suggestions but not one of the action items.
>

That's a fine question, and it made me think deeper about our options.

I think both of the following suggestions from my previous email aim to
protect from the same attacks:
a) Include version number in blinded key derivation formula
b) Include canonical onion address in descriptor

Both (a) and (b) above aim to protect against scenarios where an
attacker (without private keys) takes a legitimate onion address, tweaks
its metadata bits (version/whatever), and creates a
different-but-equivalent onion address that has the exact same behavior
as the original one (points to the same 25519 key, same descriptor,
etc.). See Alec's and Ian's emails for a demonstration of how this can
be exploited:
https://lists.torproject.org/pipermail/tor-dev/2017-April/012160.html
https://lists.torproject.org/pipermail/tor-dev/2017-April/012159.html

I'm pretty sure that both (a) and (b) defend against Alec's and Ian's
attack since:

- With (a), the different-but-equivalent onion address would produce a
  different blinded key from the original onion address. The attacker
  would not be able to forge a signature for the descriptor since they
  don't know the private part of the new blinded key, so the descriptor
  would not be accepted by the HSDir or the client.

- With (b), the different-but-equivalent onion address would work, but
  when the client fetches the descriptor, the client would verify the
  new "canonical-onion-addr" field, notice that they reached this onion
  service from another address, and reject the descriptor. It's like we
  are including an SSL CN field in our HS descriptors.

I considered (b) easier to understand and reason about, and it also
seems to protect against a wider variety of descriptor replay attacks,
and that's why I suggested we go with (b).

My main fear with (b) is that in the future we might come up with a new
load-balancing scheme of sorts that would get screwed by the
"canonical-onion-addr" field. It does not seem to pose a problem with
onionbalance or stealth-auth kind of schemes, so all is good so far.
Please let me know if you can think of a scenario where (b) is a bad idea.

> If the version number is included in the descriptor but not in the
> blinded key derivation, can a service publish descriptors for multiple
> protocol versions? Would there be a conflict if the HS directories store
> the descriptors under the same blinded key?
>

Yes it's possible to publish descs for multiple protocol versions, since
we use a different URL for each version. Quoting from spec:

   Hidden service descriptors conforming to this specification are uploaded
   with an HTTP POST request to the URL /tor/hs//publish relative to
   the hidden service directory's root, and downloaded with an HTTP GET
   request for the URL /tor/hs// where  is a base64 encoding of
   the hidden service's blinded public key and  is the protocol
   version which is "3" in this case.

Also the HSDirs store the descriptors using both the publickey and the
version as indices, so this should not be a problem.

---

Thanks for all the feedback people.

Greetings from Athens!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Action items wrt prop224 onion address encoding (was Re: Proposition: Applying an AONT to Prop224 addresses?)

[George Kadianakis]

George Kadianakis <desnac...@riseup.net> writes:

> Ian Goldberg <i...@cs.uwaterloo.ca> writes:
>
>> On Wed, Apr 05, 2017 at 10:02:07AM -0400, David Goulet wrote:
>>> Another thing about this I just thought of. This AONT construction seems 
>>> wise
>>> to use. But it's still not entirely clear to me why we need a 1bit version
>>> field. Taking this:
>>> 
>>> base64( AONT( pubkey || 0x ) || version)
>>> 
>>> If the version is 1 byte, then only the end of the address can be mangled 
>>> with
>>> and if it is, the tor client won't be able to fetch the descriptor because 
>>> of
>>> how the URL is constructed (correct version number is needed).
>>> 
>>> So I really don't see the phishing attack here being successful at all...?
>>> 
>>> Can you enlighten what attack we are trying to avoid here that we require a
>>> 1bit version field?
>>
>> I believe the danger Alec was wanting to avoid was that someone (not the
>> onion service owner) could take an existing onion address, bump the
>> version number (which wouldn't change the vanity beginning of the
>> address), and upload the very same descriptor to the resulting blinded
>> address (under the new version number).  Then the modified address would
>> work just like the original.
>>
>> As mentioned elsewhere in the thread, this is solved if that descriptor
>> contains (under the signature by the "master" onion key) the actual
>> onion address you were expected to use to get there.  Does it?  If so,
>> I think we don't have to worry about this problem at all.
>>
>
> Hello people,
>
> the AONT thread has grown to an immense size and includes all sorts of
> discussions, so I will split it into two smaller threads with just
> action items so that we move this forward ASAP (as this interacts with
> our current implementation efforts).
>
> 
>
> "Then let's include the canonical onionaddress (including version) into
> the descriptor so that clients can verify that they used the
> onionaddress that the onionservice was intending for them to use"
>
> So I guess the current suggested plan is to add an extra descriptor
> field with the onionaddress (or its hash) into the _encrypted parts_ of
> the descriptor so that clients can do this extra verification to defend
> against downgrade attacks.
>
> I think this seems like a reasonable defence here, and more safe +
> engineering-friendly than the AONT stuff (see David's email). We should
> just make sure that this plan does not interact badly with things like
> onionbalance and future name systems.
>
> Do you think this makes sense?  If yes, I will write a spec patch in the
> next few days.
>
> And I think this sums up the discussion wrt onion address encoding. I'm
> going to start a new thread about the ed25519-related suggestions that
> were thrown into this thread.
>

And here is a torspec branch that specifies this behavior:
 
https://gitweb.torproject.org/user/asn/torspec.git/commit/?h=prop224-desc-phishing

We basically add the canonical onion address in the inner encrypted
layer of the descriptor, and expect the client to verify it. I made this
feature optional in case we ever decide it was a bad idea.

Please let me know if you think this behavior is worthwhile merging upstream 
and implementing.

Thanks! :)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Action items wrt ed25519 onion address verification in prop224 (was Re: Proposition: Applying an AONT to Prop224 addresses?)

[George Kadianakis]

Ian Goldberg  writes:

> On Mon, Apr 03, 2017 at 04:40:52PM +0100, Alec Muffett wrote:
>> On 3 Apr 2017 3:48 p.m., "Ian Goldberg"  wrote:
>> 
>> The other thing to remember is that didn't we already say that
>> 
>> facebookgbiyeqv3ebtjnlntwyvjoa2n7rvpnnaryd4a.onion
>> 
>> and
>> 
>> face-book-gbiy-eqv3-ebtj-nlnt-wyvj-oa2n-7rvp-nnar-yd4a.onion
>> 
>> will mean the same thing?  So we're already past the "one (st)ring to
>> rule them all" point?
>> 
>> 
>> That's a great point, and I'm definitely interested and in favour of
>> readability.
>> 
>> How about this, though: I know that Tor doesn't want to be in the business
>> of site reputation, but what if (eg) Protonmail offers a Onion "Safe
>> Browsing" extension some day, of known-bad Onions for malware reasons?
>
> That's a quite good motivating example, thanks!
>
>> There's quite a gulf between stripping hyphens from a candidate onion
>> address and doing strcmp(), versus either drilling into the candidate
>> address to compute the alternative forms to check against the blacklist, or
>> even requiring the blacklist to be 8x larger?
>
> Yes, that's true.  I'm definitely in favour of the "multiply by L (the
> order of the group) and check that you get the identity element; error
> with 'malformed address' if you don't" to get rid of the torsion point
> problem.
>

Hello again,

this is the second subthread of the AONT thread that grew too big for
its own good, and it's about ed25519.

The topic of this subthread is the above ed25519 verification of onion
addresses that Ian suggested a few times already.

So the idea is that before you use an onionaddress (as a client or
whatever), you should extract its ed25519 pubkey and multiply it by the
group order and make sure you get back the identity element to ensure
that there are no torsion components to the key.

I'm pretty weak on crypto so I have some questions about this defence:

- Why are we doing this? Are we doing this because if we allow torsion
  components in the keys, someone could basically create multiple
  equivalent keys for each legit ed25519 key, using the Z/8Z torsion
  scalar as the tweak?

  Or is the reason to defend against small subgroup attacks? I think
  not, because from my understanding these attacks mainly apply to DH
  protocols which is not what we are doing with onion addresses.

- Is this something that we should be doing for _any_ received ed25519
  ever, even in other parts of the protocol?

- Should we do this verification also for received x25519 (DH) keys? It
  seems like RFC7748 is instead suggesting we ensure that the DH output
  is not all-zeroes. Are these two defences equivalent for our purposes?

Thanks for the help :)

(Also, please let me know if there are any other action items from the
AONT thread that I missed.)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Action items wrt prop224 onion address encoding (was Re: Proposition: Applying an AONT to Prop224 addresses?)

[George Kadianakis]

Ian Goldberg  writes:

> On Wed, Apr 05, 2017 at 10:02:07AM -0400, David Goulet wrote:
>> Another thing about this I just thought of. This AONT construction seems wise
>> to use. But it's still not entirely clear to me why we need a 1bit version
>> field. Taking this:
>> 
>> base64( AONT( pubkey || 0x ) || version)
>> 
>> If the version is 1 byte, then only the end of the address can be mangled 
>> with
>> and if it is, the tor client won't be able to fetch the descriptor because of
>> how the URL is constructed (correct version number is needed).
>> 
>> So I really don't see the phishing attack here being successful at all...?
>> 
>> Can you enlighten what attack we are trying to avoid here that we require a
>> 1bit version field?
>
> I believe the danger Alec was wanting to avoid was that someone (not the
> onion service owner) could take an existing onion address, bump the
> version number (which wouldn't change the vanity beginning of the
> address), and upload the very same descriptor to the resulting blinded
> address (under the new version number).  Then the modified address would
> work just like the original.
>
> As mentioned elsewhere in the thread, this is solved if that descriptor
> contains (under the signature by the "master" onion key) the actual
> onion address you were expected to use to get there.  Does it?  If so,
> I think we don't have to worry about this problem at all.
>

Hello people,

the AONT thread has grown to an immense size and includes all sorts of
discussions, so I will split it into two smaller threads with just
action items so that we move this forward ASAP (as this interacts with
our current implementation efforts).

From skimming the thread, this seems like the general discussion flow:

- "Let's do AONT so that no one can tweak the onion address while
  keeping the same blinded pubkey so that people can't create multiple
  onion addresses that point to the same key and look almost the same"

- "Hm, but there are no bits to tweak apart from the version field"

- "But maybe v4->v3 downgrade attacks are possible using the version
  field, so let's include the whole onionaddress (including version)
  into the blinded key derivation"

- But then maybe in 2020 an attacker is able to replay a v4 descriptor
  into an HSDir as a v3 descriptor, and then do a downgrade attack by
  persuading a victim to fetch the v3 descriptor (see Ian/Alec latest mails)

And I think then we ended up with:

"Then let's include the canonical onionaddress (including version) into
the descriptor so that clients can verify that they used the
onionaddress that the onionservice was intending for them to use"

So I guess the current suggested plan is to add an extra descriptor
field with the onionaddress (or its hash) into the _encrypted parts_ of
the descriptor so that clients can do this extra verification to defend
against downgrade attacks.

I think this seems like a reasonable defence here, and more safe +
engineering-friendly than the AONT stuff (see David's email). We should
just make sure that this plan does not interact badly with things like
onionbalance and future name systems.

Do you think this makes sense?  If yes, I will write a spec patch in the
next few days.

And I think this sums up the discussion wrt onion address encoding. I'm
going to start a new thread about the ed25519-related suggestions that
were thrown into this thread.

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] ***SPAM*** Re: Proposition: Applying an AONT to Prop224 addresses?

[George Kadianakis]

Ian Goldberg <i...@cs.uwaterloo.ca> writes:

> On Mon, Apr 03, 2017 at 02:53:17PM +0100, Alec Muffett wrote:
>> On 3 April 2017 at 13:04, George Kadianakis <desnac...@riseup.net> wrote:
>> 
>> > I'm calling it weird because I'm not sure how an
>> > attacker can profit from being able to provide two addresses that
>> > correspond to the same key, but I can probably come up with a few
>> > scenarios if I think about it.
>> 
>> 
>> Hi George!
>> 
>> I'll agree it's a weird edge case :-)
>> 
>> I think the reason my spider-sense is tingling is because years of cleaning
>> up after intrusions has taught me that sysadmins and human beings are very
>> bad at non-canonical address formats, especially where they combine them
>> with either blacklisting, or else case-statements-with-default-conditions.
>> 
>> If one creates scope for saying "the address is .onion but you can
>> actually use <foo'>.onion or <foo''>.onion which are equivalent" - then
>> someone will somehow leverage that either a) for hackery, or b) for social
>> engineering.
>> 
>> Compare:
>> 
>> * http://01770001
>> * http://2130706433
>> * http://0177.0.0.1  <- this one tends to surprise people
>> * http://127.0.0.1
>> 
>> …and the sort of fun shenanigans that can be done with those "equivalent
>> forms"
>> 
>> People who've been trained not to type [X] into their browser, might be
>> convinced to type [X']
>> 
>> It's a lot easier for people to cope with there being one-and-only-one
>> viable form for any given hostname or address-representation.
>
> But as I said to Alec in AMS, anyone on the internet can register
> "facebook.mydomain.com" and have the A record point to the same thing as
> facebook.com.  So there are always alternate names for any given
> website.  TLS, of course, is designed to protect against these
> shenanigans.
>

Hey,

sorry for the slow responses to this thread. Got lots of post-meeting
backlog to handle, and I'm also working on the various ed25519 stuff.

Specifically, I'm now working on the suggested check of multiplying any
received curve25519 point with the group order and ensuring the result
is the identity element.

> Prop224 *also* (mostly) protects against these shenanigans, because even
> if there were two onion addresses that resolved to the same pubkey, the
> daily blinded version incorporates the original onion address (not just
> the pubkey, right?  *Right?*), so the alternate address-with-same-pubkey
> won't actually point anywhere.  However, an adversary can upload a
> descriptor there; I'm not sure what the implications of that are just
> now.
>

Actually, I *don't* think that the blind factor of the derived key
incorporates the actual onion address. Citing the proposal:

  Let the basepoint be written as B. Assume B has prime order l, so
  lB=0. Let a master keypair be written as (a,A), where a is the private
  key and A is the public key (A=aB)

  To derive the key for a nonce N and an optional secret s, compute the
  blinding factor h as H(A | s, B, N), and let:

Perhaps we can add another component to h as follows:
 h = H(A, s, B, N, ONIONADDRESS)
where ONIONADDRESS is a string representation of the service's onion address.

I think this code is already implemented, but this might be worth fixing anyhow.
I'll make a ticket.

> The other thing to remember is that didn't we already say that
>
> facebookgbiyeqv3ebtjnlntwyvjoa2n7rvpnnaryd4a.onion
>
> and
>
> face-book-gbiy-eqv3-ebtj-nlnt-wyvj-oa2n-7rvp-nnar-yd4a.onion
>
> will mean the same thing?  So we're already past the "one (st)ring to
> rule them all" point?
>

I don't think we have actually decided on such a feature yet. It was
suggested but the tradeoffs are not clearly skewed to the "let's do it"
direction.

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Comments on proposal 279 (Name API)

[George Kadianakis]

Nick Mathewson  writes:

> Hi !  I'll make some comments here on the draft onion naming API at
>
> https://gitweb.torproject.org/torspec.git/tree/proposals/279-naming-layer-api.txt
>
> (Some of  these are probably things you already meant, or already said
> elsewhere.)
>

Thanks for the timely comments! I'm replying to this thread with my
thoughts, but I didn't have time to actually fix the proposal. I'll do
that in The Future.

>
>
> Section 2.1 and elsewhere:
>
> I suggest that we require all address suffixes to end with .onion;
> other TLDs are not reserved like .onion is, and maybe we shouldn't
> squat any we haven't squatted already.   I think we might also want to
> have all output addresses end with .onion too.
>
> I suggest  also that we might want to reserve part of the namespace
> for standardized namespaces and some for experimental or local ones.
> Like, if we standardize on namecoin that could be .bit.onion, but if
> we don't, it could be .bit.x.onion.
>

I have mixed feelings about keeping the .onion suffix.

One one hand it seems like The Right Thing to do, since we managed to
get .onion standarized in the IETF which comes with various
benefits. Also, starting to squat other tlds arbitrarily seems like a
silly thing to do.

However, I also dislike asking users to visit something.bit.onion
instead of something.bit, since people are not used to the second tld
having a semantic meaning, and I can imagine people getting very
confused about what it means.

Anyhow, it seems like maintaining the .onion suffix is the right
approach here.

> I finally suggest that we distinguish names that are supposed to be
> global from ones that aren't.
>
> Section 2.3:
>
> How about we require that the suffixes be distinct?  If we do that, we
> can drop this "priority" business and we can make the system's
> behavior much easier to understand and explain.
>

Definitely agreed on this simplification suggestion. The priority
feature has confused people, and it's not that useful. In the future we
could reinstall it if we consider it practical.

> Let's require that the TLDs actually begin with a dot.  (That is, I
> think that ".foo.onion" can include "bar.foo.onion", but I don't like
> the idea of "foo.onion" including "barfoo.onion".)
>

Makes sense.

>
> Section 2.3.1:
>
> Does the algorithm apply recursively?  That is, can more then one
> plugin rewrite the same address, or can one plugin rewrite its own
> output?
>
> (I would suggest "no".)
>

Agreed no. We should specify it.

> I think there should be a way for a plugin to say "This address
> definitely does not exist" and stop resolution.  Otherwise no plugin
> can be authoritative over a TLD.
>

Agreed.

> Section 2.5.1:
>
> Is the algorithm allowed to produce non-onion addresses?  Should it be?
>

I'd say no. We should specify this. 

> Must query IDs be unique?  Over what scope must they be unique? Who
> enforces that?
>

I think the NS API client should enforce that, and maybe the server
should throw an error if it's not unique.

We should specify.

> May query IDs be negative?  Can they be arbitrarily large?
>

We should specify this too.

> I think result should indeed be optional on failure.
>
> Section 2.5.1 and 2.5.2:
>
> We should specify what exactly clients and plugins will do if they
> receive an unrecognized message, or a malformed message.
>

Agreed.

> Section 2.5.3.
>
> See security notes on caching below; client-side caching can lead to
> undesirable results.
>

Agreed.

> As noted above, I agree with requiring all result addresses to be .onion.
>
> Section 3.1:
>
> I prefer the "put everything under .onion" option.   I also think that
> we should require that the second-level domain be 10 characters or
> less, to avoid confusion with existing onion addresses.
>

We should think more about this, but seems reasonable.

>
>
> General questions:
>
> I know we've done stdout/stdin for communication before, but I wonder
> if we should consider how well it's worked for us.  The portability on
> Windows can be kind of hard.
>
> Two alternatives are TCP and named pipes.
>
> Another alternative might be just using the DNS protocol and asking
> for some kind of "ONION_CNAME" record.  (DNS is ugly, but at least
> it's nice and standard.)
>

Yup, I think this is an _important_ open part of the proposal that we
should figure out sooner than later. Ideally, we should consult Nathan
or mtigas or other members of our mobile team. I wish I had done this
during the dev meeting...

TCP seems like a plausible alternative here. Unfortunately, we will have
to invent a new protocol for that tho.

>
> Security notes:
>
> I'd like to know what the browser people think about the risks here of
> (eg) probing to see whether the user has certain extensions installed
> or names mapped.  Maybe .hosts.onion should only be allowed in the
> address bar, not in HREF attributes et al?
>

Yep, David F. also mentioned this problem. We should think of how to

Re: [tor-dev] Proposition: Applying an AONT to Prop224 addresses?

[George Kadianakis]

Ian Goldberg  writes:

> On Mon, Mar 27, 2017 at 01:59:42AM -0400, Ian Goldberg wrote:
>> > To add an aside from a discussion with Teor: the entire "version" field
>> > could be reduced to a single - probably "zero" - bit, in a manner perhaps
>> > similar to the distinctions between Class-A, Class-B, Class-C... addresses
>> > in old IPv4.
>> > 
>> > Thus: if the first bit in the address is zero, then there is no version,
>> > and we are at version 0 of the format
>> > 
>> > If the first bit is one, we are using v1+ of the format and all bets are
>> > off, except that the obvious thing then to do is count the number of 1-bits
>> > (up to some limit) and declare that to be version number.  Once we're up to
>> > 3 or 4 or 7 or 8 one-bits, then shift version encoding totally.
>> > 
>> > Teor will correct me if I misquote him, but the advantage here was:
>> > 
>> > a) the version number is 1 bit, ie: small, for the forseeable / if we get
>> > it right
>> > 
>> > b) in pursuit of smallness, we could maybe dump the hash in favour of a
>> > AONT + eyeballs, which would give back a bunch of extra bits
>> > 
>> > result: shorter addresses, happier users.
>> 
>> You indeed do not require a checksum under an AONT, but you do require
>> redundancy if you want to catch typos.  Something like
>> 
>> base64( AONT( pubkey || 0x ) || version)
>> 
>> is fine.  If you want "version" to be a single bit, then the AONT would
>> have to operate on non-full bytes, which is a bit (ha!) annoying, but
>> not terrible.  In that case, "0x" would actually be 15 bits of 0,
>> and version would be 1 bit.  This would only save 1.4 base32 characters,
>> though.  If you took off some more bits of the redundancy (down to 8
>> bits?), you would be able to shave one more base32 char.  And indeed, if
>> you make the redunancy just a single byte of 0x00, then the extra 0-bit
>> for the "version" actually fits neatly in the one leftover bit of the
>> base32 encoding, I think, so the AONT is back to working on full bytes.
>> 
>> But is a single byte of redundancy enough?  It will let through one out
>> of every 256 typos.  (I thought we had spec'd 2 bytes for the checkcum
>> now, but maybe I misremember?  I'm also assuming we're using a simple
>> 256-bit encoding of the pubkey, rather than something more complex that
>> saves ~3 bits.)
>> 
>> (Heading to the airport.)
>
> OK, here are the details of this variant of the proposal.  Onion
> addresses are 54 characters in this variant, and the typo-resistance is
> 13 bits (1/8192 typos are not caught).
>
> Encoding:
>
> raw is a 34-byte array.  Put the ed25519 key into raw[0..31] and 0x
> into raw[32..33].  Note that there are really only 13 bits of 0's for
> redundancy, plus the 0 bit for the version, plus 2 unused bits in
> raw[32..33].
>
> Do the AONT.  Here G is a hash function mapping 16-byte inputs to
> 18-byte outputs, and H is a hash function mapping 18-byte inputs to
> 16-byte outputs.  Reasonable implementations would be something like:
>
> G(input) = SHA3-256("Prop224Gv0" || input)[0..17]
> H(input) = SHA3-256("Prop224Hv0" || input)[0..15]
>
> raw[16..33] ^= G(raw[0..15])
> # Clear the last few bits, since we really only want 13 bits of redundancy
> raw[33] &= 0xf8
> raw[0..15] ^= H(raw[16..33])
>
> Then base32-encode raw[0..33].  The 56-character result will always end
> in "a=" (the two unused bits at the end of raw[33]), so just remove that
> part.
>
> Decoding:
>
> Base32-decode the received address into raw[0..33].  Depending on your
> base32 decoder, you may have to stick the "a=" at the end of the address
> first.  The low two bits were unused; be sure the base32 decoder sets
> them to 0.  The next lowest bit (raw[33] & 0x04) is the version bit.
> Ensure that (raw[33] & 0x04 == 0); if not, this is a different address
> format version you don't understand.
>
> Undo the AONT:
>
> raw[0..15] ^= H(raw[16..33])
> raw[16..33] ^= G(raw[0..15])
> # Clear the last few bits, as above
> raw[33] &= 0xf8
>
> Check the redundancy by ensuring that raw[32..33] = 0x.  If not,
> there was a typo in the address.  (Note again that since we explicitly
> cleared the low 3 bits of raw[33], there are really only 13 bits of
> checking here.)
>
> raw[0..31] is then the pubkey suitable for use in Ed25519.  As before
> (and independently of the AONT stuff), you could sanity-check it to make
> sure that (a) it is not the identity element, and (b) L times it *is*
> the identity element.  (L is the order of the Ed25519 group.)  Checking
> (a) is important; checking (b) isn't strictly necessary for the reasons
> given before, but is still a sensible thing to do.  If you don't check
> (b), you actually have to check in (a) that the pubkey isn't one of 8
> bad values, not just the identity.  So just go ahead and check (b) to
> rest easier. ;-)
>
>
> This version contains two calls to SHA3, as opposed to the one such call
> in the non-AONT (but including a checksum) version.  The 

Re: [tor-dev] GSoC 17 | Name System API for Tor Onion Services

[George Kadianakis]

George Kadianakis <desnac...@riseup.net> writes:

> Pickfire <pickf...@riseup.net> writes:
>
>> Hi,
>>
>> I am Ivan Tham. Currently studying in Computer Science in APIIT Malaysia. I 
>> am
>> interested particapate in Google Summer of Code 2017 under tor organization. 
>> I
>> am interested to see Proposal 224 coming along but I would really like to see
>> [Proposal 272][0] and hope that tor hidden services can be more 
>> user-friendly.
>>
>
> Hello,
>
> there is still interest in this proposal but unfortunately it hasn't
> been revised since it was first posted on the mailing list. The mailing
> list feedback unfortunately has not been incorporated to the proposal
> yet; particularly the comments by David Fifield are very relevant and
> should be considered carefully before taking the proposal too seriously.
>
> In general, I suggest to anyone who wants to work on this proposal, to
> do it using a Tor controller instead of hacking the main C tor
> code. meejah suggested this here:
>   https://lists.torproject.org/pipermail/tor-dev/2016-October/011517.html
> and it seems like a proper solution here would involve controller events
> like NEWRESOLVE, MAPADDRESS, and plus some extra magic.
>
> I must say that this project is definitely relevant for GSoC, but it
> needs a _strong_ and _independent_ student that can handle it.
>
> Also, Tor is currently having a real life meeting so most of us are very
> busy. We plan to discuss the proposal during the meeting, so I hope to
> send a short update next week at some point if I find the time (and also
> merge it to torspec.git since it's currently missing).
>

FWIW, I merged the draft proposal from [tor-dev] to torspec, so that we
have a common point of reference. The proposal still needs improvements
before we can call it complete.

You can find it as proposal279:

https://gitweb.torproject.org/torspec.git/tree/proposals/279-naming-layer-api.txt

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] GSoC 17 | Name System API for Tor Onion Services

[George Kadianakis]

Ivan Tham <pickf...@riseup.net> writes:

> George Kadianakis <desnac...@riseup.net> wrote:
>
>> Pickfire <pickf...@riseup.net> writes:
>> 
>> > I am Ivan Tham. Currently studying in Computer Science in APIIT Malaysia. 
>> > I am
>> > interested particapate in Google Summer of Code 2017 under tor 
>> > organization. I
>> > am interested to see Proposal 224 coming along but I would really like to 
>> > see
>> > [Proposal 272][0] and hope that tor hidden services can be more 
>> > user-friendly.
>> 
>> there is still interest in this proposal but unfortunately it hasn't
>> been revised since it was first posted on the mailing list. The mailing
>> list feedback unfortunately has not been incorporated to the proposal
>> yet; particularly the comments by David Fifield are very relevant and
>> should be considered carefully before taking the proposal too seriously.
>
> Does
>
>> In general, I suggest to anyone who wants to work on this proposal, to
>> do it using a Tor controller instead of hacking the main C tor
>> code. meejah suggested this here:
>>   https://lists.torproject.org/pipermail/tor-dev/2016-October/011517.html
>
> Does that mean that I well be working on the prototype with python first and
> then convert it into C?
>

Hello,

I think a well made controller-based prototype in Python and one or two
PoC integrations of simple name systems (e.g. local hosts file, remote
hosts file) should be fine for a GSoC summer project.

If we like the controller prototype we could consider writing it in C in
the future.

>> and it seems like a proper solution here would involve controller events
>> like NEWRESOLVE, MAPADDRESS, and plus some extra magic.
>
> I don't quite understand what does that mean.
>

Please check control-spec.txt for a description of those events:
   https://gitweb.torproject.org/torspec.git/tree/control-spec.txt
It might make things clearer.

>> I must say that this project is definitely relevant for GSoC, but it
>> needs a _strong_ and _independent_ student that can handle it.
>
> I will definitely do my best for it but I will really need a mentor to help
> because I am confused by some parts of it as well.
>

Hmm. Which parts did you find confusing?

Cheers!


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] GSoC 17 | Name System API for Tor Onion Services

[George Kadianakis]

Pickfire  writes:

> Hi,
>
> I am Ivan Tham. Currently studying in Computer Science in APIIT Malaysia. I am
> interested particapate in Google Summer of Code 2017 under tor organization. I
> am interested to see Proposal 224 coming along but I would really like to see
> [Proposal 272][0] and hope that tor hidden services can be more user-friendly.
>

Hello,

there is still interest in this proposal but unfortunately it hasn't
been revised since it was first posted on the mailing list. The mailing
list feedback unfortunately has not been incorporated to the proposal
yet; particularly the comments by David Fifield are very relevant and
should be considered carefully before taking the proposal too seriously.

In general, I suggest to anyone who wants to work on this proposal, to
do it using a Tor controller instead of hacking the main C tor
code. meejah suggested this here:
  https://lists.torproject.org/pipermail/tor-dev/2016-October/011517.html
and it seems like a proper solution here would involve controller events
like NEWRESOLVE, MAPADDRESS, and plus some extra magic.

I must say that this project is definitely relevant for GSoC, but it
needs a _strong_ and _independent_ student that can handle it.

Also, Tor is currently having a real life meeting so most of us are very
busy. We plan to discuss the proposal during the meeting, so I hope to
send a short update next week at some point if I find the time (and also
merge it to torspec.git since it's currently missing).

I'm looking forward to any proposal comments, as well as responses to
the feedback received. Also, if someone can sketch an implementation
plan for the controller idea that would be great.

Sorry for the uncertainties here and good luck!
Cheers!






___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal xyz : Count Unique IP addresses in an anonymous way

[George Kadianakis]

teor  writes:

> Hi Jaskaran,
>
>> On 17 Mar 2017, at 23:42, Jaskaran Singh  wrote:
>> 
>> §2. Research
>> 
>> There are three ways to solve this problem. All the three ways are
>> actually Big Data
>> Algorithms. A few problems arises for each of these algorithms since
>> they are made for
>> big data but the data we would provide is not necessarily big.
>
> As we discussed on IRC, there's a simpler way to solve the problem of
> storing IP addresses in memory: store a (keyed) hash of the IP address
> instead.
>
> The hash can be tuned to be sufficiently hard to make brute-forcing
> impractical. (As long as each 'country' has sufficiently many IP
> addresses. And as long as the threat model excludes adversaries which
> only want to confirm a few addresses.)
>

Hey teor,

I feel like replying to this thread is basicaly moot without a precise
threat model, since it seems like each person here has a different
threat model in mind. There are many attacks and scenarios applicable
and it's unclear which ones we are trying or hoping to cover.

For example, I'm confused on how a keyed hash is better than a simple
hash (or scrypt) of the IP address (which most people agree is not a
good solution). That is, if an attacker pwns a box and steals the hash
key and the list of hashes, the attacker can easily brute-force the 2^32
keyed hashes at that point and derive the list of connected IP
addresses. What's the difference?

> The key can be rotated at a suitable interval, ensuring that past
> addresses can not be discovered by future attackers.
>

Hmm, is this attack in the threat model? And how does it work exactly?

Also how does salted hash vs normal hash make a difference here, since
even in the normal hash variant, if you memwipe/rotate the hash list
everyday, an attacker should not be able to discover past addresses.

All in all, I think it will be hard to choose a scheme here without a
precise threat model, and I don't see the original proposal making an
attempt to define that.

Greetings from Amsterdam!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [PATCH] prop224: Trivial improvements to HS ntor section

[George Kadianakis]

George Kadianakis <desnac...@riseup.net> writes:

> Hey David,
>
> please check my `prop224-ntor` torspec branch for some basic
> improvements to the HS ntor section that came up while implementing it.
>
> Here it is:
>https://gitweb.torproject.org/user/asn/torspec.git/log/?h=prop224-ntor
>

Hey again David,

here is another round of easy prop224 fixes based on our discussions
in IRC: 
https://gitweb.torproject.org/user/asn/torspec.git/log/?h=prop224-more-ntor

Namely the first commit updates our MAC definition to fit the latest
code, and the second commit fixes some more undefined variables around
the ntor part of the proposal.

Let me know if you like them and I'll merge them.

Thanks!
   

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] [PATCH] prop224: Trivial improvements to HS ntor section

[George Kadianakis]

Hey David,

please check my `prop224-ntor` torspec branch for some basic
improvements to the HS ntor section that came up while implementing it.

Here it is:
   https://gitweb.torproject.org/user/asn/torspec.git/log/?h=prop224-ntor

Let me know if you like them and I will merge them ASAP.

Cheers!


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [RFC] Directory structure of prop224 onion services

[George Kadianakis]

David Goulet <dgou...@ev0ke.net> writes:

> On 30 Jan (16:16:07), George Kadianakis wrote:
>> David Goulet <dgou...@ev0ke.net> writes:
>> 
>> > On 26 Jan (15:05:26), George Kadianakis wrote:
>> >> Hey list,
>> >
>> > Hi!
>> >
>> > First, big thanks for this write up!
>> >
>> >> 
>> >> with service-side prop224 implementation moving forward, we need to pin 
>> >> down
>> >> the directory structure of prop224 onion services. This will be very 
>> >> similar to
>> >> the current directory structure, but with some mods to facilitate 
>> >> assymetric
>> >> client authorization keys and offline keys.
>> >> 
>> >> As people have pointed out, the HS directory structure matters less after 
>> >> the
>> >> introduction of ephemeral ADD_ONION onion services, but still it's an 
>> >> important
>> >> part of onion service sysadmin UX.
>> >> 
>> >> 
>> >
>> 
>> Hey David,
>> 
>> thanks for the useful comments.
>> 
>> Please check my torspec branch `prop224-directory-format`.
>> 
>> FWIW, I agree with all the expected behavior details you noted at the
>> end of your email. I encoded some of those behaviors in the spec, but I
>> didn't provide a complete formal algorithm of how the whole process
>> works because I don't think it's spec material and also because I feel
>> that during implementation we will get new insights on how this should
>> work.
>> 
>> Let me know how you feel about the spec patch :)
>
> Good stuff! And yes, I don't think it's spec material at all but good to have
> in an Appendfix for reference. Once this file structure will be released in a
> tor version, we *must* update the man page FILES section.
>

OK, I merged this patch to torspec as well!

Thanks for the feedback everyone.

We can edit it if we figure out something is wrong; and we need to
update it anyway to add the offline keys info.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [RFC] Proposal for the encoding of prop224 onion addresses

[George Kadianakis]

Ivan Markin <t...@riseup.net> writes:

> Hi,
>
> George Kadianakis wrote:
>> I made a torspec branch that alters prop224 accordingly:
>>   
>> https://gitweb.torproject.org/user/asn/torspec.git/commit/?h=prop224-onion-address=50ffab9903880acf55fe387f4d509ecb2aa17f95
>
> It seems that SHA3 digest length is missing for onion address
> generation. I guess (?) that it supposed to be SHA3-256 but it
> definitely should be specified here.
> I think that it just a typo since there is definition of H() above.
>

OK guys, thanks for all the great feedback!

I merged my prop224 onion encoding patch to torspec just now, after
fixing the bug that Ivan mentioned above.

Hope this works for you :)

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [RFC] Directory structure of prop224 onion services

[George Kadianakis]

David Goulet <dgou...@ev0ke.net> writes:

> On 26 Jan (15:05:26), George Kadianakis wrote:
>> Hey list,
>
> Hi!
>
> First, big thanks for this write up!
>
>> 
>> with service-side prop224 implementation moving forward, we need to pin down
>> the directory structure of prop224 onion services. This will be very similar 
>> to
>> the current directory structure, but with some mods to facilitate assymetric
>> client authorization keys and offline keys.
>> 
>> As people have pointed out, the HS directory structure matters less after the
>> introduction of ephemeral ADD_ONION onion services, but still it's an 
>> important
>> part of onion service sysadmin UX.
>> 
>> So the HiddenServiceDir directory will contain the following items:
>> 
>> - "./hostname"[FILE]
>> 
>>This is a file containing the onion address of the onion service.
>> 
>>As you can see it's the same filename as in v2. Should we suffix it with 
>> v3
>>to make it clear that it's v3 onion? Would we ever have v2 and v3 onions
>>living in the same directory?
>
> I don't believe we should suffix here because for almost 10 years, users/apps
> have been exposed to "hostname" and it does make sense that it's the goto file
> for that.
>
> Current implementation doesn't allow two services in the same HiddenServiceDir
> and for prop224, the ongoing implementation doesn't allow it either. Sharing a
> directory brings all sorts of uneeded complexity. So if the directory is v3,
> everything in it will be v3.
>
>> 
>> - "./private_key_ed25519"  [FILE]
>> 
>>This is the file containing the private master ed25519 key of the onion 
>> service.
>> 
>>If offline keys are _enabled_, then this file doesn't exist and instead a
>>directory is made containing blinded keys for every day [TODO: The 
>> directory
>>format here will be specified in the future].
>
> If that file doesn't exists, the public key is needed else the service can't
> derive the .onion and create the hostname file. The offline case is an extra
> use case but I suspect we would use "public_key_ed25519" along with the
> blinded keys specific file name. (Unless we make our "tor-genkey" tool
> generate the hostname file as well. #bikesheding)
>
>> 
>> - "./client_authorized_pubkeys"   [FILE]
>> 
>>   If client authorization is _enabled_, this is a newline-separated file of
>>   " " entries for authorized clients. You can think of 
>> it
>>   as the ~/.ssh/authorized_keys of onion services.
>> 
>> - "./client_authorized_privkeys/"  [DIRECTORY]
>>   "./client_authorized_privkeys/alice" [FILE] 
>>   "./client_authorized_privkeys/bob"   [FILE]
>>   "./client_authorized_privkeys/charlie"   [FILE]
>
> Small clarification. The "" field in the the pubkey file is the
> same for the privkey file name. So if "alice" is in the pubkey file, it will
> be "alice" in this privkey directory.
>
>>   
>>   If client authorization is _enabled_ _AND_ if the hidden service is
>>   responsible for generating and distributing private keys for its clients,
>>   then this directory contains files with client's private keys. The idea is
>>   that these files can be shredded and deleted after the private key has been
>>   passed to the client. For more context here, please read the client
>>   authorization thread in [tor-dev] and see 'Appendix F' of prop224 for more
>>   details on how this works.
>
> Also, expected behavior that we should go for when implementing this within
> the "tor" code base. We could think of many ways to make this more complex
> that it could be but going *simple* is what I'm aiming for:
>
> - The torrc option HiddenServiceAuthorizeClient as to match the list of client
>   in the pubkey file in so if the pubkey file has extra entries, we error at
>   startup. With this in mind, here are the behaviors:
>
> i) if a privkey file exists but no entry in the pubkey file, add the entry to
>pubkey file as long as the client name is found in
>HiddenServiceAuthorizeClient.
>
> ii) a pubkey entries does NOT need a corresponding privkey to be used. As long
> as the client name is found in HiddenServiceAuthorizeClient.
>
> iii) if a client name is specified in the HiddenServiceAuthorizeClient option 
> but
>  NOT in the pubkey file, generate pubkey/privkey unless the privkey file
>  exists which is (i) behavior where pubkey is deriv

Re: [tor-dev] [RFC] Proposal for the encoding of prop224 onion addresses

[George Kadianakis]

chelsea komlo  writes:

> Hey!
>
>> Here is some extra pressure for you ;).
>
> :) thanks, I will try!
>
> Before starting, someone today very kindly pointed me to Prop 271, the
> naming system API for Tor Onion services. Overall, my larger concern is
> whether adding the version in the onion address makes both using and
> distributing onion addresses harder. If the long-term plan is for onion
> addresses to not be used directly, then having the version in the onion
> address is completely fine as this wouldn't present a barrier to entry
> for end users.
>
>>
>> The HSDir fetch/post URL has gone in 0.3.0 (feature freeze today in theory ;)
>> with the version in it:
>>
>> --> /tor/hs//publish
>>
>> So few things. First, if we don't have the version in the onion address, this
>> means the client needs to try to fetch the descriptor for multiple version
>> that is starting at the highest it knows and then going down as it's failing.
>> That, I'm really not too keen to this, uneeded load on the network.
>
> Yep, fair. So the idea of "fetch multiple descriptors, where a
> descriptor is for a single version," isn't viable for performance reasons.
>  
>>
>> Second thing is that HSDir might not all support the same version by the time
>> we roll out prop224 thus the importance of having it in 0.3.0 (a version
>> *before* the next gen release). Even with that, this is going to be an
>> interesting experiement to have a set of HSDir supporting v3 and a set not
>> supporting it because we kind of have this requirement of using 3 nearest
>> relays for a replica but what if one of them doesn't support v3?
>
> Yeah, that is hard. Although I'm not entirely sure how this complexity
> is correlated with how the client consumes the HS version...
>
>> Third thing, we could have a fix for this with a single descriptor supporting
>> multiple version but then this has implication outside the onion address
>> discussion and unfortunately 0.3.0 material again (that freezes today).
>>
>> So I'm eager to hear your idea on this! But it's important to keep in mind
>> that 0.3.0 has already some building blocks with some version restrictions 
>> :S.
>> Changing those would mean delaying adoption by a 6 months (and it could be
>> OK!).
>
> Yeah! So if the plan is that onion addresses will not be used directly
> by end users and there is an abstraction layer that hides things like
> version upgrade from end users, then going ahead with the current plan
> sounds good.
>
> However, if there is a chance that end users will consume onion
> addresses directly, then having this discussion seems like a good idea.
> The scenario that worries me is something like this:
>
> 1) Facebook creates a hidden service and distributes this address
> 2) A new hidden service version is created
> 3) Facebook is reluctant to upgrade because this would mean
> re-distributing a new onion address to a _lot_ of people. Also, there
> are problems of securely distributing and verifying new onion addresses-
> malicious parties could use this opportunity to distribute lookalikes,
> for example.
>

Hmm, on the above scenario, why would Tor change the version of the
onion address if the pubkey and checksum algorithm do not change? The
way I see it, the main scenario where we bump the onion address version
is if we upgrade the cryptosystem of the identity key or the checksum
algorithm. In that case, Facebook will have to migrate to another
address anyhow, so moving the version field to the HSDir layer does not
really help.

Furthermore, as David said, HS descriptors do have a version field
anyway, so we can always take version-specific decisions on the HSDir
layer without changing the onion address.

Finally, keeping a version field on the onion address, lets clients take
version-specific decisions _before_ contacting HSDirs, which is not
possible right now. The use of this is not obvious to me at this point,
but I'm sure that onion service applications can find some use. Or
it can be used by hidden services that want their clients to use an
alternative HSDir hash ring logic (e.g. increase or decrease the default
number of responsible HSDirs) by encoding this info in the version field.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [RFC] Proposal for the encoding of prop224 onion addresses

[George Kadianakis]

grarpamp  writes:

> Skimming thread...
>
> Version or not is fine, provided if you want versions you
> know you must store the bits somewhere, or ensure regex
> parser rules to recognize and match an intrinsic version
> represented by entire address format specification itself.
>
> Note onion search spiders rely on such address recognition
> and parsing. So it's not all just about the browser brain urlbar.
>
> GPU capacity hasn't hit 16 char yet, mnemonic
> brain memory has, but that's only happened based on
> address luck and/or GPU prefixing. We're more or less
> at the limits, new random bits past 16 won't matter and
> shouldn't be considered much an argument to brain relavance.
> Some other brain layer will come along, and if not, there's
> always search.
>
> If version goes in address, I'd wary against putting it last.
> A lot of things naturally sort and route and default based
> on higher order bits appear prefixing on the left.. IPv4 IPv6
> bitcoin PTR DHT filesystem unix tools... the list goes on.
> A single leading character is not a problem and gives
> plenty of bits of version capacity regardless of encoding.
> Trailing version just plain feels shaky to rely on or to
> advocate to the world as a new standard. Certainly
> not without consultation with other anonymous overlay
> projects as to their future needs and direction as well,
> or to develop such an interop standard.
>

Hm, can you please expand on this? I think I understood none of your arguments.

What's the problem with version field being in the end and tools sorting
addresses based on higher order bits? Also why does version field being
in the end makes it shaky to rely on?





___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [RFC] Proposal for the encoding of prop224 onion addresses

[George Kadianakis]

David Goulet <dgou...@ev0ke.net> writes:

> On 24 Jan (14:27:43), George Kadianakis wrote:
>> s7r <s...@sky-ip.org> writes:
>> 
>> 
>
> I like this quite a bit! Simple, easy, and trivial to understand. 56
> characters address, after that it will be the time to improve UX/UI with all
> sorts of possible tricks to make them easier to remember or copy paste or
> visualize or what not.
>
> Unless some feedback NACK this, I say push that in the proposal soon. I'll
> personally start implementing that scheme this week.
>
> Thanks!
> David
>

Hello,

I made a torspec branch that alters prop224 accordingly:
  
https://gitweb.torproject.org/user/asn/torspec.git/commit/?h=prop224-onion-address=50ffab9903880acf55fe387f4d509ecb2aa17f95

I will merge this to torspec RSN if I don't hear any grave objections.

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] [RFC] Directory structure of prop224 onion services

[George Kadianakis]

Hey list,

with service-side prop224 implementation moving forward, we need to pin down
the directory structure of prop224 onion services. This will be very similar to
the current directory structure, but with some mods to facilitate assymetric
client authorization keys and offline keys.

As people have pointed out, the HS directory structure matters less after the
introduction of ephemeral ADD_ONION onion services, but still it's an important
part of onion service sysadmin UX.

So the HiddenServiceDir directory will contain the following items:

- "./hostname"[FILE]

   This is a file containing the onion address of the onion service.

   As you can see it's the same filename as in v2. Should we suffix it with v3
   to make it clear that it's v3 onion? Would we ever have v2 and v3 onions
   living in the same directory?

- "./private_key_ed25519"  [FILE]

   This is the file containing the private master ed25519 key of the onion 
service.

   If offline keys are _enabled_, then this file doesn't exist and instead a
   directory is made containing blinded keys for every day [TODO: The directory
   format here will be specified in the future].

- "./client_authorized_pubkeys"   [FILE]

  If client authorization is _enabled_, this is a newline-separated file of
  " " entries for authorized clients. You can think of it
  as the ~/.ssh/authorized_keys of onion services.

- "./client_authorized_privkeys/"  [DIRECTORY]
  "./client_authorized_privkeys/alice" [FILE] 
  "./client_authorized_privkeys/bob"   [FILE]
  "./client_authorized_privkeys/charlie"   [FILE]
  
  If client authorization is _enabled_ _AND_ if the hidden service is
  responsible for generating and distributing private keys for its clients,
  then this directory contains files with client's private keys. The idea is
  that these files can be shredded and deleted after the private key has been
  passed to the client. For more context here, please read the client
  authorization thread in [tor-dev] and see 'Appendix F' of prop224 for more
  details on how this works.

So this is it. The above should handle most uses of onion services + client
authorization. The directory format of offline keys will be specified as we
move forward with implementation. 

Hope things here are not too controversial. Looking forward to your feedback.

In a few days, I will add a small Appendix section to prop224 with the
above, and also fix the parts of 'Appendix F' that got outdated since then.

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [RFC] Proposal for the encoding of prop224 onion addresses

[George Kadianakis]

chelsea komlo  writes:

> Hey George,
>
> Thanks for sending this and summarizing everything!
>
>> [D1] How to use version field:
>>
>>   The version field is one byte long. If we use it as an integer we can
>>   encode 256 values in it; if we use it as a bitmap we could encode
>>   properties and such.
>>
>>   My suggestion is to simply use it as an integer like Bitcoin does. So 
>> we
>>   can assign value \x01 to normal onion services, and in the future we 
>> can
>>   assign more version tags if we need to. For example, we can give a
>>   different version field to onion services in the testnet. We can also
>>   reserve a range of values for application-specific purposes.
>
> Will hidden service addresses only encode a single version?
>
> If yes to the above, only allowing a limited number of versions on the
> network at a single time might be a good idea. Otherwise we run into the
> dilemma where hidden service operators need to maintain and distribute
> multiple addresses, and users need to understand what version their Tor
> client supports (and potentially their friend's as as well, if they want
> to share a HS link).
>
> As s7r said, Bitcoin addresses are single user/single use [1], whereas
> HS addresses are multiple user/multiple use. Because of this difference
> in purpose/use, I would argue we'll need to consider circumstances such
> as version incompatibility, upgrade path, longevity, etc more strongly
> for HS addresses than for Bitcoin addresses.
>
> The idea of supporting multiple versions in a HS address was discussed
> earlier- is this still a viable scheme, or did the cons eventually
> outweigh the pros for this?
>

Hey Chelsea,

while writing the proposal, I felt like supporting multiple versions in
the version field would be more trouble than worth it. I also dislike
the fact that multiple addresses could then represent the same public
key.

Also, I doubt we will ever reach the point where we have multiple HS
versions existing simultaneously in our network that can also share the
same onion address.  This time, we will have two versions, the legacy
onion services and prop224; and their addresses are definitely incompatible.

Worst case, if the need for this becomes apparent at some point, we can
abuse the integer valued version field to encode such information (hey,
we have 256 values after all). Or perhaps this is the best case since it
means that the hidden service protocol has evolved a lot...

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [RFC] Proposal for the encoding of prop224 onion addresses

[George Kadianakis]

s7r <s...@sky-ip.org> writes:

> Hello George,
>
> George Kadianakis wrote:
>> Hello list,
>> 
>> we've had discussions over the past years about how to encode prop224 onion
>> addresses. Here is the latest thread: 
>> https://lists.torproject.org/pipermail/tor-dev/2016-December/011734.html
>> 
>> Bikeshedding is over; it's time to finally pick a scheme! My suggested scheme
>>
>> 
>> 
>
> The version field is useful and allows room for much stuff that we might
> need to do. I think it would be better to place it at the end of the
> address. I don't think all addresses should start with the same prefix
> tbh - this will make them slightly less distinguishable (as much as
> possible users should be able to differentiate onion addresses, which
> are re-usable for long term, as opposite to Bitcoin where the
> recommended way is to use 1 address 1 time, different one every time and
> the users just need to see a string that looks and reads like a Bitcoin
> address and just make sure it's copied (scanned) from/to the right place).
>

OK thanks for the useful discussion. I identified at least three feedback 
points:

+ Screw base58 it's not gonna work. We stick to base32. Usability will
  be "restored" with a proper name system.

+ Move version byte to the end of the address to avoid constant
  prefix. Moving version byte to the middle as teor suggested would
  cause forward-compatibility issues.

+ My checksum calculations were wrong. Checksum is strong! 2 bytes are enough.

And given the above, here is the new microproposal:

  onion_address = base32(pubkey || checksum || version)
  checksum = SHA3(".onion checksum" || pubkey || version)

  where:
   pubkey is 32 bytes ed25519 pubkey
   version is one byte (default value for prop224: '\x03')
   checksum hash is truncated to two bytes

  Here are a few example addresses (with broken checksum):

   l5satjgud6gucryazcyvyvhuxhr74u6ygigiuyixe3a6ysis67ororad.onion
   btojiu7nu5y5iwut64eufevogqdw4wmqzugnoluw232r4t3ecsfv37ad.onion
   vckjr6bpchiahzhmtzslnl477hdfvwhzw7dmymz3s5lp64mwf6wfeqad.onion
  
  Checksum strength: The checksum has a false negative rate of 1/65536.

  Address handling: Clients handling onion addresses first parse the
  version field, then extract pubkey, then verify checksum.

Let me know how you feel about this one. If people like it I will
transcribe it to prop224.

Thanks again Ivan, Ian, Linda, teor, s7r, Chelsea :)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [RFC] Proposal for the encoding of prop224 onion addresses

[George Kadianakis]

George Kadianakis <desnac...@riseup.net> writes:

> Hello list,
>
> 
>
> [D3] Do we like base32???
>
>   In this proposal I suggest we keep the base32 encoding since we've been
>   using it for a while; but this is the perfect time to switch if we feel
>   the need to.
>
>   For example, Bitcoin is using base58 which is much more compact than
>   base32, and also has much better UX properties than base64:
>  https://en.bitcoin.it/wiki/Base58Check_encoding#Background
>
>   If we wanted to get a more compact encoding, we could adopt base58 or
>   make our own adaptation of it. In this proposal I'm using base32 for
>   everything, but I could be persuaded that now is the time to use a 
> better
>   encoding.
>

Oops, pressed "Send" a bit too quickly as always...

Just to give you a better idea here, I did some calculations about the
compactness of base58.

It seems that if we use Bitcoin's base58 we will be able to encode a
37-byte address (32 byte pubkey, one version byte and 4 bytes of
checksum) into 51 base58 characters, instead of 60 base32 characters.

Comparison:

(base32):
 tc2dty3zowj6oyhbyb5n3a2h3luztlx22hy2cwdvn37omsv7quy7rxiysn3a.onion
 tbdczrndtadzdhb6iyemnxf7f4i6x7yojnunarlrvt2virtmrecmwgx5golq.onion

(base58):
 tkb8klf9zgwqnogidda76mzpl6tszzy36hwxmsssznydyxyb9kf.onion
 touecgu8rmjxexxipud5bdku4mkfqezyd4dz1jvhtvqvbtlvytj.onion

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] [RFC] Proposal for the encoding of prop224 onion addresses

[George Kadianakis]

Hello list,

we've had discussions over the past years about how to encode prop224 onion
addresses. Here is the latest thread: 
https://lists.torproject.org/pipermail/tor-dev/2016-December/011734.html

Bikeshedding is over; it's time to finally pick a scheme! My suggested scheme
basically follows from the discussion on that thread, and is heavily based on
the Bitcoin address format:
https://en.bitcoin.it/wiki/Base58Check_encoding

https://en.bitcoin.it/wiki/Technical_background_of_version_1_Bitcoin_addresses

Here is the suggested scheme:

  onion_address = base32(version + pubkey + checksum)
  checksum = SHA3(".onion checksum" + version + pubkey)

  where:
   pubkey is 32 bytes (ed25519)
   version is one byte
   checksum is _truncated_ to two bytes

  With the above construction onion_address ends up being 56 bytes long
  (excluding the ".onion"):

  tbi5tdxbosiotphawjyu7f5pw5tlnvbvfjrj7meskbsnwr2bqbu2t4gg.onion
  tcrdnadkefvbdm3u56kz6lfh6v5lr24fpog5vzsy4n3djr2ymueu34ws.onion
  tcdw7lwmtp5pbwj2w7wf6amxdhmc62qitj2teu376r5s2fqke4r3uiq6.onion

If people like the above suggestion, I will take the effort to engrave it in
prop224.

Here is the discussion section. Please provide feedback!

[D1] How to use version field:

  The version field is one byte long. If we use it as an integer we can
  encode 256 values in it; if we use it as a bitmap we could encode
  properties and such.

  My suggestion is to simply use it as an integer like Bitcoin does. So we
  can assign value \x01 to normal onion services, and in the future we can
  assign more version tags if we need to. For example, we can give a
  different version field to onion services in the testnet. We can also
  reserve a range of values for application-specific purposes.

[D1.1] Default version value:

  The next question is what version value to assign to normal onion
  services. In the above scheme where:

 onion_address = base32(version + pubkey + checksum)

  the value of 'version' basically determines the first two characters of
  the onion address. In Bitcoin, they've made it such that the default
  version value basically prefixes addresses with "1"; so all normal
  Bitcoin addresses start with 1 as in 14tDWDT9zqDufWZmiLqoaT9qJyHi7RRZPE

  What should we do in Tor? My suggestion is to use '\x98' as the default
  version value which prefixes all addresses with 't' (as in Tor).  Check
  the examples I cited above.

  An alternative is to turn the scheme to:
onion_address = base32(pubkey + checksum + version)
  where the version byte is at the end with no effect at usability.

  A heavier alternative would be to have two bytes of version so that we
  can just prefix them all with 'tor'...

[D2] Checksum strength:

  In the suggested scheme we use a hash-based checksum of two bytes (16 
bits).
  This means that in case of an address typo, we have 1/65536 probability
  to not detect the error (false negative). It also means that after 256
  typos we will have 50% probability to miss an error (happy birthday!).

  I feel like the above numbers are pretty good given the small checksum 
size.

  The alternative would be to make the checksum four bytes (like in
  Bitcoin).  This would _greatly_ increase the strength of our checksum but
  it would also increase our address length by 4 base32 characters (and
  also force us to remove leading padding from base32 output). This is how
  these 60-character addresses look like:

 tc2dty3zowj6oyhbyb5n3a2h3luztlx22hy2cwdvn37omsv7quy7rxiysn3a.onion
 tbdczrndtadzdhb6iyemnxf7f4i6x7yojnunarlrvt2virtmrecmwgx5golq.onion
 tc6pcgyorusw3jj5tosxakmcwfmcend2q4g2qnbjtkhuuh4dcgvs4rl4rdaa.onion

  You probably don't notice the size difference compared to the
  56-character addresses, which perhaps is an argument for adopting a four
  byte checksum. Let me know what you think about this.

[D3] Do we like base32???

  In this proposal I suggest we keep the base32 encoding since we've been
  using it for a while; but this is the perfect time to switch if we feel
  the need to.

  For example, Bitcoin is using base58 which is much more compact than
  base32, and also has much better UX properties than base64:
 https://en.bitcoin.it/wiki/Base58Check_encoding#Background

  If we wanted to get a more compact encoding, we could adopt base58 or
  make our own adaptation of it. In this proposal I'm using base32 for
  everything, but I could be persuaded that now is the time to use a better
  encoding.

Let me know what you think!

Thanks :)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 274: A Name System API for Tor Onion Services

[George Kadianakis]

grarpamp  writes:

> Always wondered how naming is relevant,
> for example, IPv6 with OnionCat as a deterministic
> form of naming. So now we propose a 'naming' layer.
> Which should not also support IPv6 addressing?
> Is not IPv6, subsequent to the 80bit scheme,
> merely a name on top of onions? ie: If we develop
> a 'naming' layer, can we not develop an IPv6 layer?

What do you mean by "develop an IPv6 layer"?

A naming layer would provide human-memorable names; IPv6 addresses are
not supposed to be human memorable.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Remaining 4 bits in prop224 addresses

[George Kadianakis]

Jesse V  writes:

> On 12/06/2016 11:27 AM, David Goulet wrote:
>> We had little discussion but some of us agree for sure on having bits for the
>> version number. That will tell a tor client to fetch the right descriptor
>> instead of trying all version that have the same type of public key (.onion
>> address). We currently have I believe 4 bit left which is only 16 values so 
>> we
>> could extend to one more byte here so have more room.
>
> I'm curious if we ever ran into this issue with the current HS protocol.
> What type of changes would warrant a new address that that could not be
> solved with a patch to the tor binary? We also need to consider the
> difficulty of distributing a one-character-different address against the
> difficulty of transitioning the network to the new descriptors. People
> get very entrenched to their onion address, bookmark them, and some even
> issue SSL certs for them.
>
> Let's say we added another character, so that we have 9 bits free. Would
> would be the consequence of using all 9 bits for a checksum? We could
> solve the version/descriptor issue using a naming system and simply
> point the name to a newer onion address. It's something to consider.
>
>> Second thing that is possible, like you stated above, is a checksum.
>> Unfortunately, I haven't thought much about this nor know the "state of the
>> art of small-checksum" but definitely something to dig through! Jessie, if 
>> you
>> feel like it, I welcome any analysis you can do on checksum here and some
>> proposal about it. (Only if you want to :).
>
> I'm not fluent in the arts of small checksums, but it seems to me that
> we do have some benefit of using the first N bits of SHA2(version +
> edDSA_address) as the checksum. I may not have time to write a full
> proposal, but even with a small number of bits we do have a decent
> chance of catching typos, which is the whole point. Obviously, this
> chance will get better as you add more bytes, but prop224 addresses are
> already fairly long and we should weigh the usability impact against the
> probability of typos.
>

Hello people and happy new year :)

I think at this point the best way forward would be for someone to take
initiative and write a Tor proposal on how onion addresses should be
encoded/represented. This way we will have something concrete that we
can discuss and work with.

Anyone interested? If not, I will get to it in a few months.

peace
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] prop224: What should we do with torrc options?

[George Kadianakis]

David Goulet  writes:

> On 06 Dec (10:09:57), teor wrote:
>> 
>> > On 6 Dec. 2016, at 09:56, David Goulet  wrote:
>> > 
>> > 
>> >
>> 
>> Here's a suggested strategy:
>> * at load time, validate the HS options as if v2 is the default, and
>>   validate them as if v3 is the default, and fail if either validation
>>   fails.
>> * then, act on the HS options as if v2 is the default, and also act as
>>   if v3 is the default, and fail if either action fails.
>>   (We need to do this because we don't discover some option issues
>>   until runtime, such as whether the directory can be created.)
>> * then, when each consensus is downloaded, publish whichever descriptor
>>   is the default in the consensus (if the HS config does not specify
>>   a specific version).
>
> This is a reasonable way to proceed considering we use a consensus param to
> know which version of default HS to create. I see this as more of an
> engineering problem that can be solved.
>
> Which what I would like us to decide on if we think that consensus param
> controlling the default version is a good idea or not. If we think yes, we can
> pull it off, if not everything is simpler :).
>
> So just to be clear, I'm behind you on the concern of making sure we validate
> the options on launch instead of failing at consensus download. There are ways
> we can address that like you outlined above.
>

Hello,

I agree that this is more of an engineering problem that can be
solved. I also like the agility that the consensus parameter gives us.

That said when we get close to implementing this feature, IMO we should
estimate how long it will take us to implement this consensus parameter
logic, and evaluate whether it's worth doing. This feature might be an
evening of work, but it could also be two annoying weeks of testing and
debugging crazy networking and consensus fetching edge cases, so we
should make sure we don't sign up for too much craziness.

The alternative would be that we don't do the consensus parameter, and
instead we just change the version parameter in alpha/stable releases
when we feel it's ready. This will be a slower and more gradual
deployment which is not necessarily a terrible thing when we are talking
about huge features like prop224.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Remaining 4 bits in prop224 addresses

[George Kadianakis]

Jesse V  writes:

> Hello all,
>
> I've been closely following the other Proposal 224 threads regarding the
> next-generation of onion services. I'm glad to see that we have a
> timeline and plan for migrating the network. One unresolved point is
> what to do with the remaining 4 bits in the longer addresses. Section
> 1.2 in the 224 document states "Note that since master keys are 32 bytes
> long, and 52 bytes of base 32 encoding can hold 260 bits of information,
> we have four unused bits in each of these names." It seems a waste for
> these to be zeroed out. The four bits could also be used to hold
> client-side flags, but I'm not aware of any applicable client settings
> that could be used here. I suggest that we use them as a checksum.
> (wasn't this proposed in Arlington?)
>
> Since speed isn't a priority, aside from Adler-32 or some CRC function,
> we could also hash the 32-byte key and use the first four bits of the
> hash.  I think a checksum is best because it helps ensure data integrity
> when the address is shared online, copy/pasted, or physically written
> down. Bitcoin addresses contain a checksum as well for exactly this
> reason. They use a combination of SHA-256 and RIPEMD-160 to compute the
> checksum component.
> Source:
> 1)
> https://en.bitcoin.it/wiki/Technical_background_of_version_1_Bitcoin_addresses
> 2) https://bitcoin.stackexchange.com/questions/32353/
>
> What do we think about a checksum, or do we have other plans here? I ask
> because once we nail down the address format, I can add support for 224
> into my Onion Name System.
>

Thanks for bringing up this topic Jesse.

I'd be interested in both a version field and a checksum to be part of
the encoding of the onion address. I also don't mind extending the
encoding by a character or two if that will make it more useful (there
is little difference between 54 and 56 characters).

WRT version field, should it be a single value/bitmap or should it be
able to denote support for multiple versions?

WRT checksum, how much of the address can we protect using a checksum of
1 or 2 bytes? My error-correcting-codes fu is a bit rusty, and I'm not
sure how many errors we can detect/correct. Anyone can do some digging
here and prepare a table, so that we can take an informed decision?
Perhaps the bitcoin community has already done this for us.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] sketch: An alternative prop224 authentication mechanism based on curve25519

[George Kadianakis]

George Kadianakis <desnac...@riseup.net> writes:

> Nick Mathewson <ni...@torproject.org> writes:
>
>> [ text/plain ]
>> Hi!  I thought I'd write this up while it was fresh in my mind.  It
>> could be used as an alternative method to the current proposed client
>> authentication mechanism.  We could implement both, or just this, or
>> just the other.
>>
>> My description here will be a bit terser than we'd want in a proper
>> proposal, but I wanted to share it.
>>
>> This design is based on George Kadianakis's client authentication
>> design; it won't make sense unless you've read it.
>>
>
> OK people,
>
> I have a more mature torspec branch now for your eyes and only.  Please
> see branch `prop224_client_auth_4` in my torspec repo:
>
> https://gitweb.torproject.org/user/asn/torspec.git/log/?h=prop224_client_auth_4
>
> The changes are based on the feedback and discussion on this thread.
>
> The only real changes from `prop224_client_auth_3` is that it increases
> the max descriptor size to 50k, and it removes the username/password
> intro-level authorization.
>
> Please let me know of anything that seems off, or anything that can make
> the proposal more readable. Otherwise, we should merge this upstream and
> move forward with fixing the already merged prop224 HSDir code.
>
> Thanks!

Hello,

I merged the above patch to torspec.git.

Thanks for the feedback and helpful comments everyone!



___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [PATCH] torspec: Keep proposal-status.txt up to date

[George Kadianakis]

s7r <s...@sky-ip.org> writes:

> George Kadianakis wrote:
>> Hello,
>> 
>> I was in a train yesterday and decided to do some torspec
>> housekeeping. So I updated the proposals/proposal-status.txt file which
>> includes a summary of every proposal.
>> 
>> You can find my changes here:
>> 
>> https://gitweb.torproject.org/user/asn/torspec.git/commit/?h=proposal-status-update=45dd81c42f8d7f451947759d49f0e60f86ec9172
>> 
>> Please let me know if you see anything wrong with it. Otherwise, I will
>> merge this to torspec soon since it's probably better than the status quo.
>> 
>> Of course, there are still things we can do to make proposal-status.txt
>> better, and also I bet some of the old proposals have changed status
>> since they were written. I updated a few of them, but I didn't do all of
>> them.
>> 
>> Cheers!
>
> Lgtm.
>
> Only I don't see a status for:
> 272 Listed routers should be Valid, Running, and treated as such
>
> (if this is intentional, pls ignore rest looks nice and clean)

Thanks for that. Addressed in a followup commit.


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] [PATCH] torspec: Keep proposal-status.txt up to date

[George Kadianakis]

Hello,

I was in a train yesterday and decided to do some torspec
housekeeping. So I updated the proposals/proposal-status.txt file which
includes a summary of every proposal.

You can find my changes here:

https://gitweb.torproject.org/user/asn/torspec.git/commit/?h=proposal-status-update=45dd81c42f8d7f451947759d49f0e60f86ec9172

Please let me know if you see anything wrong with it. Otherwise, I will
merge this to torspec soon since it's probably better than the status quo.

Of course, there are still things we can do to make proposal-status.txt
better, and also I bet some of the old proposals have changed status
since they were written. I updated a few of them, but I didn't do all of
them.

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Scheduling future Tor proposal reading groups

[George Kadianakis]

Hello people,

in the beginning of 2016 we started organizing little-t-tor proposal
reading groups in IRC, where we would discuss the current status of Tor
proposals and coordinate on how to move them forward. You can see a list
of previous such meetings here:
 
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/MeetingSchedule#Proposalreviewmeetings

Unfortunately at some point, 2016 started showing its true self, and we kind of
stopped doing those meetings in March. But they were useful, and we should
probably start doing them again!

I went through the mailing list and found a few interesting subjects that could
benefit from group discussion. Here are some ideas:

- Post-Quantum key exchanges for Tor 

  There are a few proposals falling under this topic that were developed in the
  past months. Here are a few:

https://gitweb.torproject.org/torspec.git/tree/proposals/263-ntru-for-pq-handshake.txt

https://gitweb.torproject.org/torspec.git/tree/proposals/269-hybrid-handshake.txt

https://gitweb.torproject.org/torspec.git/tree/proposals/270-newhope-hybrid-handshake.txt
https://lists.torproject.org/pipermail/tor-dev/2016-October/011553.html

  I'm far from an expert on this field, so I'm not sure about the current
  status of this project and the right direction to approach it. However, it
  seems that there have been enough developments here lately that a group
  discussion might be useful.

- A name system API for Tor

  This is a proposal suggesting a single API that allows us to integrate secure
  name systems with Tor hidden services:
https://lists.torproject.org/pipermail/tor-dev/2016-October/011514.html

  The proposal received useful feedback in and out of the mailing list. It
  seems that implementing the proposal as part of a Tor controller might be an
  easier way to test it. Some discussion on future directions might be helpful
  here, as this is something that will be needed sooner than later.

- New topics in Next Gen Hidden Services

  We've done multiple IRC meetings on prop224, but it keeps on growing as it's
  being developed. Here are a few topics that might be worth discussing as a 
group:
  - Control port API for hidden services 
(https://trac.torproject.org/projects/tor/ticket/20699)
  - torrc UX for hidden services 
(https://lists.torproject.org/pipermail/tor-dev/2016-November/011661.html)
  - torrc/control UX for hidden service client auth 
(https://lists.torproject.org/pipermail/tor-dev/2016-November/011617.html)
  - UX for offline keys (https://trac.torproject.org/projects/tor/ticket/18098)
  - UX of hidden services on Tor Browser

- Prop271: Another algorithm for guard selection

  We've also done a few IRC meetings on future guard algorithms, but we are now
  closer to completing that project than ever. After we have a few results and
  statistics of the new guard algorithm, it might be worth scheduling a meeting
  to discuss how well it works and ways to improve it.

- And here are some more misc projects that might be worth discussing further:
  - The Tor browser sandbox that Yawning is developing and UX implications?
  - prop273: Exit relay pinning for web services ?


Please let me know if you are excited talking about any of these topics (feel
free to +1 in private email if you feel it's too spammy), or if there are any
topics I forgot to mention.

If we get enough participation we can schedule the first such meeting for 
December.

Thanks!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] sketch: An alternative prop224 authentication mechanism based on curve25519

[George Kadianakis]

Nick Mathewson  writes:

> [ text/plain ]
> Hi!  I thought I'd write this up while it was fresh in my mind.  It
> could be used as an alternative method to the current proposed client
> authentication mechanism.  We could implement both, or just this, or
> just the other.
>
> My description here will be a bit terser than we'd want in a proper
> proposal, but I wanted to share it.
>
> This design is based on George Kadianakis's client authentication
> design; it won't make sense unless you've read it.
>

OK people,

I have a more mature torspec branch now for your eyes and only.  Please
see branch `prop224_client_auth_4` in my torspec repo:
   
https://gitweb.torproject.org/user/asn/torspec.git/log/?h=prop224_client_auth_4

The changes are based on the feedback and discussion on this thread.

The only real changes from `prop224_client_auth_3` is that it increases
the max descriptor size to 50k, and it removes the username/password
intro-level authorization.

Please let me know of anything that seems off, or anything that can make
the proposal more readable. Otherwise, we should merge this upstream and
move forward with fixing the already merged prop224 HSDir code.

Thanks!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] sketch: An alternative prop224 authentication mechanism based on curve25519

[George Kadianakis]

David Goulet <dgou...@ev0ke.net> writes:

> [ text/plain ]
> On 15 Nov (16:29:33), George Kadianakis wrote:
>> Nick Mathewson <ni...@torproject.org> writes:
>> 
>> 
>> 
>> Hello,
>> 
>> I worked some more on prop224 client authorization. I have a draft
>> torspec patch at prop224_client_auth_3 in my repo:
>> 
>> https://gitweb.torproject.org/user/asn/torspec.git/commit/?h=prop224_client_auth_3
>
> I personally like this very much. Apart from minor syntax issues and spacing,
> I think this scheme is getting to a point where it's simple and efficient.
>
> I think the introduce auth section (INTRO-AUTH) needs to be edited more though
> because we are still mentionning password and ed25519 authentication but I
> understand that waiting until we all agree on the client auth mechanism is
> wise.
>
>> 
>> I ended up using the x25519 scheme described above by Nick.
>> 
>> I also ended up dodging the UX questions raised on this thread, by only
>> specifying the Tor protocol level details, and leaving the out-of-band
>> HS<->client protocol mostly unspecified. I believe that this out-of-band
>> protocol and configuration details can be figured out in the future, and
>> we should not block on them right now.
>
> Yes, I believe this is fine. Note that tor-keygen tool is getting another
> _very_ important use case here that is the key generation on client side.
>
>> 
>> I also added some more high-level details on client auth in the intro
>> section, which should be useful to people who read the proposal for the
>> first time.
>> 
>> In the appendix, I added a section with some rough ideas on how the
>> torrc configuration could work, and how a control port interface could
>> work. This is just a sketch so far, and we can get more specific as we
>> get closer to implementation fot this feature.
>
> Looks good to me. The naming will _most_ probably change as we already have
> those names in torrc with a different API but the format to me is simple and
> easy to use.
>
> And the control port commands are _very_ important as it would allow TBB to
> easily add client auth with some UI instead of hot hacking torrc file. And
> this would be a giant step foward in usability of client auth.
>
>> 
>> Finally, I've been assuming AES128 for the STREAM() cipher so that the
>> size computations performed earlier on this thread are still
>> accurate. If we want to pump it to AES256, the IVs and the descriptor
>> cookie ciphertexts will double in size for each authorized client. If we
>> actually want to go with AES256, we need to peform new computations
>> about the padding and max descriptor size. If you people like the rest
>> of the patch here, I can do the calculations again. 
>
> Which might also change our maximum size of 40k we put there.
>
> One note. I haven't seen a maximum value of client we allow. Did I miss it or
> it's also maybe too early to put there as we are still discussing the
> specifics. Which makes me think that maybe that 40k limit is also a bit
> arbitrary for now based on that.
>

Hello,

I understand your concern, and hence I performed the descriptor size
calculations again based on the encoding format of the latest branch
(prop224_client_auth_3). You asked for it!

First of all, in the end of this email I inline a real prop224 HS descriptor
based on the prop224_client_auth_3 torspec branch, and _without any encryption_.
Use that descriptor as a guidance, or to double-check my calculations.

To cut to the chase, here are my current recommendations regarding descriptor
padding and maximum size:

- The superencrypted blob (between "superencrypted\n-BEGIN MESSAGE-"
  and "-END MESSAGE-") should be padded with NULs to the nearest
  multiple of 10k bytes before it gets encrypted and base64ed. This is the same
  as what is specified in prop224_client_auth3.

- Max total descriptor size accepted by HSDirs is 50k bytes. We should also
  control this with a consensus parameter, in case it ever ends up being too
  small or too big. In prop224_client_auth_3 it is suggested that the max size
  should be 40k bytes, but actually I think 50k bytes is a better number given
  the new scheme and the 10k padding. Please see below.

- We always add fake "auth-client" entries to reach the nearest multiple of 16.

Now here are some observations:

a) The superencrypted blob gets padded to nearest multiple of 10k bytes before
   it gets base64ed. If you base64-encode 30k bytes you get back 40k base64
   bytes, which is basically the max size of superencrypted blob we can support
   if the max descriptor size is 50k bytes. The remaining 10k bytes are left

Re: [tor-dev] sketch: An alternative prop224 authentication mechanism based on curve25519

[George Kadianakis]

Nick Mathewson  writes:

> [ text/plain ]
> Hi!  I thought I'd write this up while it was fresh in my mind.  It
> could be used as an alternative method to the current proposed client
> authentication mechanism.  We could implement both, or just this, or
> just the other.
>
> My description here will be a bit terser than we'd want in a proper
> proposal, but I wanted to share it.
>
> This design is based on George Kadianakis's client authentication
> design; it won't make sense unless you've read it.
>
> =
>
> Let every client generate a curve25519 keypair, and tell the hidden
> service operator about the public key.  This keypair takes the place
> of the long-term shared secret.  For some client C, denote the secret
> key as x_X and the public key as X_C.
>
> For every descriptor, the hidden service generates a fresh keypair  Y>, and includes Y in the the outer encrypted layer.
>
> Now, for each client, the hidden service computes curve25519(X_C, y)
> and uses this as the input for two KDF functions. Call these outputs
> K1_C and K2_C.  The hidden service generates an auth-client line for
> each client as follows:
>
>"auth-client" SP client-id SP encrypted-cookie
>
> This is the same as in George's proposal, except that client-id is
> derived from a truncated version of K1_C, and the encrypted-cookie
> portion is encrypted based on K2_C.
>
>
> When the client receives the descriptor, it decrypts the outer layer,
> then sees the value of Y that the hidden server advertised.  It
> computes curve25519(Y, x_c), and derives K1_C and K2_C.  It uses K1_C
> to find the appropriate entry on the list, and then uses K2_C to
> decrypt it and find the descriptor cookie.
>
> =
>
> Advantages:
>   * managing public keys can be easier than managing shared secrets.
>   * The encoding is slightly shorter, since no IV is needed, since K2
> is different every time.
>   * probably others?
>
> Disadvantages:
>   * Curve25519 costs more computationally than non-public-key operations
>   * probably others?
>
>

Hello,

I worked some more on prop224 client authorization. I have a draft
torspec patch at prop224_client_auth_3 in my repo:

https://gitweb.torproject.org/user/asn/torspec.git/commit/?h=prop224_client_auth_3

I ended up using the x25519 scheme described above by Nick.

I also ended up dodging the UX questions raised on this thread, by only
specifying the Tor protocol level details, and leaving the out-of-band
HS<->client protocol mostly unspecified. I believe that this out-of-band
protocol and configuration details can be figured out in the future, and
we should not block on them right now.

I also added some more high-level details on client auth in the intro
section, which should be useful to people who read the proposal for the
first time.

In the appendix, I added a section with some rough ideas on how the
torrc configuration could work, and how a control port interface could
work. This is just a sketch so far, and we can get more specific as we
get closer to implementation fot this feature.

Finally, I've been assuming AES128 for the STREAM() cipher so that the
size computations performed earlier on this thread are still
accurate. If we want to pump it to AES256, the IVs and the descriptor
cookie ciphertexts will double in size for each authorized client. If we
actually want to go with AES256, we need to peform new computations
about the padding and max descriptor size. If you people like the rest
of the patch here, I can do the calculations again. 

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] sketch: An alternative prop224 authentication mechanism based on curve25519

[George Kadianakis]

teor <teor2...@gmail.com> writes:

> [ text/plain ]
>
>> On 11 Nov. 2016, at 04:18, George Kadianakis <desnac...@riseup.net> wrote:
>> 
>> George Kadianakis <desnac...@riseup.net> writes:
>> 
>>> [ text/plain ]
>>> Nick Mathewson <ni...@torproject.org> writes:
>>> 
>>>> [ text/plain ]
>>>> Hi!  I thought I'd write this up while it was fresh in my mind.  It
>>>> could be used as an alternative method to the current proposed client
>>>> authentication mechanism.  We could implement both, or just this, or
>>>> just the other.
>>>> 
>>>> My description here will be a bit terser than we'd want in a proper
>>>> proposal, but I wanted to share it.
>>>> 
>>>> This design is based on George Kadianakis's client authentication
>>>> design; it won't make sense unless you've read it.
>>>> 
>>>> =
>>>> 
>>>> Let every client generate a curve25519 keypair, and tell the hidden
>>>> service operator about the public key.  This keypair takes the place
>>>> of the long-term shared secret.  For some client C, denote the secret
>>>> key as x_X and the public key as X_C.
>>>> 
>>>> For every descriptor, the hidden service generates a fresh keypair <y,
>>>> Y>, and includes Y in the the outer encrypted layer.
>>>> 
>>>> Now, for each client, the hidden service computes curve25519(X_C, y)
>>>> and uses this as the input for two KDF functions. Call these outputs
>>>> K1_C and K2_C.  The hidden service generates an auth-client line for
>>>> each client as follows:
>>>> 
>>>>   "auth-client" SP client-id SP encrypted-cookie
>>>> 
>>>> This is the same as in George's proposal, except that client-id is
>>>> derived from a truncated version of K1_C, and the encrypted-cookie
>>>> portion is encrypted based on K2_C.
>>>> 
>>>> 
>>>> When the client receives the descriptor, it decrypts the outer layer,
>>>> then sees the value of Y that the hidden server advertised.  It
>>>> computes curve25519(Y, x_c), and derives K1_C and K2_C.  It uses K1_C
>>>> to find the appropriate entry on the list, and then uses K2_C to
>>>> decrypt it and find the descriptor cookie.
>>>> 
>>>> =
>>>> 
>>>> Advantages:
>>>>  * managing public keys can be easier than managing shared secrets.
>>>>  * The encoding is slightly shorter, since no IV is needed, since K2
>>>> is different every time.
>>>>  * probably others?
>>>> 
>>>> Disadvantages:
>>>>  * Curve25519 costs more computationally than non-public-key operations
>>>>  * probably others?
>>>> 
>>>> 
>>> 
>>> Hey Nick,
>>> 
>>> thanks for the feedback on the torspec patch and this scheme. I'll wait
>>> to figure out if we want to use this scheme before updating the torspec
>>> patch further.
>>> 
>>> This proposed scheme seems like a good idea. It basically switches the
>>> identity of authed clients from symmetric keys to x25519 keys, which
>>> seems more intuitive and correct.
>>> 
>>> I wonder if clients could/should use the same keypair for intro-layer
>>> authentication as well. Section 3.4.2 currently specifies how to do
>>> intro-layer auth using ed25519 keys, so perhaps we can use a single
>>> curve25519 key (and switch it between ed25519 and x25519) to perform
>>> both descriptor-layer and intro-layer authentication?
>>> 
>>> WRT the suggested scheme, I think my biggest concern here is the UX, and
>>> specifically this phrase
>>> 
>>>  "Let every client generate a curve25519 keypair, and tell the hidden
>>>  service operator about the public key".
>>> 
>>> The client auth UX currently involves the HS operator passing
>>> credentials to the clients, not the other way around. This is good for
>>> UX since it means that all configuration is performed by the HS operator
>>> (who can be assumed technical), instead of its clients (who should be
>>> assumed to be non-technical). All clients need to do is copy-paste the
>>> auth line to their torrc. I feel like this UX should be maintained,
>>> except if there is a very strong reason to break it.
>>> 
>> 
>> OK, 

Re: [tor-dev] sketch: An alternative prop224 authentication mechanism based on curve25519

[George Kadianakis]

George Kadianakis <desnac...@riseup.net> writes:

> [ text/plain ]
> Nick Mathewson <ni...@torproject.org> writes:
>
>> [ text/plain ]
>> Hi!  I thought I'd write this up while it was fresh in my mind.  It
>> could be used as an alternative method to the current proposed client
>> authentication mechanism.  We could implement both, or just this, or
>> just the other.
>>
>> My description here will be a bit terser than we'd want in a proper
>> proposal, but I wanted to share it.
>>
>> This design is based on George Kadianakis's client authentication
>> design; it won't make sense unless you've read it.
>>
>> =
>>
>> Let every client generate a curve25519 keypair, and tell the hidden
>> service operator about the public key.  This keypair takes the place
>> of the long-term shared secret.  For some client C, denote the secret
>> key as x_X and the public key as X_C.
>>
>> For every descriptor, the hidden service generates a fresh keypair <y,
>> Y>, and includes Y in the the outer encrypted layer.
>>
>> Now, for each client, the hidden service computes curve25519(X_C, y)
>> and uses this as the input for two KDF functions. Call these outputs
>> K1_C and K2_C.  The hidden service generates an auth-client line for
>> each client as follows:
>>
>>"auth-client" SP client-id SP encrypted-cookie
>>
>> This is the same as in George's proposal, except that client-id is
>> derived from a truncated version of K1_C, and the encrypted-cookie
>> portion is encrypted based on K2_C.
>>
>>
>> When the client receives the descriptor, it decrypts the outer layer,
>> then sees the value of Y that the hidden server advertised.  It
>> computes curve25519(Y, x_c), and derives K1_C and K2_C.  It uses K1_C
>> to find the appropriate entry on the list, and then uses K2_C to
>> decrypt it and find the descriptor cookie.
>>
>> =
>>
>> Advantages:
>>   * managing public keys can be easier than managing shared secrets.
>>   * The encoding is slightly shorter, since no IV is needed, since K2
>> is different every time.
>>   * probably others?
>>
>> Disadvantages:
>>   * Curve25519 costs more computationally than non-public-key operations
>>   * probably others?
>>
>>
>
> Hey Nick,
>
> thanks for the feedback on the torspec patch and this scheme. I'll wait
> to figure out if we want to use this scheme before updating the torspec
> patch further.
>
> This proposed scheme seems like a good idea. It basically switches the
> identity of authed clients from symmetric keys to x25519 keys, which
> seems more intuitive and correct.
>   
> I wonder if clients could/should use the same keypair for intro-layer
> authentication as well. Section 3.4.2 currently specifies how to do
> intro-layer auth using ed25519 keys, so perhaps we can use a single
> curve25519 key (and switch it between ed25519 and x25519) to perform
> both descriptor-layer and intro-layer authentication?
>
> WRT the suggested scheme, I think my biggest concern here is the UX, and
> specifically this phrase
>
>   "Let every client generate a curve25519 keypair, and tell the hidden
>   service operator about the public key".
>
> The client auth UX currently involves the HS operator passing
> credentials to the clients, not the other way around. This is good for
> UX since it means that all configuration is performed by the HS operator
> (who can be assumed technical), instead of its clients (who should be
> assumed to be non-technical). All clients need to do is copy-paste the
> auth line to their torrc. I feel like this UX should be maintained,
> except if there is a very strong reason to break it.
>

OK, I think sooner or later we needed to have a UX discussion about HS
client authorization!

The current rend-spec.txt specifies the torrc options for client auth in
sections 2.3/2.4. We don't do this in prop224 yet, but I think it's
useful because figuring out the torrc format now will help us understand
if the scheme we are making is useful.

Here is a rough proposed UX for prop224 client auth, using Nick's idea
from the top post of this thread. FWIW, I've used the same torrc option
naming as we currently have but we should probably introduce new ones.

-- Hidden Service configuration

A hidden service uses the following torrc line to specify authed clients:

 HiddenServiceAuthorizeClient  ,,...

If this option is configured, Tor generates client authorization data
and creates a new "hs_client_auth.txt" file which contains lines with the
following forma

Re: [tor-dev] sketch: An alternative prop224 authentication mechanism based on curve25519

[George Kadianakis]

Nick Mathewson  writes:

> [ text/plain ]
> Hi!  I thought I'd write this up while it was fresh in my mind.  It
> could be used as an alternative method to the current proposed client
> authentication mechanism.  We could implement both, or just this, or
> just the other.
>
> My description here will be a bit terser than we'd want in a proper
> proposal, but I wanted to share it.
>
> This design is based on George Kadianakis's client authentication
> design; it won't make sense unless you've read it.
>
> =
>
> Let every client generate a curve25519 keypair, and tell the hidden
> service operator about the public key.  This keypair takes the place
> of the long-term shared secret.  For some client C, denote the secret
> key as x_X and the public key as X_C.
>
> For every descriptor, the hidden service generates a fresh keypair  Y>, and includes Y in the the outer encrypted layer.
>
> Now, for each client, the hidden service computes curve25519(X_C, y)
> and uses this as the input for two KDF functions. Call these outputs
> K1_C and K2_C.  The hidden service generates an auth-client line for
> each client as follows:
>
>"auth-client" SP client-id SP encrypted-cookie
>
> This is the same as in George's proposal, except that client-id is
> derived from a truncated version of K1_C, and the encrypted-cookie
> portion is encrypted based on K2_C.
>
>
> When the client receives the descriptor, it decrypts the outer layer,
> then sees the value of Y that the hidden server advertised.  It
> computes curve25519(Y, x_c), and derives K1_C and K2_C.  It uses K1_C
> to find the appropriate entry on the list, and then uses K2_C to
> decrypt it and find the descriptor cookie.
>
> =
>
> Advantages:
>   * managing public keys can be easier than managing shared secrets.
>   * The encoding is slightly shorter, since no IV is needed, since K2
> is different every time.
>   * probably others?
>
> Disadvantages:
>   * Curve25519 costs more computationally than non-public-key operations
>   * probably others?
>
>

Hey Nick,

thanks for the feedback on the torspec patch and this scheme. I'll wait
to figure out if we want to use this scheme before updating the torspec
patch further.

This proposed scheme seems like a good idea. It basically switches the
identity of authed clients from symmetric keys to x25519 keys, which
seems more intuitive and correct.
  
I wonder if clients could/should use the same keypair for intro-layer
authentication as well. Section 3.4.2 currently specifies how to do
intro-layer auth using ed25519 keys, so perhaps we can use a single
curve25519 key (and switch it between ed25519 and x25519) to perform
both descriptor-layer and intro-layer authentication?

WRT the suggested scheme, I think my biggest concern here is the UX, and
specifically this phrase

  "Let every client generate a curve25519 keypair, and tell the hidden
  service operator about the public key".

The client auth UX currently involves the HS operator passing
credentials to the clients, not the other way around. This is good for
UX since it means that all configuration is performed by the HS operator
(who can be assumed technical), instead of its clients (who should be
assumed to be non-technical). All clients need to do is copy-paste the
auth line to their torrc. I feel like this UX should be maintained,
except if there is a very strong reason to break it.

I think your proposal could be slightly tweaked to fit the above UX, if
the HS generates the client's private keys, and then passes them to the
clients. The rest works the same. Perhaps we can also give the option
for clients to generate the keys themselves if they so want.

Having the HS generate its client's private keys, kind of beats the
purpose of having assymetric crypto keys, since we basically treat them
as shared secrets. However, I don't think this changes the threat model
in any significant way, as the hidden service can be seen a trusted
party in some sense. Some more thinking needs to be done here.

The way I see it, by using this proposed scheme, we get a nicer
cryptographic scheme (symmetric keys kind of suck) and a more compact
auth-client format, for the price of a bit more engineering (generating
ephemeral keypairs and encoding them in descriptors). I think this might
be worth it.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Revisiting prop224 client authorization

[George Kadianakis]

David Goulet <dgou...@ev0ke.net> writes:

> [ text/plain ]
> On 17 Oct (13:35:24), George Kadianakis wrote:
>> George Kadianakis <desnac...@riseup.net> writes:
>> 
>> > [ text/plain ]
>> > Hello,
>> >
>> > we've reached the point in prop224 development where we need to pin down
>> > the precise cell formats, so that we can start implementing them. HS
>> > client authorization has been one of those areas that are not yet
>> > finalized and are still influencing cell format.
>> >
>> > Here are some topics based on special's old notes, plus some further
>> > recent discussion with David and Yawning.
>> >
>> 
>> Hello again,
>> 
>> I read the feedback on the thread and thought some more about this. Here
>> are some thoughts based on received feedback. A torspec branch coming
>> soon if people agree with my points below.
>> 
>> I'd also like to introduce a new topic of discussion here:
>>
>> 
>> 
>> > a) I think the most important problem here is that the authorization-key 
>> > logic
>> >in the current prop224 is very suboptimal. Specifically, prop224 uses a
>> >global authorization-key to ensure that descriptors are only read by
>> >authorized clients. However, since that key is global, if we ever want 
>> > to
>> >revoke a single client we need to change the keys for all clients. The
>> >current rend-spec.txt does not suffer from this issue, hence I adapted 
>> > the
>> >current technique to prop224.
>> >
>> >Please see my torspec branch `prop224_client_auth` for the proposed 
>> > changes:
>> >
>> > https://gitweb.torproject.org/user/asn/torspec.git/log/?h=prop224_client_auth
>> >
>> >Some further questions here:
>> >
>> >i) Should we fake the client-auth-desc-key blob in case client 
>> > authorization
>> >   is not enabled? Otherwise, we leak to the HSDir whether client auth 
>> > is
>> >   enabled. The drawback here is the desc size increase (by about 330 
>> > bytes).
>> >
>> >   Alternatively, we can try to put it in the encrypted part of the
>> >   descriptor. So that we require subcredential knowledge to access the
>> >   encrypted part, and then client_auth_cookie knowledge to get the
>> >   encryption key to decrypt the intro points etc. I feel that this
>> >   double-encryption design might be too annoying to implement, but 
>> > perhaps
>> >   it's worth it?
>> >
>> 
>> Seems like people preferred the double-encryption idea here, so that we
>> reveal the least amount of information possible in the plaintext part of
>> the desc.
>> 
>> I think this is a reasonable point since if we put the auth keys in the
>> plaintext part of the descriptor, and we always pad (or fake clients) up
>> to N authorized clients, it will be obvious to an HSDir if a hidden
>> service has more than N authorized clients (since we will need to fake
>> 2*N clients then).
>> 

Hello,

I worked some more on the descriptor part of client authorization and
prepared a torspec patch. You can find it at `prop224_client_auth_2` in
my repo:
   
https://gitweb.torproject.org/user/asn/torspec.git/commit/?h=prop224_client_auth_2=a85ffa341cc6c493ad031972f466a6dd5d8510e2

Based on received feedback, I went with the double-layer encryption
style, where the first layer is encrypted using the HS pubkey and the
second layer is encrypted using the descriptor cookie. Inside the first
layer plaintext is the information for authorized clients to derive the
descriptor cookie.

I also included the padding suggestions from my previous post that
should help with hiding the number of client auths.

I'd like feedback on the following:

a) Do you like the descriptor format and logic? Can we make it nicer or
   easier to implement? 

b) Do you like the proposal format? Is it messy and/or hard to
   understand? Ideas on how can it be improved?

   I think we have reached the point where every subsystem of prop224 is
   complex enough to warrant its own proposal, but I'm resisting the
   urge to dig into this rabbithole right now.

c) Is the descriptor cookie encryption format good enough Namely:
 encrypted_cookie = STREAM(iv, client_auth_cookie) XOR descriptor_cookie

d) Current changes: I changed "authentication-required" to
   "intro-auth-required" in the descriptor, to make it more clear its
   about introduction-layer authentication.

Feedback on the patch and the above points is very much welcome!

BTW, I'm not done with this thread yet, there are still some more points
that need to be handled wrt client authorization. But this spec patch is
the most important and lengthiest of them all, so let's get it out of
the way first.

Thanks!

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Revisiting prop224 client authorization

[George Kadianakis]

George Kadianakis <desnac...@riseup.net> writes:

> [ text/plain ]
> Hello,
>
> we've reached the point in prop224 development where we need to pin down
> the precise cell formats, so that we can start implementing them. HS
> client authorization has been one of those areas that are not yet
> finalized and are still influencing cell format.
>
> Here are some topics based on special's old notes, plus some further
> recent discussion with David and Yawning.
>

Hello again,

I read the feedback on the thread and thought some more about this. Here
are some thoughts based on received feedback. A torspec branch coming
soon if people agree with my points below.

I'd also like to introduce a new topic of discussion here:

d) Should we introduce the concept of stealth auth again?

   IIUC the current prop224 client auth solutions are not providing all
   the security properties that stealth auth did. Specifically, if Alice
   is an ex-authorized-client of a hidden service and she got revoked,
   she can still fetch the descriptor of a hidden service and hence
   learn the uptime/presense of the HS. IIUC, with stealth auth this was
   not previously possible.

> a) I think the most important problem here is that the authorization-key logic
>in the current prop224 is very suboptimal. Specifically, prop224 uses a
>global authorization-key to ensure that descriptors are only read by
>authorized clients. However, since that key is global, if we ever want to
>revoke a single client we need to change the keys for all clients. The
>current rend-spec.txt does not suffer from this issue, hence I adapted the
>current technique to prop224.
>
>Please see my torspec branch `prop224_client_auth` for the proposed 
> changes:
>
> https://gitweb.torproject.org/user/asn/torspec.git/log/?h=prop224_client_auth
>
>Some further questions here:
>
>i) Should we fake the client-auth-desc-key blob in case client 
> authorization
>   is not enabled? Otherwise, we leak to the HSDir whether client auth is
>   enabled. The drawback here is the desc size increase (by about 330 
> bytes).
>
>   Alternatively, we can try to put it in the encrypted part of the
>   descriptor. So that we require subcredential knowledge to access the
>   encrypted part, and then client_auth_cookie knowledge to get the
>   encryption key to decrypt the intro points etc. I feel that this
>   double-encryption design might be too annoying to implement, but perhaps
>   it's worth it?
>

Seems like people preferred the double-encryption idea here, so that we
reveal the least amount of information possible in the plaintext part of
the desc.

I think this is a reasonable point since if we put the auth keys in the
plaintext part of the descriptor, and we always pad (or fake clients) up
to N authorized clients, it will be obvious to an HSDir if a hidden
service has more than N authorized clients (since we will need to fake
2*N clients then).

---

WRT protocol, I guess the idea here is that if client auth is enabled,
then we add some client authorization fields in the top of the encrypted
section of the descriptor, that can be used to find the client-auth
descriptor encryption key. Then we add another client-auth-encrypted
blob inside the encrypted part, which contains the intro points etc. and
is encrypted using the descriptor encryption key found above.

So the first layer is encrypted using the onion address, and the second
layer is encrypted using the client auth descriptor key. This won't be
too hard to implement, but it's also different from what's currently
coded in #17238.

Do people feel OK with this?

Also, what should happen if client auth is not used? Should we fall back
to the current descriptor format, or should we fake authorized clients
and add a fake client-auth-encrypted-blob for uniformity? Feedback is
welcome here, and I think the main issue here is engineering time and
reuse of the current code.

---

Now WRT security, even if we do the double-encryption thing, and we
consider an HSDir adversary that knows the onion address but is not an
authorized client,we still need to add fake clients, otherwise that
adversary will know the exact number of authorized clients. So fake
clients will probably need to be introduced anyhow.

As David pointed out, this all boils down to how much we pad the
encrypted part of the descriptor, otherwise we always leak info. If we
are hoping for a leakless strategy here, we should be generous with our
padding.

Let's see how much padding we need:

- Each intro point adds about 1.1k bytes to the descriptor (according to
  david).

- Each block of 16 authorized clients adds about 1k bytes to the
  descriptor (according to the format described below).

- Apart from intro points and authorized clients, the rest of the
  descriptor is not t

[tor-dev] Revisiting prop224 client authorization

[George Kadianakis]

Hello,

we've reached the point in prop224 development where we need to pin down
the precise cell formats, so that we can start implementing them. HS
client authorization has been one of those areas that are not yet
finalized and are still influencing cell format.

Here are some topics based on special's old notes, plus some further
recent discussion with David and Yawning.

a) I think the most important problem here is that the authorization-key logic
   in the current prop224 is very suboptimal. Specifically, prop224 uses a
   global authorization-key to ensure that descriptors are only read by
   authorized clients. However, since that key is global, if we ever want to
   revoke a single client we need to change the keys for all clients. The
   current rend-spec.txt does not suffer from this issue, hence I adapted the
   current technique to prop224.

   Please see my torspec branch `prop224_client_auth` for the proposed changes:
   https://gitweb.torproject.org/user/asn/torspec.git/log/?h=prop224_client_auth

   Some further questions here:

   i) Should we fake the client-auth-desc-key blob in case client authorization
  is not enabled? Otherwise, we leak to the HSDir whether client auth is
  enabled. The drawback here is the desc size increase (by about 330 bytes).

  Alternatively, we can try to put it in the encrypted part of the
  descriptor. So that we require subcredential knowledge to access the
  encrypted part, and then client_auth_cookie knowledge to get the
  encryption key to decrypt the intro points etc. I feel that this
  double-encryption design might be too annoying to implement, but perhaps
  it's worth it?

   ii) Should we use the descriptor ASCII format to encode all the
   client-auth-desc-key data? Or is that weird binary format OK?

   iii) Should we use a fresh IV for each ENCRYPTED_DESC_COOKIE? rend-spec.txt
does not do that, and IIUC that's OK because it uses a fresh key for
every encryption (even though the plaintext and IV is the same).

b) Do we want a NACK from the HS for when client authorization fails? Currently
   the only way for a client to learn that they are not authorized (or that
   their keys changed or got revoked) is that they never complete the 
rendezvous.

   To achieve that we would need a whole new cell (INTRODUCE_SERVICE_NACK) from
   the hidden service all the way to the client. Worth it?

c) Another suggestion here by special, is to introduce an additional
   layer of access control at the HSDir request level, such that HSDirs
   don't even serve descriptors to clients that do not prove knowledge
   of a pre-shared fetch key.

   This way unauthorized clients cannot even learn presense information of
   hidden services. This might be quite useful for applications like Ricochet,
   who want to hide their presense from revoked clients. 

   Of course this assumes that the HSDir is honest and will honor the fetch key
   protocol. However, even if the HSDir is dishonest we are just back to the
   current security level.

   We started sketching out a solution in the bottom of these notes:
 https://people.torproject.org/~asn/hs_notes/client_auth.jpg
   but the solution is not trivial to implement and we are not sure whether
   it's worth complicating the protocol further (e.g. we need to design a way
   for apps like Ricochet to get access to the fetch key).

d) It might be worthwhile padding the encrypted part of INTRODUCE1 to obscure
   whether client authorization is in place.

As you can see I have mainly worked on point (a) which I consider the
most urgent. I welcome feedback on all points, so that we move forward
with the design here.

Thanks :)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Proposal 274: A Name System API for Tor Onion Services

[George Kadianakis]

Hello list,

we've been busy discussing secure name systems for the past few months,
and how we could integrate them with Tor.

We had a few people ask us for precise interfaces, so here is an attempt
for a modular name system API. It's largely based on the PT API, which
has been pretty successful.

The proposal was brainstormed on a Sunday morning in Seattle between
Yawning, David and me. I wrote the actual proposal contents so any
innacuracies, impossibilities and errors are mine :)

If people like it and are interested in implementing it, we can draft
out an implementation plan for any person who wants to go for it. But
for now, please consider this proposal text as a draft that is meant to
be improved from this list's feedback. 

Have fun and looking forward to your comments (and code).

---


Filename: 274-naming-layer-api.txt
Title: A Name System API for Tor Onion Services
Author: George Kadianakis, Yawning Angel, David Goulet
Created: 04-Oct-2016
Status: Draft

  Table Of Contents:

1. Introduction
1.1. Motivation
1.2. Design overview and rationale
2. System Specification
2.1. System overview [SYSTEMOVERVIEW]
2.2. System Illustration
2.3. System configuration [TORRC]
2.3.1. Tor name resolution logic
2.4. Name system initialization [INITPROTOCOL]
2.5. Name resolution using NS API
2.5.1. Message format
2.5.2. RESOLVED status codes
2.5.3. Further name resolution behavior
2.6. Cancelling a name resolution request
2.7. Launching name plugins [INITENVVARS]
2.8. Name plugin workflow [NSBEHAVIOR]
2.8.1. Name plugin shutdown [NSSHUTDOWN]
2.9. Further details of stdin/stdout 
communication
2.9.1. Message Format
3. Discussion
3.1. Using second-level domains instead of tld
3.2. Name plugins handling all tlds '*'
3.3. Deployment strategy
3.4. Miscellaneous discussion topics
4. Acknowledgements
A.1: Example communication Tor <-> name plugin 
[PROTOEXAMPLE]
A.2: Example plugins [PLUGINEXAMPLES]

1. Introduction

   This proposal specifies a modular API for integrating name systems with Tor.

1.1. Motivation

   Tor onion service addresses are decentralized and self-authenticated but
   they are not human-memorable (e.g. 3g2upl4pq6kufc4m.onion). This is a source
   of poor usability, since Internet users are familiar with the convenient
   naming of DNS and are not used to addresses being random text.

   In particular, onion addresses are currently composed of 16 random base32
   characters, and they look like this:

  3g2upl4pq6kufc4m.onion
  vwakviie2ienjx7t.onion
  idnxcnkne4qt76tg.onion
  vwakviie2ienjx6t.onion

   When Proposal 224 get deployed, onion addresses will become even
   bigger: 53 base32 characters. That's:

llamanymityx4fi3l6x2gyzmtmgxjyqyorj9qsb5r543izcwymle.onion
lfels7g3rbceenuuqmpsz45z3lswakqf56n5i3bvqhc22d5rrsza.onion
odmmeotgcfx65l5hn6ejkaruvai222vs7o7tmtllszqk5xbysola.onion
qw3yvgovok3dsarhqephpu2pkiwzjdk2fgdfwwf3tb3vgzxr5kba.onion

   Over the years Tor users have come up with various ad-hoc ways of handling
   onion addresses. For example people memorize them, or use third-party
   centralized directories, or just google them everytime.

   We believe that the UX problem of non-human-memorable problems is not
   actually solved with the above ad-hoc solutions and remains a critical
   usability barrier that prevents onion services from being used by a wider
   audience.

1.2. Design overview and rationale

   During the past years there has been lots of research on secure naming and
   various such systems have been proposed (e.g. GNS, Namecoin, etc.).

   Turns out securely naming things is a very hard research problem, and hence
   none of the proposed systems is a clear winner: all of them carry various
   trade-offs. Furthermore, none of the proposed systems has seen widespread use
   so far, which makes it even harder to pick a single project.

   Given the plenitude of options, one approach to decide which system
   is best is to make various decent name systems available and let the
   Tor community and the sands o

Re: [tor-dev] Joining Tor Project's software infrastructure

[George Kadianakis]

Paul Syverson <paul.syver...@nrl.navy.mil> writes:

> [ text/plain ]
> On Tue, Oct 04, 2016 at 12:54:55PM -0700, George Kadianakis wrote:
>> 
>> BTW, monitor this list, there will soon be a proposal specifying a Tor
>> API for naming systems like OnioNS, similar to the one we have for
>> pluggable transports.
>> 
>
> There will? Was this discussed at tordev? How did I miss this? Who's
> working on it?  Is there a draft anywhere?
>
> (Feel free to respond to tor-dev. I just didn't want to spam the list
> if good general answers to these were not quite ready.)
>

Hey Paul :)

It was discussed on Sunday, the last day of the hackfest, so that's why
you missed it :(

Here are some spoilers:
   https://people.torproject.org/~asn/hs_notes/name_transport_plugin.jpg

I hope to work on the proposal ASAP.

[CCing tor-dev just for completeness]
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Joining Tor Project's software infrastructure

[George Kadianakis]

Jesse V  writes:

> [ text/plain ]
> Hey everyone,
>
> Although it hasn't been obvious, I actually have been making some
> serious progress on my Onion Name System (OnioNS) project. The paper was
> accepted into PoPETS, the design is finally stable, and the software is
> looking really promising, although not yet ready for use. I've focused
> on improving the maturity of the software and set up the following:
>
> * PGP-signed commits using my torproject identity
> * Continuous integration via Travis CI
> * Use clang-analyzer and cppcheck static analysis scanners
> * Automated enforcement of code styling via clang-format
> * Inclusion of a manpage and command-line help menu
> * Packaging for Debian (Wheezy+) on my own machine
> * Automated building for Ubuntu (14.04+) on Launchpad
> * Protection against memory leaks and buffer overflows
> * Software is split to minimize installation footprint
> * Use of CMake, which should help the port to Windows
> * No RPATH hacks
> * Support for release and debug builds
> * Support for Clang and GCC; all warnings enabled for debug
> * No reliance on OpenSSL (CVE-2016-6309, are you serious?!)
> * Use of well-maintained libraries for networking and parsing
> * Minimal dependencies: Botan, JsonCpp, libcurl, libmicrohttpd
> * Non-native dependencies included as git submodules
>
> At this point, I am interested in taking the next step and try to
> integrate with the Tor Project infrastructure. I am looking for
> information on the following:
>
> 1) How do I upload packages to deb.torproject.org?
> 2) How do I join Gitian to generate reproducible builds?
> 3) How do I set up an Onion Name System component in Trac so that I can
> migrate bug tracking from Github to Trac?
> 4) How do I set up a repository on gitweb.torproject.org? I'm more
> comfortable using Github, but there's no need to centralize git.
>
> I have a LDAP account, which should help with some of these. I wasn't
> able to find much useful documentation on these topics, so I would
> appreciate some help. Also, if anyone has any experience with RPM
> packaging, I would like to figure that out. I haven't had too much luck
> with Alien.
>
> As I approach a proper release, my overall goal is to improve the
> maturity, trust, usability of the OnioNS software. Please let me know
> how I can accomplish the above tasks.
>

Hello Jesse,

glad to hear you are still working on the OnioNS project, and happy to
hear that the paper got accepted in PoPETS. We have great plans for
hidden service naming layers, and it's great to see more people working
on this topic. Thanks for updating the wiki as well.

FWIW, I feel that using Github (or any other third-party git) for code
hosting and ticket tracking is fine for now. If you still want a
personal repository on gitweb, check out these docs:
https://help.torproject.org/tsa/

I'm also not sure what's the exact policy on gitian and
deb.torproject.org, and whether third-party applications (like OnioNS)
can be placed there.

Personally, I'm hesitant of giving the impression that Tor officially
supports any of the proposed naming systems so early. I'd feel more
confident about them after they get deployed and used by the wider
community.

On this topic, what's your deployment strategy? If you really really
need Tor project infrastructure to achieve it, let's talk about it.

BTW, monitor this list, there will soon be a proposal specifying a Tor
API for naming systems like OnioNS, similar to the one we have for
pluggable transports.

Cheers!

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] How to query HS hostname from control port

[George Kadianakis]

meejah  writes:

> [ text/plain ]
> Jesse V  writes:
>
>> TL;DR: Please let me know how to fetch the hostname of my hidden service
>> from Tor's control port.
>
> There are two types of onion services: "on disk" ones configured via
> torrc/SETCONF and the HiddenServiceDir and related options **or**
> "ephemeral" onions, created via the recently added ADD_ONION command.
>
> In the latter case, you can list them with GETINFOs on "onions/current"
> or "onions/detached" depending on the options when you created them.
>

I guess a side question here is why those GETINFO commands only return the
ephemeral onion services and not all of them.

We should probably provide a common interface for getting the hostname of onion
services, regardless of whether they are ephemeral or permanent. Also, by
making this a control port command, Tor libraries can fetch it without doing
weird filesystem IO.

Anyone who wants to open a ticket here, or has a counter argument? :)

> For the "HiddenServiceDir" onions, the only way to get the onion address
> that I know of is by reading the "hostname" file.
>
> Can you describe what you're trying to do?
>
> -- 
> meejah
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [tor-talk] Onion Service introduction point selection

[George Kadianakis]

Muhammad Hammad Mazhar  writes:

> [ text/plain ]
> I am looking into Tor onion services as part of my research, and am 
> interested in how an Onion Service selects its introduction points, in 
> particular where in source code does it do so. I am aware of rendservice.c in 
> the source code, but any pointers are welcome.
>
> Please move this mail to the requisite list if required.
>

You were very close!

Please see function rendservice.c:rend_consider_services_intro_points() and
related code.

[Moved to [tor-dev] from [tor-talk])
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] prop224: Ditching key blinding for shorter onion addresses

[George Kadianakis]

Jeremy Rand <jeremyr...@airmail.cc> writes:

> [ text/plain ]
> George Kadianakis:
>> Lunar <lu...@torproject.org> writes:
>> 
>>> [ text/plain ]
>>> George Kadianakis:
>>>> this is an experimental mail meant to address legitimate usability concerns
>>>> with the size of onion addresses after proposal 224 gets implemented. It's
>>>> meant for discussion and it's far from a full blown proposal.
>>>
>>> Taking a step back here, I believe the size of the address to be a
>>> really minor usability problem. IPv6 adressses are 128 bits long, and
>>> plenty of people in this world now access content via IPv6. It's not a
>>> usability problem because they use a naming—as opposed to
>>> addressing—scheme to learn about the appropriate IPv6 address.
>>>
>> 
>> That's true. Naming systems are indeed the way to go wrt UX. The future sucks
>> if our users are supposed to use 24 (or 56) random characters as addresses.
>> 
>> That said, the current IPv6 naming scheme (DNS) is far from perfect as
>> well. Tor would never use it (or any other system with similar threat model).
>> 
>> Furthermore, all the _secure naming systems_ that have been suggested have
>> their own tradeoffs. They are either centralized, or they use blockchains, or
>> they require money, or they require a whole network/community to exist, or 
>> they
>> have annoying name-squatting issues, or they require a non-anonymous
>> registration, or they save HS history on disk, or their protocol is three 
>> times
>> more complicated than Tor itself, or ...
>>   
>> So it's not like we have the perfect solution on the naming scheme right now.
>> We likely need plenty of trial experimentation before we decide on one (or
>> multiple) naming cheme becoming the official.
>> 
>> We really need to start serious work in this area ASAP! Maybe let's start by
>> making a wiki page that lists the various potential solutions (GNS, Namecoin,
>> Blockstack, OnioNS, etc.)?
>
> I'd be happy to provide feedback on the Namecoin section of such a wiki
> page.
>

Hello people interested in this topic,

I made a wiki page for Naming Systems  here:
  https://trac.torproject.org/projects/tor/wiki/doc/OnionServiceNamingSystems

Feel free to start adding information and links and make it look nicer.

Let's try to build a good knowledge base that will help us take informed
decisions. Please try to maintain some sort of consistent structure through the
document.

(In the unlikely case where the doc gets out of hand, I will try to find some
time to curate it.)

Thanks! :)


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] prop224: Ditching key blinding for shorter onion addresses

[George Kadianakis]

Lunar <lu...@torproject.org> writes:

> [ text/plain ]
> George Kadianakis:
>> this is an experimental mail meant to address legitimate usability concerns
>> with the size of onion addresses after proposal 224 gets implemented. It's
>> meant for discussion and it's far from a full blown proposal.
>
> Taking a step back here, I believe the size of the address to be a
> really minor usability problem. IPv6 adressses are 128 bits long, and
> plenty of people in this world now access content via IPv6. It's not a
> usability problem because they use a naming—as opposed to
> addressing—scheme to learn about the appropriate IPv6 address.
>

That's true. Naming systems are indeed the way to go wrt UX. The future sucks
if our users are supposed to use 24 (or 56) random characters as addresses.

That said, the current IPv6 naming scheme (DNS) is far from perfect as
well. Tor would never use it (or any other system with similar threat model).

Furthermore, all the _secure naming systems_ that have been suggested have
their own tradeoffs. They are either centralized, or they use blockchains, or
they require money, or they require a whole network/community to exist, or they
have annoying name-squatting issues, or they require a non-anonymous
registration, or they save HS history on disk, or their protocol is three times
more complicated than Tor itself, or ...
  
So it's not like we have the perfect solution on the naming scheme right now.
We likely need plenty of trial experimentation before we decide on one (or
multiple) naming cheme becoming the official.

We really need to start serious work in this area ASAP! Maybe let's start by
making a wiki page that lists the various potential solutions (GNS, Namecoin,
Blockstack, OnioNS, etc.)?

> While I do think we should think of nicer representation for the new
> addresses than base32, and we should adress that, working on a naming
> system sounds like an easier way out to improve onion services
> usability than asking people to remember random addresses (be them 16 or
> 52 characters-long).
>

Agreed that base32 is not very good as far as encoding goes.

Regardless of the naming system that we need to deploy, we should also
acknowledge that we need a good address encoding scheme anyway, as not all HSes
will use the naming system. This is also the case with IP addresses: many
people mess with IP addresses everyday (for config files, etc.)

I recently read this paper from USENIX which is a UX study on fingerprint
encodings:

https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/dechand
We might be able to learn somethign from it. For example, they found out that
alphanumeric encodings (like base32) make for the worst experience when
compared to other encodings (pure numeric, unrelated words, sentences, etc.).

Finally, if we acknowledge that at least some people will have to mess with the
raw addresses themselves, another benefit of the addresses being small is that
they are easier to verify by hand (verifying a 24 characters fingerprint, is
easier than verifying a 54 characters fingerprint).


> (I now plenty of people who type “riseup” in the Google search bar of
> their browser to access their mailbox… They don't even want to/can't remember
> an URL. Hardly a chance they will remember an onion address, whatever
> its size.)
>

Indeed. We need a good naming scheme if we ever hope for onion services to
become widespread.

> Maybe it would be worthwhile to ask the UX team for input on the topic?
>

I think that would be a good idea indeed.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Alternative Implementations of Tor

[George Kadianakis]

Alexander Færøy  writes:

> [ text/plain ]
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> Hello.
>
> Over the past year I've been hacking, on and off, on an implementation
> of Tor in the Erlang programming language. The project started out after
> I met Linus Nordberg at the Erlang User Conference in the summer of 2015
> - -- a couple of weeks before the CCC Camp in Germany. Linus and I
> discussed what it would take to get a very basic implementation of a Tor
> relay up and running in Erlang.
>
> The project has been named Talla, which is still open for change --
> especially if it collides with an already established project within the
> wider Tor development community. I was unable to find any name clashes
> myself, but I may very well have overlooked something.
>
> The parts of Talla that is implemented in "pure" Erlang is going to be
> licensed under a two-clause BSD license. Talla also contains a "module"
> which is using Tor's ed25519 ref10 implementation (Thanks to Yawning for
> a great amount of help there) -- that module is licensed under the same
> license as Tor itself.
>
> I'm now at the point where things are slowly starting to take form. The
> important pieces of the code have so far been kept available only to a
> tiny group of close friends that I trust not to share the code until I,
> and possibly other people, considers that the code is stable enough for
> the wilderness of the wider internet.
>
> There is, to my knowledge, currently only one implementation of Tor that
> is actively in use on the production network, which is the C
> implementation. I'm aware of a Haskell implementation made by Galois,
> which to me mostly seemed like it was designed to be building blocks for
> writing more specialized clients and doing research with the Tor
> network. Last time I looked, the Haskell implementation's main function
> was doing a DNS lookup through a circuit within the Tor network and then
> quitting. I was also told there had been an implementation in Go that
> have had activity on the production network, but that project was
> abandoned by its maintainer.
>
> In general I'm a bit uncertain about the "best practices" of dealing with a
> third party Tor implementation, which Talla is.
>
> I'm writing this email to receive suggestions, comments, and possibly
> creative ideas about the following:
>
> 1. What is the general criteria set from the Tor project's perspective
>on when it is acceptable to make alternative Tor implementations
>available to the general public?
>
>I'm currently testing Talla using Chutney with a mixture of NTOR and
>pre-NTOR Tor daemons running (inspired by one of the configuration
>files in the Chutney repository, which referred to a 'tor-old'
>binary).
>
>My current plan is to stabilize Talla further until my gut feeling is
>that I can try to announce a single, middle, relay to the production
>Tor network. This relay will, of course, have a platform-string set
>to something easily identifiable like "Talla 0.0.1 (...)" and the
>contact-string set to a valid method of reaching out to me with, in
>its announced server descriptor. I will closely monitor that things
>are going as I expect and probably turn it off shortly after the
>test, when I have seen that my code isn't too "crashy" -- this will
>most likely be repeated a number of times until I'm satisfied with
>the results.
>
>Could I do more to ensure that the people caring for the network as a
>whole wont fear me pressing the start-button here?
>
> 2. I will not do any classical releases (as in packagable .tar.gz) until
>I'm past the point where my gut feelings are telling me that my code is
>reasonably stable for the production network of Tor.
>
>I will, in a very visible location, request that no distribution
>developers makes any packages of the code until there is a release.
>
>I think this is already the norm, but I guess being explicit won't
>hurt.
>
> 3. I will write, also in a visible location, a warning that the code is
>not production ready and that people should probably stick to running a
>Tor relay using the official Tor daemon and point to the installation
>instructions on torproject.org.
>
> 4. Not have any installation documentation and hope that Erlang is still
>an esoteric enough language to make people pass by without trying :-)
>
> 5. Talla will not have any references to the directory authorities that
>are currently used for the Tor production network. This means that
>anyone who is interested in running Talla will have to explicitly set
>the directory authority servers in Talla's configuration file.
>
>This will allow people who want to toy around with it together with
>Chutney to be easily able to do that.
>
> Why am I asking all these questions now, when I could just wait until
> Talla is ready? In two weeks there will be a smaller 

Re: [tor-dev] Tor and Namecoin

[George Kadianakis]

George Kadianakis <desnac...@riseup.net> writes:

> [ text/plain ]
> Jeremy Rand <jeremyr...@airmail.cc> writes:
>
>> [ text/plain ]
>> Hello Tor devs,
>>
>> Namecoin is interested in collaboration with Tor in relation to
>> human-readable .onion names; I'm reaching out to see how open the Tor
>> community would be to this, and to get feedback on how exactly the
>> integration might work.
>>
>> The new hidden service spec is going to substantially increase the
>> length of .onion names, which presents usability concerns.  Namecoin
>> provides a way to resolve a human-readable .bit name to a .onion name.
>> Another benefit of Namecoin is that it provides a way to lookup TLS
>> fingerprints for clearnet .bit sites, which reduces the risk of MITM
>> attacks on clearnet websites from malicious or compromised CA's.
>>
>> 
>>
>> There are a few options I can think of for integrating this with Tor for
>> .onion naming.  One would be to modify OnioNS to call the Namecoin SPV
>> client.  This would concern me because OnioNS is in C++, which
>> introduces the risk of memory safety vulnerabilities.  Another would be
>> to use an intermediate proxy like Yawning's or-ctl-filter.  A third
>> option would be to try to get external name resolution implemented in
>> Tor itself, which I believe Jeff Burdges has suggested in the past.  If
>> Option A or B is used, any solution would need to pass the stream
>> isolation info to the SPV client.
>>
>
> Hello Jeremy,
>
> I'm a big noob when it comes to blockchains, namecoin, SPV clients and such, 
> so
> I'm mainly going to focus on how to integrate this with Tor.
>
> It seems to me that a plausible way to kickstart this big project would be to
> make an unofficial add-on for TBB that can do the namecoin dance. People can
> then install it and experiment with it. If it fits the Tor use case well, then
> a community might be formed that will push this project forward even more.
>
> If it's an optional add-on, we also don't have to worry that much about the
> 400MB blockchain size, since it's gonna be optional and only people who want 
> it
> will have to download it. This way we can learn how much of a problem the
> download size is anyway (it seems to me like a total blocker for people in
> non-western fast-internet countries).
>
> That's why I would suggest experimenting with the first two approaches you
> mentioned that don't require a modification to Tor's protocol.
>

On this front, please check out Nick's new mail showing how to integrate
external name resolvers into Tor:

https://lists.torproject.org/pipermail/tor-dev/2016-August/011253.html
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Tor and Namecoin

[George Kadianakis]

Jeremy Rand  writes:

> [ text/plain ]
> Hello Tor devs,
>
> Namecoin is interested in collaboration with Tor in relation to
> human-readable .onion names; I'm reaching out to see how open the Tor
> community would be to this, and to get feedback on how exactly the
> integration might work.
>
> The new hidden service spec is going to substantially increase the
> length of .onion names, which presents usability concerns.  Namecoin
> provides a way to resolve a human-readable .bit name to a .onion name.
> Another benefit of Namecoin is that it provides a way to lookup TLS
> fingerprints for clearnet .bit sites, which reduces the risk of MITM
> attacks on clearnet websites from malicious or compromised CA's.
>
> 
>
> There are a few options I can think of for integrating this with Tor for
> .onion naming.  One would be to modify OnioNS to call the Namecoin SPV
> client.  This would concern me because OnioNS is in C++, which
> introduces the risk of memory safety vulnerabilities.  Another would be
> to use an intermediate proxy like Yawning's or-ctl-filter.  A third
> option would be to try to get external name resolution implemented in
> Tor itself, which I believe Jeff Burdges has suggested in the past.  If
> Option A or B is used, any solution would need to pass the stream
> isolation info to the SPV client.
>

Hello Jeremy,

I'm a big noob when it comes to blockchains, namecoin, SPV clients and such, so
I'm mainly going to focus on how to integrate this with Tor.

It seems to me that a plausible way to kickstart this big project would be to
make an unofficial add-on for TBB that can do the namecoin dance. People can
then install it and experiment with it. If it fits the Tor use case well, then
a community might be formed that will push this project forward even more.

If it's an optional add-on, we also don't have to worry that much about the
400MB blockchain size, since it's gonna be optional and only people who want it
will have to download it. This way we can learn how much of a problem the
download size is anyway (it seems to me like a total blocker for people in
non-western fast-internet countries).

That's why I would suggest experimenting with the first two approaches you
mentioned that don't require a modification to Tor's protocol.

Specifically, if you can build a PoC with Yawning's or-ctl-filter that's what I
would go for, since I'm not actually sure what's the current state of the
OnioNS code, and how easy it will be to plug namecoin in there. What would be
the procedure for third-party people with TBB to install namecoin + 
or-ctl-filter?
Would it be a painful UX?

FWIW, I'm also not sure what's the state of Jeff Burdges' name resolution idea,
whether there are any plans on moving forward, and whether it would fit the
Namecoin API.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] prop224: Ditching key blinding for shorter onion addresses

[George Kadianakis]

ban...@openmailbox.org writes:

> [ text/plain ]
> On 2016-07-29 17:26, George Kadianakis wrote:
>> Hello people,
>> 
>> this is an experimental mail meant to address legitimate usability 
>> concerns
>> with the size of onion addresses after proposal 224 gets implemented. 
>> It's
>> meant for discussion and it's far from a full blown proposal.
>> 
>> Anyway, after prop224 gets implemented, we will go from 16-character 
>> onion
>> addresses to 52-character onion addresses. See here for more details:
>> 
>> https://gitweb.torproject.org/torspec.git/tree/proposals/224-rend-spec-ng.txt#n395
>> 
>> This happens because we want the onion address to be a real public key, 
>> and not
>> the truncated hash of a public key as it is now. We want that so that 
>> we can do
>> fun cryptography with that public key. Specifically, we want to do key 
>> blinding
>> as specified here:
>> 
>> https://gitweb.torproject.org/torspec.git/tree/proposals/224-rend-spec-ng.txt#n1692
>> 
>
>
> Speaking out of turn here:
>
> Why not integrate kernelcorn's OnioNS project and keep all the current 
> security properties?
>
> OnioNS addresses are much more user friendly than even the shorter 
> .onion addresses.

Hello bancfc,

AFAIK, the OnioNS project was never actually finished nor deployed.

It also has various engineering/deployment issues that have not been addressed
and it requires a whole infrastructure/community to work.

In general, I'm open to DNS-like approaches for hidden services, but if we can
also improve the UX situation on the protocol layer, that seems like a win to 
me :)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] prop224: Ditching key blinding for shorter onion addresses

[George Kadianakis]

Nick Mathewson <ni...@alum.mit.edu> writes:

> [ text/plain ]
> On Fri, Jul 29, 2016 at 11:26 AM, George Kadianakis
> <desnac...@riseup.net> wrote:
>> So basically in this scheme, HSDirs won't be able to verify the signatures of
>> received descriptors.
>>
>> The obvious question here is, is this a problem?
>
> I'm not sure I fully understand, so here's a couple of quick questions
> before I look more deeply.  (I'm assuming that descriptors are indexed
> by their ephemeral address here.  If that's wrong and they're indexed
> by something other than than ephemeral address, my analysis is wrong.)
>
>
> 1) In your scheme, how does descriptor replacement work?  In the
> current scheme, if my introduction points change, I can upload a new
> descriptor.  In this scheme, it seems that either _anyone_ can upload
> a new descriptor with the same ephemeral address (which is insecure),
> or descriptors cannot be replaced (which is problematic).
>
> 2) Even if descriptors can't be replaced, there's still a problem:
> What stops an attacker from racing the hidden service to try to upload
> their own competing descriptor with the same ephemeral address?
>

Oops, you are absolutely right! :)

Seems like security property (e) from the previous email was much more
important than I thought! The HSDir must indeed be able to validate that a
descriptor was signed using a particular ephemeral key (which then maps to a
particular identity key).

OK, here is another brainstormy attempt at shortening onion addresses
post-prop224, while maintaining the same security properties:

Can we use HSDirs to store encrypted identity keys for HSes, along with the
encrypted descriptors? When clients query the HSDir, they fetch both the
encrypted identity key and the encrypted descriptor, and use the former to
decrypt the latter.

Let me try to make this more formal by stating some definitions from prop224:

A is the identity public key of the HS
A_t is the ephemeral blinded key of the HS for time period t
E(desc) is the encrypted HS descriptor using key H(t||A) exactly as in 
prop224

Now let's introduce some new notions:

- onion_address = H(A || "onion_address") truncated to 128 bits

onion_address is the string that clients need to know to connect to an 
HS.

- query_key = H(onion_address || t || "query_key")

query_key is used to query the HSDir for HS information. Specifically 
for
the encrypted HS identity key and the encrypted descriptor of the HS.

- encryption_key = H(onion_address || t || "encryption_key")

encryption_key is used to encrypt the identity key A of the HS so that 
it's
hidden from entities who don't know the onion_address (like HSDirs).

- identity_key_ciphertext = E_encryption_key(A)

identity_key_ciphertext is the identity key of the HS encrypted using 
encryption_key.

Now for the protocol part:

In current prop224, the HS uploads to the HSDir its ephemeral blinded key
A_t, and its encrypted descriptor E(desc). That remains the same in this
scheme, but the HS also uploads the query_key and identity_key_ciphertext
to the HSDir. The HSDir verifies the descriptor signature as normal, and
uses query_key as the index for all that information.

Now, a client who wants to connect to that HS, uses the onion_address to
derive the query_key, and sends query_key to the HSDir. The HSDir then
checks its index, and returns to the client the encrypted descriptor
E(desc) and also the identity_key_ciphertext.

The client at this point derives encryption_key using the onion_address,
and decrypts the identity_key_ciphertext. After the client does so, the
client knows A and can decrypt and verify the descriptor ciphertext as
normal.

Does this make sense?

The idea is that we use HSDirs for PKI along with descriptor storage. We
basically encrypt the identity key of the HS using the onion_address and store
it in the HSDirs. We never reveal onion_address to the HSDirs.

Please let me know in what way this scheme is completely broken!

(I've received many concerns about the future size of onion addresses, and
hence I feel obligated to hit my head against the wall some more.)

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] prop224: Ditching key blinding for shorter onion addresses

[George Kadianakis]

Hello people,

this is an experimental mail meant to address legitimate usability concerns
with the size of onion addresses after proposal 224 gets implemented. It's
meant for discussion and it's far from a full blown proposal.

Anyway, after prop224 gets implemented, we will go from 16-character onion
addresses to 52-character onion addresses. See here for more details:
  
https://gitweb.torproject.org/torspec.git/tree/proposals/224-rend-spec-ng.txt#n395

This happens because we want the onion address to be a real public key, and not
the truncated hash of a public key as it is now. We want that so that we can do
fun cryptography with that public key. Specifically, we want to do key blinding
as specified here:
  
https://gitweb.torproject.org/torspec.git/tree/proposals/224-rend-spec-ng.txt#n1692

As I understand it the key blinding scheme is trying to achieve the following 
properties:
a) Every HS has a permanent identity onion address
b) Clients use an ephemeral address to fetch descriptors from HSDir
c) Knowing the ephemeral address never reveals the permanent onion address
c) Descriptors are encrypted and can only be read by clients that know the 
identity onion key
d) Descriptors are signed and verifiable by clients who know the identity onion 
key
e) Descriptors are also verifiable in a weaker manner by HSDirs who know the 
ephemeral address

In this email I'm going to sketch a scheme that has all above properties except 
from (e).

The suggested scheme is basically the current HSDir protocol, but with clients
using ephemeral addresses for fetching HS descriptors. Also, we truncate onion
address hashes to something larger than 80bits.

Here is a sketch of the scheme:

--

Hidden service Alice has a long-term public identity key: A
Hidden service Alice has a long-term private identity key: a

The onion address of Alice, as in the current scheme, is a truncated H(A).
So let's say: onion_address = H(A) truncated to 128 bits.

The full public key A is contained in Alice's descriptor as it's currently the 
case.

When Alice wants to publish a descriptor she computes an ephemeral address
based on the current time period 't': ephemeral_address = H(t || onion_address)

Legitimate clients who want to fetch the descriptor also do the same, since
they know both 't' and 'onion_address'.

Descriptors are encrypted using a key derived from the onion_address. Hence,
only clients that know the onion_address can decrypt it.

Descriptors are signed using the long-term private key of the hidden service,
and can be verified by clients who manage to decrypt the descriptor.

---

Assuming the above is correct and makes sense (need more brain), it should
maintain all the security properties above except from (e).

So basically in this scheme, HSDirs won't be able to verify the signatures of
received descriptors.

The obvious question here is, is this a problem?

IIUC, having the HSDirs verify those signatures does not offer any additional
security, except from making sure that the descriptor signature was actually
created using a legitimate ed25519 key. Other than that, I don't see it
offering much.

So, what does this additional HSDir verification offer? It seems like a weak
way to ensure that no garbage is uploaded on the HSDir hash ring. However, any
reasonable attacker will put their garbage in a descriptor and sign it with a
random ed25519 key, and it will trivially pass the HSDir validation.

So do we actually care about this property enough to introduce huge onion
addresses to the system?

Please discuss and poke holes at the above system.

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Implementation plan for prop271 "Another algorithm for guard selection"

[George Kadianakis]

Hello people,

our new guard algorithm proposal (prop271) is getting finalized and we are 
planning to
move to implementation soon. You can find the proposal here:
  
https://gitweb.torproject.org/torspec.git/tree/proposals/271-another-guard-selection.txt

You can also find the STRIKE team's implementation of proposal 259 here:
https://github.com/twstrike/tor_for_patching
Ideally we should borrow as much code as possible from their implementation.

To move forward with engineering, here follows an initial thirteen step
implementation plan for proposal 271.

Where it made sense, I tried to provide links to current functions that can be
used, or to code that the STRIKE team wrote and can be reused or studied.

Let me know what I should make clearer or where I should go further in depth.

---

Design decisions:

An initial design decision here is whether we should make a new file for this
feature or use the old entrynodes.c file? A related question here is whether
both algos should co-exist, or should the new one replace the old? I leave both
decisions to the person who writes the code.

---

Implementation plan:

1) Create main guard selection state data structure (instead of using global 
variables)
** Ideally there should be no global variables, so that in the future we can 
have multiple parallel guardsets (e.g. prop247)
** Thoughtworks: guard_selection_t

2) Sample SAMPLED_GUARDS from GUARDS
** Fill in state for each sampled guard (added_on_date, etc.)
** Save/load SAMPLED_GUARDS to/from state
** Unittests for sampling/saving/loading
** Thoughtworks function: fill_sampled_guards_from_entrynodes()
** Thoughtworks unittest: test_fill_in_sampled_set()

3) Derive FILTERED_GUARDS / USABLE_FILTERED_GUARDS from SAMPLED_GUARDS
** The filtering function can be called at any point (e.g. ON_CONSENSUS)
** Write unittests ensuring correctness of filtering
** See populate_live_entry_guards()
** Thoughtworks function: filter_set() / filter_sampled()

4) Maintain set of PRIMARY_GUARDS
** Functions to extract set of (live) primary guards out of FILTERED_GUARDS
** Unittest
** Thoughtworks function: retry_primary_guards() / next_primary_guard()
** Thoughtworks unittests: test_next_primary_guard()

5) Selecting guards for circuits
** Meant to replace choose_random_entry_impl()
** See section SELECTING in prop271
** Add new circuit states to or_circuit_t
** Implement the guard selection logic
** Unittests on circuit state machine
** Unittests on guard selection logic

6) Update guard state when a circuit fails/succeeds
** Section ON_FAIL and ON_SUCCESS
** See entry_guard_register_connect_status()
** Unittest

7) Maintain CONFIRMED_GUARDS
** Add guard to CONFIRMED_GUARDS when circuit succeeds
** Fill in state info for each confirmed guard (confirmed_on_date, etc.)
** Migration from old state format to new
** Thoughtworks unittest: test_used_guards_parse_state()

8) Retry schedule for guards (section RETRYING)
** See entry_is_time_to_retry()
** Unittest

9) Update list of waiting circuits (section UPDATE_WAITING)
** Implementation of guard "higher priority" ordering (see CONFIRMED)
** Unittest

10) Update state whenever we get a new consensus (section ON_CONSENSUS)
** Update SAMPLED_GUARDS elements (section SAMPLED)
** Remove obsolete/expired guards
** See entry_guards_compute_status() / remove_obsolete_entry_guards()
** Thoughtworks function: entry_guards_update_profiles()

11) Bridge support
** Thoughtworks function: fill_in_from_bidges()

12) Separate state instances when EntryNodes/ExcludeNodes/etc are used
** See prop271 MEANINGFUL_RESTRICTION_FRAC etc.

13) Glue things together!
** Add clean and meaningful logging so that we can *heavily* field test the 
feature in our machines
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [network-team] [doodle poll] Meeting to discuss guard proposal draft status

[George Kadianakis]

Nick Mathewson <ni...@alum.mit.edu> writes:

> [ text/plain ]
> On Tue, Jul 5, 2016 at 11:44 PM, isis agora lovecruft
> <i...@torproject.org> wrote:
>> George Kadianakis transcribed 1.6K bytes:
>>> Hello!
>>>
>>> With Nick we settled on Thursday at 14:00 UTC as the time for this meeting!
>>> The plan is to finish the meeting in strictly less than 90 minutes.
>>
>> Hey George, Hey Everyone,
>>
>> If we do it Thursday at 16:00 UTC (or later), I could attend, but not before,
>> since I will be on a flight.  If it works out better for everyone else to 
>> have
>> it at 14:00 instead, then that's also fine with me and please just let me 
>> know
>> afterwards what I can do to help.
>
> Does anybody object to 1600 UTC on Thursday?

Hm. I can't do 16:00 UTC this Thursday :(





___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [network-team] [doodle poll] Meeting to discuss guard proposal draft status

[George Kadianakis]

Nick Mathewson  writes:

> [ text/plain ]
> Hi!
>
> We should do proposal discussion meetings again.  In particular, we
> should talk about this draft I've been working on for a revised
> version of the twstrike team's revised version of Isis's revised
> version of the guard selection and rotation algorithm.
>
> I've been tracking my work on ticket #19468:
>   https://trac.torproject.org/projects/tor/ticket/19468
>
> There's some commentary happening on gitlab:
>   https://gitlab.com/asn/torspec/merge_requests/2
>
> I expect I'll do another round of revision by Wednesday.
>
> I'm trying to schedule a time for us to get together to review this,
> so that we can divide it up into tickets to implement it, ideally
> based on the twstrike code that we have for the prior revision.
>

Hello!

With Nick we settled on Thursday at 14:00 UTC as the time for this meeting!
The plan is to finish the meeting in strictly less than 90 minutes.

#tor-dev is the place :)

See you there!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Improving hidden services on mobile phones

[George Kadianakis]

Hey list,

just for the record, during the past weeks we've been discussing how to improve
hidden services hosted on mobile phones in this here trac ticket:
   https://trac.torproject.org/projects/tor/ticket/16387
Discussion has also spilled over ticket #18620.

The discussion also has a design aspect as some suggested improvements would
require protocol changes, hence I'm informing this list in case some people are
interested in jumping in and participating.

For example, for some use cases it might make sense to have an HS -> IP cell
INTRODUCE2_ACK that acknowledges the receipt of INTRODUCE2 cells, so that the
IP knows that the intro circuit is active and the HS received the introduction
request before sending out its own INTRODUCE_ACK to the client. That's because
mobile hidden services are unstable and their intro circuits break frequently,
causing reachability issues. See the second part of
https://trac.torproject.org/projects/tor/ticket/16387#comment:8 for more 
background.

Have a good day :)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Revisiting prop224 overlap descriptor logic and descriptor lifetimes

[George Kadianakis]

George Kadianakis <desnac...@riseup.net> writes:

> [ text/plain ]
> Roger Dingledine <a...@mit.edu> writes:
>
>> [ text/plain ]
>> On Mon, Jun 13, 2016 at 03:48:39PM +0300, George Kadianakis wrote:
>>> The main issue for me right now is that I can't recall how this helps with
>>> clock skewed clients, even though that was a big part of our discussion in 
>>> Montreal.
>>> 
>>> Specifically, I think that clients (and HSes) should determine the set of
>>> responsible HSDirs (i.e. the current time period) based on the 
>>> "valid-after" of
>>> their latest consensus, instead of using their local clock. This way, as 
>>> long
>>> as the client's skewed clock is good enough to verify the latest consensus, 
>>> the
>>> client will have a consistent view of the network and SRV (assuming an
>>> honest/updated dirguard). I tried to clarify this a bit in commit 465156d, 
>>> so
>>> please let me know if it's not a good idea.
>>
>> Interesting idea! I think I like it. You're right that in Montreal
>> we were thinking in terms of client clocks, and we might be able to
>> reduce the problem (both in frequency and in magnitude) by considering
>> the time in the last consensus we have.
>>
>> Another argument in favor of using the last consensus is that we will be
>> picking the "relays that are closest to the right location in the hash
>> ring" out of our last consensus already. (That is not a strong argument
>> in favor though, I think, since in theory there won't be so much churn
>> in a day that all of the relays in our last consensus will become wrong.)
>>
>> All of this said, it seems like you are basing your arguments on some
>> expectations about how clients handle consensuses that have surprising
>> dates in them (surprising either because the client's clock is skewed,
>> or because their directory guard gave them the wrong consensus). How
>> *do* clients handle these situations? If we could get the intended /
>> expected behavior written down, then we would have a better chance of
>> identifying bugs in it that we can then fix.
>>
>
> I agree that we should get the intended/expected behavior written down!
>

A few days have passed and I still feel that using the latest consensus
valid_after time is a more robust way for taking decisions on how to perform
the HS protocol, than using the local client clock. After all, the whole Tor
protocol relies on having a good consensus (for HSDirs, SRV, etc.), so you
can't go very far with a bad consensus anyway.

On the subject of clock skewed clients, I opened ticket #19460 with a few
suggestions for improving the handling of consensuses with surprising dates.
In general, I feel that with the #19460 suggestion implemented, the system can
be accomodating towards slightly clock skewed clients both in the forward and
backwards directions.

Also, we have logs in place to warn people that their clocks are ultra skewed
(based on the received consensus date). We also have mechanisms in place to
ensure that we refetch a consensus when the current consensus date is too far
off (see update_consensus_networkstatus_downloads()). Now whether all these
mechanisms and logs work properly in all cases is something we need to test
extensively.

Of course, using the consensus valid_after time is not bulletproof either:
there are various edge cases where this can have bad results. For example,
imagine a world where the real time is 07:00UTC, but Alice is a 10 hours
backwards-skewed client whose local time is 21:00UTC. Imagine that Alice starts
up Tor with an old consensus with valid_after 20:00UTC (because her dirguard
lied, or because Alice had that consensus cached). In this case, Alice will not
realize that the consensus is hella old, and will try to use it. She will then
compute the wrong set of HSDirs, and fail the HS protocol. This case is
plausible in theory but also quite hard to protect against, since both Alice
and her consensus had wrong but convenient times.

All in all, I feel that using the consensus valid_after time for time period
related calculations seems reasonable at this point, but we should do more
testing (ideally automated) as we implement the relevant parts.

> Here is an initial attempt at figuring out the current Tor behavior when
> handling consensuses with surprising dates. More work is required here.
>
>> For example, do I as a client just ignore and discard a consensus from
>> 6 hours in the future? I don't remember the answer, so I can't do a good
>> job at analyzing your proposed change.
>>
>
> In general, the relevant time checks seem to happen at
> 

Re: [tor-dev] Revisiting prop224 overlap descriptor logic and descriptor lifetimes

[George Kadianakis]

Roger Dingledine <a...@mit.edu> writes:

> [ text/plain ]
> On Mon, Jun 13, 2016 at 03:48:39PM +0300, George Kadianakis wrote:
>> The main issue for me right now is that I can't recall how this helps with
>> clock skewed clients, even though that was a big part of our discussion in 
>> Montreal.
>> 
>> Specifically, I think that clients (and HSes) should determine the set of
>> responsible HSDirs (i.e. the current time period) based on the "valid-after" 
>> of
>> their latest consensus, instead of using their local clock. This way, as long
>> as the client's skewed clock is good enough to verify the latest consensus, 
>> the
>> client will have a consistent view of the network and SRV (assuming an
>> honest/updated dirguard). I tried to clarify this a bit in commit 465156d, so
>> please let me know if it's not a good idea.
>
> Interesting idea! I think I like it. You're right that in Montreal
> we were thinking in terms of client clocks, and we might be able to
> reduce the problem (both in frequency and in magnitude) by considering
> the time in the last consensus we have.
>
> Another argument in favor of using the last consensus is that we will be
> picking the "relays that are closest to the right location in the hash
> ring" out of our last consensus already. (That is not a strong argument
> in favor though, I think, since in theory there won't be so much churn
> in a day that all of the relays in our last consensus will become wrong.)
>
> All of this said, it seems like you are basing your arguments on some
> expectations about how clients handle consensuses that have surprising
> dates in them (surprising either because the client's clock is skewed,
> or because their directory guard gave them the wrong consensus). How
> *do* clients handle these situations? If we could get the intended /
> expected behavior written down, then we would have a better chance of
> identifying bugs in it that we can then fix.
>

I agree that we should get the intended/expected behavior written down!

Here is an initial attempt at figuring out the current Tor behavior when
handling consensuses with surprising dates. More work is required here.

> For example, do I as a client just ignore and discard a consensus from
> 6 hours in the future? I don't remember the answer, so I can't do a good
> job at analyzing your proposed change.
>

In general, the relevant time checks seem to happen at
networkstatus_get_reasonably_live_consensus() and not during consensus parsing.
That function is then called by router_have_minimum_dir_info() during
bootstrapping. If that function returns NULL, then Tor will get stuck at
"Boostrapping 25%: Loading networkstatus consensus".

Here is the basic logic of networkstatus_get_reasonably_live_consensus():

--
  #define REASONABLY_LIVE_TIME (24*60*60)
  if (consensus &&
  consensus->valid_after <= now &&
  now <= consensus->valid_until+REASONABLY_LIVE_TIME)
return consensus;
  else
return NULL;
--

And here are the scenarios:

Case #1: Handling consensuses with old dates

 If a client receives a consensus with an old date (i.e. the client's clock
 is skewed forward), the consensus will get verified just fine and Tor
 won't even log about the skew (XXX maybe we should fix this?)

 However when networkstatus_get_reasonably_live_consensus() gets reached,
 Tor will refuse to handle any consensuses whose valid_until date has
 expired by more than 24 hours.

Case #2: Handling consensuses with future dates

 If a client receives a consensus with a valid_after in the future
 (i.e. the client's clock is skewed backwards), the consensus will get
 verified fine and a log will appear about the skew ("Our clock is N hours
 behind the time published in the consensus yada yada...")

 However, when networkstatus_get_reasonably_live_consensus() gets reached,
 Tor will refuse to handle any consensuses whose valid_after date is in the
 future.

We see that while Tor consensus handling is quite flexible towards forward
skewed clocks (case #1), it's actually quite strict towards backward skewed
clocks (case #2). We might want to rethink how this should work, if we are
serious about supporting clock skewed clients. After all, handling consensuses
with future dates is safer than handling consensuses with older dates (which
are replayable).

I also wonder if we can consider the above problem orthogonal wrt prop224.
After all the problem here is on the consensus handling layer, and affects all
current clients and not just HS clients. We should first figure out exactly how
well the current Tor behavior works with the sugges

Re: [tor-dev] Revisiting prop224 overlap descriptor logic and descriptor lifetimes

[George Kadianakis]

David Goulet <dgou...@ev0ke.net> writes:

> [ text/plain ]
> On 13 Jun (15:48:39), George Kadianakis wrote:
>> Hello people,
>> 
>> I invite you to check out another round of time period-related prop224 spec
>> changes, based on our discussions in Montreal. These new changes simplify the
>> overlap descriptor publishing logic, and improve the caching lifetime of
>> descriptors in HSDirs.
>> 
>> You can find them in my branch `prop224-montreal-timeperiods` or here:
>> 
>> https://gitweb.torproject.org/user/asn/torspec.git/log/?h=prop224-montreal-timeperiods
>
> Couple of things.
>
> Section 2.2.2., about this TODO:
>
> [TODO: Control republish period using a consensus parameter?]
>
> Right now, we have RendPostPeriod for such a thing and some random added to
> it. As we discussed, a service changing that value will make it different from
> all others and thus more noticeable. But, we cleared out some uses cases where
> it could be useful such as a service load balancing and republishing a new
> descriptor often to change its intro points or keys.
>

I think HSes rotating intro points or keys should publish a new descriptor
regardless of the value of RendPostPeriod. This is not mentioned in prop224 tbh
(maybe it should), but this is also what little-t-tor does currently (it marks
the descriptor as dirty when rotating intro points).

> Making this a consensus params is a good idea imo but we should also provide
> an option to override it. Maybe it could make sense to _only_ have the option
> to change it if you are a NonAnonymous service for instance?
>
> Section 2.2.2.1:
>
> [TODO: What to do when we run multiple hidden services in a single host?]
>
> This could be quite "obvious" at the Guard. Building at least 12 short live
> circuits is a give away here that I'm running HSes. Apart from adding some
> random offset for each HS (which even then...), I'm not sure how to address
> this. Even now, we just upload all decriptors at the same RendPostPeriod.
> Maybe it's not too big of a problem?
>

Indeed, I'm also unsure on how to handle this properly.

> Section 2.2.5:
>
> "Hidden services MUST also keep their introduction circuits alive..."
>
> Does that mean service keeps them open (and the keys) until the descriptor
> expires on the HSDir? That is a service uploads a desc at 23:00 but then at
> 00:00 it creates a new descriptor using the new SRV so it should keep the
> intro points open until 02:00 (23:00 + 3 hours lifetime)?
>

Yes, that's what I mean. I will try to add an example to the proposal to make
it more clear.

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Onionboat: Using Docker for easy hidden services

[George Kadianakis]

Hello segfault,

someone linked me to this today:
   
https://nonconformity.net/2016/06/10/onionboat-using-docker-for-easy-tor-hidden-services/
   https://github.com/jheretic/onionboat

Forwarding it because of being potential relevant to your GSoC project (Tails 
server).

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Revisiting prop224 overlap descriptor logic and descriptor lifetimes

[George Kadianakis]

Hello people,

I invite you to check out another round of time period-related prop224 spec
changes, based on our discussions in Montreal. These new changes simplify the
overlap descriptor publishing logic, and improve the caching lifetime of
descriptors in HSDirs.

You can find them in my branch `prop224-montreal-timeperiods` or here:

https://gitweb.torproject.org/user/asn/torspec.git/log/?h=prop224-montreal-timeperiods

The main issue for me right now is that I can't recall how this helps with
clock skewed clients, even though that was a big part of our discussion in 
Montreal.

Specifically, I think that clients (and HSes) should determine the set of
responsible HSDirs (i.e. the current time period) based on the "valid-after" of
their latest consensus, instead of using their local clock. This way, as long
as the client's skewed clock is good enough to verify the latest consensus, the
client will have a consistent view of the network and SRV (assuming an
honest/updated dirguard). I tried to clarify this a bit in commit 465156d, so
please let me know if it's not a good idea.

Am I missing something wrt clock skewed clients here? If yes, can someone
demonstrate the effects of these changes with an example, so that I can clarify
the proposal further?

Feedback is welcome!
If I receive positive feedback, I will merge this in torspec.git ASAP.

Thanks!


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Improving PT documentation on website

[George Kadianakis]

Hello Nima,

I see you said this on IRC yesterday while referring to the pluggable transport
docs: "I'm gonna be working on some of these documentations this month. do you
have any suggestions on how we can improve them?"

I agree that this is extremely high priority right now and I'm very glad to see
you working on it. I feel that this is far more important than designing new
protocols at this point. The way I see it, our main issues with PTs at this
point is documentation and UX.

I imagine you are talking about 
https://www.torproject.org/docs/pluggable-transports.html.en
and maybe also the wiki page: 
https://trac.torproject.org/projects/tor/wiki/doc/PluggableTransports 

Some thoughts:

- With regards to the pluggable-transports.html.en page:
-- we should cleanup the PT list in the website. only include deployed PTs. 
   maybe even remove list entirely and move it to wiki. if that page is only 
for users, they don't need to know info about each PT.
-- add some sort of intuitive diagram similar to: 
https://www.torproject.org/images/obfsproxy_diagram.png
-- link to the new video as well
-- IMO we need a PT logo to give PTs a visual identity. When I wrote those 
pages, I used the moustached-onion logo. Do we have a better one?

- We also need up-to-date instructions on how to setup obfsbridges.
-- Maybe we need https://www.torproject.org/projects/obfsproxy.html.en but for 
obfs4
-- Here are the instructions that need to be updated: 
https://www.torproject.org/projects/obfsproxy-debian-instructions.html.en#instructions
 https://www.torproject.org/projects/obfsproxy-instructions.html.en#instructions
-- (but obfs5 is coming soon as well... PT deployment startegy not very good.)

- The wiki page is also quite bloated by now, and it's unclear what the 
audience is. 
-- The wiki page should also have links to user/bridge instructions, since
   that's where most people end up looking.
-- It might be worth cleaning it up, or splitting it in multiple pages.
   especially so since we link it from the main PT page.

I'm not very good with UX and web design stuff, but the above list would be the
things I would start addressing first...

Let me know what you think!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Onioncat and Prop224

[George Kadianakis]

str4d  writes:

> [ text/plain ]
> On 27/04/16 22:31, grarpamp wrote:
>> On 4/25/16, Tim Wilson-Brown - teor  wrote:
>>>
 On 22 Apr 2016, at 17:03, grarpamp  wrote:

 FYI: The onioncat folks are interested in collaborating
 with tor folks regarding prop224.

 https://gitweb.torproject.org/torspec.git/tree/proposals/224-rend-spec-ng.txt
>>>
>>> I'm interested in what kind of collaboration onioncat would like to do on
>>> prop224, next-generation hidden services.
>>> It would be great to work this out in the next few weeks, as we're coding
>>> parts of the proposal right now.
>> 
>> Yep :) And I know Bernhard was hoping to get in touch with Roger
>> on this before long.
>> 
>> Basically, prop224 HS being wider than 80 bits will break onioncat's
>> current HS onion <---> IPv6 addressing mechanism.
>> 
>> They're looking at various backward compatibility options, as well
>> as possibly making side use of the HSDir DHT, or even integrating
>> more directly with the tor client.
>> 
>
> Just FYI, I recently migrated all of I2P's spec proposals to the
> website, and came across a seven-year-old proposal that Bernhard wrote
> about improving I2P support in GarliCat:
>
> https://geti2p.net/spec/proposals/105-garlicat-name-translation
>
> I don't know how well it has aged, but given that Tor is now facing the
> same issues that I2P has, perhaps it can be of some use if resurrected
> from the dead :)
>

Hello,

disclaimer: I've never used onioncat and I'm not even sure what are the use
cases for it. I just read the docs on its webpage in an attempt to understand
the system and maybe contribute to this thread. I probably don't understand the
system sufficiently yet.

Starting with a question wrt the "Scalable Solution" for I2P, I did not exactly
understand how the DNS approach would work for garlicat.  Would GC do the DNS
request? Or the application? And to which DNS server? Some GC-specific DNS
server?  The solution seems plausible, but it requires the extra DNS protocol
round trip.

BTW, if the application-layer protocol supports DNS, can't we just fake the DNS
protocol part and resolve the DNS query inside onioncat? So, Alice is in
onioncat as "aliceonioncathiddenservice.onion" and she passes her onion to Bob
who wants to torrent with her through onioncat. Alice gives
"aliceonioncathiddenservice.onion" to her friend Bbo, and Bob tries to connect
to it through his application (e.g. a torrent client). Bob's application will
probably do a DNS query for the onion address, which onioncat can intercept (?).
At that point, onioncat can register the onion address as the destination,
and pass back a fake IPv6 address to the torrent client. From that point and
on, the torrent client will use the fake IPv6 address, and onioncat will
reroute the connection through the onion circuit to
"aliceonioncathiddenservice.onion".

I'm not sure if the above idea will work, but even if it does, it's worse than
the current onioncat, since it requires the application-layer protocol to
support DNS.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Latest state of the guard algorithm proposal (prop259) (April 2016)

[George Kadianakis]

Fan Jiang <fanji...@thoughtworks.com> writes:

> [ text/plain ]
> 2016年4月22日 上午4:54，"George Kadianakis" <desnac...@riseup.net>写道：
>>
>> Fan Jiang <fanji...@thoughtworks.com> writes:
>>
>> > [ text/plain ]
>> > On Thu, Apr 21, 2016 at 4:32 AM, George Kadianakis <desnac...@riseup.net
>>
>> > wrote:
>> >
>> > 
>
> It seems like we come to a point that most of prop259 can be stable for a
> while, we are going to do some cleanup in this implementation and spec. I
> think next week we can ask more people to review, does that sounds OK?
>

Sounds good!

> BTW, About your segmentfault I couldn't reproduce, maybe related with your
> torrc/state file?

I think it's some sort of dangling pointer error. e->chosen_by_version seems to
be referring to a corrupted memory address.

Not sure if it's caused by my state file, but in general the prop259 branch
should work well with any old state file without crashing.

Here is some gdb output from the segfault:

---
Apr 22 17:08:21.161 [notice] Tor v0.2.8.1-alpha-dev (git-d7ed996b2aba9f10) 
running on Linux with Libevent 2.0.21-stable, OpenSSL 1.0.1k and Zlib 1.2.8.
Apr 22 17:08:21.161 [notice] Tor can't help you if you use it wrong! Learn how 
to be safe at https://www.torproject.org/download/download#warning
Apr 22 17:08:21.161 [notice] This version is not a stable Tor release. Expect 
more bugs than usual.
Apr 22 17:08:21.161 [notice] Read configuration file 
"/home/f/Computers/tor/mytor/../confs/prop259".
Apr 22 17:08:21.164 [notice] Opening Socks listener on 127.0.0.1:

Program received signal SIGSEGV, Segmentation fault.
__strchr_sse2 () at ../sysdeps/x86_64/multiarch/../strchr.S:32
32  ../sysdeps/x86_64/multiarch/../strchr.S: No such file or directory.
(gdb) up
#1  0x5567eb25 in guards_update_state (next=0x559c3f40, 
next@entry=0x559c35e8, guards=guards@entry=0x559c4620, 
config_name=config_name@entry=0x5570c3be "UsedGuard") at 
src/or/prop259.c:1155
1155!strchr(e->chosen_by_version, ' ')) {
(gdb) p/x e
$1 = 0x559c42d0
(gdb) p/x e->chosen_by_version
$2 = 0x41373230
(gdb) x/s chosen_by_version
No symbol "chosen_by_version" in current context.
(gdb) x/s e->chosen_by_version
0x41373230: 
---

and here is my state file in case it matters:

---
EntryGuard jaures4 5CF8AFA5E4B0BB88942A44A3F3AAE08C3BDFD60B DirCache
EntryGuardAddedBy 5CF8AFA5E4B0BB88942A44A3F3AAE08C3BDFD60B 0.2.5.12 2016-01-11 
02:54:36
EntryGuard SGGSUK4 38F423A4320380FFE32DB60B72E7457CD6E3F096 DirCache
EntryGuardAddedBy 38F423A4320380FFE32DB60B72E7457CD6E3F096 0.2.5.12 2016-01-25 
08:20:31
EntryGuard aTomicRelayFR1 25EF027A85BAA044048AD1D635AF8583DB88C08F DirCache
EntryGuardAddedBy 25EF027A85BAA044048AD1D635AF8583DB88C08F 0.2.5.12 2016-02-06 
01:04:20
TorVersion Tor 0.2.5.12 (git-3731dd5c3071dcba)
LastWritten 2016-02-07 18:18:11
---
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Latest state of the guard algorithm proposal (prop259) (April 2016)

[George Kadianakis]

Fan Jiang <fanji...@thoughtworks.com> writes:

> [ text/plain ]
> On Thu, Apr 21, 2016 at 4:32 AM, George Kadianakis <desnac...@riseup.net>
> wrote:
>
>> Fan Jiang <fanji...@thoughtworks.com> writes:
>>
>> > [ text/plain ]
>> > Hi,
>> >
>> >> Hello Fan and team,
>> >>
>> >> 
>> >>
>> > Sounds great, that can simplify the logic a lot, I've done the change, no
>> > more pending_dir_guard.
>> >
>>
>> Hm. Can't you also remove pending_guard? What's the point of it now?
>>
>>
> Pending guard was actually for sampled_guard, say when we are in
> SAMPLED_GUARDS_STATE
> we don't wan't to make the algo return a new random guard
> picked_by_banwith, we want to keep the same one
> until the callback of first hop comes. We can make it specific for
> sampled_guards after checking the state.
> Does this make sense?
>

Hello,

I'm a bit confused. I can't find this SAMPLED_GUARDS_STATE in the spec or code.

IIUC, we are picking guards from SAMPLED_GUARDS to be used based on their
bandwidth? If yes, I'm not sure if that's needed.

Since we already sampled by bandwidth when we originally put the guards in
SAMPLED_GUARDS, I don't think we also need to sample by bandwidth when we pick
guards from SAMPLED_GUARDS to be used.

Instead you could treat SAMPLED_GUARDS as an ordered FIFO list and always
return the top element when you are asked to sample for it. Then you wouldn't
need to keep track of 'pending_guard' anymore. That seems simpler.

Did I understand the problem correctly? Do you find any security issues with my
suggestion?

> BTW, looking at your e54551adbfd5be4bee795df10f925b53fc9e730d I suggest you
>> also use entry_is_live() with the ENTRY_NEED_UPTIME and ENTRY_NEED_CAPACITY
>> flags always enabled. So that it always returns Stable && Fast guards.
>>
>>
> Yes, I have done this change.
>

Saw the commit. Cheers.

> We should also look at how ENTRY_ASSUME_REACHABLE and ENTRY_NEED_DESCRIPTOR
>> are
>> used in the rest of the code, to see if we should enable them or not
>> ourselves.
>>
>> >> Never saw this before, will look into it.
>> >
>> > - There is a memleak on 'extended' in filter_set().
>> >>
>> >>   In general, I feel that logic in that function is a bit weird. The
>> >> function
>> >>   is called filter_set() but it can actually end up adding guards to the
>> >> list.
>> >>   Maybe it can be renamed?
>> >>
>> >>
>> > Split it up to filter_set & expand_set probably can make this clear.
>> >
>> > - What's up with the each_remaining_by_bandwidth() function name?
>> >>
>> >>
>> > I guess it should be iterate_remaining_guards_by_bandwidth.
>> >
>>
>> Better. Or sample_guard_by_bandwidth() ? Or get_guard_by_bandwidth() ?
>>
>> IIUC that function is probalistically sampling from the 'guards' set, so
>> it's
>> not really iterating it.
>>
>>
> We are actually pick and removing guards from remaining_set in this
> function,
> And I saw the filter_set used this function wrong which has been fixed,
> so maybe `pop` is better than `get`.
>
> Another thing:
> Do we still need decide_num_guards to define the n_primary_guards? and it's
> the only remaining one is using for_directory.
>

No, let's not use decide_num_guards to define the number of primary guards. The
original purpose of decide_num_guards was completely different, and it does not
apply to prop259 very well.

I think it's safe to ignore decide_num_guards for now.

Thanks!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Revisiting prop224 time periods and HS descriptor upload/downloads

[George Kadianakis]

David Goulet <dgou...@ev0ke.net> writes:

> [ text/plain ]
> On 13 Apr (15:34:54), George Kadianakis wrote:
>> David Goulet <dgou...@ev0ke.net> writes:
>> 
>> > [ text/plain ]
>> > On 12 Apr (16:01:32), George Kadianakis wrote:
>> >> David Goulet <dgou...@ev0ke.net> writes:
>> >> 
>> >> > [ text/plain ]
>> >> > On 11 Apr (14:42:02), George Kadianakis wrote:
>> >> >> David Goulet <dgou...@ev0ke.net> writes:
>> >> >> 
>> >> >> > [ text/plain ]
>> >> >> > On 04 Apr (19:13:39), George Kadianakis wrote:
>> >> >> >> Hello,
>> >> >> >> 
>> >> >> >> during March we discussed the cell formats of prop224:
>> >> >> >>   
>> >> >> >> https://lists.torproject.org/pipermail/tor-dev/2016-March/010534.html
>> >> >> >> 
>> >> >> >> The prop224 topic for this month has to do with the way descriptors 
>> >> >> >> get
>> >> >> >> uploaded and downloaded, how this is scheduled using time periods 
>> >> >> >> and how the
>> >> >> >> shared randomness subsystem interacts with all that.
>> >> >> >> 
>> >> >> >> 
>
>> 
>> Here are some things left to be done:
>> 
>> - Specify *when* hidden services upload descriptors. Do they do it hourly, or
>>   only when a change has happened. Both approaches leak information to the
>>   HSDir (the former leaks uptime, the latter leaks intro point changes).
>
> I'm more comfortable right now with the HS uploading every RendPostPeriod
> (default: 1 hour). Even if the descriptor content doesn't change, it should
> increment the revision-counter.
>
> I'm more and more convinced that making RendPostPeriod _not_ configurable is
> also something we should do and thus allowing us to have the HSDir use that
> value instead (maybe).
>

Hello,

I pushed some more changes to my `prop224-timeperiods-1` branch.

As discussed above, I specified that HSes should upload their descriptors
periodically here:
  
https://gitweb.torproject.org/user/asn/torspec.git/commit/?h=prop224-timeperiods-1=c5309b7d6422daea980ed7ff401ec99f4fca58da

>> 
>> - Specify behavior of hidden services and clients with regards to time 
>> periods
>>   and the use of SRVs as discussed in
>> https://lists.torproject.org/pipermail/tor-dev/2016-April/010757.html
>
> I'm happy with your approach if you find mine a bit more complicated. They
> both result in the _same_ behavior anyway. Altough, in terms of code, for each
> SRV value, we need to keep the valid-after and the valid-until time in our SRV
> data structure (which we don't right now with current prop250 code).
>

I also specified the behavior of hidden services and clients in this commit:
  
https://gitweb.torproject.org/user/asn/torspec.git/commit/?h=prop224-timeperiods-1=e64c87b3685a40704f1108563c1c8341864bb71d

Please let me know how that section can become cleaner if you have any ideas.

I think this covers up the time period related changes for now. If you people
think that my `prop224-timeperiods-1` branch looks good and that I didn't
forget of anything else, I will ask Nick to review it and then merge it to 
torspec.

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] prop224: HSDir caches question with OOM

[George Kadianakis]

Tim Wilson-Brown - teor <teor2...@gmail.com> writes:

> [ text/plain ]
>
>> On 20 Apr 2016, at 07:22, David Goulet <dgou...@ev0ke.net> wrote:
>> 
>> On 18 Apr (13:18:25), George Kadianakis wrote:
>>> Tim Wilson-Brown - teor <teor2...@gmail.com> writes:
>>> 
>>>> [ text/plain ]
>>>> 
>>>>> On 16 Apr 2016, at 05:47, David Goulet <dgou...@ev0ke.net> wrote:
>>>>> 
>>>>> Hi!
>>>>> 
>>>>> (For the following, I'm only talking about HS directory.)
>>>>> 
>>>>> Here is my conundrum. I was working on plugging the new cache for 
>>>>> proposal 224
>>>>> into our OOM. I stumbled upon a fun problem.
>>>>> 
>>>>> 
>>>> 
>>>> We should add entries to a time-sorted list when they are created, 
>>>> otherwise this algorithm is O(n^2).
>>>> 
>>>> A third alternative is that we can iterate through each time period:
>>>> Set K to the oldest expected descriptor age in hours, minus 1 hour
>>>> Deallocate all entries from Cache A that are older than K hours
>>>> Deallocate all entries from Cache B that are older than K hours
>>>> Set K to K - 1 and repeat this process
>>>> 
>>>> This algorithm is O(Kn), which is ok as long as K is small.
>>>> This carries a slight risk of over-deallocating cache entries. Which is OK 
>>>> at OOM time.
>>>> I like this one, because it's simple, performant, and doesn't need any 
>>>> extra memory allocations.
>>>> 
>>> 
>>> I feel this is a reasonable approach here. It will probably have to be 
>>> combined
>>> with time-sorted lists as teor suggested, otherwise each of those steps will
>>> require walking through the whole cache every time.
>
> Yes, that's why I said O(K*n), which is alright as long as K is small.
> If we keep sorted lists (warn: v2 code changes), then the algorithm is O(n) 
> instead.
>
>>> 
>>> BTW, before it says "Set K to K - 1 and repeat this process" we should also
>>> check whether our memory issue has been addressed and if so then we can 
>>> exit,
>>> right?
>> 
>> Yeah I would do that. Every time we cleanup a cache, we check if we've 
>> reached
>> our bytes cutoff.
>
> Yes, we can terminate the algorithm as soon as we reach our goal.
>
>>> In that case, maybe we can add the same check between these two checks:
>>>> Deallocate all entries from Cache A that are older than K hours
>>>> Deallocate all entries from Cache B that are older than K hours
>>> so that maybe sometimes we can get away with just removing stuff from the 
>>> v2 cache.
>>> 
>>> Are there any other issues with the above approach?
>> 
>> While implementing this solution, I found one annoying issue :).
>> 
>> In order to compute K, I use the maximum lifetime of a cache entry. So
>> basically if my maximum lifetime in hours is M=10 then K will start at 10 - 
>> 1.
>> (Where 1 hour is basically the "assumed" HS post period.)
>
> Note that we can decrement K by a multiple of the post period - for example,
> it can be 3 hours: 9, 6, 3, 0.
>
> This makes the algorithm run faster (as it would iterate through the unsorted 
> v3
> list 11 times, rather than 33 times), at the cost of sometimes deallocating 
> some
> more recent descriptors in that 3 hour period.
>
>> However, our v2 and v3 cache have different lifetimes.
>> 
>>v2: 48 hours is the max lifetime (REND_CACHE_MAX_AGE) plus the maximum
>>clock skew of 24 hours (REND_CACHE_MAX_SKEW) == 72 hours.
>> 
>>v3: We've settled on 33 hours which is 24h time period, 6 hours of overlap
>>period and 3 hours of maximum clock skew.
>> 
>> So somehow, we need to "unify" K to fit both caches. Some options (some I do
>> not like):
>> 
>> 1) We could iterate over both caches at first and find out the oldest
>>   descriptor which requires a first pass of O(n) + O(m). Use that value for
>>   our starting point of K. But then inherently, we'll most of the time wipe 
>> v2
>>   descriptors since they have a long lifetime. (maybe good!?)
>
> I think that clients are just as likely to re-use descriptors that are an 
> hour old,
> regardless of the maximum lifetimes of v2 and v3 descriptors.
>
> So I don't care if we take out more v2 descriptors.
>

I feel this is a fine way to do this.

Since

Re: [tor-dev] Latest state of the guard algorithm proposal (prop259) (April 2016)

[George Kadianakis]

Fan Jiang  writes:

> [ text/plain ]
> Hi,
>
>> Hello Fan and team,
>>
>> I think I'm not a big fan of the pending_guard and pending_dir_guard
>> concept. To me it seems like a quick hack that tries to address fundamental
>> issues with our algorithm that appeared when we tried to adapt the
>> proposal to
>> the tor codebase.
>>
>>
> Yeah agree, this pending_guard hack was trying to avoid some implementation
> problem, we need to redesign.
> I haven't got any good idea about this, that will be nice if you already
> got some thoughts.
>
>
>> I think one of the main issues with the current algorithm structure is that
>> _one run of the algorithm_ can be asked to _setup multiple circuits_, and
>> each
>> of those circuits has different requirements for guards. That is, since we
>> do
>> filtering on START based on the requirements of circuit #1, this means
>> that any
>> other circuits that appear before END is called, also have to adapt to the
>> requirements of circuit #1. This is obvious in the code since we use
>> guard_selection->for_directory throughout the whole algorithm run, even
>> though
>> for_directory was just the restriction of circuit #1.
>>
>> Specifically about the pending_guard trick, I feel that it interacts in
>> unpredictable ways with other features of the algorithm. For example,
>> consider
>> how it interacts with the primary guards heuristic. It could be time for
>> the
>> algorithm to reenter the primary guards state and retry the top guards in
>> the
>> list, but because of the pending_guard thing we actually return the 15th
>> guard
>> to the circuit.
>>
>> IMO we should revisit the algorithm so that one run of the algorithm can
>> accomodate multiple circuits by design and without the need for hacks.
>> Here is
>> an idea towards that direction:
>>
>>   I think one very important change that we can do to simplify things is to
>>   remove the need to filter guards based on whether they are dirguards,
>> fast,
>>   or stable. My suggestion here would be to *only* consider guards that are
>>   dirguards _and_ fast _and_ stable. This way, any circuit that appears
>> will be
>>   happy with all the guards in our list and there is no need to do the
>>   pending_dir_guard trick. See [0] on why I think that's safe to do.
>>
>>   This is easy to do in the current codebase. You just need to call
>>   entry_is_live() with need_uptime, need_capacity and for_directory all
>>   enabled (instead of flags being 0).
>>
>>   If you do the above, your sampled guard set will be able to accomodate
>> any
>>   circuit that comes its way and that should simplify logic considerably.
>>
>>
> Sounds great, that can simplify the logic a lot, I've done the change, no
> more pending_dir_guard.
>

Hm. Can't you also remove pending_guard? What's the point of it now?

BTW, looking at your e54551adbfd5be4bee795df10f925b53fc9e730d I suggest you
also use entry_is_live() with the ENTRY_NEED_UPTIME and ENTRY_NEED_CAPACITY
flags always enabled. So that it always returns Stable && Fast guards.

We should also look at how ENTRY_ASSUME_REACHABLE and ENTRY_NEED_DESCRIPTOR are
used in the rest of the code, to see if we should enable them or not ourselves.

>> Never saw this before, will look into it.
>
> - There is a memleak on 'extended' in filter_set().
>>
>>   In general, I feel that logic in that function is a bit weird. The
>> function
>>   is called filter_set() but it can actually end up adding guards to the
>> list.
>>   Maybe it can be renamed?
>>
>>
> Split it up to filter_set & expand_set probably can make this clear.
>
> - What's up with the each_remaining_by_bandwidth() function name?
>>
>>
> I guess it should be iterate_remaining_guards_by_bandwidth.
>

Better. Or sample_guard_by_bandwidth() ? Or get_guard_by_bandwidth() ?

IIUC that function is probalistically sampling from the 'guards' set, so it's
not really iterating it.


Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] prop224: HSDir caches question with OOM

[George Kadianakis]

Tim Wilson-Brown - teor  writes:

> [ text/plain ]
>
>> On 16 Apr 2016, at 05:47, David Goulet  wrote:
>> 
>> Hi!
>> 
>> (For the following, I'm only talking about HS directory.)
>> 
>> Here is my conundrum. I was working on plugging the new cache for proposal 
>> 224
>> into our OOM. I stumbled upon a fun problem.
>> 
>> 
>
> We should add entries to a time-sorted list when they are created, otherwise 
> this algorithm is O(n^2).
>
> A third alternative is that we can iterate through each time period:
> Set K to the oldest expected descriptor age in hours, minus 1 hour
> Deallocate all entries from Cache A that are older than K hours
> Deallocate all entries from Cache B that are older than K hours
> Set K to K - 1 and repeat this process
>
> This algorithm is O(Kn), which is ok as long as K is small.
> This carries a slight risk of over-deallocating cache entries. Which is OK at 
> OOM time.
> I like this one, because it's simple, performant, and doesn't need any extra 
> memory allocations.
>

I feel this is a reasonable approach here. It will probably have to be combined
with time-sorted lists as teor suggested, otherwise each of those steps will
require walking through the whole cache every time.

BTW, before it says "Set K to K - 1 and repeat this process" we should also
check whether our memory issue has been addressed and if so then we can exit,
right?

In that case, maybe we can add the same check between these two checks:
> Deallocate all entries from Cache A that are older than K hours
> Deallocate all entries from Cache B that are older than K hours
so that maybe sometimes we can get away with just removing stuff from the v2 
cache.

Are there any other issues with the above approach?

PS: We could also in theory imagine writing those descriptors on disk, and only
caching the ones that we have received requests for lately. This way we
would not utilize our RAM that much...


___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Latest state of the guard algorithm proposal (prop259) (April 2016)

[George Kadianakis]

Fan Jiang  writes:

> [ text/plain ]
> Thanks for the insights.
>
>
>> >> It seems like the latest version of prop259 was posted a few weeks ago:
>> >>
>> https://lists.torproject.org/pipermail/tor-dev/2016-March/010625.html
>> >>
>> > 
>> >
>> >> A few things:
>> >>
>> >> a) Are there still proposal design decisions that need to be taken and
>> we
>> >> are
>> >>unclear on? I admit I'm a bit lost in the latest [tor-dev] thread, so
>> >> maybe
>> >>I can be of help somehow here?
>> >>
>> >> There are still some issues, like for_directory may leads to maintain
>> two
>> > sets of
>> > sampled_set independently, which is not yet defined clearly in proposal.
>> >
>>
>> Hm, how come this is happening?
>>
>> I would think that for_directory would now be just another filter (like the
>> ipv6 one etc.) that can be applied on top of the sampled list.
>>
> The problem here is for_directory is a parameter of function call, that
> means we can't do filter
> before the call happens. Now the filter action only happens at START stage.
> Maybe do a check before we return the selected guard (if not valid, then
> continue picking new one)
> can be a solution.
>

Hello,

hm, that's an interesting problem...

So we learn whether a circuit is for_directory when
choose_random_entry_prop259() is called, but at that point the prop259
algorithm has already STARTed and its filtering parameters have been
determined?

So if some part of tor calls choose_random_entry_prop259() quickly twice, first
with for_directory set, and the second time with for_directory unset, the guard
algorithm will proceed in both cases with for_directory being set? Because the
context has been set in the first call to choose_random_entry_prop259()?

This seems like a problem to me, and I'm not sure how to solve it well...

One way could be to use the 'cpath_build_state_t *state' in
choose_random_entry_prop259() to be able to understand whether each call is
about a different circuit or not. Then you could start a separate algorithm
invocation (so new START) for each new circuit you see.

But then I'm not sure how to do this without each separate algorithm call
wrecking up the sampled_guards and used_guards lists... In the dev meeting in
Valencia, we discussed with Ola and Reinaldo about using locking and blocking
for this, but I'm not sure how much that would impact the performance...

Do you guys have any plans here?

---



BTW, I also noticed that you have various global structures in prop259.c, like
entry_guard_selection and used_guards and sampled_guards. I see you've been
working hard on un-globalizing the entry_guards list, but I'm not sure if there
is value to that if all those other structures are global. Do you have plans
for making those structures local to each specific algorithm instance?

Cheers!

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Revisiting prop224 time periods and HS descriptor upload/downloads

[George Kadianakis]

David Goulet <dgou...@ev0ke.net> writes:

> [ text/plain ]
> On 12 Apr (16:01:32), George Kadianakis wrote:
>> David Goulet <dgou...@ev0ke.net> writes:
>> 
>> > [ text/plain ]
>> > On 11 Apr (14:42:02), George Kadianakis wrote:
>> >> David Goulet <dgou...@ev0ke.net> writes:
>> >> 
>> >> > [ text/plain ]
>> >> > On 04 Apr (19:13:39), George Kadianakis wrote:
>> >> >> Hello,
>> >> >> 
>> >> >> during March we discussed the cell formats of prop224:
>> >> >>   https://lists.torproject.org/pipermail/tor-dev/2016-March/010534.html
>> >> >> 
>> >> >> The prop224 topic for this month has to do with the way descriptors get
>> >> >> uploaded and downloaded, how this is scheduled using time periods and 
>> >> >> how the
>> >> >> shared randomness subsystem interacts with all that.
>> >> >> 
>> >> >> 
>> >> >> 
>> >
>> > The logic I sketched out above makes it that we would need parameters (from
>> > the consensus) like so (or hardcode them):
>> >
>> > - TIME_PERIOD_ROTATION_TIME (currently 12:00)
>> >
>> 
>> [Second email with some more thoughts]
>> 
>> BTW, currently in prop224 the TIME_PERIOD_ROTATION_TIME is at 00:00 because 
>> of
>> the following paragraph:
>> 
>>Time periods start with the Unix epoch (Jan 1, 1970), and are
>>computed by taking the number of whole minutes since the epoch and
>>dividing by the time period. So if the current time is 2013-11-12
>>13:44:32 UTC, making the seconds since the epoch 1384281872, the
>>number of minutes since the epoch is 23071364.  If the current time
>>period length is 1500 (the default), then the current time period
>>number is 15380. It began 15380*1500*60 seconds after the epoch at
>>2013-11-11 20:00:00 UTC, and will end at (15380+1)*1500*60 seconds
>>after the epoch at 2013-11-12 21:00:00 UTC.
>> 
>> I wonder what's the best way to change this to start at 12:00. 
>> 
>> We could in theory compute the "number of whole minutes since the epoch plus 
>> 12
>> hours" and use that in the division, but that would be a bit ugly... Is 
>> there a
>> more elegant thing to do?
>> 
>> We could also in theory change the shared random value generation to happen 
>> at
>> 12:00, and then have TIME_PERIOD_ROTATION_TIME naturally start at 00:00, but
>> this requires changing prop250. Could it be worth it? :/
>
> I think it's fine we keep the start time at 12:00 here. It's just an offset
> from the start of the epoch. Furthermore, adding a "rotation time" makes it
> that we we can control where everything started which doesn't have to be the
> epoch time at 00:00.
>
> We can find the start of the TP with those two (rotation time and lifetime)
> and then divide that time value by the lifetime to get the nth time period.
>
> Also, controling the rotation time is good to have for chutney testing with
> much more smaller timings.
>

OK, I posted a torspec branch with some initial changes based on the
discussions of this thread at 'prop224-timeperiods-1':

https://gitweb.torproject.org/user/asn/torspec.git/log/?h=prop224-timeperiods-1

Specifically, wrt time periods and the start time, I introduced a "rotation
time offset" of 12 hours to the epoch calculation. You can see it here:
 
https://gitweb.torproject.org/user/asn/torspec.git/commit/?h=prop224-timeperiods-1=6499cbf1023806f75d567503ad0ab8625a3876a7

Please let me know if the technique can be simplified or improved or needs 
better wording.
We need this mechanic to be clear and easy to understand and implement!

>> 
>> 
>> 
>> 
>> Currently the proposal says:
>> 
>>Hidden service directories should accept descriptors [...] and retain them
>>for at least [TODO: how much?] minutes after the end of the period.
>> 
>> but that means that HSDirs need to keep track of when the period ends, and
>> whether a descriptor was uploaded for the current time period or for the
>> overlay period...
>
> I think this will make things much more complicated. IMO, the HSDir should
> _only_ rely on the revision counter and an expiry time and not trying to try
> and guess the lifetime of a descriptor from the service perspective.
>
> However, here is an idea. Considering teor's argument about HS fingerprinting,
> we should make the upload happen regurlarly so having RendPostPeriod
> cus

Re: [tor-dev] Revisiting prop224 time periods and HS descriptor upload/downloads

[George Kadianakis]

David Goulet <dgou...@ev0ke.net> writes:

> [ text/plain ]
> On 11 Apr (14:42:02), George Kadianakis wrote:
>> David Goulet <dgou...@ev0ke.net> writes:
>> 
>> > [ text/plain ]
>> > On 04 Apr (19:13:39), George Kadianakis wrote:
>> >> Hello,
>> >> 
>> >> during March we discussed the cell formats of prop224:
>> >>   https://lists.torproject.org/pipermail/tor-dev/2016-March/010534.html
>> >> 
>> >> The prop224 topic for this month has to do with the way descriptors get
>> >> uploaded and downloaded, how this is scheduled using time periods and how 
>> >> the
>> >> shared randomness subsystem interacts with all that.
>> >> 
>> >> Here are some discussion topics. Lots of text on the first two, less text 
>> >> on the rest:
>> >> 
>> >> 
>> >> 
>> >>   In any case, this is how this might look like:
>> >> 
>> >> 
>> >> +--+
>> >> |  |
>> >> | 00:00  12:00   00:00   12:00   00:00   12:00 |
>> >> | SRV#1  TP#1SRV#2   TP#2SRV#3   TP#3  |
>> >> |  |
>> >> |   $ |---$-==|---$-==||
>> >> |overlap12   overlap23 |
>> >> |  |
>> >> +--+
>> >> 
>> >>   Legend:[TP#1 = Time Period #1]
>> >>  [SRV#1 = Shared Random 
>> >> Value #1]
>> >> 
>> >> 
>> >> 
>> 
>> How else could we simplify this logic?
>
> It seems simple enough. Maybe the algorithm I sketched out above makes it
> simpler? Maybe not!... It's basically the _same_ end results as you.
>

Yes, both approaches seem equivalent.

> The logic I sketched out above makes it that we would need parameters (from
> the consensus) like so (or hardcode them):
>
> - TIME_PERIOD_ROTATION_TIME (currently 12:00)
>
> - TIME_PERIOD_[LIFETIME | SPAN | DURATION] (currently 24h)
>
> - SHARED_RANDOM_VALUE_[CREATION | ROTATION]_TIME (currently 00:00)
>
> - SHARED_RANDOM_VALUE_[LIFETIME | SPAN | DURATION] (currently 24h)
>
> I doubt we can go simpler than that. Both algorithms have one single check
> ending in two outcomes that is either use previous or current.
>

So, should we update prop250 and add SHARED_RANDOM_VALUE_CREATION_TIME and
SHARED_RANDOM_VALUE_LIFETIME as consensus parameters?

>> 
>> >> 
>> >> 
>> >>   + HSDir behavior
>> >> 
>> >> Currently the spec says the following:
>> >> 
>> >>  Hidden service directories should accept descriptors at least [TODO:
>> >>  how much?] minutes before they would become valid, and retain them
>> >>  for at least [TODO: how much?] minutes after the end of the period.
>> >> 
>> >> After discussion with David, we thought of chopping off the first 
>> >> part of
>> >> that paragraph and not imposing any such weak restrictions for 
>> >> accepting
>> >> descriptors (see #18332).
>> >> 
>> >> We still have not decided about the second part of that paragraph, 
>> >> that is
>> >> how long descriptors should be retained after the end of the period. 
>> >> We
>> >> currently think clock skew is the only thing that can bring clients 
>> >> to the
>> >> wrong HSDir after the end of the period. Maybe an hour is OK? David
>> >> suggested 12 hours. The current Tor is doing 48 hours... Any ideas?
>> >
>> > It should at least be 24 hours (maximum possible) with an adjustment of at 
>> > the
>> > _very_ least the overlap period. If the overlap period is 6 hours, we can 
>> > then
>> > add the "maximum clock skew" we think is reasonable and we would end up 
>> > with
>> > an OK value imo.
>> >
>> > Descriptor maximum lifetime:24 hours
>> > Overlap period span:6 hours (taken from your diagram)
>> > Maximum accept

Re: [tor-dev] Revisiting prop224 time periods and HS descriptor upload/downloads

[George Kadianakis]

David Goulet <dgou...@ev0ke.net> writes:

> [ text/plain ]
> On 04 Apr (19:13:39), George Kadianakis wrote:
>> Hello,
>> 
>> during March we discussed the cell formats of prop224:
>>   https://lists.torproject.org/pipermail/tor-dev/2016-March/010534.html
>> 
>> The prop224 topic for this month has to do with the way descriptors get
>> uploaded and downloaded, how this is scheduled using time periods and how the
>> shared randomness subsystem interacts with all that.
>> 
>> Here are some discussion topics. Lots of text on the first two, less text on 
>> the rest:
>> 
>> 
>> 
>>   In any case, this is how this might look like:
>> 
>> 
>>+--+
>>|  |
>>| 00:00  12:00   00:00   12:00   00:00   12:00 |
>>| SRV#1  TP#1SRV#2   TP#2SRV#3   TP#3  |
>>|  |
>>|   $ |---$-==|---$-==||
>>|overlap12   overlap23 |
>>|  |
>>+--+
>> 
>>   Legend:[TP#1 = Time Period #1]
>>  [SRV#1 = Shared Random 
>> Value #1]
>> 
>> 
>> 
>> - So now that we have ironed out the time period stuff slightly, let's 
>> discuss
>>   the behavior that hidden services, clients and HSDirs should inherit.
>> 
>>   This email is quite long already so I'm going to go with examples, instead 
>> of
>>   formal specification. However, this stuff needs to go formally in the
>>   proposal IMO, so any help in formalizing it would be great.
>> 
>>   + Hidden Service behavior:
>> 
>> Example 1: Our hidden service boots up at 14:00 of TP#1. In this case, we
>>  are nowhere close to the overlap period, so the hidden service should 
>> just
>>  publish its TP#1 descriptor to the HSDir hash ring using SRV#1 (which at
>>  that point should be in consensuses as "shared-rand-current-value").
>> 
>>  The hidden service might also want to calculate its overlap OFFSET (as
>>  specified in [TIME-OVERLAP]) and schedule a time callback for publishing
>>  its TP#2 descriptors.
>> 
>> Example 2: Our hidden service boots up at 03:00 of TP#1. That's outside 
>> of
>>  the overlap period again, but this time the hidden service needs to use 
>> the
>>  SRV from "shared-rand-previous-value" because the SRV was rotated at 
>> midnight.
>> 
>> Example 3: Our hidden service boots up at 09:00 of TP#1. That's inside 
>> the
>>  overlap period, so the hidden service should calculate its overlap
>>  OFFSET and compare it with the current time.
>> 
>>  If it has not passed, then we are in the exact same case as Example 2.
>> 
>>  If the overlap OFFSET _has_ passed, then the hidden service needs to act
>>  as Example 2, and _also_ publish its TP#2 descriptors to a second set of
>>  HSDirs using SRV#2.
>> 
>> I think these are all the cases for the hidden service, but I would like 
>> to
>> formalize this in a way that can be written in the spec. Particularly, 
>> I'm
>> not sure how to formalize which SRV to pick at a given time point.
>
> It sounds simple as:
>
> "If we are before to the overlap period, use the time period shared random
> value (TP1 == SRV1). If we are in the overlap period, upload two descriptors
> using _both_ SRVs."
>
> Plausible?
>

I'm not sure it's so simple. As it is now, there is no indicator connecting
time periods with shared random values, so "TP1 == SRV1" might make sense to us
but it's not something that can be implemented. How does the client know
whether to use "shared-rand-previous-value" or "shared-rand-current-value"?

Here is an idea:

"A hidden service uploading its normal descriptor using a consensus with
 valid-after between 12:00UTC (inclusive) and 00:00UTC (exlusive), uses the
 _current_ SRV. A hidden service uploading its normal descriptor using a
 consensus with valid-after between 00:00UTC (inclusive) and 12:00UTC
 (exclusive), uses the _previous_ SRV.

 A hidden service uploading its overlap descriptor

Re: [tor-dev] Reviewing prop224 fixes

[George Kadianakis]

David Goulet <dgou...@ev0ke.net> writes:

> [ text/plain ]
> On 04 Apr (13:07:29), George Kadianakis wrote:
>> John Brooks <john.bro...@dereferenced.net> writes:
>> 
>> > [ text/plain ]
>> > (Thread forked from [tor-dev] Notes from the prop224 proposal reading 
>> > group)
>> >
>>  
>>
>> OK I addressed the comments above in my branch `prop224-fixes`.
>> I also ripped out the MAINT_INTRO command as was discussed. 
>> 
>> Please see: 
>> https://gitweb.torproject.org/user/asn/torspec.git/log/?h=prop224-fixes
>> 
>> (I didn't change the key type thing you pointed out. Not sure if changing it 
>> to
>> the old type / len / key architecture would make things better. Please let me
>> know if you decide it will.)
>> 
>> > 442f0b3791797ebbac3feb2bffb87318fe8d84 "Clarify prop224 use of shared 
>> > random”
>> >
>> > Seems like we will need a lot more detail on how the shared random values 
>> > are used
>> > for the hash ring, the process for switching to the new SRV, and so on. Is 
>> > somebody
>> > planning to write that up? Has it all been decided yet?
>> >
>> 
>> Agreed. Looking at the time period logic is next on my stack, and my plan is 
>> to
>> make another thread similar to this one.
>> 
>> >> 
>> >> Stuff I need to look into next:
>> >> - Can we simplify the backwards compat logic?
>> >> - Should we add extensions to the rendezvous cells (at the cost of 
>> >> failing backwards compat)?
>> >> - Address more TODOs (there are a bunch of hard ones in there).
>> >> - Clean up some messy sections.
>> >> - Figure out the fate of UPDATE-KEYS-SUBCMD (see my previous mail).
>> >
>> > Happy to discuss any of these any time. My list right now is:
>> >
>> > - Look at onion hostnames, figure out the extra 4 bits and potentially a 
>> > checksum
>> > - Fix client authentication
>> > - Thinking more about denial of service, especially on hsdirs
>> >
>> 
>> Sounds good. Let me know when you have thoughts or patches.
>
> I'm happy with the current changes! I say you can merge them upstream in
> torspec or wait for a nickm/roger ack :)
>

OK, I merged the spec changes from this thread upstream.

They should be on the HEAD of https://gitweb.torproject.org/torspec.git .

Thanks for the feedback!

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev