Re: [tor-dev] onion v2 deprecation plan?

[grarpamp]

> for v3 support, HSFETCH won't be ncessary...

N...

HSFETCH 

is an absolutely necessary control function now critical to operation of a
variety of onionland search / index / status / webhosting / research services,
and any other basic commandline checks that have zero wish to be
spawning heavy browser / wget / specific application apparatus or wasting
resources establishing TCP connections to the services themselves,
and for debugging tor client and network itself independant from
all such other tools, and helpful for "Why can't I connect" questions
from users, etc.

Further, a minimum request and result option semantics of
HSFETCH regarding the above were roughly specified previously...

HSFETCH  [fetch_mode] [output_each] [raw_desc]
[sync_here] [update_cache]
 as integrated with vN-DescId and SERVER= elements

General operation
- fetch mode: 0 default client resolution method, 1 force from network [3 only,
  do not recurse to cache], 2 force from cache [4 only, do not recurse
to network]
- source of the returned descriptor: network, or local cache
- network may have different data than cache so [update_cache] or not,
default yes
- result status of the fetch [a]: ok found, 404 not found, network
failure + why: reason
- status of the descriptor [a]: ok good, bad + why: crypto failed, expired,
  parsing data type, truncated, corrupt, etc
- decoding and printing out all fields of the returned descriptor,
  optionally also print the whole descriptor as received, regardless of any bad
  status, in hex [raw_desc] to further help debugging and other uses
For each HSDir by respective HSDir identifier [output_each=true]
- above result status of the fetch for each HSDir queried (typically six)
- above status of the descriptor from each responding HSDir
- above decoding and printing for each responding HSDir [output_each=verbose]

[a] These should be tagged 0 if all sources / HSDirs were ok, or 1 if an
ok descriptor was ultimately able to be returned to the fetch but some of
the sources / HSDirs had negatives thus pointing to further investigation,
or 2 if none were ok (and print the status if all six had same status, or
print "various" if they varied).

Since nodes may be performing other tasks in parallel, even over the
same onions, and due to various mandatory modes and sources being
selected, and network delays, it is very helpful and unambiguous if the
output is selectably synchronus to this command and console [sync_here],
thus leaving the HS_DESC / HS_DESC_CONTENT event streams doing their
thing if so enabled possibly on / for other control connections / monitors.
A command instance identifier should be added to be returned to match
up with respective event logstream (which would otherwise perhaps be
identified as "socks / tunnel / trans / dns / natd / etc" for those sources),
but that does not replace utility of [sync_here] for lesser capable
users or cases.

Future applications may (and perhaps already do) use HSDir descriptor
mechanism as their own datastore via HSPOST, for which similar
HSFETCH models would be useful, thus good to consider herein,
certainly as an extensible.

There were a tickets and emails somewhere on these concepts so
that they could be further and better implemented. Some elements
were committed, others remain :)

> It is the main reason it hasn't been done that is for a lack of use case. The
> HSFETCH for v3 is much more tricky in terms of engineering and interface thus
> we decided to implement it on the "need it basis".
>
> If OnionBalance actually needs it for v3, then we should try to get that on
> the roadmap.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[teor]

> On 30 Apr 2018, at 23:55, David Goulet  wrote:
> 
>> On 28 Apr (09:34:09), teor wrote:
>> 
>> On 28 Apr 2018, at 04:48, Damian Johnson  wrote:
>> 
 OnionBalance requires STEM support for V3
>>> 
>>> Hi Alec, would you mind clarifying what you need from Stem? As far as
>>> I'm aware Stem supports v3 onion service creation...
>>> 
>>> https://trac.torproject.org/projects/tor/ticket/25124
>>> 
>>> See the 'version 3' note at the end of...
>>> 
>>> https://stem.torproject.org/api/control.html#stem.control.Controller.create_ephemeral_hidden_service
>>> 
>>> I'm unaware of the ball being in my court on any v3 Onion Service
>>> stuff. If there's something I should have on my radar then please let
>>> me know!
>> 
>> OnionBalance requires Stem support for v3 HSFETCH
>> But Stem requires Tor support for v3 HSFETCH
>> https://trac.torproject.org/projects/tor/ticket/25417
> 
> H I was told before the v3 control port work by Donnacha that for v3
> support, HSFETCH won't be ncessary...
> 
> It is the main reason it hasn't been done that is for a lack of use case. The
> HSFETCH for v3 is much more tricky in terms of engineering and interface thus
> we decided to implement it on the "need it basis".
> 
> If OnionBalance actually needs it for v3, then we should try to get that on
> the roadmap.

I didn't know there was any alternative to HSFETCH.

If OnionBalance can do without it, then we don't need it on the roadmap.

T
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[David Goulet]

On 28 Apr (09:34:09), teor wrote:
> 
> On 28 Apr 2018, at 04:48, Damian Johnson  wrote:
> 
> >> OnionBalance requires STEM support for V3
> > 
> > Hi Alec, would you mind clarifying what you need from Stem? As far as
> > I'm aware Stem supports v3 onion service creation...
> > 
> > https://trac.torproject.org/projects/tor/ticket/25124
> > 
> > See the 'version 3' note at the end of...
> > 
> > https://stem.torproject.org/api/control.html#stem.control.Controller.create_ephemeral_hidden_service
> > 
> > I'm unaware of the ball being in my court on any v3 Onion Service
> > stuff. If there's something I should have on my radar then please let
> > me know!
> 
> OnionBalance requires Stem support for v3 HSFETCH
> But Stem requires Tor support for v3 HSFETCH
> https://trac.torproject.org/projects/tor/ticket/25417

H I was told before the v3 control port work by Donnacha that for v3
support, HSFETCH won't be ncessary...

It is the main reason it hasn't been done that is for a lack of use case. The
HSFETCH for v3 is much more tricky in terms of engineering and interface thus
we decided to implement it on the "need it basis".

If OnionBalance actually needs it for v3, then we should try to get that on
the roadmap.

Cheers!
David

-- 
1dKuWYP7Si7mjurnrG6vfHiQfiibnDK5yH7HDBggBeM=


signature.asc
Description: PGP signature
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[teor]

Hi nusenu,

I've just finished catching up with this thread, ticket changes,
and the IRC discussion overnight.

> On 28 Apr 2018, at 09:30, teor  wrote:
> 
> On 28 Apr 2018, at 05:56, nusenu  wrote:
> 
>>> And also we will be able to reduce the
>>> requirements for becoming an HSDir which will strengthen and make our
>>> network more robust.
>>> 
>>> That said, I think we are unfortunately still far from deprecating v2
>>> onions:
>> 
>> 
>> thanks for listing all these relevant ticket IDs, I pasted your email into
>> trac and I made them child tickets of
>> https://trac.torproject.org/projects/tor/ticket/25955
>> 
>> lets try to link all relevant tickets there
> 
> Please don't, that confuses our workflow.
> A lot of these tickets are unrelated to v3 onion service features in Tor.
> 
> If you want to link all these tickets somewhere, open another ticket.

I'm sorry, that was what you did.

But using the cyperpunks account made a few network team members
nervous. When we see accounts we don't know making sweeping
changes, we tend to revert them. That's what happened in this case.

You're welcome to use the cuperpunks account for any changes, but
please check with us next time before making large changes.

In this case, I'm not sure if we want a parent ticket, or a tag.

A tag might be more useful, because we use parent tickets for closely
coupled tasks. And when there are too many layers of parents, people
get confused.

T

___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[teor]

On 28 Apr 2018, at 04:48, Damian Johnson  wrote:

>> OnionBalance requires STEM support for V3
> 
> Hi Alec, would you mind clarifying what you need from Stem? As far as
> I'm aware Stem supports v3 onion service creation...
> 
> https://trac.torproject.org/projects/tor/ticket/25124
> 
> See the 'version 3' note at the end of...
> 
> https://stem.torproject.org/api/control.html#stem.control.Controller.create_ephemeral_hidden_service
> 
> I'm unaware of the ball being in my court on any v3 Onion Service
> stuff. If there's something I should have on my radar then please let
> me know!

OnionBalance requires Stem support for v3 HSFETCH
But Stem requires Tor support for v3 HSFETCH
https://trac.torproject.org/projects/tor/ticket/25417

T___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[Jonathan Marquardt]

On Fri, Apr 27, 2018 at 04:03:00PM -0400, grarpamp wrote:
> a) If defined as shifting v3 to be "provisioned by default" via docs
> and function, while *continuing to support v2* functionality
> on the network, there's no problem, everyone is happy.
> b) While v2 and v3 do share some capabilities, since v2 and v3
> do offer their own exclusive subset of capabilities to users
> that cannot currently be found in the opposing version,
> *removing v2 support* is a definite issue.

I think that, before making v3 the default, all features from v2 like 
HidServAuth should be implemented and should have been around for a couple of 
Tor versions.

Also, what would happen to an old Tor instance with v2 onion services 
configured after the upgrade? These onion services should definitely not 
automatically be switched to v3, as it could break many configurations on 
systems with automatic software updates. I suggest that, if Tor sees an onion 
service configured in torrc and if there's no "HiddenServiceVersion 3" in 
torrc and there are v2 keys in the HiddenServiceDir, it should continue to 
serve a v2 service.

But leaving a note in the log about v3 services if there's a v2 configured 
could be a good thing, I think.

> v3 is a nice advancement and is needed by lots of users
> for what it can do for them, just as v2 is.

Totally agreed.
-- 
OpenPGP Key: 47BC7DE83D462E8BED18AA861224DBD299A4F5F3
 https://www.parckwart.de/pgp_key


signature.asc
Description: PGP signature
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[nusenu]

> Yes indeed. The sooner we deprecate v2 the sooner we can stop worrying
> about malicious HSDirs.

yes, that was indeed the motivation for my email (mostly because I see how much 
time
goes into the constant detection and rejection of these relays - not by me)

> And also we will be able to reduce the
> requirements for becoming an HSDir which will strengthen and make our
> network more robust.
> 
> That said, I think we are unfortunately still far from deprecating v2
> onions:


thanks for listing all these relevant ticket IDs, I pasted your email into
trac and I made them child tickets of
https://trac.torproject.org/projects/tor/ticket/25955

lets try to link all relevant tickets there



-- 
https://mastodon.social/@nusenu
twitter: @nusenu_



signature.asc
Description: OpenPGP digital signature
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[Alec Muffett]

On 27 April 2018 at 19:48, Damian Johnson  wrote:

> > OnionBalance requires STEM support for V3
>
> Hi Alec, would you mind clarifying what you need from Stem? As far as
> I'm aware Stem supports v3 onion service creation...
> ...
> I'm unaware of the ball being in my court on any v3 Onion Service
> stuff. If there's something I should have on my radar then please let
> me know!


Hi Damian!

That's awesome, and good to know - I first wrote that text a few months ago
(on the basis of David's comments in that ticket) and haven't revised it
since, so I am heartened to see progress.

However I am also not the best person to say what else will be needed,
because that would probably be Donncha re: the future of OnionBalance for
v3.

At the moment in OnionBalance the v2 Introduction Points of the
(predetermined) worker onions are passively scraped from the HSDir; the
descriptors are parsed, the IPs are blended and re-combined and re-signed
under the master onion key (eg: nytimes3xbfgragh) and then published back
to the HSDirs, with the result that NewYorkTimes-browsing clients end up
communicating with 1 of N possible "worker" daemons, thereby sharing the
load.

My understanding - and please jump in, if I am wrong - is that the
synthesis of a v3 descriptor which blends the introduction points of
several independent v3 "worker" Tor daemons will be a more complex affair
than the existing process, because (?) the "worker" tor daemons will
somehow have to be more actively involved - apparently they may have to
sign the v3 Introduction Points themselves, though I am not sure how that
will work for a blended descriptor? Multiple/distinct signatures, perhaps?
The last opportunity I had to speak with anyone (re: this) was more than 1
year ago, so I am rusty, and I apologise if I have gotten some details
wrong.

So: OnionBalance relies heavily upon Stem (
https://github.com/DonnchaC/onionbalance/tree/develop/onionbalance) and I
am not qualified to say what, if any, additional v3 Stem features will be
useful or outstanding to support the descriptor-blending that is needed for
loadbalanced configurations.

But OnionBalance for v3 will certainly be necessary. :-)

-a

-- 
http://dropsafe.crypticide.com/aboutalecm
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[grarpamp]

On Fri, Apr 27, 2018 at 8:01 AM, Alec Muffett  wrote:
> OnionBalance

Forgot to include this in the former list of
common / useful onion tools, thanks.

If anyone knows of others, feel free to add to thread.

> OTOH, I have been performance testing simultaneous regular-expression
> matching of v2/3 addresses, and so far this is the winner:
>
>   "\\b([a-z2-7]{16}(?:[a-z2-7]{40})?\\.onion)\\b"

Did you post the results of the various RE engines and RE's somewhere?
Some of the onion services out there might find it useful in their backend work.

> ...and it's already in the codebase at
> https://github.com/alecmuffett/eotk/blob/master/templates.d/nginx.conf.txt#L299
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[Damian Johnson]

> OnionBalance requires STEM support for V3

Hi Alec, would you mind clarifying what you need from Stem? As far as
I'm aware Stem supports v3 onion service creation...

https://trac.torproject.org/projects/tor/ticket/25124

See the 'version 3' note at the end of...

https://stem.torproject.org/api/control.html#stem.control.Controller.create_ephemeral_hidden_service

I'm unaware of the ball being in my court on any v3 Onion Service
stuff. If there's something I should have on my radar then please let
me know!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[George Kadianakis]

Jonathan Marquardt  writes:

> On Wed, Apr 25, 2018 at 04:58:36PM -0400, grarpamp wrote:
>> In onionland, there seems to be little knowledge of v3, thus little worry
>> about v2 in cases where v3 would actually apply to benefit, that's bad.
>
> v3 onion services just seem like a way worse deal to the average user and 
> the unknowledgeable admin. Mainly because the addresses are way too long. I 
> can remember a couple of v2 addresses, but not a single v3 address. So that's 
> just bad advertising from the start.
>

IMO if you have the ability to memorize v2 addresses by heart, you are
already not an average user. Average users just google most things they
try to visit.

That said, I do share your concerns, and that's why I mentioned that
finding a solution to the onion name issue is a priority before v3 can
go mainstream (or even v2).

> Before at least Facebook, DuckDuckGo, The New York Times, the Debian Project 
> and even the Tor Project themselves (!) have rolled out their v3 onion 
> services, one shouldn't even think about deprecating HSv2. It's going to be 
> around for many years to come, taking for them just as long to vanish as an 
> SSL version, I think, unfortunately.

Agreed. We are indeed a long way from deprecating HSv2 :)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[Alec Muffett]

It's not just about getting the protocol stack right, but also the
ancillary software environment; people keep asking me for "V3 support in
EOTK" and my stock response is this:

 BEGIN 
OnionBalance requires STEM support for V3, before it can be updated
(possibly a substantial rewrite will be needed) to support the new format
onions. It's not only a matter of "longer addresses" but also a matter of
cross-signing the descriptors to support new-style cryptography, so in fact
it might be safest to create a new, separate OnionBalance for V3.

So: STEM needs updating and testing for V3, and then OnionBalance needs to
support the new STEM library and encryption. Then (for me) EOTK needs to
support the new OnionBalance.

I am not expecting a solution to ship until 2019, earliest.
 END 

...and that's even without refactoring the other bits of EOTK to address
the changes when STEMv3 lands.


OTOH, I have been performance testing simultaneous regular-expression
matching of v2/3 addresses, and so far this is the winner:

  "\\b([a-z2-7]{16}(?:[a-z2-7]{40})?\\.onion)\\b"

...and it's already in the codebase at
https://github.com/alecmuffett/eotk/blob/master/templates.d/nginx.conf.txt#L299

- alec :-)

-- 
http://dropsafe.crypticide.com/aboutalecm
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[Jonathan Marquardt]

On Wed, Apr 25, 2018 at 04:58:36PM -0400, grarpamp wrote:
> In onionland, there seems to be little knowledge of v3, thus little worry
> about v2 in cases where v3 would actually apply to benefit, that's bad.

v3 onion services just seem like a way worse deal to the average user and 
the unknowledgeable admin. Mainly because the addresses are way too long. I 
can remember a couple of v2 addresses, but not a single v3 address. So that's 
just bad advertising from the start.

Before at least Facebook, DuckDuckGo, The New York Times, the Debian Project 
and even the Tor Project themselves (!) have rolled out their v3 onion 
services, one shouldn't even think about deprecating HSv2. It's going to be 
around for many years to come, taking for them just as long to vanish as an 
SSL version, I think, unfortunately.
--
OpenPGP Key: 47BC7DE83D462E8BED18AA861224DBD299A4F5F3
 https://www.parckwart.de/pgp_key


signature.asc
Description: PGP signature
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[George Kadianakis]

nusenu  writes:

> Hi,
>
> even though you are probably years away from deprecating onion v2 services
> it is certainly good to have a clear plan.
>
> I'm asking because the sooner onion v2 are deprecated the sooner some 
> people can stop worrying about malicious HSDirs.
>

Yes indeed. The sooner we deprecate v2 the sooner we can stop worrying
about malicious HSDirs. And also we will be able to reduce the
requirements for becoming an HSDir which will strengthen and make our
network more robust.

That said, I think we are unfortunately still far from deprecating v2
onions:

The first actual step to v2 deprecation, is to make v3 the default
version.  But to get there, we first need to solve various bugs and
issues with the current v3 system (#25552, #22893, #23662, #24977,
etc.).  We also need to implement various needed features, like offline
keys (#18098), client-authorization (#20700 ; WIP 
https://github.com/torproject/tor/pull/36),
control port commands like HSFETCH (#25417) and revive onionbalance for
v3.  We might also want to consider possible improvements to the UX of
long onion names (like #24310) 
(https://blog.torproject.org/cooking-onions-names-your-onions).

After we do most of the above, we can turn the switch to make v3 the
default, and then we need to wait some time for most of the users to
migrate from v2 to v3. After that we can initiate the countdown, and
eventually deprecate v2s for real.

It's hard to provide an actual timeline for the above right
now. However, we are currently applying for some onion-service-related
grants, and hopefully if we get them we will have the funding to
accelerate the development pace.

Cheers!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[Damian Johnson]

Hi nusenu, thanks for bringing this up! Filed tickets for a couple
things we should sort out before deprecating v2...

https://trac.torproject.org/projects/tor/ticket/25918
https://trac.torproject.org/projects/tor/ticket/25920
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[grarpamp]

On Wed, Apr 25, 2018 at 2:22 PM, nusenu  wrote:
> even though you are probably years away from deprecating onion v2 services
> it is certainly good to have a clear plan.
>
> I'm asking because the sooner onion v2 are deprecated the sooner some
> people can stop worrying about malicious HSDirs.

The publisher of the onion is the one who decides which to use
and what level of concern / tech applies to their use case.
In onionland, there seems to be little knowledge of v3, thus little worry
about v2 in cases where v3 would actually apply to benefit, that's bad.
And mooting of v3 in cases where it doesn't much benefit use case.
Rather than removing v2 support from the code [1]...
- the risk / benefit / tradeoffs / ux / uses of v2 vs v3 should be widely
publicized to educate about v3.
- common tools and applications such as ricochet, onionshare,
onioncat, onionvpn, bittorrent, securedrop, control, stem, nyx, etc
should add v3 support, thus also feeding back into education.
- some future release should change the default
documentation and operation inflection from v2 to v3.
At that point if a user goes to create an onion, everything
should lead to and result in them having created a v3,
other than a standalone v2 reference section in manpage on how
to create v2 onions, and continue v2 dirservices support, etc [1].

[1] v2 does have legitimate long term use cases exists,
there's a ticket opened for extended nonremoval support of v2.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev