Re: [tor-dev] Release new version of stem

[Damian Johnson]

> > That is tricky as stem is not maintained anymore and therefore
> > deprecated.
>
> That is a pitty. It would be nice if you can actually make this obvious on the
> git repo like "currently stem is unmaintained and therefore deprecated"

Hiya. Just a minor terminology correction: Stem is unmaintained but
not deprecated. The later implies that there's a replacement. There's
other options [1] but for most of Stem's features it's the only game
in town.

> maybe search for people in the community to take over - maybe ask people from
> onionshare?

If someone's serious about it we can talk but properly maintaining
Stem is a lot of work [2][3].

> Especially as stem is still marked as the way to interact with tor via python.

As far as I'm aware it is.

Cheers! -Damian

[1] 
https://stem.torproject.org/faq.html#are-there-any-other-controller-libraries
[2] https://github.com/torproject/stem/issues/97
[3] https://github.com/torproject/stem/issues/77
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Can "ExcludeNodes" be used multiple times in torrc?

[Damian Johnson]

Hi nusenu, tor's manual says [1]...

"To split one configuration entry into multiple lines, use a single
backslash character (\) before the end of the line. Comments can be
used in such multiline entries, but they must start at the beginning
of a line."

I just double checked and it works...


torrc


ControlPort 9051

ExcludeNodes \
  # nickname: moria1
  9695DFC35FFEB861329B9F1AB04C46397020CE31, \
  # nickname: tor26
  847B1F850344D7876491A54892F904934E4EB85D



% cat demo.py
from stem.control import Controller

with Controller.from_port() as controller:
  controller.authenticate()
  print(controller.get_conf('ExcludeNodes'))

% python demo.py
9695DFC35FFEB861329B9F1AB04C46397020CE31,847B1F850344D7876491A54892F904934E4EB85D



[1] 
https://2019.www.torproject.org/docs/tor-manual.html.en#_the_configuration_file_format
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] How do Ed25519 relay IDs look like?

[Damian Johnson]

> I hope we can agree to use the same format in all places.

Thanks nusenu, that's a great summary. Honestly I doubt that
deprecating RSA keys is on anyone's visible horizon, and by extension
RSA-based fingerprints will remain our canonical identifiers for the
foreseeable future.

That leaves our present default position as "ed25519 public identity
keys are a base64 encoded descriptor field that has no relationship to
fingerprints, but might become the basis for them in the future".

That said, I'm happy to discuss this topic further if Nick or the
Network team would like to do so.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] How do Ed25519 relay IDs look like?

[Damian Johnson]

> Isn't using "fingerprint" not a bit misleading since it is not the output of
> a hash function but the ed25519 master public key itself?

Hi nusenu, that's fair. We've begun to conflate a couple concepts here...

* Relay operators, controllers, DirPorts, etc all require a canonical
relay identifier. They don't care how it's derived as long as it's
unique to the relay.

* Relays publish a public ed25519 key. This is an implementation
detail that isn't of interest to the above populations.

I'd advise against attempting to rename "fingerprint". That hasn't
gone well for hidden services [1]. But with that aside, relay
identifiers and the representation of ed25519 public keys don't
necessarily need to be one and the same.

[1] https://trac.torproject.org/projects/tor/ticket/25918
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] How do Ed25519 relay IDs look like?

[Damian Johnson]

> The way I understand it is this: Relay fingerprints are based on the
> RSA key, which will go away eventually. The canonical identifier will
> be the identity. We should start that transition

Thanks Sebastian. In that case we should put more thought into this
because fingerprints are foundational to our control and directory
specifications. Commands, events, descriptors... really everything
reference relays by fingerprint (or optionally sometimes nickname).
Migrating to a new identifier is no small task.

First, I'd advise that we call these 'v2 fingerprints' so it's clear
that we intend to substitute these anywhere traditional fingerprints
are used.

Second, I would advise against truncated base64 identifiers.
Fingerprints are 40 character hex. master-key-ed25519's base64 value
can include slashes (such as
"yp0fwtp4aa/VMyZJGz8vN7Km3zYet1YBZwqZEk1CwHI") which will be
problematic for DirPort urls, GETINFO commands, etc.

The simplest solution would be to simply hexify these values. This
will raise our fingerprint length from 40 to 64 characters which will
slightly impact DirPorts [1], but otherwise I don't anticipate a
problem with such a replacement.



import base64


def hexify_id(ed25519_identifier):
  binary_id = base64.b64decode(ed25519_identifier +
((len(ed25519_identifier) % 4) * '='))
  return bytes.hex(binary_id).upper()


identifier = 'yp0fwtp4aa/VMyZJGz8vN7Km3zYet1YBZwqZEk1CwHI'
print('the hex id of "%s" is "%s"' % (identifier, hexify_id(identifier)))



% python3.7 demo.py
the hex id of "yp0fwtp4aa/VMyZJGz8vN7Km3zYet1YBZwqZEk1CwHI" is
"CA9D1FC2DA7869AFD53326491B3F2F37B2A6DF361EB75601670A99124D42C072"



Cheers! -Damian

[1] At most 96 server or extrainfo descriptors can be downloaded from
DirPorts via their fingerprint due to a limitation on the url length
by squid proxies...

https://gitweb.torproject.org/stem.git/commit/?id=871a957f

Maybe this is no longer relevant? If it is then raising the
fingerprint length from 40 to 64 will reduce this maximum to 60 (which
seems fine to me).
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] How do Ed25519 relay IDs look like?

[Damian Johnson]

> I was wondering why the base64 string is 43 characters long for a 32byte 
> Ed25519 key.
> 32*8/6=42

That is because tor drops trailing '=' from base64 encoded values
within descriptors. Some fields indicate this within the spec, others
don't.

https://gitweb.torproject.org/stem.git/tree/stem/util/str_tools.py#n98

> I'd like to use "ed25519 identity" or even just "identity" here going
> forward.

Gotcha. The name of 'identity' makes me wonder how this relates to
relay fingerprints, which are the canonical identifier we use.

Regardless, the more we can standardize the terminology we use the
less confusing these fields will be.

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] So Windows is Adding a Package Manager.....

[Damian Johnson]

Welcome Microsoft, to the cutting edge of the 90s! ;P

Kidding aside this could be a great improvement. I'm rooting for them. This
is constrained to Windows installers (exe or msi) but hopefully they expand
this in the future. Distributing interpreted applications (python, ruby,
etc) have always been a PITA. If they make a first rate package manager
like apt or yum this could dramatically improve distributability to their
platform.

Thanks for pointing this out. We distribute Tor Browser as an exe so that
certainly could be a good candidate.


On Sun, May 24, 2020 at 3:55 PM Keifer Bly  wrote:

>
>
> Hi all,
>
>
>
> So based on this article here,
>
>
>
> https://docs.microsoft.com/en-us/windows/package-manager/
>
>
>
> Windows is getting a package manager in the near future,  including
> winget, a Linux like tool that can install and manage packages. With this
> in might  it be possible to add tor as a manageable package for Windows
> (for auto upgrades, install via this method, etc.)?
>
>
>
> Thx.
>
>
>
>
>
> Sent from Mail  for
> Windows 10
>
>
>
>
> --
> [image: Avast logo] 
>
> This email has been checked for viruses by Avast antivirus software.
> www.avast.com 
>
> <#m_7575364686860156035_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Stem and Nyx bug tracking moved

[Damian Johnson]

Hi all. Stem and Nyx's issue tracking has moved from Trac to GitHub.
In the future please file tickets at...

https://github.com/torproject/stem/issues/
https://github.com/torproject/nyx/issues/

Thanks!

PS. All existing tickets have moved. These trac components will be
disabled to cut down on confusion...

  https://trac.torproject.org/projects/tor/ticket/32832
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Raising exceptions in add_event_listener() threads (was Re: HSv3 descriptor work in stem)

[Damian Johnson]

Thanks George, this is a great question! I've expanded our tutorial to
hopefully cover this a bit better...

https://stem.torproject.org/tutorials/tortoise_and_the_hare.html#advanced-listeners

You can trivially print exceptions within your listener if that is all
you care about...



import time
import traceback

from stem.control import EventType, Controller


def broken_handler(event):
  try:
raise ValueError('boom')
  except:
print(traceback.format_exc())

with Controller.from_port() as controller:
  controller.authenticate()
  controller.add_event_listener(broken_handler, EventType.BW)
  time.sleep(2)



% python demo.py
Traceback (most recent call last):
  File "demo.py", line 9, in broken_handler
raise ValueError('boom')
ValueError: boom

Traceback (most recent call last):
  File "demo.py", line 9, in broken_handler
raise ValueError('boom')
ValueError: boom



... but if your event handler has grown sophisticated enough to make
this a significant issue I'd suggest a producer/consumer model as the
tutorial now demonstrates.

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] New python Tor client implementation

[Damian Johnson]

> 1. A number of python implementations I've found on the internet had
> broken dependencies or didn't work at all. So, I decided to create a
> fully functional Tor client with the bare minimum of dependencies.

Ah! Gotcha. For what it's worth I'm in complete agreement with you on
that. I designed Stem to avoid any hard dependencies [1] for the exact
same reason.

> 2. Up until now I thought Stem was just a library to control the Tor
> process. As it said at front of https://stem.torproject.org:
> "With it you can use Tor's control protocol to script against the Tor
> process". Unfortunately API docs didn't help me to see that certain Tor
> primitives were indeed implemented.

Makes sense. Stem's client functionality is relatively new. I
implemented enough to download descriptors [2], but there's still a
lot of room to grow [3].

Cheers! -Damian

[1] https://stem.torproject.org/faq.html#does-stem-have-any-dependencies
[2] https://gitweb.torproject.org/stem.git/commit/?id=820881a
[3] 
https://2019.www.torproject.org/getinvolved/volunteer.html.en#pythonTorClient
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] New python Tor client implementation

[Damian Johnson]

Thanks James, this is awesome! Very well done. Out of curiosity why
sidestep any use of Stem? It supports encoding/decoding most tor cell
types...

Yours: https://github.com/torpyorg/torpy/blob/master/torpy/cells.py
Mine: https://gitweb.torproject.org/stem.git/tree/stem/client/cell.py

If you'd care to integrate any of this functionality into Stem I'd be
delighted to work with you.

Cheers! -Damian

On Thu, Oct 17, 2019 at 7:05 AM James Brown  wrote:
>
> Hello all,
>
> Recently I finished a pure python implementation of the Tor client. It's 
> called torpy (https://github.com/torpyorg/torpy).
>
> It offers handy API, supports v2 hidden services with "basic" and "stealth" 
> authorization protocol. Works with python 3.6+.
> It has no dependencies on the original C Tor client and Stem. For more 
> information please take a look at README on github.
>
> Here is a quick example of how to use the library:
> ```python
> from torpy import TorClient
>
> hostname = 'ifconfig.me' # or onion-services as well, for example 
> 'http://facebookcorewwwi.onion'
> tor = TorClient()
>
> # Choose random guard node and create 3-hops circuit
> with tor.create_circuit(3) as circuit:
># Create tor stream to host
>with circuit.create_stream((hostname, 80)) as stream:
># Now we can communicate with host
>stream.send(b'GET / HTTP/1.0\r\nHost: %s\r\n\r\n' % hostname.encode())
>recv = stream.recv(1024)
> ```
>
> Please list torpy project at 
> https://trac.torproject.org/projects/tor/wiki/doc/ListOfTorImplementations
>
> It would be nice if you try the client. I look forward to any feedback.
>
> Cheers.
>
>
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] HSv3 descriptor work in stem

[Damian Johnson]

Thanks George! Yup, work on that branch is in progress:

https://gitweb.torproject.org/user/atagar/stem.git/log/?h=hsv3

On Thu, Oct 17, 2019 at 8:01 AM George Kadianakis  wrote:
>
> Damian Johnson  writes:
>
> >>Can I use `_descriptor_content()` to do that? Or should I call
> >>`_descriptor_content()` to generate the whole thing _without_ the
> >>sig, and then do the signature computation on its result and
> >>concatenate it after?
> >
> > Hi George. Yup, to create a signed descriptor we create the bulk of
> > the content then append the signature. Server and extrainfo
> > descriptors already do this so I suspect you can do something
> > similar...
> >
> > https://gitweb.torproject.org/stem.git/tree/stem/descriptor/server_descriptor.py#n902
> > https://gitweb.torproject.org/stem.git/tree/stem/descriptor/__init__.py#n1388
> >
> > Will this do the trick?
> >
> > PS. Sorry about the duplicate. Hit reply rather than reply-all
> > forgetting that you included the list.
>
> Thanks for the reply Damian! That was super useful!
>
> The current state of affairs can be found here: 
> https://trac.torproject.org/projects/tor/ticket/31823#comment:1
> (just in case you didn't check IRC that day)
>
> peace
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] HSv3 descriptor work in stem

[Damian Johnson]

>Can I use `_descriptor_content()` to do that? Or should I call
>`_descriptor_content()` to generate the whole thing _without_ the
>sig, and then do the signature computation on its result and
>concatenate it after?

Hi George. Yup, to create a signed descriptor we create the bulk of
the content then append the signature. Server and extrainfo
descriptors already do this so I suspect you can do something
similar...

https://gitweb.torproject.org/stem.git/tree/stem/descriptor/server_descriptor.py#n902
https://gitweb.torproject.org/stem.git/tree/stem/descriptor/__init__.py#n1388

Will this do the trick?

PS. Sorry about the duplicate. Hit reply rather than reply-all
forgetting that you included the list.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] (no subject)

[Damian Johnson]

Further posts from vignesh will require moderator approval. Sorry
folks for the noise. :)

On Thu, Sep 26, 2019 at 12:01 AM vignesh kannan
 wrote:
>
> Please sent me a video that how to get dark web please
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] TBB Memory Allocator choice fingerprint implications

[Damian Johnson]

> If someone is going to criticize other people's work and dismiss it as
> nearly useless without even somewhat informing themselves about it,
> they should expect to be called out on that. You might find my reply
> offensive, but I found the email that I was replying to extremely
> offensive and I had to subscribe to this list and figure out how to
> send a reply to a past email from the archive to defend the value of
> my work. It wasn't fair or accurate criticisms or comparisons.

Hi Daniel. Further posts from you will require moderator approval.

Personally I'm baffled what made you so upset. From what I can tell
Tom didn't even claim harden_malloc is a bad project, just that
benchmarks should be redone with Firefox's jemalloc forking in mind
and that he doesn't expect benefits from integrating it via LD_PRELOAD
for our use case (though to be honest this thread got too wonky for me
- not my specialty).

Regardless, contentious technical discussions on this list are fine
but personal attacks are not.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Hack

[Damian Johnson]

Hi Pordeshi, this is a list for development related discussions
regarding tor. Your question is both off topic and... um... illegal.
Further posts from you will require moderator approval.


On Fri, Jul 26, 2019 at 7:18 PM Pordeshi Pothik
 wrote:
>
> Hi,I want to learn how to hack facebook.
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] The Tor project as a GSoC voucher for Whonix?

[Damian Johnson]

Thanks teor! Quite side note: it would be helpful if such questions
are broached earlier in the process. Google has been calling for org
applications for several weeks now. Asking roughly a day before the
deadline creates last minute confusion and reduces the chances of an
affirmative response.

On Tue, Feb 5, 2019 at 8:06 AM teor  wrote:
>
> Dear Whonix Community,
>
> On February 5, 2019 2:20:18 PM UTC, iry  wrote:
> >
> >
> >teor:
> >> Dear Whonix Community,
> >>
> >> On February 4, 2019 11:52:44 PM UTC, iry  wrote:
> >> Dear Tor Developers,
> >>
> >> Whonix is applying to be a Google Summer of Code organization this
> >> year. I am writing on behalf of Whonix to ask if the Tor project
> >> could be a voucher for Whonix. Specifically, in the application
> >> form, it asks:
> >>
> > If you are a new organization to GSoC, is there a Google
> > employee or previously participating organization who will
> > vouch for you? If so, please enter their name, contact email,
> > and relationship to your organization.
> >>
> >> Whonix community would be really appreciated if Tor can be our
> >> voucher this year. And please feel free to contact me to provide
> >> the information described above anytime before the deadline
> >> (February 6, 2019 at 20:00 UTC).
> >>
> >> Thank you very much!
> >>
> >> Cheers, iry
> >>
> >> I have forwarded your request to the Tor Core Contributors.
> >>
> >> Some of us were at a hackfest and FOSDEM last week. A few are still
> >> working or travelling. So I am not sure how many of us will check
> >> our emails in the next 36 hours.
> >>
> >> I hope we will have a response for you around 24 hours from now.
> >>
> >
> >Thank you so much for your help, teor!
> >
> >We really appreciate it!
>
> Tor won't be able to vouch for Whonix for GSoC 2019. I understand that you 
> are on a tight deadline, so I wanted to let you know as soon as possible.
>
> I have received a range of responses from Tor Core Contributors over the past 
> few hours. A number of people raised concerns about the Whonix community's 
> culture, and whether it would be a good experience for students.
>
> I hope that we can give a more detailed and helpful response later. But it 
> may take us some time, because we are still working through our internal 
> community processes.
>
> T
>
> --
> teor
> --
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Archiving old Components in Trac

[Damian Johnson]

Hi teor, 'Core Tor/Erebus' is mine. Feel free to drop it. That's the
name of a prospective web UI for relays but I won't be getting to it
in the near future.

No need to archive it. It doesn't have any tickets. We can always
re-add it if I cobble together time to make it a thing.

Thanks for cleaning these up!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Bine - Go library for using and embedding Tor into applications

[Damian Johnson]

Thanks Chad, great work! Added Bine to our list of controller libraries...

https://stem.torproject.org/faq.html#are-there-any-other-controller-libraries


On Fri, May 18, 2018 at 9:56 AM, Chad Retz  wrote:
> I posted it on /r/TOR but figured I'd post it here too:
> https://github.com/cretz/bine
>
> I built it because I have plans to use it in another project. In
> general it does what Stem does. I did incorporate V3 onion service
> generation in there and am happy to say it publishes much faster.
> Also, I have the option to statically compile Tor using the fairly new
> embedding API (i.e. tor_api.h). I've only tested it on Windows (see
> https://github.com/cretz/tor-static) but I figure that's the hardest
> of them to get working anyways. Thoughts/questions welcome.
>
> Chad
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[Damian Johnson]

> OnionBalance requires STEM support for V3

Hi Alec, would you mind clarifying what you need from Stem? As far as
I'm aware Stem supports v3 onion service creation...

https://trac.torproject.org/projects/tor/ticket/25124

See the 'version 3' note at the end of...

https://stem.torproject.org/api/control.html#stem.control.Controller.create_ephemeral_hidden_service

I'm unaware of the ball being in my court on any v3 Onion Service
stuff. If there's something I should have on my radar then please let
me know!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Standardizing the 'onion service' name

[Damian Johnson]

Hi all, teor suggested engaging the list with #25918 so here we go!
Ticket #25918 has a couple goals...

1. Provide a tracking ticket for the rename effort.
2. Come to a consensus on if we should move forward with "onion
service" or revert back to "hidden service". The limbo we've been in
for months is confusing for our users and we should standardize on a
name.

Here's the ticket...



A recent post on tor-dev@ just got me thinking about the roadblocks we
have for v2 deprecation. There's a couple I don't believe we're
following in trac so lets fix that.

For me the biggest is its name. Renaming takes work, and we attempted
to rename hidden services in v3 without investing the time to make it
happen. We should either fix that or revert to to the old name. To
move forward we need to...

Have OnionService aliases for controller commands, events, descriptor
fields, and anything else referencing 'HS' or 'HiddenService'.

Speaking of which, how do we plan to replace abbreviations? Having an
'OSFETCH' or 'OS_CREATED' event doesn't exactly have the same ring as
their HS counterparts. ;P

Adjust all our docs to be consistent about the name.

Renaming takes work. Lesson I learned from Nyx is that it works best
if you draw a line in the sand and stand by it. With Nyx, version 2.0
is called Nyx (you won't find any docs saying otherwise) and version
1.x is the legacy 'arm' project.

If I was in your shoes I'd opt for the same. Either prioritize the
aliases and be firm that v3 are 'Onion Services' or abort the rename.
Otherwise this will live in a confusing dual-named limbo land
indefinitely. ;P

Cheers! -Damian

PS. Stem and Nyx have stuck with the old name ("hidden services") and
will continue to do so until tor's standardized this.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] onion v2 deprecation plan?

[Damian Johnson]

Hi nusenu, thanks for bringing this up! Filed tickets for a couple
things we should sort out before deprecating v2...

https://trac.torproject.org/projects/tor/ticket/25918
https://trac.torproject.org/projects/tor/ticket/25920
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Notes from 12 April 2018 Simple Bandwidth Scanner Meeting

[Damian Johnson]

> Good question!
>
> While I didn't necessarily work on sbws while at my place of work, I
> couldn't rationalize that it is unrelated to my day job. Thus I need to
> get permission from my employer in order to release sbws.
>
> I've already submitted that paperwork and expect to get it back in about
> 2 weeks.
>
> It is licensed under CC0 and is therefore in the public domain (or will
> be... depending on how you want to interpret the situation).
>
> Fun side note: little-t-tor itself had to go through the same process.
>
> Sorry for the inconvenience.

A! Mystery solved. Makes complete sense - thanks Matt!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Notes from 12 April 2018 Simple Bandwidth Scanner Meeting

[Damian Johnson]

> https://github.com/pastly/simple-bw-scanner/blob/master/docs/source/specification.rst
> (ask Pastly for access)

Hi Matt, why is this repo read restricted? I was idly curious to see
the code of sbws and was surprised it's effectively closed source.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] DocTor Check for <3 bw auth votes available

[Damian Johnson]

> I get the feeling that I'm annoying you so I'll try to stop that by avoiding 
> any further
> such DocTor requests.

Hi nusenu, I'm not annoyed in the least - sorry if I gave that impression!

My read is that you asked for a notification when bandwidth
authorities are missing, which we already have. It's a simple
messaging change to say "we have X of Y" and if you'd care to send
that patch I'd be happy to merge it.

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] DocTor Check for <3 bw auth votes available

[Damian Johnson]

Hi nusenu, the notice pretty clearly says that one isn't present...

"NOTICE: The following directory authorities are not reporting
bandwidth scanner results: gabelmoo"
https://lists.torproject.org/pipermail/tor-consensus-health/2018-February/008624.html

You're right that it doesn't tell you the number of authorities that
vote. If you'd care to add that to the message then feel free.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] DocTor Check for <3 bw auth votes available

[Damian Johnson]

Hi nusenu, hi teor. We already have a check that we have the expected
bandwidth authorities...

https://gitweb.torproject.org/doctor.git/tree/consensus_health_checker.py#n541

... and that they're reasonably in sync...

https://gitweb.torproject.org/doctor.git/tree/consensus_health_checker.py#n744
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] monitoring significant drops of flags in dirauth votes

[Damian Johnson]

> It might be smartest to just put in an exception for moria1's
> HSDir votes, since we know it's being different.

Suppressed any notices for HSDir flags. Also fixed the time based
suppression for the check (it should have sent one notice a day rather
than one an hour).
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] [prop-meeting] [prop#285] "Directory documents should be standardized as UTF-8"

[Damian Johnson]

> For the metrics tools there are some guidelines on this we can follow:
> https://docs.oracle.com/javase/tutorial/i18n/text/design.html. The other
> language would be Python (for stem), but Python developers have probably
> got a good understanding of unicode/str/bytes by now. (In Python 3: when
> using UTF-8, BOM will not be stripped and will be interpreted as data,
> and you can have a NUL in a str).

Hi Iain. Actually, for Stem I'm really looking forward to this too.
Stem has special handling for the contact and platform fields (iirc
the only spot non-ascii content can presently appear). Stem's parsers
and API will be simplified once everything is uniformly utf-8. :P

Possibly a stupid question but any reason not to require the whole
descriptor document to be printable characters?
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] monitoring significant drops of flags in dirauth votes

[Damian Johnson]

> thanks for implementing the new check so fast.

No problem! Thanks for suggesting it.

> This is also very useful but slightly different from what I had in mind,
> because it would not trigger if dirauths upgrade from A to B in the
> same hour and most exits, guards or hsdirs are gone due to a bug in version B.

This should catch a bug with B unless every authority upgrades to B in
the same hour. Otherwise we'd get an alert - either because the
majority is B and the remaining A votes are out of band, or the
consensus is made with A and authorities that upgraded to B are
different.

Is there another check in particular that you'd like? One gotcha is
that checks that require state (such as comparing with the last hour's
consensus) is a bit more work.

> I tried to find something related to this in the 0.3.3.x changelogs
> because moria1 (the affected dirauth) is the only one running tor alpha
> but I didn't find anything related to a change in what is required
> to earn the HSDir flag. Has there been any change related to how
> HSDir is assigned that would explain that significant difference?

For what it's worth I started with alarming when authorities differed
more than 20% from the consensus but it was a bit noisier...

[consensus-health] NOTICE: longclaw had 3100 HSDir flags in its vote
but the consensus had 2583
[consensus-health] NOTICE: moria1 had 756 HSDir flags in its vote but
the consensus had 2583
[consensus-health] NOTICE: moria1 had 1397 Guard flags in its vote but
the consensus had 1761
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] monitoring significant drops of flags in dirauth votes

[Damian Johnson]

Thanks nusenu! Nice idea, added it to DocTor...

https://gitweb.torproject.org/doctor.git/commit/?id=8945013

It gives a notice if flags issued by an authority are 50% different
from the conensus. Presently there's only one instance of that...

[consensus-health] NOTICE: moria1 had 756 HSDir flags in its vote but
the consensus had 2583

On Sun, Feb 11, 2018 at 1:21 AM, nusenu  wrote:
> Hi Damian and Tom,
>
> Roger discovered that dannenberg did not include any exit flags
> in certain votes anymore [1].
>
> It would be great if we would detect and notify about such events in the 
> future.
>
> I see two places where this could be added:
>
> DocTor:
> a new check that alerts on events where a certain dir auth does
> either no longer include certain flags (guard, hsdir, exit, ..) at all
> or better: if the amount of relays with a certain flag significantly dropped
> by xx % from one vote to the next.
>
> consenus-health graphs:
> we have nice graphs per dirauth and bwauth, if we would have
> per-dirauthvote-per-flag (mainly guard, exit, hsdir - we have already running)
> graphs as well we could spot such events (and even trends)
> better. (btw: what caused there recent flat-line in graphs on 2018-02-03 - 
> 2018-02-05)
>
> What do you think?
>
>
> thanks for considering it,
> nusenu
>
>
> [1] 
> https://lists.torproject.org/pipermail/tor-relays/2018-February/014480.html
>
>
> --
> https://mastodon.social/@nusenu
> twitter: @nusenu_
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Stem ORPort protocol support

[Damian Johnson]

> So, there is this already -- not sure how "complete" it is though (and
> looks like hasn't seen commits for 2+ years) but might have useful code:

Thanks meejah! Took a peek but they both look pretty old and it's
unclear to me how complete either got. If there's something in
particular you think is worthwhile integrating I'm all ears.

> For any of these efforts, writing a "Tor protocol library" that
> *doesn't* do any I/O would be the most useful; then other Python tools
> can benefit from the protocol support without being tied to "threads" or
> to a particular async framework.

Actually, this uses a similar pattern as the rest of Stem in that cell
packing/unpacking is independent of the threaded socket. Just as you
could use stem.response for controller message parsing in txtorcon,
you can use stem.client.cell for cell packing/unpacking with a twisted
application too. That said, thanks for the reminder to keep this in
mind as we go.

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Stem ORPort protocol support

[Damian Johnson]

Hi all. Over the last few months Tim and I have been collaborating on
Python support for the ORPort protocol. With it you can download
descriptors without a DirPort, and possibly fancier things in the
future like full circuit construction.

Tim put together a wonderful proof of concept called Endosome...

  https://github.com/teor2345/endosome

... and I just finished integrating it into Stem...

  https://gitweb.torproject.org/stem.git/tree/stem/client/__init__.py

With stem.client you can now download descriptors...

  import stem.client

  with stem.client.Relay.connect('127.0.0.1', 12345, [3]) as relay:
circ = relay.create_circuit()
circ.send('RELAY_BEGIN_DIR', stream_id = 1)
desc = circ.send('RELAY_DATA', 'GET /tor/server/authority
HTTP/1.0\r\n\r\n', stream_id = 1).data
circ.close()

print(desc)

When run this looks like...

  % python demo.py
  HTTP/1.0 200 OK
  Date: Wed, 07 Feb 2018 18:42:41 GMT
  Content-Type: text/plain
  Content-Encoding: identity
  Expires: Fri, 09 Feb 2018 18:42:41 GMT

  router Unnamed 97.113.177.53 12345 0 23456
  identity-ed25519
  -BEGIN ED25519 CERT-
  AQQABm/qAazUltT1iUUbIMw8VNNhGb50FDHKJz6S94FLQNxL0LObAQAgBAAapbO9
  iLFD0l9SEiEMFQWIT2VnbLyCZKvbrxTs5ULC1l1hQPoui6Y/lEd3yjrQhIs/vl6R
  1S6FbwSFDmiXOzq47mFrse4C71ht3TpLOD0F3wiyjWtsqU1k7iPmmpejUgs=
  -END ED25519 CERT-
  master-key-ed25519 GqWzvYixQ9JfUhIhDBUFiE

I'd like to emphasize this is still very alpha. The API isn't set in
stone and there's no doubt quite a few rough edges. However, I wanted
the list to be aware just in case anyone would care to build on it. I
plan to draw a line at 'download descriptors through ORPorts' but I'd
be delighted to help others if there's more ambitious directions
they'd care to go (potentially all the way up to a Python Tor client,
similar to Orchid).

Now that we've reached this milestone I'm taking a break to focus on
Stem support for v3 Onion Services for a bit. However, when I come
back the next things on my dance card are...

  a. Support ORPort downloads in the stem.descriptor.remote module.
  b. More integ tests so Stem can be used as a tool for testing tor's ORPort.
  c. Give more thought to the API we'd like to vend.
  d. Brainstorm a GSoC project idea that expands these capabilities.

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] stem support for v3 ephemeral onion services

[Damian Johnson]

Great! Thanks David. Juggling some other things but I'll try to give
this a pass in the next week or two.


On Wed, Jan 31, 2018 at 12:56 PM, David Goulet  wrote:
> On 25 Jan (08:10:00), David Goulet wrote:
>> On 25 Jan (06:50:30), teor wrote:
>> >
>> > > On 25 Jan 2018, at 05:14, Micah Lee  wrote:
>> > >
>> > > Now that Tor Browser 7.5 is released and includes the tor 0.3.2 series,
>> > > which supports next generation onion services, I would love to make
>> > > OnionShare use these by default. Here is the issue [1].
>> > >
>> > > OnionShare is written in python3 and relies on stem to communicate with
>> > > the Tor controller. Although tor now supports v3 ephemeral onion
>> > > services, stem still doesn't. Just take make sure, here's a quick test,
>> > > with Tor Browser 7.5 (providing tor 0.3.2.9) open in the background, and
>> > > using stem 1.6.0:
>> >
>> > Here is the current status of v3 onion control port support:
>> >
>> > This proposal is listed as "Finished":
>> > https://gitweb.torproject.org/torspec.git/tree/proposals/284-hsv3-control-port.txt
>> >
>> > And the code was merged into 0.3.3 (not 0.3.2):
>> > https://trac.torproject.org/projects/tor/ticket/20699
>> >
>> > Stem typically tracks Tor's control spec.
>> > But the changed commands have not been integrated into control-spec.txt:
>> > https://trac.torproject.org/projects/tor/ticket/24847
>>
>> This one is on me and I apologize... I haven't took the time to make a proper
>> control-spec.txt patch from the prop#284.
>>
>> I'll put this on my priority list so when 0.3.3 goes stable (~ April 15th),
>> we'll have a up to date control spec for Stem to work with.
>
> Hi!
>
> Ok the proposal 284 has landed in the control-spec.txt officially. Could be
> some fix up here and there once atagar looks at it ;) but overall the proposal
> is officially closed and merged.
>
> https://gitweb.torproject.org/torspec.git/commit/?id=6bd0a699a0856240480a4bc339722db142beec0a
>
> David
>
>>
>> Cheers!
>> David
>>
>> --
>> 6Swl3Ho3WPfuToRBmU/VqJyeGkbf9yd3uucj/BJVoKk=
>
>
>
>> ___
>> tor-dev mailing list
>> tor-dev@lists.torproject.org
>> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
>
> --
> aVYoXwD2iAuCmaVVnSs4Ds40PAgernzXRM/WFwLLoEU=
>
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] New Authority File (was: Re: New Fallback Directory File Format)

[Damian Johnson]

> We've merged a new list of fallbacks in the version 2.0.0 format.

Hi Tim. Last couple weeks I've been working on a Fallback and Endosome
branch for Stem. Just finished and merged the former...

https://gitweb.torproject.org/stem.git/commit/?id=ea2752c

Stem now supports the v2 format, has additional test coverage, and is
synced with Tor's current fallback file.

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] New Fallback Directory File Format

[Damian Johnson]

> Done!
>
> * the file now starts with a type and a version line:
>   /* type=fallback */
>   /* version=2.0.0 */
> * extrainfo is mandatory (occasionally we won't get a descriptor, so
>   we'll warn and mark the relay extrainfo=0)
> * each fallback entry ends with /* = */

Sweet, thanks Tim!

> * is 6 extra info caches (up from 4) enough in a list of 150?

Hmmm. Can't say on Stem's side I have an opinion on this. It doesn't
rely on fallback directories for anything so extrainfo caches aren't a
concern.

> * do you want the delimiter before the first fallback entry as well?

Stem doesn't care about a delimiter before the first entry but that
seems like a good idea so we have a clear separation between comments
and the start of the machine readable section.

A detail Stem does care about is that the last entry ends with a
delimiter. If it doesn't that's fine, but code is a tad simpler if we
ensure it does. :)

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Did tor-wiki-changes ML break?

[Damian Johnson]

> Could the torproject provide you with a VM to run that service?

Hi nusenu. My point is that I'm providing it because it's convenient
for me since I already run an r2e instance for my personal news feeds.
I'd rather not maintain two r2e cron jobs.

If someone else would care to run it on a tor VM that's fine with me.
Another option would be for tor to provide a VM where I host this
*and* my personal feeds. I'm fine with that, though does feel a bit
funky. :)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] New Fallback Directory File Format

[Damian Johnson]

> Do you want me to add an explicit end of record comment, or is the
> comma sufficient?

Hi Tim. I'd rather not rely on just a comma. I can easily see us
tweaking the layout so 'expect a line with only a comma' breaks.

I actually like both of the other suggested options: having
'extrainfo=0' so that's explicit *and* a delimiter. For instance...

/* == */

... or whatever. That would both help with parsing and make the file
nicer to read.

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] New Fallback Directory File Format

[Damian Johnson]

Hi Tim, added preliminary Stem parsing support for the v2 fallback format...

  https://gitweb.torproject.org/stem.git/commit/?id=ea55eaa

Few thoughts are...

  * It would be nice if the 'extrainfo=' lines were mandatory since I
need a delimiter between the entries.
  * Maybe we should start the document with a format version number?

Cheers! -Damian



On Fri, Dec 22, 2017 at 6:53 AM, teor  wrote:
> Hi all,
>
> Some time in the next few weeks, the Tor fallback directory mirror file
> format will change. This affects stem and Relay Search, which parse this
> file.
>
>
> Change Description
>
> Here is a list of changes to the file format:
> * the "weight" line has been removed, and replaced with a Tor config
>   default (#24679, #24681)
> * the comma that separates fallback C strings is now on its own line
> * a "nickname" comment has been added (#24600)
> * an optional "extrainfo" comment has been added (#22759)
>
> The added fields will be populated with placeholders until the list is
> rebuilt (#22271). This will hopefully happen some time in the next few
> weeks.
>
>
> Requesting More Extra Info Caches
>
> There are only a few fallbacks that cache extra-info documents.
> I checked 67, and only 4 cached extra-info documents.
>
> atagar, do you want me to ask some fallback operators to set
> DownloadExtraInfo 1?
>
> What number or proportion would you like?
>
> (We allow approximately 25% of fallbacks to go down before we start to rebuild
> the list. In the worst case, this can mean that ~40% are down at some point.)
>
>
> Example Entries
>
> A sample entry in the new format, using actual relay info:
>
> "5.9.110.236:9030 orport=9001 id=0756B7CD4DFC8182BE23143FAC0642F515182CEB"
> " ipv6=[2a01:4f8:162:51e2::2]:9001"
> /* nickname=rueckgrat */
> /* extrainfo=1 */
> ,
>
> The current fallback file in the new format, with placeholders:
> https://github.com/teor2345/tor/blob/ticket22759_tree/src/or/fallback_dirs.inc
>
> A small sample fallback file in the new format, with actual relay info:
> https://trac.torproject.org/projects/tor/attachment/ticket/22759/fallback_dirs_new_format.inc
>
> Please let me know if you would like any changes to the format.
>
> T
>
> --
> Tim Wilson-Brown (teor)
>
> teor2345 at gmail dot com
> PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
> ricochet:ekmygaiu4rzgsk6n
> xmpp: teor at torproject dot org
> 
>
>
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Did tor-wiki-changes ML break?

[Damian Johnson]

> Would it help if we moved this to some other infrastructure?

If someone else would like to maintain it then sure. Karsten
originally ran it, then I took it over when he shut down his r2e
instance since I already run one for my news feeds anyway.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Did tor-wiki-changes ML break?

[Damian Johnson]

Oh! Interesting, just got a dump of r2e emails. Guess it came back on
its own without a router reboot. Neat!


On Tue, Dec 19, 2017 at 10:54 AM, Damian Johnson <ata...@torproject.org> wrote:
> Hi nusenu. Yup, I was just thinking about that. CenturyLink did some
> work on my apartment's connection yesterday that knocked it offline.
> Probably just needs the router to be rebooted but I'm visiting with
> family through new years so the r2e instance will be unavailable until
> then.
>
> More annoyingly now I don't get Path of Exile news. Blah. >:(
>
>
> On Tue, Dec 19, 2017 at 10:03 AM, nusenu <nusenu-li...@riseup.net> wrote:
>> Hi Damian,
>>
>> it appears to be the case that wiki changes are no longer send to this ML.
>> Could you have a look?
>> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-wiki-changes
>>
>> thank you!
>> nusenu
>>
>>
>>
>> --
>> https://mastodon.social/@nusenu
>> twitter: @nusenu_
>>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Did tor-wiki-changes ML break?

[Damian Johnson]

Hi nusenu. Yup, I was just thinking about that. CenturyLink did some
work on my apartment's connection yesterday that knocked it offline.
Probably just needs the router to be rebooted but I'm visiting with
family through new years so the r2e instance will be unavailable until
then.

More annoyingly now I don't get Path of Exile news. Blah. >:(


On Tue, Dec 19, 2017 at 10:03 AM, nusenu  wrote:
> Hi Damian,
>
> it appears to be the case that wiki changes are no longer send to this ML.
> Could you have a look?
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-wiki-changes
>
> thank you!
> nusenu
>
>
>
> --
> https://mastodon.social/@nusenu
> twitter: @nusenu_
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 284: Hidden Service v3 Control Port

[Damian Johnson]

> Not entirely true actually, if we do that, the old Stem won't be able to
> pickup the descriptor ID from new Tor... So how do you suggest to proceed with
> backward compat? Just a new field like "DESCRIPTOR_ID=" and we leave the
> "DescriptorID" in duplicating the information for v2 descriptors? Kinda seems
> weird.

Hi David. Meejah was right that there seems to be a misunderstanding
here. I only commented about it because the DESCRIPTOR_ID was part of
your proposal. If it's just citing an existing part of the spec then
no worries. :)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 284: Hidden Service v3 Control Port

[Damian Johnson]

> What do you propose exactly?

Hi David. What I mean is that having an optional positional field...

MyEvent Field1 Field2 [Field3] Key1=Value1

... means we cannot ever add more positional fields in the future. For
example...

MyEvent Field1 Field2 [Field3] [Field4] Key1=Value1

... would be ambiguous if the third field is Field3 or Field4 since
they're both optional. We also could not add new mandatory positional
fields...

MyEvent Field1 Field2 Field4 [Field3] Key1=Value1

... because it would be ambiguous if the third field was Field4 with a
new version of tor or Field3 with an old one.

> I can't really change the "DescriptorID" to a
> key=value format. So, you think I should just not extend that field and use a
> new "key=value" for it?

Why not? Does the DescriptorID have equal signs in it? If so then you
could also make this be a mandatory positional field with a filler
value like 'none' if unavailable.

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Fwd: Nyx 2.0 Release

[Damian Johnson]

Hi tor-dev. Sorry for the cross post but while tor-relays@ is the
perfect spot to announce Nyx, tor-dev@ is the traditional place to
unveil Stem.

I'm pleased to announce Stem 1.6, the accumulation of a full year of
improvements for our controller library...

https://stem.torproject.org/change_log.html#version-1-6

Besides features such as descriptor creation and ed25519 support the
main highlight for this release is performance tuning. Descriptor
parsing is ~25% faster and low-level control socket handling got some
special attention.

Cheers! -Damian

-- Forwarded message --
From: Damian Johnson <ata...@torproject.org>
Date: Mon, Nov 6, 2017 at 3:41 PM
Subject: Nyx 2.0 Release
To: tor-rel...@lists.torproject.org


Hi all, after years of being in the works I'm pleased to announce Nyx!
A long overdue modernization of arm.

http://blog.atagar.com/nyx-release-2-0/
https://nyx.torproject.org/

Even more important for our controller space at large, Nyx is coming
hand-in-hand with Stem 1.6. A full year of improvements that include
descriptor creation support, ed25519 certificates, performance tuning,
and much, much more...

https://stem.torproject.org/change_log.html#version-1-6

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Proposal 284: Hidden Service v3 Control Port

[Damian Johnson]

Hi David, great proposal! Sorry I'm juggling too many things right now
to really really review it. Quick skim though looks great. One quick
thought is that the HS_DESC event has an optional positional argument
(DescriptorID). This is fine *but* I'd advise against it since it will
prevent you from ever adding more positional arguments in the future.
Making it a key=value argument instead sidesteps this.


On Mon, Nov 6, 2017 at 6:59 AM, David Goulet  wrote:
> Hi everyone,
>
> Attached is the proposal draft for the hidden service v3 contro port
> specification.
>
> The idea with this proposal is to _only_ extend the current commands and
> events to v3. Nothing new is added. We can think of more things to add after
> but for now, I wanted a baseline to start with that is only extending what
> exists.
>
> Any kind of feedbacks is welcome! :)
>
> Cheers!
> David
>
> --
> Zu3IyL4LcdnKNkQIZqEqaTNUapUEJFdEcN02dPwo5FQ=
>
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Summer 2017 Internship to Create a Bridge Bandwidth Scanner

[Damian Johnson]

> Hey Damian!
>
> Sorry, in my nearsightedness, having only ever had used Stem for parsing, I
> had not realised that enough of the control protocol was implemented in Stem
> to do this without using txtorcon!  Please feel free to reword the posting (or
> suggest a change) that you think would more accurately reflect this.

No problem in the least. :P

For what it's worth Stem is a complete implementation of the control
and directory specification. That is to say, everything in there
*should* be supported (I keep an eye on all spec commits to ensure we
continue to match everything tor has). If Stem's missing something
then that's a bug I'd love to know about.

Blog post looks great. I'd probably only suggest a couple small tweaks...

* s/also outsourceable in Python to txtorcon/also outsourceable in
Python to txtorcon or stem
* Drop the paragraph after it since it doesn't seem to add much.

It's probably not worth including in the blog post but for what it's
worth the following list all the controller and descriptor parser libs
I'm aware of...

https://stem.torproject.org/faq.html#are-there-any-other-controller-libraries
https://stem.torproject.org/tutorials/mirror_mirror_on_the_wall.html#are-there-any-other-parsing-libraries

> Also, I would be delighted to co-mentor with you (and also meejah, if the
> intern decides to go the txtorcon route), that sounds great!

Great!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Summer 2017 Internship to Create a Bridge Bandwidth Scanner

[Damian Johnson]

> The last example on the page it seems like you're just "really hoping"
> that the next stream to open is the one you want to map onto the new
> circuit, or am I missing something?

Hi meejah. Yup, you're right. To keep things simple it's just
attaching the next stream. If you were using your tor client for other
things too this wouldn't be a safe assumption and the caller would
need additional checks that it's the stream they want.

Python's builtin connection modules really suck, hence our need for
SocksiPy or PycURL. Maybe python3's asyncio (the builtin which seems
intended to replace twisted) has better options. I certainly wouldn't
be surprised if what Twisted provides is better than the builtins we
had on python 2.7.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Summer 2017 Internship to Create a Bridge Bandwidth Scanner

[Damian Johnson]

Hi Isis. For what it's worth if it turns out to be based on Stem I'd
be delighted to help mentor and/or do code reviews.

Also, I agree with Tim. The present wording makes it sound like
txtorcon is the only game in town when it comes to custom circuit
construction. Lots of options, and even if they don't do it in python
it's not hard. :)


On Thu, Jun 15, 2017 at 2:07 PM, teor  wrote:
>
>> On 16 Jun 2017, at 03:49, isis agora lovecruft  wrote:
>>
>> Hello all!
>>
>> I have made a brief post on our blog to announce an exciting intership
>> opportunity we have available!
>>
>> https://blog.torproject.org/blog/summer-2017-internship-create-bridge-bandwidth-scanner
>
> stem also does circuit construction via the control port, so the
> intern can avoid doing anything twisted... if they want.
>
> There's even TorCtl, but it's old and unmaintained.
>
> T
>
> --
> Tim Wilson-Brown (teor)
>
> teor2345 at gmail dot com
> PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
> ricochet:ekmygaiu4rzgsk6n
> xmpp: teor at torproject dot org
> 
>
>
>
>
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Default bridges that are not publishing statistics

[Damian Johnson]

> You asked your bridge for its server descriptor and it gave it to you.
>
> Did you try:
>  print 
> controller.get_extrainfo_descriptors("8B5F0BD647B3C4AF2C57F148FF6A1FB8B695B0AE")
>
> https://stem.torproject.org/api/descriptor/remote.html#stem.descriptor.remote.DescriptorDownloader.get_extrainfo_descriptors

Hi teor, minor correction: you link to the DescriptorDownloader's method
but cite it as being part of Controller. This is why David didn't find it
later. The Controller can fetch some descriptor types. For instance...

https://stem.torproject.org/api/control.html#stem.control.Controller.get_server_descriptor
https://stem.torproject.org/tutorials/mirror_mirror_on_the_wall.html#can-i-get-descriptors-from-the-tor-process

However, the Controller doesn't provide extrainfo descriptors. This is
for a couple reasons...

1. The control protocol only allows it to be retrieved by the descriptor
   digest rather than relay fingerprint, which is a pita...

   https://gitweb.torproject.org/torspec.git/tree/control-spec.txt#n562

2. This isn't what you want anyway. Fetching decriptors from the tor
   process only gets you the cached descriptors which isn't what folks
   generally want. So you're right that the DescriptorDownloader is
   the way to go.

> I don't know how to find a server that caches *all* bridge extra infos.
> Maybe you should try running these queries against the bridge authority?

Bridge descriptors are not public like normal descriptors. To work of
course the tor client can retrieve it if you know the bridge address
but beyond that I think the only thing available is the sanitized
descriptors from CollecTor...

https://collector.torproject.org/recent/bridge-descriptors/

> I think the documentation at:
> https://stem.torproject.org/api/descriptor/remote.html#stem.descriptor.remote.Query
> is out of date: as far as I know, newer stem versions try fallback
> directories rather than authorities.

Unfortunately it doesn't. I had to revert that because they lack
extrainfo docs...

https://gitweb.torproject.org/stem.git/commit/?id=758f632

> Thanks again for your suggestions. Passing endpoints= doesn't seem to do
> anything.
> >>> import stem.descriptor.remote
> >>> BIFROEST = ("37.218.247.217", 80)
> >>> print 
> list(stem.descriptor.remote.get_server_descriptors("C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4",
>  endpoints=(BIFROEST,)))
> []

For what it's worth stem has authority information so this can also be
done with...

>>> import stem.descriptor.remote
>>> bifroest = stem.descriptor.remote.get_authorities()['Bifroest']
>>> print 
>>> list(stem.descriptor.remote.get_server_descriptors("C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4",
>>>  endpoints=((bifroest.address, bifroest.dir_port),)))

> It also doesn't work when trying the fingerprint of another default
> bridge, or of one I just got from bridge.torproject.org.

Alas, this is getting into areas I'm not too familiar with. To prevent
enumaration
I'm sure bridges are handled specially.

> Atagar might be able to help with the stem side of things.

Happy to help if there's any other stem questions.

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Leekspin and Stem descriptor creation

[Damian Johnson]

Hi Isis. For kicks and giggles this weekend I added the ability for
Stem to create descriptors...

https://stem.torproject.org/tutorials/mirror_mirror_on_the_wall.html#can-i-create-descriptors
https://gitweb.torproject.org/stem.git/commit/?id=b2a54ad

This is something Stem has always been able to do but it was buried in
our testing helpers. Now descriptor subclasses have a static create()
and content() method.

This *doesn't* yet support signing descriptors. I'm looking into that
next. To that end I took a peek today at leekspin
(https://pypi.python.org/pypi/leekspin) and... damn. Beautiful work!
Besides the crypto are there any other aspects of your library you
think we should incorporate?

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] txtorcon versioning

[Damian Johnson]

Hi meejah. This sounds like a good move since txtorcon makes so many
small incremental releases. I like symantic versioning for stem [1],
but I've only made six releases thus far. Not nineteen. :P

Symantic versioning provides a clear way of indicating what upgrades
are safely backwards compatible and which aren't. In the above you say
"The onion service APIs *will* change" but also "At one point, I
thought of breaking a few now-regrettable APIs. However, I will not do
this." - are you changing APIs or not?

If you are then Calendar Versioning will make it tricker for your
users to figure out when you're doing so.

[1] https://stem.torproject.org/change_log.html#versioning

On Wed, Apr 12, 2017 at 1:25 PM, meejah  wrote:
>
> I will soon release the next version of txtorcon with a ton of cool new
> features. This will be called 0.19.0. More details:
>
>http://timaq4ygg2iegci7.onion/releases.html
>
> Going forward, versioning will switch to a "CalVer.org" variant. At one
> point, I thought of breaking a few now-regrettable APIs.
>
> However, I will not do this.
>
> The next version after 0.19.0 will be 17.x.y
> Changes in any existing APIs will be done by first introducing the new
> thing, deprecating the old thing and eventually removing the old
> thing. The new documentation's "programming guide" includes some notes
> on API stability. I have not to date broken/changed any existing
> API. Also at this point nothing is deprecated (but there are "preferred"
> APIs).
>
> The onion services APIs *will* change for the 17.x release.
>
> New code should follow the recommendations in the programming
> guide. Existing code will continue to work for the forseeable future.
>
> thanks,
> meejah
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Control-port filtering: can it have a reasonable threat model?

[Damian Johnson]

Hi Nick. Just a quick note that something I've wanted from time to
time is a 'make the control port read-only' option so only GETINFO,
GETCONF, events, etc would work. Yes, these could be used to
deanonymize a user, but it could provide assurance the controller
doesn't tamper with tor. This has been of interest to me since nyx
(aka arm) is primarily a read-only monitor and this could provide
users with an assurance that it's not doing anything to their tor
instance.

Besides that, 'makes the control port read-only' is a pretty straight
forward, simple to understand capability for a torrc option to have.

Cheers! -Damian

On Mon, Apr 3, 2017 at 11:41 AM, Nick Mathewson  wrote:
> Hi!
>
> As you may know, the Tor control port assumes that if you can
> authenticate to it, you are completely trusted with respect to the Tor
> instance you have authenticated to.  But there are a few programs and
> tools that filter access to the Tor control port, in an attempt to
> provide more restricted access.
>
> When I've been asked to think about including such a feature in Tor in
> the past, I've pointed out that while filtering commands is fairly
> easy, defining a safe subset of the Tor control protocol is not.  The
> problem is that many subsets of the control port protocol are
> sufficient for a hostile application to deanonymize users in
> surprising ways.
>
> But I could be wrong!  Maybe there are subsets that are safer than others.
>
> Let me try to illustrate. I'll be looking at a few filter sets for example.
> =
> Filters from https://github.com/subgraph/roflcoptor/filters :
>
> 1. gnome-shell.json
>
> This filter allows "SIGNAL NEWNYM", which can potentially be used to
> deanonymize a user who is on a single site for a long time by causing
> that user to rebuild new circuits with a given timing pattern.
>
> 2. onioncircuits.json
>
> Allows "GETINFO circuit-status" and "GETINFO stream-status", which
> expose to the application a complete list of where the user is
> visiting and how they are getting there.
>
> 3. onionshare-gui.json
>
> Allows "SETEVENTS HS_DESC", which is exposes to the application every
> hidden service which the user is visiting.
>
> 4. ricochet.json
>
> Allows "SETEVENTS HS_DESC", for which see "onionshare-gui" above.
>
> 5. tbb.json
>
> Allows "SETEVENTS STREAM" and "GETINFO circuit-status", for which see
> "onioncircuits" above.
>
> =
> Filters from 
> https://git-tails.immerda.ch/tails/tree/config/chroot_local-includes/etc/tor-controlport-filter.d
> :
>
> 1. onioncircuits.yml
>
> See onioncircuits.json above; it allows the same GETINFO stuff.
>
> 2. onionshare.yml
>
> As above, appears to allow HS_DESC events.  It allows "GETINFO
> onions/current", which can expose a list of every onion service
> locally hosted, even those not launched through onionshare.
>
> 3. tor-browser.yml
>
> As "tbb.json" above.
>
> 4. tor-launcher.yml
>
> Allows setconf of bridges, which allows the app to pick a hostile
> bridge on purpose.  Similar issues with Socks*Proxy.  The app can also
> use ReachableAddresses to restrict guards on the .
>
> Allows SAVECONF, which lets the application make the above changes
> permanent (for as long as the torrc file is persisted)
> =
>
> So above, I see a few common patterns:
>   * Many restrictive filters still let the application learn enough
> about the user's behavior to deanonymize them.  If the threat model is
> intended to resist a hostile application, then that application can't
> be allowed to communicate with the outside world, even over Tor.
>
>   * Many restrictive filters block SETCONF and SAVECONF.  These two
> changes together should be enough to make sure that a hostile
> application can only deanonymize _current_ traffic, not future Tor
> traffic. Is that the threat model?  It's coherent, at least.
>
>   * Some applications that care about their own onion services
> inadvertantly find themselves informed about everyone else's onion
> services.  I wonder if there's a way around that?
>
>   * The NEWNYM-based side-channel above is a little scary.
>
>
> And where do we go forward from here?
>
> The filters above seem to have been created by granting the
> applications only the commands that they actually need, and by
> filtering all the other commands.  But if we'd like filters that
> actually provide some security against hostile applications using the
> control port, we'll need to take a different tactic: we'll need to
> define the threat models that we're trying to work within, and see
> what we can safely expose under those models.
>
> Here are a few _possible_ models we could think about, but I'd like to
> hear from app developers and filter authors and distributors more
> about what they think:
>
>  A. Completely trusted controller.  (What we have now)
>
>  B. Controller is untrusted, but is blocked from exfiltrating information.
> B.1. Controller can't connect to the network at all.
> B.2. Controller can't connect 

Re: [tor-dev] GSoC 2017 - Project "Crash Reporter for Tor Browser"

[Damian Johnson]

>> P.S. Have I to send proposal to GSoc as draft?
>
> I don't know the answer to this, but hopefully Damian does?

It would be useful if you uploaded a draft to the site, but really the
only hard requirement is that the proposal is uploaded before the
deadline. ;)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] GSoC 2017 - Questions

[Damian Johnson]

Hi Krishna, absolutely! We love having new volunteers be it through
GSoC or not. Hell, most of us got our start outside the program. ;)

I'll leave the crypto parallelism questions to Nick, George, David,
and others far more knowledgeable of the core tor codebase than me.


On Mon, Mar 27, 2017 at 6:17 PM, Krishna Shukla
 wrote:
> Hey,
>
> I'm Krishna Shukla, I'm studying a bachelors of computer science at the
> University of Queensland.
> I guess the relevant subjects I've studied so far covers C and Unix
> programming, Computer Networks, Algorithms and Data Structures, and
> Programming in the large. (got a high distinction in all the above)
>
> My most important question is if I could work on a project but not actually
> be apart of GSoC? - I am unfortunately ineligible as my brother works as a
> Security Engineer at Google Sydney. And if the above is okay would it also
> be okay to not have to strictly abide by their timeline as I don't actually
> have holidays during this time in Australia but I'd like to contribute in my
> free time nonetheless!
>
> As for projects themselves I'm really interested in the relay crypto
> parallelism and the hidden service crypto parallelism. And I have a couple
> of questions regarding them.
>
> For the relay crypto parallelism I wanted to know what is there left to be
> done? When I looked at the tickey #1749 someone called towelenee made a few
> patches that already made it multi threaded, were these changes just not
> accepted? Also wanted to know if specific knowledge about circuit
> cryptography was required? As I know of it, but I certainly cannot make my
> own fully homomorphic cryptosystem, is it more in the steps of the system
> has already been made, it just needs to be parallelised correctly?
>
> It also states the code is written to expect immediate responses, I'm not
> sure what you mean by that, after all there is always a slight delay, and if
> it becomes multi threaded we can never know what is running what when, so is
> it more someone is waiting at the other end of a socket and needs it ASAP,
> or is it internally things want the answer quickly (in which case I don't
> know how to solve it other than uses mutexes which is probably not so okay)?
>
> I am interested in the hidden service crypto parallelism in its own right,
> but I was also thinking weather it would be a feasible idea to combine the
> two projects and create a multi-threaded decryption library that could be
> linked to both the tor relay and the hidden services (could release it as a
> cryptosystem library, all the fully homorphic cryptosystem libraries I found
> used GPL licenses and thus not compatible with tors), or are their
> requirements too far apart?
>
> Also I was wondering how the Ahmia automated blacklisting was planned to
> work? As in how would a list of child abuse sites be fetched? Honestly I
> don't actually know python, I've worked with it and Django before in a
> hackathon once, but I cannot claim any real knowledge in it, but at the same
> time I am passionate about the topic of child abuse, and I think if I can
> help reduce its demand in anyway by making it harder to find, I'd say it's
> some good added to the world.
>
> Apologies about the long mail guys,
> Krishna Shukla
>
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] GSoC 2017 - Project "Onionoo"

[Damian Johnson]

ExoneraTor is in the same boat. The metrics team are the ones that are
unavailable so everything they own (Onionoo, Atlas, Metrics-Lib,
ExoneraTor, etc) would be poor choices.

You're more than welcome to propose your own project idea but if so
you'll need to find a mentor in our community. If you'd care for
projects with mentors already available then please see...

https://www.torproject.org/getinvolved/volunteer.html.en#Coding

On 3/14/17, Nur-Magomed <nmag...@gmail.com> wrote:
> It's a pity. Could you provide me some guide lines and additional materials
> for project "ExoneraTor",  I also interested to work with that
>
> 2017-03-14 22:27 GMT+03:00 Damian Johnson <ata...@torproject.org>:
>
>> Hi Nur-Magomed. Unfortunately the folks maintaining Onionoo are
>> unavailable to mentor this summer. I'd suggest looking into another
>> subproject.
>>
>>
>>
>> On Tue, Mar 14, 2017 at 10:59 AM, Nur-Magomed <nmag...@gmail.com> wrote:
>> > Hi!
>> >
>> > I’m Nur-Magomed Dzhamiev, 4th year student from institute of
>> > information
>> > technology (speciality: computer security) of North-Caucasus Federal
>> > University. I have experience with Java, SQL, Web (HTML, CSS,
>> > JavaScript,
>> > PHP) and also I designed protocol based on JSON for Android app.
>> >
>> >
>> > I would love to contribute to the project "Onionoo". Please provide me
>> some
>> > guide lines and additional materials for study and get a clear
>> understanding
>> > about the mentioned project. Thank you!
>> >
>> >
>> > Regards
>> >
>> > Nur-Magomed
>> >
>> >
>> > ___
>> > tor-dev mailing list
>> > tor-dev@lists.torproject.org
>> > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>> >
>> ___
>> tor-dev mailing list
>> tor-dev@lists.torproject.org
>> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>>
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] GSoC 2017 - Project "Onionoo"

[Damian Johnson]

Hi Nur-Magomed. Unfortunately the folks maintaining Onionoo are
unavailable to mentor this summer. I'd suggest looking into another
subproject.



On Tue, Mar 14, 2017 at 10:59 AM, Nur-Magomed  wrote:
> Hi!
>
> I’m Nur-Magomed Dzhamiev, 4th year student from institute of information
> technology (speciality: computer security) of North-Caucasus Federal
> University. I have experience with Java, SQL, Web (HTML, CSS, JavaScript,
> PHP) and also I designed protocol based on JSON for Android app.
>
>
> I would love to contribute to the project "Onionoo". Please provide me some
> guide lines and additional materials for study and get a clear understanding
> about the mentioned project. Thank you!
>
>
> Regards
>
> Nur-Magomed
>
>
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] GSOC Contribution Into and Help

[Damian Johnson]

Hi Anshul. Second person today regarding Ahmia. :P

For Ahmia Juha will be your best point of contact. Cc-ed him here. If
he doesn't respond to you in a few days give me a nudge and I'll nag
him. ;)

Cheers! -Damian

On 3/12/17, Anshul Malik  wrote:
> ​​
> Hi there!
>
> I am Anshul, computer science undergrad from NIT, Kurukshetra, India.
> I really like using tor, always wanted to contribute somhow, and here
> google gives me a chance.
>
> I am a good programmer, have been doing projects with node, recently also
> started with python django. I have also contributed to firefox devtools.
>
> I wish to contribute to *Ahima- Hidden Service Search*
>
> I saw ideas list on the page, I can definitely work on
> - Automate blacklisting
> - Add hidden services function
> - Elastic Search
>
> Just need a headstart, Please let me know where can I find contribution
> guide and some more information about Ahima, I am currently studying the
> architecture and codebase from github.
>
> Thanks
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] GSoC 17- Intro

[Damian Johnson]

Hi Pasan. For Ahmia Juha will be your best point of contact. Cc-ed him
here. If he doesn't respond to you in a few days give me a nudge and
I'll nag him. ;)

Cheers! -Damian

On 3/12/17, Pasan Chamikara  wrote:
> Hi,
> I am Pasan Chamiekara, a final year undergraduate in BSc in Biological
> Science at University of Colombo and a final year undergraduate at Sri
> Lanka Institute
> of Information Technology (BSc Hons in IT specializing in Cyber Security).
> (Simply a dual degree expectee) . I do pocess 7 months of completed
> internship at Sri Lanka CERT | CC as an
> Intern - Information Security.
> My main research interests are in automated malware analysis, Intrusion
> detection and Neural Networking.
>
> I do pocess strong skills in html, css, php, javascript, python, C, C++
> ,java, ASM and intermediate level skills in Android,scala.
>
> I am interested in the following components,
> *Ahmia - Hidden Service Search*
>
> Regards
> Pasan
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Tor in Google Summer of Code 2017

[Damian Johnson]

Oops! My bad. Great catch, thanks for pointing that out.


On Tue, Mar 7, 2017 at 10:53 AM, Jaskaran Singh <jvsg1...@gmail.com> wrote:
> Hi Damian,
>
> On Tuesday 07 March 2017 11:54 PM, Damian Johnson wrote:
>
>> Finally, write down your project idea using our template [5] and submit
>> your application to Google before March 25th [6].
>
> I think the deadline is April 3 this year.
>
> Regards,
> Jaskaran
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Tor in Google Summer of Code 2017

[Damian Johnson]

Interested in coding on Tor and getting paid for it by Google? If you
are a student, we have good news for you: we have been accepted as a
mentoring organization for Google Summer of Code 2017!

  https://summerofcode.withgoogle.com/organizations/5675829762719744/

Here's the facts: GSoC gives you the opportunity to work on your own
Tor-related coding project with one of the Tor developers as your
mentor. Your mentor will help you when you're stuck and guide you in
becoming part of the Tor community. Google pays you for the
three months of your project, so that you can focus on coding and
don't have to worry about how to pay your bills.

  https://www.torproject.org/getinvolved/volunteer.html.en#Projects

Did we catch your attention? These are your next steps: Go look at
the Google Summer of Code FAQ [1] to make sure you are eligible to
participate. Have a look at our ideas list [2] to see if one of those
projects matches your interests. If there is no project on that list
that you'd want to work on, read the documentation on our website [3]
and make up your own! Come to #tor-dev on OFTC [4] or let us know about
your project idea here. Communication is essential to success in the
summer of code, and we're unlikely to accept students we haven't heard
from before reading their application. So really, speak up on this list
or come to our IRC channel and talk to us!

Finally, write down your project idea using our template [5] and submit
your application to Google before March 25th [6].

We are looking forward to discussing your project idea with you!

[1] https://developers.google.com/open-source/gsoc/faq
[2] https://www.torproject.org/getinvolved/volunteer.html.en#Coding
[3] https://www.torproject.org/docs/documentation.html.en#UpToSpeed
[4] https://www.torproject.org/about/contact.html.en#irc
[5] https://www.torproject.org/about/gsoc.html.en#Template
[6] https://developers.google.com/open-source/gsoc/timeline
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] GSoC 2017 - my name is Marcin

[Damian Johnson]

Hi Marcin, glad you want to get involved! We won't know until the 27th
if we've been accepted into this year's program or not but always
delighted for students to get involved early. We have a handful of
project ideas and points of contact listed on...

https://www.torproject.org/getinvolved/volunteer.html.en#Projects

But you're of course more than welcome (and even encouraged!) to
brainstorm projects of your own.

Cheers! -Damian


On Mon, Feb 13, 2017 at 12:37 PM, Marcin Czarnecki
 wrote:
> Hello all,
>
> tl;dr can go on about Tor and anonymity and privacy related things for
> hours, give me a chance
>
> I am studying Computer Science at the University of Brighton (in the UK). I
> am interested in participating in Google Summer of Code 2017 with the Tor
> project. I am a web security and anonymity enthusiast, tried to work with
> the Tor Project for GSoC 2016 but it did not work out.
>
> I have not contributed to the Tor's code base but have been an active, semi
> advanced user for past 3 years. Have been experimented with setting up
> exit-nodes (ended up with my box being considered the #1 most evil machine
> in the world :)), creating Tor-clearnet gateways, hosting own HS, trying to
> exploit other HSes, playing with encryption etc. I think that gave a me a
> lot of insight of how Tor works (still amazed and finding new things when
> reading docs).
>
> I am planning to base my proposal on the idea on a easier way of hosting HS
> so it would become more accessible to non tech savvy Tor users plus
> enforcing good security practices when hosting a HS (got some experience and
> cool ideas [citation needed]).
>
> My main skills are in Linux and back-end web development but I am also
> interested in scripting and automation as well as encryption. Can talk about
> Bitcoin for hours (security, anonymity, etc.), thinking about using that as
> my other project proposal as this should be the currency of Tor and more
> integrated into Tor (Tor Browser extension?). When it comes to programming I
> feel confident with PHP, Python and HTML, CSS, JS stuff, and know a bit of
> other languages but I am a fast learner!
>
> When it comes to tor related projects, two years ago I did this small
> crawler with web interface for searching (#2 most evil machine) in Python:
> https://github.com/marncz/torcrawler
>
> Cool feature on which I stopped working was HTTP headers fingerprinting
> which could led to grouping Hidden Services by hosts providers.
>
> Feel free to contact me:
> email: marnczarnecki[at]gmail.com
> IRC at OFTC: marncz
>
> Many thanks,
> Marcin
>
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] non-anonymous ephemeral onion services with stem

[Damian Johnson]

> Excellent. I'm pretty sure this will work, but can you confirm? If I'd
> like to use a non-anonymous ephemeral onion service, would code that's
> something like this work, assuming c is a Controller?
>
> c.set_conf('HiddenServiceSingleHopMode', 1)
> c.set_conf('HiddenServiceNonAnonymousMode', 1)
> c.create_ephemeral_hidden_service(8080)
>
> And when other processes connect to the Tor control port and run
> create_ephemeral_hidden_service, those onion services wouldn't be
> non-anonymous?

Good question. The non-anonymous torrc options are pretty clunky to
use. In part this is by design because the authors wanted to
discourage their use.

I thought those torrc options could only be set prior to tor starting
up (like DisableDebuggerAttachment), but on reflection the manual
doesn't say that so maybe that's not the case? However, seems you also
need to set 'SOCKSPort 0'...

https://www.torproject.org/docs/tor-manual.html.en#HiddenServiceNonAnonymousMode

If you call the above SETCONF does tor give any indication that you
need to set the SOCKSPort too? If not then it feels like it should
since that's pretty unintuitive.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] non-anonymous ephemeral onion services with stem

[Damian Johnson]

Oops, great catch - thanks Micah! Added a note saying how to use them...

"Version 1.5: Added support for non-anonymous services. To do so set
'HiddenServiceSingleHopMode 1' and 'HiddenServiceNonAnonymousMode 1'
in your torrc."


On Wed, Dec 28, 2016 at 8:54 AM, Micah Lee  wrote:
> The stem documentation for create_ephemeral_hidden_service [1] says:
> "Changed in version 1.5.0: Added support for non-anonymous services."
>
> But I can't figure out to actually use this feature. There doesn't seem
> to be a new argument to say if you want your onion service to be
> non-anonymous.
>
> It also says, "Changed in version 1.5.0: Added the basic_auth argument."
> But there's a new basic_auth argument you can pass into the function to
> use that.
>
> [1]
> https://stem.torproject.org/api/control.html#stem.control.Controller.create_ephemeral_hidden_service
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Stem Release 1.5

[Damian Johnson]

Damn this was long overdue. I'm delighted to announce Stem 1.5.2, the
accumulation of seventeen months of improvements.

What is *Stem *, you ask? For those who
aren't familiar with it Stem is a Python library for interacting with Tor.
With it you can script against your relay, descriptor data, or even write
applications similar to Nyx and Vidalia.

So what's new in this release? Short answer: a lot.

For full details *see our release announcement*
!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] strange ARM results

[Damian Johnson]

>> Gotcha. Definitely a refresh bug then. Sorry about that. :(
>
> No need to apologize, I think Stem/Nyx/Arm is wonderful software. Very
> useful, thank you (and all contributors) for writing it!
>
>> Ahh, think I know the issue. Nyx requires the copy of Stem from its
>> git repo too...
>>
>> https://git.torproject.org/stem.git
>>
>
> SUCCESS!
>
> Nyx updates the circuits page, the data is in-sync with tor-prompt and
> seems to be correct.

Thanks Rob! And glad to hear it's working!
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] strange ARM results

[Damian Johnson]

> If I run tor-prompt and ARM side by side, the tor-prompt results are
> updated when I do a GETINFO circuit-status. The ARM circuits page only
> updates after a restart.

Gotcha. Definitely a refresh bug then. Sorry about that. :(

>> > ARM version 1.4.5 seems to be the latest version. I checked out NYX but
>> > failed to get it running (Unable to load nyx's internal configurations:
>> > [Errno 21] Is a directory: '/home/rob/src/nyx/nyx/settings')
>>
>> Interesting! How did you attempt to run it? Nyx is under active
>> development so quite possible I buggered something up but can't say
>> I've seen this one. Please provide the exact commands you ran - for
>> instance...
>>
>> % git clone http://dccbbv6cooddgcrq.onion/nyx.git
>> % cd nyx
>> % ./run_nyx
>
> I did:
>
> git clone https://git.torproject.org/nyx.git
> cd nyx
> ./run_nyx

Ahh, think I know the issue. Nyx requires the copy of Stem from its
git repo too...

https://git.torproject.org/stem.git

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] strange ARM results

[Damian Johnson]

Hi Rob. I suppose it's possible arm is having a refresh issue but
can't say there's a known bug around that. To double check try running
tor-prompt and giving it 'GETINFO circuit-status'...

https://stem.torproject.org/tutorials/down_the_rabbit_hole.html

This is the command arm uses to get the circuit information.

> ARM version 1.4.5 seems to be the latest version. I checked out NYX but
> failed to get it running (Unable to load nyx's internal configurations:
> [Errno 21] Is a directory: '/home/rob/src/nyx/nyx/settings')

Interesting! How did you attempt to run it? Nyx is under active
development so quite possible I buggered something up but can't say
I've seen this one. Please provide the exact commands you ran - for
instance...

% git clone http://dccbbv6cooddgcrq.onion/nyx.git
% cd nyx
% ./run_nyx
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Adding depictor/stem to Jenkins

[Damian Johnson]

Hi Tom, just food for thought but another option would be a cron task
that pulls the repos and runs that if there's a change. That's what I
do for stem's website so it reflects the changes I push.

Cheers! -Damian

On Tue, Jul 5, 2016 at 5:31 AM, Tom Ritter  wrote:
> Hi all,
>
> Hoping someone can help me out here... I'd like to add a job to
> jenkins that runs the depictor command (`python write_website.py`)
> whenever a commit is made to the dev repo master branch[0] OR stem's
> master branch. (If I could only have one I'd pick stem's.)
>
> Historically, one of the reasons consensus-health has lagged a little
> bit has been because in the past there was some tiny change in Stem
> that wound up breaking it. I don't think that's happened lately, but
> it's caused me to be conservative about upgrading stem, and usually
> the changes I need are the latest one added to master.  A jenkins job
> would give me a bit more confidence and enable a bit more proactivity
> on fixing it for breaking changes.
>
> Who runs jenkins? What would I need to do to set this up?  The good
> news is there's only three commands needed to go from zero to
> index.html:
>
> git clone https://git.torproject.org/user/tom/depictor.git
> cd depictor
> git clone https://git.torproject.org/stem.git
> ./write_website.py
>
> -tom
>
> [0] https://gitweb.torproject.org/user/tom/depictor.git/
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] DescripTor 1.2.0 is released

[Damian Johnson]

Hi Karsten, congrats on the release! I gotta admit, first thing I
wondered when I saw this was 'what is DescripTor and why does it have
a name that will be so easily confused with the documents it
fetches?'.

Quick peek at the readme seems to indicate this is the DirPort
fetching capabilities of metrics-lib? Is this an effort to slit
metrics-lib up into smaller libraries?

Cheers! -Damian


On Tue, May 31, 2016 at 12:17 PM, Karsten Loesing
 wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hello devs,
>
> I just released DescripTor 1.2.0:
>
> https://dist.torproject.org/descriptor/1.2.0/
>
> - From the change log:
>
> # Changes in version 1.2.0 - 2016-05-31
>
>  * Medium changes
>- Include the hostname in directory source entries of consensuses
>  and votes.
>- Also accept \r\n as newline in Torperf results files.
>- Make unrecognized keys of Torperf results available together with
>  the corresponding values, rather than just the whole line.
>- In Torperf results, recognize all percentiles of expected bytes
>  read for 0 <= x <= 100 rather than just x = { 10, 20, ..., 90 }.
>- Rename properties for overriding default descriptor source
>  implementation classes.
>- Actually return the signing key digest in network status votes.
>- Parse crypto parts in network status votes.
>- Document all public parts in org.torproject.descriptor and add
>  an Ant target to generate Javadocs.
>
>  * Minor changes
>- Include a Torperf results line with more than one unrecognized
>  key only once in the unrecognized lines.
>- Make "consensus-methods" line optional in network statuses votes,
>  which would mean that only method 1 is supported.
>- Stop reporting "-END .*-" lines in directory key
>  certificates as unrecognized.
>- Add code used for benchmarking.
>
> In particular the full rewrite of Javadocs was painful but hopefully
> useful to people here, not necessarily just DescripTor users but
> anyone working with Tor network data.  Here's the compiled web version
> until DescripTor has its own website:
>
> https://people.torproject.org/~karsten/volatile/descriptor-docs-2016-05-31/
>
> Many thanks to iwakeh for helping with most of these changes!
>
> All the best,
> Karsten
>
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
>
> iQEcBAEBAgAGBQJXTeNUAAoJEC3ESO/4X7XB208IAKwyJF2jgjvuREfYKkI9UFva
> ZoMbXgITXfTNJ4hXSc5x30jMw56xVummaWMMgBKwscPyhJAdngUjxQt//8ZOwFx/
> hezHbRGxxRZquiROvIMW1mLIfnvSnkZAVL6tPuQmiKfqcR2ExMs3KCZsdCcfI/KR
> ZB9tHnGsSqhME+XPxQNAhT/OgBNnaq4Y7WFMhLuOHDm4/sCIjeoeix8aF1ve27ue
> FdOvaxjY1iBipxNdKkup5SXmL1tBmQ7bwTV59EduLatq+30tMnE7Xyat9MeQMyGI
> Bs9/5h+f29zREzyPp6kPd/m0eN1udvXF8nqa34QqAk0YHECE1BoIRDRo7qkixx4=
> =xs8m
> -END PGP SIGNATURE-
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Summer of Privacy Application | Rohit Dua

[Damian Johnson]

> I recently knew about this years Tor summer of code. Although the date of
> application is gone, I would like to ask if I could still manage to get into
> the program.

Hi Rohit. Jury's still out if we'll be running SoP again this year (at
present we're just focused on GSoC which will possibly replace SoP -
we'll see). If you're looking for a paid internship then unfortunately
I don't have any tips for you, but if you're simply looking to help
we're always happy to get new contributors!

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] GSoC'16 proposal: the Torprinter project (a Panopticlick-like website)

[Damian Johnson]

> Hi Pierre, on first glance looks like a nice proposal. Just a heads up
> though that to be considered there needs to be a prospective mentor
> for this project. Reaching out to tor-dev@ is a great first step and
> hopefully it'll do the trick, but if it doesn't try asking on the
> #tor-dev irc channel.

Oops, my bad. Missed that this was Panopticlick - gave GeKo a nudge to
take a peek. :)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Interest in GSOC 2016

[Damian Johnson]

Hi Akash. Good spot to start is to take a look a the following pages
and think what sort of project would strike your fancy.

https://www.torproject.org/about/gsoc.html.en
https://www.torproject.org/getinvolved/volunteer.html.en#Coding


On Wed, Mar 9, 2016 at 12:58 AM, AKASH DAS  wrote:
> Hii Community,
>
> I am a great enthusiast in network security and is very keen to increase my
> knowledge in this field and I think this is a way I can learn more things.
> So, I want to do something for this org and wanna contribute to it. I am
> currently a second year  undergrad CSE student in Indian Institute Of
> Information Technology, Sricity.
> So please can anyone guide me so that I can get into it as soon as possible
> and start contributing.
>
> Regards,
> Akash Das
> Indian Institute Of Information Technology, Sricity
> System Administrator
>
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Tor in Google Summer of Code 2016

[Damian Johnson]

Interested in coding on Tor and getting paid for it by Google? If you
are a student, we have good news for you: we have been accepted as a
mentoring organization for Google Summer of Code 2016!

  https://summerofcode.withgoogle.com/organizations/6655685232164864/

Here's the facts: GSoC gives you the opportunity to work on your own
Tor-related coding project with one of the Tor developers as your
mentor. Your mentor will help you when you're stuck and guide you in
becoming part of the Tor community. Google pays you 5,500 USD for the
three months of your project, so that you can focus on coding and
don't have to worry about how to pay your bills.

  https://www.torproject.org/getinvolved/volunteer.html.en#Projects

Did we catch your attention? These are your next steps: Go look at
the Google Summer of Code FAQ [1] to make sure you are eligible to
participate. Have a look at our ideas list [2] to see if one of those
projects matches your interests. If there is no project on that list
that you'd want to work on, read the documentation on our website [3]
and make up your own! Come to #tor-dev on OFTC [4] or let us know about
your project idea here. Communication is essential to success in the
summer of code, and we're unlikely to accept students we haven't heard
from before reading their application. So really, speak up on this list
or come to our IRC channel and talk to us!

Finally, write down your project idea using our template [5] and submit
your application to Google before March 25th [6].

We are looking forward to discussing your project idea with you!

[1] https://developers.google.com/open-source/gsoc/faq
[2] https://www.torproject.org/getinvolved/volunteer.html.en#Coding
[3] https://www.torproject.org/docs/documentation.html.en#UpToSpeed
[4] https://www.torproject.org/about/contact.html.en#irc
[5] https://www.torproject.org/about/gsoc.html.en#Template
[6] https://developers.google.com/open-source/gsoc/timeline
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] tor-wiki-changes is dead. Alternatives?

[Damian Johnson]

>>> https://lists.torproject.org/pipermail/tor-wiki-changes/2015-December/017387.html
>>
>> thanks for that url.
>
> Oh, looks like the list is pushing emails again.

https://lists.torproject.org/pipermail/tor-wiki-changes/2016-February/017444.html
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Nyx Project Ideas

[Damian Johnson]

Hi wonderful relay operators. It's GSoC season again, where students
can be funded to make open source projects like Tor even better!

Nyx (previously known as arm [1]) has been my main focus this last
year and is inching ever closer to release. For those unfamiliar with
it, Nyx is an ncurses monitor for Tor relays providing a bandwidth
graph, event log, connections, config editor, and more.

Rather than add new features my work has focused on making Nyx simpler
and faster, but GSoC provides us an opportunity to do even more. So
I'm curious - what do you want from an ncurses monitor? The answer may
be 'keep it simple'. Feature creep does us no favors. But if there's a
good fit I'd love to mentor a project that makes your lives even
better!

I'm not overly fond of the ideas I've had so far...

* Windows support. This poses a few challenges. [2]
* When running multiple tor instances on a single system connect to
them all, aggregating the information.

So anything come to mind?

Cheers! -Damian

[1] https://www.atagar.com/arm/
[2] https://trac.torproject.org/projects/tor/wiki/doc/arm#Windows
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Is tor-wiki-changes ML dead?

[Damian Johnson]

https://lists.torproject.org/pipermail/tor-wiki-changes/2015-December/017387.html


On Sat, Feb 6, 2016 at 1:41 PM, nusenu  wrote:
> Hi,
>
> tor-wiki-changes doesn't seem to send out any emails?
>
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-wiki-changes
>
> https://trac.torproject.org/projects/tor/ticket/18262
>
>
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Git users, enable fsck by default!

[Damian Johnson]

> I suspect that setting things globally (in your ~/.gitconfig)
>   git config --global --add  transfer.fsckobjects true
>   git config --global --add  fetch.fsckobjects true
>   git config --global --add  receive.fsckobjects true
> might also work.  (However, I haven't verified it.)

You only need the first (transfer.fsckobjects). According to both the
thread and git's man page the other two adopt its value ("If not set,
the value of transfer.fsckObjects is used instead.").
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Comparing Stem, metrics-lib, and zoossh

[Damian Johnson]

Hi Karsten, hi Philipp, added these benchmarks to our site...

https://stem.torproject.org/tutorials/mirror_mirror_on_the_wall.html#are-there-any-other-parsing-libraries

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Comparing Stem, metrics-lib, and zoossh

[Damian Johnson]

Yikes, thanks for getting these Karsten! I don't think we should omit
the earlier results since the python community is still very much
split between 2.7 and 3.x. I'll include both so users know they can
upgrade their interpreter to get a nice little speed boost.

Thanks!


On Fri, Jan 15, 2016 at 5:43 AM, Karsten Loesing <kars...@torproject.org> wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 14/01/16 17:22, Damian Johnson wrote:
>> Oh, forgot to talk about compression. You can run the stem script
>> against compressed tarballs but python didn't add lzma support
>> until python 3.3...
>>
>> https://stem.torproject.org/faq.html#how-do-i-read-tar-xz-descriptor-archives
>>
>>  I suppose we could run over bz2 or gz tarballs, or upgrade python.
>> But can't say the compressed benchmark is overly important.
>
> I just ran all the Stem measurements using Python 3, which now
> includes xz tarballs.  The table below contains all results:
>
> server-descriptors-2015-11.tar.xz:
>  - metrics-lib: 0.334261 ms
>  - Stem[**]: 0.63 ms (188%)
>
> server-descriptors-2015-11.tar:
>  - metrics-lib: 0.28543 ms
>  - Stem: 1.02 ms (357%)
>  - Stem[**]: 0.63 ms (221%)
>
> server-descriptors-2015-11/:
>  - metrics-lib: 0.682293 ms
>  - Stem: 1.11 ms (163%)
>  - Stem[**]: 1.03 ms (151%)
>  - Zoossh: 0.458566 ms (67%)
>
> extra-infos-2015-11.tar.xz:
>  - metrics-lib: 0.274610 ms
>  - Stem[**]: 0.46 ms (168%)
>
> extra-infos-2015-11.tar:
>  - metrics-lib: 0.2155 ms
>  - Stem: 0.68 ms (316%)
>  - Stem[**]: 0.42 ms (195%)
>
> consensuses-2015-11.tar.xz:
>  - metrics-lib: 255.760446 ms
>  - Stem[**]: 913.12 ms (357%)
>
> consensuses-2015-11.tar:
>  - metrics-lib: 246.713092 ms
>  - Stem: 1393.10 ms (565%)
>  - Stem[**]: 876.09 ms (355%)
>
> consensuses-2015-11/:
>  - metrics-lib: 283.910864 ms
>  - Stem: 1303.53 ms (459%)
>  - Stem[**]: 873.45 ms (308%)
>  - Zoossh: 83 ms (29%)
>
> microdescs-2015-11.tar.xz[*]:
>  - metrics-lib: 0.099397 ms
>  - Stem[**]: 0.33 ms (332%)
>
> microdescs-2015-11.tar[*]:
>  - metrics-lib: 0.066566 ms
>  - Stem: 0.66 ms (991%)
>  - Stem[**]: 0.34 ms (511%)
>
> [*] The microdescs* tarballs contain microdesc consensuses and
> microdescriptors, but I only cared about the latter; what I did is
> extract tarballs, delete microdesc consensuses, and re-create and
> re-compress tarballs
>
> [**] Run with Python 3.5.1
>
> Is Python 3 really that much faster than Python 2?  Should we just
> omit Python 2 results from this comparison?
>
> All the best,
> Karsten
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
>
> iQEcBAEBAgAGBQJWmPd6AAoJEJD5dJfVqbCrW2IIAL7KyVxDbLczXjtzgwLxFjzw
> s9AjhRILb4cBUwr4N4bFAe6x2rXT5w0dEOweMqjcki7IQ4+/gcjok3PLvT6z6lUW
> 5pKHppU8OmaZItARvGRNlDxWt4E2SSP597GwTWr7rPwwjRRjXmqNPrWAUzq1eteB
> S8os9M2whsEntfUF+aPmZbu2oNzJYdnOL/B139MA72nuo9d6no3CXyTFfvT4a9kV
> K8vg1w54yDtyp15+uVGaJjfbQRJdPRmpjzkSntngnvSL098g1Rq7coRARMrIJ4BB
> 8+WjqtoU5IlnuMS3U/aC/FaXFWLz0vHoXci33ZP+kwmX4GywC1mC/QGbvinlkPk=
> =WQF6
> -END PGP SIGNATURE-
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Comparing Stem, metrics-lib, and zoossh

[Damian Johnson]

> What you could try, though, is extend Zoossh to parse tarballs rather
> than directories.  This is more than 2 times faster in metrics-lib,
> and it doesn't clutter your hard disk with thousands or millions of
> tiny files.

For what it's worth processing tarballs rather than flat files made a
huge difference for Stem as well (tempted to say it was a 5x
improvement). Since you care so much about Zoossh's speed this could
be a great way to make it even faster. :)
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Comparing Stem, metrics-lib, and zoossh

[Damian Johnson]

> This was Stem commit c01a9cda4e7699c7f4bd642c8e81ed45aab7a29b and
> Python version 2.7.10.

Great, thanks! Also what was the metrics-lib and zoossh commits?

> Or should we add these performance tests for metrics-lib, Stem, and
> Zoossh to their own repository that also comes with scripts to fetch
> data from CollecTor?  (Not sure if this is a smart thing to do, but I
> figured I should ask before adding things to the metrics-lib repository.)

Sinister plan #572 is that I'll add these results and scripts to the
page we have with the library comparison. Probably tomorrow. If you'd
care to clean up the metrics-lib examples that would be great.
Otherwise I'll include what we have and you can send me patches later
with what you'd like.

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Comparing Stem, metrics-lib, and zoossh

[Damian Johnson]

Nice! Few questions...

* Where are your metrics-lib scripts used for the benchmarks? Should
be easy for me to write stem counterparts once I know what we're
running. I'll later be including our demo scripts with the benchmarks
later so if possible comments would be nice so they're good examples
for newcomers to our libraries.

* Which exact tarballs are you parsing? It would be useful if we ran
all our benchmarks on the same host with the same data.

* Please take note somewhere of the metric-lib commit id used since
I'll want to include that later when we add the results to our site.

Sorry I didn't get to this for the task exchange. Been focusing on Nyx
so quite a few things have fallen off my radar.

Cheers! -Damian


On Sun, Jan 3, 2016 at 9:56 AM, Karsten Loesing  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hi Damian,
>
> I'm digging out this old thread, because I think it's still relevant.
>
> I started writing some performance evaluations for metrics-lib and got
> some early results.  All examples read a monthly tarball from
> CollecTor and do something trivial with each contained descriptor that
> requires parsing them.  Here are the average processing times by type:
>
> server-descriptors-2015-11.tar.xz: 0.334261 ms
> server-descriptors-2015-11.tar: 0.285430 ms
> extra-infos-2015-11.tar.xz: 0.274610 ms
> extra-infos-2015-11.tar: 0.215500 ms
> consensuses-2015-11.tar.xz: 255.760446 ms
> consensuses-2015-11.tar: 246.713092 ms
> microdescs-2015-11.tar.xz[*]: 0.099397 ms
> microdescs-2015-11.tar[*]: 0.066566 ms
>
> [*] The microdescs* tarballs contain microdesc consensuses and
> microdescriptors, but I only cared about the latter; what I did is
> extract tarballs, delete microdesc consensuses, and re-create and
> re-compress tarballs
>
> These evaluations were all run on a Core i7 with 2GHz using an SSD as
> storage.
>
> Any surprises in these results so far?
>
> Would you want to move forward with the comparison and also include
> Stem?  (And, Philipp, would you want to include Zoossh?)
>
> All the best,
> Karsten
>
>
> On 01/10/15 09:28, Karsten Loesing wrote:
>> Hello Philipp and iwakeh, hello list,
>>
>> Damian and I sat down yesterday at the dev meeting to talk about
>> doing a comparison of the various descriptor-parsing libraries with
>> respect to capabilities, run-time performance, memory usage, etc.
>>
>> We put together a list of things we'd like to compare and tests
>> we'd like to run that we thought we'd want to share with you.
>> Damian and I will both be working on these for metrics-lib for a
>> short while and then switch to Stem.  Please feel free to join us
>> in these effort. The result is supposed to live on Stem's home page
>> unless somebody comes up with a better place.
>>
>> Thanks!
>>
>> All the best, Damian and Karsten
>>
>>
>> On 30/09/15 10:57, Karsten Loesing wrote:
>>> 1. capabilities - supported descriptor types - all the ones on
>>> CollecTor's formats.html - hidden service descriptors (have an
>>> agreed @type for that) - getting/producing descriptors - reading
>>> from file/directory - reading from tarballs - reading from
>>> CollecTor's .xz-compressed tarballs - fetching from CollecTor -
>>> downloading from directories (authorities or mirrors) -
>>> generating (for unit test) - recognizing @type annotation -
>>> inferencing from file name - keeping reading history - user
>>> documentation - validation (format, crypto, successful
>>> sanitization) - packages available - how much usage by (large)
>>> applications
>>
>>> 2. performance (CPU time, memory overhead) - compression:
>>> .xz-compressed tarballs/decompressed tarballs/plain-text -
>>> descriptor type: consensus, server descriptor, extra-info
>>> descriptor, microdescriptors - validation: on or off (allows
>>> lazy loading)
>>
>>> 3. tests by descriptor type - @type server-descriptor 1.0 -
>>> Stem's "List Outdated Relays" - average advertised bandwidth -
>>> fraction of relays that can exit to port 80 - @type extra-info
>>> 1.0 - sum of all written and read bytes from
>>> write-history/read-history - number of countries from which v3
>>> requests were received - @type network-status-consensus-3 -
>>> average number of relays with Exit flag - @type
>>> network-status-vote-3 - Stem's "Votes by Bandwidth Authorities" -
>>> @type dir-key-certificate-3 - @type
>>> network-status-microdesc-consensus-3 1.0 - @type microdescriptor
>>> 1.0 - look at single microdesc cons and microdescs, compile list
>>> of extended families - fraction of relays that can exit to port
>>> 80 - @type network-status-2 1.0 - @type directory 1.0 - @type
>>> bridge-network-status - @type bridge-server-descriptor - @type
>>> bridge-server-descriptor 1.0 - @type bridge-extra-info 1.3 -
>>> @type bridge-pool-assignment - @type tordnsel 1.0 - @type torperf
>>> 1.0
>>
>>> 4. action items - get in touch with Dererk for packaging
>>> metrics-lib for Debian
>>
>>
>>
>
> -BEGIN PGP SIGNATURE-
> 

Re: [tor-dev] Feedback on CollecTor web redesign

[Damian Johnson]

I agree with David. I like the change, especially the table (nice
work!). Though a lot of the text is well into TL;DR territory. If the
welcome message could be half the size that would help, and the
formats have some redundancy. For instance, hidden service descriptors
and bridge pool annotations have just one sub-header so they could be
combined with their top level header.

Cheers! -Damian


On Tue, Oct 20, 2015 at 12:37 PM, David Goulet  wrote:
> On 20 Oct (21:31:38), Karsten Loesing wrote:
>> Hi devs,
>>
>> I just finished a redesign of the CollecTor website and would
>> appreciate your feedback:
>>
>>   https://metrics.torproject.org/index2.html
>
> I think you mean: https://collector.torproject.org/index2.html
>
> :)
>
> I like it. There is quite a bit of text and information pass the Data
> Formats section but what I really enjoy now (from which I got annoyed
> from the original design) is the two buttons for recent/ and archive/
> descriptors. Way easier to get then before where I had to go in a
> subsection and then click recent/ in the middle of the paragraph.
>
> Cheers!
> David
>
>>
>> For reference, the old CollecTor website is still available here:
>>
>>   https://metrics.torproject.org/index.html
>>
>>   https://metrics.torproject.org/formats.html
>>
>> Let me add that I'm not a web designer, and let me prove that
>> statement by telling you that all HTML on that website was
>> written using vim.  I'm more than happy to accept patches or
>> suggestions, though I'll likely ask you to explain them to me.
>>
>> Thanks!
>>
>> All the best,
>> Karsten
>> ___
>> tor-dev mailing list
>> tor-dev@lists.torproject.org
>> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Feedback on CollecTor web redesign

[Damian Johnson]

> I agree with David. I like the change, especially the table (nice
> work!). Though a lot of the text is well into TL;DR territory. If the
> welcome message could be half the size that would help, and the
> formats have some redundancy. For instance, hidden service descriptors
> and bridge pool annotations have just one sub-header so they could be
> combined with their top level header.

Hmmm, now that I think about this some more I wonder if the two of us
should do some collaboration on this. Stem's descriptor page and
CollecTor both provide differing general information...

Stem - https://stem.torproject.org/tutorials/mirror_mirror_on_the_wall.html

  * What is a descriptor?
  * Where do descriptors come from?
  * Brief summary of the most important descriptor types
  * Library comparison we're working on

CollecTor - https://collector.torproject.org/index2.html

  * Detailed listing of all descriptor types
  * Reference users on where to get 'em

Perhaps we should combine parts of both for a "New to descriptors?
Lets give you a very simple, short description of them". Your table
would be perfect for that.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Comparing Stem, metrics-lib, and zoossh

[Damian Johnson]

> Damian and I sat down yesterday at the dev meeting to talk about doing
> a comparison of the various descriptor-parsing libraries with respect
> to capabilities, run-time performance, memory usage, etc.

Hi Karsten, started moving this forward with the easy bit: a table
comparing capabilities. Mind taking a peek? Philipp, is this accurate
for Zoossh?

https://stem.torproject.org/tutorials/mirror_mirror_on_the_wall.html#are-there-any-other-parsing-libraries

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Comparing Stem, metrics-lib, and zoossh

[Damian Johnson]

> The only thing that's wrong is that zoossh can detect types by looking
> at @type.

Thanks! Fix pushed, it'll show up in a few minutes.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Trac priorities and severities

[Damian Johnson]

Hi David. Personally I find the split severity and priority redundant
but that's fine (milestone, tor version, and other fields are only
applicable to core tor - it's easy to ignore yet another field).
However, in this case you made it mandatory by giving it a default.
Mind making priority optional?

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Let's identify which measurement-related tools need work when relays switch from RSA identities to ed25519 identities

[Damian Johnson]

> It's true, dropping the fingerprint is quite invasive and might break
> things.  But that's why we're making plans now to make this transition
> as smooth as possible.
>
> However, I don't think that we can get away by just replacing the
> existing 20 byte long RSA key digest with a 20 byte long hash of the
> new, 32 byte long ed25519 key.  There are probably cryptographic
> reasons for that.  But in addition to that, there should be a time
> when relays use both fingerprints in parallel so that directory
> authorities and other tools can build a map from old to new
> identities.  Otherwise, relays would lose all their history only
> because they are switching from RSA to ed25519, and wouldn't that be
> sad?  There might be more reasons that I'm currently not thinking of.
>  But: I'm not making this call, I'm just thinking about the possible
> impact of this change and which code needs to be updated.
>
> How about we talk more about this in Berlin together with Nick?  Maybe
> take a look at the list I wrote and think of other code that will need
> work when this change actually takes place?

Certainly, sounds good! For what it's worth my concern is about the
fingerprint's longstanding role as the relay identifier. Some spots
off the top of my head...

* The control spec allows you to query descriptors by fingerprint or nickname.
* Circuit events and most other things citing relays do so by
fingerprint or fingerprint~nickname.
* Atlas, Globe, and just about every tool that allows you to look up
relay information does so by fingerprint.
* In Stem anything related to identifying a relay does so by fingerprint.

No doubt there's many, many other things that'll break too. This is
the main identifier of a relay and I'd expect mucking with it to break
all the things. Hence my interest in an in-place replacement instead.
;)

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Troubleshooting Tor 0.2.7.1+ ephemeral hidden services

[Damian Johnson]

Hi Micah, on first glance looks fine. For what it's worth here's the
example I got working a while back...

https://stem.torproject.org/tutorials/over_the_river.html#ephemeral-hidden-services

Cheers! -Damian


On Tue, Sep 8, 2015 at 4:58 PM, Micah Lee  wrote:
> I'm trying to add support for ephemeral hidden services to OnionShare.
> They'll solve a wide variety of problems related to OnionShare using a
> system Tor instead of Tor browser, and running in environments like
> Tails or Whonix. I'd rather never write the hidden service info to disk
> anyway.
>
> I noticed that Tor Browser 5.5a2 comes with Tor 0.2.7.2-alpha, so it's
> easy to test. But so far I'm not having any success, at least with using
> stem.
>
> Before I open a bug report, either against stem or against tor, am I
> doing this right? I've made a simple test project:
>
> https://github.com/micahflee/ephemeral-hs-test
>
> It successfully appears to create the hidden service, but it never
> succeeds in connecting to itself. I also can't connect to it using Tor
> Browser.
>
> --
> Micah Lee
>
>
> ___
> tor-dev mailing list
> tor-dev@lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Let's identify which measurement-related tools need work when relays switch from RSA identities to ed25519 identities

[Damian Johnson]

>> Hi Karsten. Quick question: with the switchover are relay fingerprints
>> going away? That is to say, server descriptors no longer have a...
>>
>> fingerprint D203 4DDF 1275 A234 4F66 9935 A3EF B908 FFC7 AE9A
>>
>> ... line, and router status entries don't have it on their dir-source?
If not, what in particular are being dropped?
>>
>> Cheers! -Damian
>
> yes, I think that's the plan.  Please see proposal 220, Sections 8 and
> 9 for some more details.

Thanks Karsten. Now that I know this isn't a completely foolish
question adding tor-dev@ back in.

If we're dropping the fingerprint that's a lot more invasive.
Fingerprints are relay's canonical identity, and simply dropping them
will break... a lot. I'd suggest instead relays should continue to
have a fingerprint but that it's the 40 character hex hash of the
ed25519 identity.

On a side note it would be nice to have a spec patch before changing
things this time. Sections 8 and 9 are a fine summary, but it's not
clear to me precisely how the descriptors are changing from it. They
certainly don't say to me "we're dropping the fingerprint field". If
that change went out without a spec patch first I'd borrow Mr. Potato
Head's angry eyes to stare at its author.

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Get Stem and zoossh to talk to each other

[Damian Johnson]

 Why are you surprised? Python is a very, very slow language. My very simple 
 unoptimized C++ parsing code is more than 10x faster than Stem (parsing from 
 memory). Go has some nice string parsing features, so zoossh should have an 
 easy time to get fast and simple parsing code.

Because with either implementation (Stem or Zoossh) I expected disk IO
to be the lion's share of the time. Both do lazy parsing so the
earlier results didn't make sense. Being cached in memory this makes
more sense.

One other thought is that the DocumentHandler.ENTRIES is telling Stem
to read the content to provide you with individual RouterStatusEntry
instances. If zoossh isn't parsing out individual entries then that
may skew the comparison. Reading server descriptors would be a better
test in this regard.

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Get Stem and zoossh to talk to each other

[Damian Johnson]

  Ideally, zoossh should do the heavy lifting as it's implemented in a
  compiled language.

 This is assuming zoossh is dramatically faster than Stem by virtue of being
 compiled. I know we've discussed this before but I forget the results - with
 the latest tip of Stem (ie, with lazy loading) how do they compare? I'd 
 expect
 time to be mostly bound by disk IO, so little to no difference.

 zoossh's test framework says that it takes 36364357 nanoseconds to
 lazily parse a consensus that is cached in memory (to eliminate the I/O
 bottleneck).  That amounts to approximately 27 consensuses a second.

 I used the following simple Python script to get a similar number for
 Stem:

 with open(file_name) as consensus_file:
 for router in stem.descriptor.parse_file(consensus_file,
 'network-status-consensus-3 1.0',
 document_handler = stem.descriptor.DocumentHandler.ENTRIES):
 pass

 This script manages to parse 24 consensus files in ~13 seconds, which
 amounts to 1.8 consensuses a second.  Let me know if there's a more
 efficient way to do this in Stem.

Interesting! First thought is 'wonder if zoossh is even reading the
file content'. Couple quick things to try are...

with open(file_name) as consensus_file:
  consensus_file.read()

... to see how much time is disk IO verses parsing. Second is to try
doing something practical (say, count the number of relays with the
exit flag). Stem does some bytes = unicode normalization which might
account for some difference but other than that I'm at a loss for what
would be taking the time.

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Relay Dashboard Code Review

[Damian Johnson]

Hi Cristobal, again apologies for the delay in getting you feedback. Just gave
RWSD a whirl and worked great - nice work!

From a UI perspective only feedback on what we have so far is...

  * Both graphs just have the title of 'Graph component'. Better would be to
say if it's inboud or outbound.

  * Graph's y-axis should indicate its units.

  * Personally I kinda like Vidalia's line graph better than bar graphs for
this, but obviously just something to experiment with. Lots of options.


https://upload.wikimedia.org/wikipedia/commons/e/ef/Tor-non-exit-relay-bandwidth-usage.jpg

  * Only having fifteen seconds of information is pretty short. Actually, if
you raise it to something high (like several minutes of data) then you
might want to consider prepopulating bandwidth information from before
RWSD started. Here's how Nyx does it...

https://gitweb.torproject.org/nyx.git/tree/nyx/graph_panel.py#n244


Unfortunately I'm still a bit of an Angular noob so I can't comment much on
those bits. Arturo: do you have any thoughts?

Otherwise, here's some feedback on the code!



run_client.py


 23 settings = dict(
 24 template_path=os.path.join(os.path.dirname(__file__),
client/templates),
 25 static_path=os.path.join(os.path.dirname(__file__),
client/static),
 26 autoescape=None,
 27 )
 28 handlers = [
 29 (r/, MainHandler),
 30 ]
 31 cyclone.web.Application.__init__(self, handlers, **settings)

Very minor thing, but little odd using dict() that way. This could also be...

  handlers = [
(r/, MainHandler),
  ]

  cyclone.web.Application.__init__(self, handlers, {
template_path: os.path.join(os.path.dirname(__file__), client/templates),
static_path: os.path.join(os.path.dirname(__file__), client/static),
autoescape: None,
  })

Again, minor and up to you. run_server.py has the same. This is similar to
what I do all throughout my codebases...

  https://gitweb.torproject.org/nyx.git/tree/nyx/graph_panel.py#n71



 37 reactor.listenTCP(int(config.get('client.port')), Application())

Actually, you don't need the int() if you provide a default. If you call...

  config.get('client.port', 9051)

... then the config will give you an int, and emit a warning if the config
has something else...

  https://stem.torproject.org/api/util/conf.html#stem.util.conf.Config.get

Same in run_server.py.


run_server.py


 35 if(dual_mode()):

Good idea on this config option. In python you don't use parentheses for
conditionals, unless you're doing some grouping.

Speaking of which, might I suggest baking pyflakes and pep8 checks into your
project? Even if you don't have unit tests yet these can speed development
by catching minor issues, and help the codebase have a more uniform style.

Stem provides a couple utilities that should make this easy. Here's an
example of using them...

  https://gitweb.torproject.org/nyx.git/tree/run_tests.py

Oh, and it'll catch the trailing whitespaces that's slipped in. ;)

  % grep -R ' $' rwsd/* | grep -v 'Binary file' | wc -l
  17



 45 @uses_settings
 46 def main(config, dual = True):

The 'dual = True' argument is unused. You use the dual_mode() util function
instead.



 87 except:
 88 log.msg(unable to attach listeners, controller off)
 89 except:
 90 log.msg(unable to start tor controller from rwsd)

_conn_listener() is a neat approach. I like it.

Generally it's good idea to both only catch the exception types you need, and
include the exception in the message so there's some hint for troubleshooting.
The second, for instance, is in effect just calling stem.connection.connect(),
which only raises ValueErrors (for invalid arguments)...

  https://stem.torproject.org/api/connection.html#stem.connection.connect

Seeing how you use it here you probably want Controller.from_port() instead.
As we discussed earlier connect() prints errors to stdout and returns None
if it can't connect. However, you're not checking if it returns None so
the second try/catch will hit an AttributeError.

I suspect there *might* be a bug around double attaching listeners when tor is
connected then reconnected, but not sure.


util/__init__.py


 49 def msg(message, config, **attr):

Unused, but feel free to leave it alone if you plan on using it later.

Re: [tor-dev] Relay Dashboard Code Review

[Damian Johnson]

   def get_websockets(ws_type = None):
 websocket = WEBSOCKETS.get(ws_type, [])
 return websocket if websocket else None

 How about just:

 def get_websockets(ws_type=None):
 return WEBSOCKETS.get(ws_type, None)

Thanks meejah for the additional feedback! For this last one as I
understand it WEBSOCKETS contains values that are empty lists, and he
wants this function to return None when that's the case. That's why I
did it the way above.
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Get Stem and zoossh to talk to each other

[Damian Johnson]

Hi Philipp, sorry about the delay! Spread pretty thin right now. Would you
mind discussing more about the use cases, and give a mockup for what this
new domain specific language would look like in practice?

My first thought is would such a language be useful enough to be worth
investing time to learn?. I've made lots of things that flopped because
they didn't serve a true need, and while a domain specific language for
descriptors sounds neat I'm not sure if I'm seeing a need for it.

Roger occasionally asks me to write one-off scripts to answer questions
about the tor network, such as how do the votes of dirauth X compare with Y?
or how many relays are unmeasured by the bandwidth auths?...

  https://stem.torproject.org/tutorials/examples/compare_flags.html
  
https://stem.torproject.org/tutorials/examples/votes_by_bandwidth_authorities.html

These questions generally take me fifteen minutes or so to answer. Yes, yes,
I'm the author of Stem so that's skewed. But still, the descriptor APIs are
simple enough that anyone should be able to do much the same with only a basic
knowledge of Python.

 Ideally, zoossh should do the heavy lifting as it's implemented in a
 compiled language.

This is assuming zoossh is dramatically faster than Stem by virtue of being
compiled. I know we've discussed this before but I forget the results - with
the latest tip of Stem (ie, with lazy loading) how do they compare? I'd expect
time to be mostly bound by disk IO, so little to no difference.

 1. Let zoossh do the data filtering and then return a list of files that
are then parsed again by Stem.  That's easy to implement, but can be
quite inefficient if the filtering step still returns plenty of data.

Yup, agreed. This plan would essentially be to double parse the results and
I'd expect it to be far slower than using either library alone.

 2. Have some IPC mechanism that passes objects from zoossh to Stem.
Objects could be serialized in some way to minimize unnecessary
parsing.  While that might be the most efficient option for now, it
probably requires too much work.

Again agreed. Theoretically possible - you could make a blank Stem descriptor
object, then populate its attributes with the zoossh parsed results. However,
this would require you to maintain the hand-built conversion function. And
again, I'm also doubtful it would yield a performance benefit in practice.

None of this is to say 'don't give it a shot'. If you think this would be a
fun project then by all means dig in! Just voicing my two cents that our
efforts might be better spent elsewhere. ;)

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] Relay Web Dashboard - Summer of Privacy

[Damian Johnson]

 is this strictly a one dashboard for one tor daemon or will this also be
 usable to monitor multiple tor instances without running multiple
 dashboard instances in parallel?

 It's being developed as one tor instance - one dashboard instance,
 although it'd be a nice feature to include what you mention into
 **future development**.

Hi nusenu. This is something I've long wanted to do with Nyx, and
definitely something for long term consideration. It's a valid use
case, especially for gigabit relay families that run multiple
instances to saturate their connection. We did this, for instance,
with Amunet. But that said, not a feature for the initial release.

 Would that include data that is not available via the controlport,
 like exit/guard probability, (measured bw), cw and cw fraction
 (available via onionoo) or is that not something that you are planing
 to include?

Think I'm gonna need to say 'patches welcome' for these. First step
would be to add them to Stem. Anything in Onionoo is easy to bring in
(it's chiefly just distilled descriptor information), but I'm not
positive about the exit/guard probabilities. Is it a simple ratio? Or
do tor clients do something more sophisticated? If it's something
simple I'd be happy to take a patch adding it to the Consensus class.

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Re: [tor-dev] additional triggers for DocTor?

[Damian Johnson]

Hi nusenu. DocTor's current sybil checker [1] is quite simple and
there's a lot of directions in which it can be improved. You and
others are more than welcome to improve it - patches very welcome. :)

[1] https://gitweb.torproject.org/doctor.git/tree/sybil_checker.py


On Fri, Jun 5, 2015 at 9:38 AM, nusenu nus...@openmailbox.org wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA512

 Hi Damian,

 does DocTor's design allow for triggers that alert if a certain amount
 of relays signed up within one day or within a week (instead of within
 one hour)?

 Another trigger could be the last_restarted timestamp.

 If that makes sense I would file feature requests on trac and provide
 some possible thresholds.

 Example of an event that would trigger then (2015-04-01):
 https://lists.torproject.org/pipermail/tor-talk/2015-April/037384.html
 -BEGIN PGP SIGNATURE-

 iQIcBAEBCgAGBQJVcdCSAAoJEFv7XvVCELh0l5IP/R0NeXImuchqastZcg4T4PK7
 Qy5VmFrWBdG9AOnypxIMgzVF5/M+hKjRgbbAp9/VrRXqpbFCgnChnQvqyJcW1nB0
 AYGfqEUfavTBthbtTfq6g43BLBZke0olBMVFlKHaAYTidXMabrY0OlZ3QjdHt1Sx
 DovQ/zGVX263VR1YZimXom1xokCDm7J51vPURjx9BLrBQ2zdEGvOWWwdT+w1CiaK
 iH4I0AGUPZfLxxLU7rCorFAIV2AyKTT97EN9lQlD3TtPmoDiZdhPthR2MRKVruEP
 3Kt76VDMRPro6oKo2d3sk5UFuyh0Hg+yqJhTJhjdZ0w/ZQtbIjlkHbnfSqgthEPL
 fi2XKQrX1BSfbQpsig70FRFHgKEQVzplXLlca9o1sn52GrwdY8fzNLc0p8xZiFSc
 xoognuyER4IgLP10c2QsYSoCx50yfpYOxDZIEFVIelDDKGIqVQzR/rGRwBfXBcn/
 598SPhka+h0BK/NudG/k5qRlsdWpQdygeAfDBUUO6AxHaIH2P96Pe1CfxNkoMGX4
 TWZ2+pTXye6wdiFhdhDYGPuCtmOgI5GQ5alHEvsk3s7/n0YitRRNHNiNe3SGw7UN
 hX6HQuWmxD5J/D1fcj5gC/gJZxkeluTRO6iOP1AoNAqP9XGgQeT7uwhL+FzT65V5
 GlMIH20AeBn2FWIesaxf
 =Ia4Q
 -END PGP SIGNATURE-
 ___
 tor-dev mailing list
 tor-dev@lists.torproject.org
 https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Damian's Status Report - May 2015

[Damian Johnson]

Squirrels! Five newborn squirrels are in our tree! And they... oh right, code.
Talk about code, Damian, like a professional.

... awww, but damn they're cute.


Stem Release 1.4.1


Though not as adorable as baby squirrels the new release of Stem is still
pretty sweet. Ephemeral hidden services, substantially faster descriptor
parsing, and a host of other improvements can all be yours with a simple
upgrade!

  https://blog.torproject.org/blog/stem-release-14


Nyx Log Panel Rewrite


No, Nyx isn't done but its log panel is! This is the third major curses
component to be rewritten, and I'm delighted to say it now sucks less!

How did it suck, you ask? Though casual users may not have noticed arm had
a laughably inefficient O(n^3) approach to log deduplication. This was in
fact so terrible arm simply turned off deduplication when it started
bogging down the system too much.

This is just one of they myriad of improvements, but in Nyx deduplication
is O(n^2) and can easily be made linear (... though fuzzy matching makes
this grosser than I'd like so first seeing if further improvement is
warranted).



Few other noteworthy things were...

* Thanks to DonnchaC Stem's tor message parsing is now a dramatic 300x faster
  when using python3!

  https://github.com/DonnchaC/stem/pull/1

* Wrote a simple bandwidth scanner to exemplify manual circuit construction
  for our tutorials...

  
https://stem.torproject.org/tutorials/to_russia_with_love.html#custom-path-selection

* Sambuddha both made our tutorial examples downloadable (#10411) and more
  easily testable (#16191)!

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[tor-dev] Simple Relay Scanner Example

[Damian Johnson]

Hi Roger, earlier this week you asked me for a simple
bandwidth-authority style scanner. This dovetailed nicely with a task
that's been on my todo list for ages to make a tutorial for manual
path selection so here ya go!

https://stem.torproject.org/tutorials/to_russia_with_love.html#custom-path-selection

Cheers! -Damian
___
tor-dev mailing list
tor-dev@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev