Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  closed
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:  fixed
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  1.15
Parent ID:  #33321 | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+
Changes (by acute):

 * actualpoints:  0.85 => 1.15


Comment:

 Adding my 0.3 actual points to yours :)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  closed
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:  fixed
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.85
Parent ID:  #33321 | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+
Changes (by karsten):

 * status:  merge_ready => closed
 * resolution:   => fixed
 * actualpoints:  0.45 => 0.85


Comment:

 Thanks a lot for the detailed explanation of how `PYTHONPATH` works. I had
 somehow assumed that the `onionperf` script would be found by specifying
 the OnionPerf directory in `PYTHONPATH`. When specifying both paths it
 works as expected.

 Thanks, also, for rebasing and providing all patches as a single path.
 That's quite a bit easier for me to review and merge.

 Pushed to master. Finally! Closing.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  merge_ready
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.45
Parent ID:  #33321 | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+

Comment (by acute):

 Replying to [comment:18 karsten]:
 > Almost there! Two further questions:
 >
 >  1. What if I want to run `onionperf` from another directory than
 `onionperf/onionperf`? I tried passing another relative or absolute path
 to `PYTHONPATH=`, but all I get is: `/usr/bin/python3: can't find
 '__main__' module in 'onionperf'`.
 You also need to specify the absolute/relative path to the onionperf
 script as well in that case, for example:

 `PYTHONPATH=/home/dev/onionperf python3
 /home/dev/onionperf/onionperf/onionperf`

 Just as a note, you can copy the script itself anywhere else on the system
 and this will still work, as long as you point the PYTHONPATH to the
 onionperf directory.
 {{{
 cp /home/dev/onionperf/onionperf/onionperf /home/dev/op
 PYTHONPATH=/home/dev/onionperf python3 /home/dev/op
 }}}
 or even directly:
 {{{
 PYTHONPATH=/home/dev/onionperf /home/dev/op
 }}}
 ...since the python3 interpreter is in a sensible place anyway.
 Hope this helps!

 >
 >  2. Do you mind posting a new branch with phw's branch rebased to
 current master and with your two patches (the one on #33433 and the one
 above) on top? I would do this myself, but I'm having trouble applying
 your patches, and I don't want to claim authorship of these commits.

 This is now here:
 https://github.com/ana-cc/onionperf/tree/acute/30586

 Thank you!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  merge_ready
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.45
Parent ID:  #33321 | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+

Comment (by karsten):

 Almost there! Two further questions:

  1. What if I want to run `onionperf` from another directory than
 `onionperf/onionperf`? I tried passing another relative or absolute path
 to `PYTHONPATH=`, but all I get is: `/usr/bin/python3: can't find
 '__main__' module in 'onionperf'`.

  2. Do you mind posting a new branch with phw's branch rebased to current
 master and with your two patches (the one on #33433 and the one above) on
 top? I would do this myself, but I'm having trouble applying your patches,
 and I don't want to claim authorship of these commits.

 Thanks!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  merge_ready
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.45
Parent ID:  #33321 | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+

Comment (by acute):

 Replying to [comment:16 karsten]:

 > However, I'm still having difficulties with building and running
 OnionPerf without using `pip`: The command `python3 setup.py install
 --user` still runs `pip` in the background, and I'm unclear how to set
 `PYTHONPATH` in order to avoid running that command in the first place.
 Once I know how to do that I can merge this branch (and apply the #33433
 patch on top of it).
 Apologies, I should have added this.
 Once all the dependencies are installed for Onionperf, this is as simple
 as:

 {{{
 git clone https://git.torproject.org/onionperf.git
 cd onionperf/onionperf
 PYTHONPATH=.. python3 onionperf
 }}}
 >
 > Another minor issue is that `python3-setuptools` needs to be listed as
 required package in the instructions now. I can add that to `README.md`,
 if that's the only remaining issue.

 I've attached a patch for README.md that includes `python3-setuptools` in
 the instructions, and also includes the instructions above on how to run
 OP without installing it - should be a quick review :)
 >
 > Thanks for your patience! :)
 Thanks very much, sorry for taking longer than anticipated.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  merge_ready
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.45
Parent ID:  #33321 | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+
Changes (by acute):

 * Attachment "0001-Adds-setuptools-and-alternative-installation-
 instruc.patch" added.


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  merge_ready
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.45
Parent ID:  #33321 | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+

Comment (by karsten):

 Okay, I agree with the general approach to have different steps for
 researchers and developers and for ourselves.

 However, I'm still having difficulties with building and running OnionPerf
 without using `pip`: The command `python3 setup.py install --user` still
 runs `pip` in the background, and I'm unclear how to set `PYTHONPATH` in
 order to avoid running that command in the first place. Once I know how to
 do that I can merge this branch (and apply the #33433 patch on top of it).

 Another minor issue is that `python3-setuptools` needs to be listed as
 required package in the instructions now. I can add that to `README.md`,
 if that's the only remaining issue.

 Thanks for your patience! :)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  merge_ready
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.45
Parent ID:  #33321 | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+

Comment (by phw):

 Replying to [comment:13 acute]:
 > Replying to [comment:10 karsten]:
 > > Thanks for these patches! And thanks for copying metrics-team; I had
 not seen the earlier updates (oh, Trac...).
 > >
 > > So, I tried out this branch without installing all dependencies via
 `sudo apt install python3-*`, and it starts downloading packages via
 `pip`. In our Java projects we have always avoided using package managers
 and only relied on packages provided by the system. We would give up this
 principle here, but are we sure we want to do that? How are we doing this
 in other Python projects at Tor?
 >
 > I was wondering whether there is a distinction here that we need to make
 - how do {researchers, developers} install and use onionperf on their
 machines vs how we deploy onionperf in Tor.
 [[br]]
 That's a good point. For third parties, installation should be quick,
 easy, and work the way it does for other Python tools. For ourselves, we
 do whatever makes the most sense for us – and that doesn't have to be
 documented in the README.md.

 On an unrelated note: I added a commit to my patch set that adds the
 pandas and seaborn requirement:
 https://github.com/NullHypothesis/onionperf/compare/defect/30586

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  merge_ready
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.45
Parent ID:  #33321 | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+
Changes (by acute):

 * actualpoints:  0.35 => 0.45


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  merge_ready
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.35
Parent ID:  #33321 | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+

Comment (by acute):

 Replying to [comment:10 karsten]:
 > Thanks for these patches! And thanks for copying metrics-team; I had not
 seen the earlier updates (oh, Trac...).
 >
 > So, I tried out this branch without installing all dependencies via
 `sudo apt install python3-*`, and it starts downloading packages via
 `pip`. In our Java projects we have always avoided using package managers
 and only relied on packages provided by the system. We would give up this
 principle here, but are we sure we want to do that? How are we doing this
 in other Python projects at Tor?

 I was wondering whether there is a distinction here that we need to make -
 how do {researchers, developers} install and use onionperf on their
 machines vs how we deploy onionperf in Tor.
 For general purpose use/development, using python-setuptools (which is a
 standard way of packaging python modules and uses pip to manage
 dependencies) and then running onionperf on the command line is entirely
 reasonable.

 For 'official' deployments, not so much. There, it makes more to install
 dependencies directly from Debian, and use a systemd service to run
 Onionperf. The actual 'python3 setup.py install' step is not even
 necessary, as you can just run onionperf directly from the git repository,
 as long as the PYTHONPATH is set. This is mostly what the ansible workflow
 does.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  merge_ready
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.35
Parent ID:  #33321 | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+

Comment (by karsten):

 With #33258 being merged we'll also have to add python3-pandas and
 python3-seaborn to the list of requirements.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  merge_ready
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.35
Parent ID: | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+

Comment (by karsten):

 Thanks for these patches! And thanks for copying metrics-team; I had not
 seen the earlier updates (oh, Trac...).

 So, I tried out this branch without installing all dependencies via `sudo
 apt install python3-*`, and it starts downloading packages via `pip`. In
 our Java projects we have always avoided using package managers and only
 relied on packages provided by the system. We would give up this principle
 here, but are we sure we want to do that? How are we doing this in other
 Python projects at Tor?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  merge_ready
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.35
Parent ID: | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+
Changes (by acute):

 * cc: metrics-team (added)
 * actualpoints:  0.25 => 0.35


Comment:

 Adding my 0.1 points to yours, and CC'ing metrics-team!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  merge_ready
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.25
Parent ID: | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+
Changes (by phw):

 * status:  needs_review => merge_ready
 * actualpoints:   => 0.25


Comment:

 Replying to [comment:7 acute]:
 > I've pushed a minor fixup commit onto my github branch to remove the
 duplication of the requirements:
 >
 > https://github.com/ana-
 cc/onionperf/commit/893306c97bff0526c42ee97dda789ebf310304be.patch
 >
 > With this applied, it is good to merge. Phw, if you are happy with this
 you can set this to merge-ready!
 [[br]]
 That's a useful patch, thanks! I force-pushed it to my GitHub branch,
 making this ready for merge now.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  needs_review
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:
Parent ID: | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+

Comment (by acute):

 I've pushed a minor fixup commit onto my github branch to remove the
 duplication of the requirements:

 https://github.com/ana-
 cc/onionperf/commit/893306c97bff0526c42ee97dda789ebf310304be.patch

 With this applied, it is good to merge. Phw, if you are happy with this
 you can set this to merge-ready!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  needs_review
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:
Parent ID: | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+
Changes (by phw):

 * status:  assigned => needs_review


Comment:

 I tried to find undocumented modules by running:
 {{{
 find onionperf/ -name "*.py" | xargs grep -h import | sort | uniq
 }}}
 This only yielded nose and nose-cov, which are used for tests.

 I also tried to find unused modules by running:
 {{{
 grep --exclude-dir=venv -ri MODULE
 }}}
 This yielded Twisted, cycler, decorator, pyparsing, python-dateutil, pytz,
 six, and zope.interface.

 Here's my patch set:
 https://github.com/NullHypothesis/onionperf/compare/defect/30586

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:
Parent ID: | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59-must
---+
Changes (by phw):

 * owner:  metrics-team => phw
 * status:  reopened => assigned


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30586 [Metrics/Onionperf]: requirements are not included in setup.py

[Tor Bug Tracker & Wiki]

#30586: requirements are not included in setup.py
---+--
 Reporter:  irl|  Owner:  metrics-team
 Type:  defect | Status:  reopened
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:
Parent ID: | Points:  0.5
 Reviewer: |Sponsor:  Sponsor59
---+--
Changes (by gaba):

 * keywords:   => metrics-team-roadmap-2020
 * points:   => 0.5
 * sponsor:   => Sponsor59


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs