Re: [tor-bugs] #29369 [Metrics/Onionperf]: Fix message logging and filtering

[Tor Bug Tracker & Wiki]

#29369: Fix message logging and filtering
-+-
 Reporter:  irl  |  Owner:  phw
 Type:  defect   | Status:  closed
 Priority:  Medium   |  Milestone:
Component:  Metrics/Onionperf|Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:  metrics-team-roadmap-2020, metrics-  |  Actual Points:  1.0
  team-roadmap-2020-june |
Parent ID:  #33321   | Points:  1.0
 Reviewer:  phw  |Sponsor:
 |  Sponsor59-must
-+-
Changes (by karsten):

 * status:  merge_ready => closed
 * resolution:   => fixed
 * actualpoints:  1 => 1.0


Comment:

 Thanks for those suggestions, I included both of them. Rebased, squashed,
 pushed to master, and now closing as fixed. Thanks again!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29369 [Metrics/Onionperf]: Fix message logging and filtering

[Tor Bug Tracker & Wiki]

#29369: Fix message logging and filtering
-+-
 Reporter:  irl  |  Owner:  phw
 Type:  defect   | Status:
 |  merge_ready
 Priority:  Medium   |  Milestone:
Component:  Metrics/Onionperf|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-team-roadmap-2020, metrics-  |  Actual Points:  1
  team-roadmap-2020-june |
Parent ID:  #33321   | Points:  1.0
 Reviewer:  phw  |Sponsor:
 |  Sponsor59-must
-+-
Changes (by phw):

 * status:  needs_review => merge_ready
 * actualpoints:  0.5 => 1


Comment:

 Two comments:
 * `-MM-YY` should probably be `-MM-DD`.
 * The string "onionperf.analysis.json.xz" shows up thrice in the save
 function. We could use a variable instead, e.g., `base_filename =
 "onionperf.analysis.json.xz"`.

 Feel free to address these comments as you see fit. I don't think this
 patch requires another review round, so I'm setting it to merge ready and
 adding my 0.5 points.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29369 [Metrics/Onionperf]: Fix message logging and filtering

[Tor Bug Tracker & Wiki]

#29369: Fix message logging and filtering
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  needs_review
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.5
Parent ID:  #33321 | Points:  1.0
 Reviewer:  phw|Sponsor:  Sponsor59-must
---+
Changes (by karsten):

 * actualpoints:  0.4 => 0.5


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29369 [Metrics/Onionperf]: Fix message logging and filtering

[Tor Bug Tracker & Wiki]

#29369: Fix message logging and filtering
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  needs_review
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.4
Parent ID:  #33321 | Points:  1.0
 Reviewer:  phw|Sponsor:  Sponsor59-must
---+
Changes (by karsten):

 * status:  assigned => needs_review
 * reviewer:   => phw


Comment:

 Not quite, but it's going in the direction I had in mind. Please take a
 look at
 
[https://gitweb.torproject.org/user/karsten/onionperf.git/commit/?h=task-29369=251f4576e97287b00a1136c08ef3fde5de92c901
 squash commit 251f457 in my task-29369 branch].

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29369 [Metrics/Onionperf]: Fix message logging and filtering

[Tor Bug Tracker & Wiki]

#29369: Fix message logging and filtering
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.4
Parent ID:  #33321 | Points:  1.0
 Reviewer: |Sponsor:  Sponsor59-must
---+

Comment (by phw):

 Replying to [comment:12 karsten]:
 > Without the date prefix the output file written at midnight will
 overwrite the output file from one day before. Do you prefer if I write
 this patch early next week and you review it?
 [[br]]
 I think I now understand. Is this (untested) code what you had in mind?
 https://github.com/NullHypothesis/onionperf/compare/defect/29369

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29369 [Metrics/Onionperf]: Fix message logging and filtering

[Tor Bug Tracker & Wiki]

#29369: Fix message logging and filtering
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.4
Parent ID:  #33321 | Points:  1.0
 Reviewer: |Sponsor:  Sponsor59-must
---+

Comment (by karsten):

 Without the date prefix the output file written at midnight will overwrite
 the output file from one day before. Do you prefer if I write this patch
 early next week and you review it?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29369 [Metrics/Onionperf]: Fix message logging and filtering

[Tor Bug Tracker & Wiki]

#29369: Fix message logging and filtering
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.4
Parent ID:  #33321 | Points:  1.0
 Reviewer: |Sponsor:  Sponsor59-must
---+

Comment (by phw):

 Replying to [comment:8 karsten]:
 > I didn't work on the implementation yet and would like to leave this to
 others, also as a way to sanity check the concept. If I were to implement
 this I'd probably avoid using the `date_filter` parameter when doing the
 nightly analysis and introduce some sort of `date_prefix` parameter. We
 should just use that `date_filter` by default for all analysis files, and
 we might consider deprecating the `date_prefix` parameter in the user
 interface. When testing this, we'll have to ensure that it works for the
 nightly case as well as the reprocessing case.
 [[br]]
 I'm afraid I don't follow. Don't we accomplish our goal by simply omitting
 the date filter when analysing the results? I don't understand what the
 date prefix should do here.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29369 [Metrics/Onionperf]: Fix message logging and filtering

[Tor Bug Tracker & Wiki]

#29369: Fix message logging and filtering
---+
 Reporter:  irl|  Owner:  phw
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.4
Parent ID:  #33321 | Points:  1.0
 Reviewer: |Sponsor:  Sponsor59-must
---+
Changes (by phw):

 * cc: metrics-team (added)
 * owner:  metrics-team => phw


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29369 [Metrics/Onionperf]: Fix message logging and filtering

[Tor Bug Tracker & Wiki]

#29369: Fix message logging and filtering
---+
 Reporter:  irl|  Owner:  metrics-team
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.4
Parent ID: | Points:  1.0
 Reviewer: |Sponsor:  Sponsor59-must
---+
Changes (by karsten):

 * status:  accepted => assigned
 * owner:  karsten => metrics-team
 * points:  1 => 1.0
 * actualpoints:   => 0.4


Comment:

 After having looked at past logs and thinking about possible fixes I came
 up with a rather trivial fix: let's give up on using a strict date filter
 when analyzing logs and simply include all transfers and control events
 found in the parsed log files.

 Example: op-ab's log files from 2019-01-13 contain measurements from
 2019-01-12 and -13, because log rotation didn't happen at 2019-01-12
 23:59:59. With the current code, the analysis file produced at 2019-01-13
 23:59:59 is called `2019-01-13.onionperf.analysis.json.xz` and contains
 measurements from 2019-01-13 only. After the suggested change the file
 would still have that file name but contain all measurements from
 2019-01-12 and -13 as found in the log files.

 (A possible alternative that I had been thinking about before was that the
 analysis of a single pair of log files could produce more than one
 analysis files depending on how many dates are covered in the log files.
 This can get nasty, though. We'd have to consider all sorts of edge cases
 where two log files contain measurements for the same date. But really, we
 don't have to make sure that measurements go into analysis files with a
 specific date. Let's just not do this anymore.)

 I didn't work on the implementation yet and would like to leave this to
 others, also as a way to sanity check the concept. If I were to implement
 this I'd probably avoid using the `date_filter` parameter when doing the
 nightly analysis and introduce some sort of `date_prefix` parameter. We
 should just use that `date_filter` by default for all analysis files, and
 we might consider deprecating the `date_prefix` parameter in the user
 interface. When testing this, we'll have to ensure that it works for the
 nightly case as well as the reprocessing case.

 Returning to metrics-team.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29369 [Metrics/Onionperf]: Fix message logging and filtering

[Tor Bug Tracker & Wiki]

#29369: Fix message logging and filtering
---+
 Reporter:  irl|  Owner:  karsten
 Type:  defect | Status:  accepted
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:
Parent ID: | Points:  1
 Reviewer: |Sponsor:  Sponsor59-must
---+
Changes (by karsten):

 * status:  reopened => accepted
 * owner:  metrics-team => karsten


Comment:

 I started looking into this.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29369 [Metrics/Onionperf]: Fix message logging and filtering

[Tor Bug Tracker & Wiki]

#29369: Fix message logging and filtering
---+--
 Reporter:  irl|  Owner:  metrics-team
 Type:  defect | Status:  reopened
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:
Parent ID: | Points:  1
 Reviewer: |Sponsor:  Sponsor59
---+--

Comment (by karsten):

 See #30238 for a (near) duplicate. What happened there is that op-ab did
 not rotate its log files on 2019-01-13 00:00:00 and also did not produce
 an analysis file for 2019-01-12. This is a bit unfortunate, because there
 have been quite a few measurements on that day. We'll have to think about
 that case when we work on this ticket to avoid it in the future. (No need
 to recreate the analysis file for 2019-01-12, that's long ago.)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #29369 [Metrics/Onionperf]: Fix message logging and filtering

[Tor Bug Tracker & Wiki]

#29369: Fix message logging and filtering
---+--
 Reporter:  irl|  Owner:  metrics-team
 Type:  defect | Status:  reopened
 Priority:  Medium |  Milestone:
Component:  Metrics/Onionperf  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:
Parent ID: | Points:  1
 Reviewer: |Sponsor:  Sponsor59
---+--
Changes (by gaba):

 * keywords:   => metrics-team-roadmap-2020
 * points:   => 1
 * sponsor:   => Sponsor59


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs