Re: [tor-bugs] #25815 [Metrics/Onionoo]: Speed up hourly updater performance

[Tor Bug Tracker & Wiki]

#25815: Speed up hourly updater performance
-+--
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  closed
 Priority:  Medium   |  Milestone:
Component:  Metrics/Onionoo  |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by karsten):

 * status:  assigned => closed
 * resolution:   => fixed


Comment:

 With #32660 being resolved a few days ago and #25848 being resolved 19
 months ago, the main two suggestions are now implemented. I think this is
 optimized enough for the moment. Closing.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25815 [Metrics/Onionoo]: Speed up hourly updater performance

[Tor Bug Tracker & Wiki]

#25815: Speed up hourly updater performance
-+--
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  assigned
 Priority:  Medium   |  Milestone:
Component:  Metrics/Onionoo  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by iwakeh):

 * owner:  iwakeh => metrics-team
 * status:  accepted => assigned


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25815 [Metrics/Onionoo]: Speed up hourly updater performance

[Tor Bug Tracker & Wiki]

#25815: Speed up hourly updater performance
-+--
 Reporter:  karsten  |  Owner:  iwakeh
 Type:  enhancement  | Status:  accepted
 Priority:  Medium   |  Milestone:
Component:  Metrics/Onionoo  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by iwakeh):

 Replying to [comment:5 karsten]:
 > I tried out switching to Jackson, and it turns out that we can save 30%
 of overall time (for writing details documents). That's almost 4 minutes
 every hour. Sounds like a good idea to me.

 Yes, this is a good idea!  I added a child ticket for this task, which can
 easily be treated independently to possibly additional changes.  All Gson-
 Jackson discussion can be moved there.  A first step for sure.

 I look further and post findings here.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25815 [Metrics/Onionoo]: Speed up hourly updater performance

[Tor Bug Tracker & Wiki]

#25815: Speed up hourly updater performance
-+--
 Reporter:  karsten  |  Owner:  iwakeh
 Type:  enhancement  | Status:  accepted
 Priority:  Medium   |  Milestone:
Component:  Metrics/Onionoo  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by iwakeh):

 * owner:  metrics-team => iwakeh
 * status:  assigned => accepted


Comment:

 Taking a look, too.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25815 [Metrics/Onionoo]: Speed up hourly updater performance

[Tor Bug Tracker & Wiki]

#25815: Speed up hourly updater performance
-+--
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  assigned
 Priority:  Medium   |  Milestone:
Component:  Metrics/Onionoo  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by karsten):

 * owner:  karsten => metrics-team
 * status:  accepted => assigned


Comment:

 I tried out switching to Jackson, and it turns out that we can save 30% of
 overall time (for writing details documents). That's almost 4 minutes
 every hour. Sounds like a good idea to me.

 I uploaded my branch with temporary commits
 [https://gitweb.torproject.org/user/karsten/onionoo.git/log/?h=task-25815
 here]. I did ''not'' verify that all outputs are correct with regard to
 unicode character escapes and character encodings in general. I only
 looked at performance.

 For the moment, I'm giving this ticket back to metrics-team. I might grab
 it back next week, unless somebody else takes it first.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25815 [Metrics/Onionoo]: Speed up hourly updater performance

[Tor Bug Tracker & Wiki]

#25815: Speed up hourly updater performance
-+--
 Reporter:  karsten  |  Owner:  karsten
 Type:  enhancement  | Status:  accepted
 Priority:  Medium   |  Milestone:
Component:  Metrics/Onionoo  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by iwakeh):

 A switch to Jackson might lead to other benefits, which at first will also
 cause some work as gson behaves differently.  (See an
 [https://trac.torproject.org/projects/tor/ticket/13574#comment:3 older
 suggestion].)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25815 [Metrics/Onionoo]: Speed up hourly updater performance

[Tor Bug Tracker & Wiki]

#25815: Speed up hourly updater performance
-+--
 Reporter:  karsten  |  Owner:  karsten
 Type:  enhancement  | Status:  accepted
 Priority:  Medium   |  Milestone:
Component:  Metrics/Onionoo  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 So, I ran some performance measurements here. In particular, I used a
 recent backup from one of the public Onionoo hosts and ran an update. By
 doing so I identified two possible areas of improvements:

  1. While writing details documents, we spend almost 40% of CPU time on
 (de-)serializing JSON documents. We might want to try out another JSON
 library than Gson. A quick search says that Jackson might be a lot faster.
 Maybe we can save 10% or 20% of the overall time here.

  2. Also while writing details documents, we read almost all details
 status files and (re-)write a corresponding details document file. That's
 almost 1M files and takes roughly 55% of the overall CPU time, including
 the 40% for JSON handling. If we can find a way to exclude files that
 don't need to be updated, we might be able to save another 10% or 20% of
 overall time. (The overall savings are related to what we do to JSON
 handling.)

 I'd like to keep this ticket for a while longer and dive deeper into these
 two ideas. But I thought I'd share these first results to hear early if
 they're a bad idea for some reason.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25815 [Metrics/Onionoo]: Speed up hourly updater performance

[Tor Bug Tracker & Wiki]

#25815: Speed up hourly updater performance
-+--
 Reporter:  karsten  |  Owner:  karsten
 Type:  enhancement  | Status:  accepted
 Priority:  Medium   |  Milestone:
Component:  Metrics/Onionoo  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by karsten):

 * owner:  metrics-team => karsten
 * status:  new => accepted


Comment:

 I'll investigate this some more.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25815 [Metrics/Onionoo]: Speed up hourly updater performance

[Tor Bug Tracker & Wiki]

#25815: Speed up hourly updater performance
-+--
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/Onionoo  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--

Comment (by karsten):

 [[Image(onionoo-start-end.png, 700px)]]

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25815 [Metrics/Onionoo]: Speed up hourly updater performance

[Tor Bug Tracker & Wiki]

#25815: Speed up hourly updater performance
-+--
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/Onionoo  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by karsten):

 * Attachment "onionoo-start-end.png" added.


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

[tor-bugs] #25815 [Metrics/Onionoo]: Speed up hourly updater performance

[Tor Bug Tracker & Wiki]

#25815: Speed up hourly updater performance
-+--
 Reporter:  karsten  |  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/Onionoo  |Version:
 Severity:  Normal   |   Keywords:
Actual Points:   |  Parent ID:
   Points:   |   Reviewer:
  Sponsor:   |
-+--
 When looking at Onionoo's logs with irl the other day we noticed that the
 hourly updater sometimes takes as long as 45 minutes.

 We obviously need to be careful not to let this time get closer to 60
 minutes or even exceed that time.

 I just looked a little bit closer at the logs and made a graph. It shows
 three phases of the hourly updater:
  - before: from starting the updater to writing details documents;
  - write: writing details documents;
  - after: from after writing details documents to the end.

 The graph compares total runtimes on the x axis with the three phases on
 the y axis. I'm reading two things in this graph.
  - Writing details documents really takes a long time. It's never faster
 than 15 minutes and can take longer than 30 minutes in some cases.
  - Whenever an hourly update takes longer than, say, 40 minutes, it's not
 just the details documenting that takes longer than usual, but also the
 phase before takes longer. I guess the system is just generally slower at
 those times.

 I'd say let's look at details document writing for optimizing runtime.
 Maybe we can decide early that we don't have to update a file. But that's
 just guessing, we should add more logging to that code, as right now the
 logs are silent for those 30 to 45 minutes.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs