Re: [tor-bugs] #25259 [Metrics/Statistics]: Tune advbwdist module of metrics-web

[Tor Bug Tracker & Wiki]

#25259: Tune advbwdist module of metrics-web
+--
 Reporter:  iwakeh  |  Owner:  metrics-team
 Type:  enhancement | Status:  closed
 Priority:  Medium  |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  | Resolution:  fixed
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--
Changes (by karsten):

 * status:  needs_review => closed
 * resolution:   => fixed


Comment:

 Squashed, and rebased and pushed to master. Closing. Thanks!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25259 [Metrics/Statistics]: Tune advbwdist module of metrics-web

[Tor Bug Tracker & Wiki]

#25259: Tune advbwdist module of metrics-web
+--
 Reporter:  iwakeh  |  Owner:  metrics-team
 Type:  enhancement | Status:  needs_review
 Priority:  Medium  |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--

Comment (by karsten):

 Replying to [comment:4 iwakeh]:
 > Please find [https://gitweb.torproject.org/user/iwakeh/metrics-
 web.git/commit/?h=task-25259 another commit] tweaking memory usage and
 processing time a little more.  The result is identical to the result from
 the current master branch.

 Great! I copied the new R file to the server and will let it run this
 afternoon. If that succeeds, I'll squash and merge to master. Thanks!

 > Anyway, future changes should either split the input data (as suggested
 in comment:1) or mive the module to java.  R is just always 'pieces' that
 don't scale well together.

 Full ack.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25259 [Metrics/Statistics]: Tune advbwdist module of metrics-web

[Tor Bug Tracker & Wiki]

#25259: Tune advbwdist module of metrics-web
+--
 Reporter:  iwakeh  |  Owner:  metrics-team
 Type:  enhancement | Status:  needs_review
 Priority:  Medium  |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--

Comment (by iwakeh):

 Please find [https://gitweb.torproject.org/user/iwakeh/metrics-
 web.git/commit/?h=task-25259 another commit] tweaking memory usage and
 processing time a little more.

 Anyway, future changes should either split the input data (as suggested in
 comment:1) or mive the module to java.  R is just always 'pieces' that
 don't scale well together.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25259 [Metrics/Statistics]: Tune advbwdist module of metrics-web

[Tor Bug Tracker & Wiki]

#25259: Tune advbwdist module of metrics-web
+--
 Reporter:  iwakeh  |  Owner:  metrics-team
 Type:  enhancement | Status:  needs_review
 Priority:  Medium  |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--

Comment (by karsten):

 Those look like good tweaks. I copied the R file to the server and will
 let it run tonight. If that succeeds (and I think it should), I'll merge
 to master tomorrow. Thanks!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25259 [Metrics/Statistics]: Tune advbwdist module of metrics-web

[Tor Bug Tracker & Wiki]

#25259: Tune advbwdist module of metrics-web
+--
 Reporter:  iwakeh  |  Owner:  metrics-team
 Type:  enhancement | Status:  needs_review
 Priority:  Medium  |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--
Changes (by iwakeh):

 * status:  new => needs_review


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25259 [Metrics/Statistics]: Tune advbwdist module of metrics-web

[Tor Bug Tracker & Wiki]

#25259: Tune advbwdist module of metrics-web
+--
 Reporter:  iwakeh  |  Owner:  metrics-team
 Type:  enhancement | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--

Comment (by iwakeh):

 Please review [https://gitweb.torproject.org/user/iwakeh/metrics-
 web.git/commit/?h=task-25259 these tuning steps].
 From the commit comment:
 Processing advbwdist-validafter.csv (350M) took 150 seconds and used up to
 7G.
 Performing pre-processing separately, helping R by defining read types,
 and avoiding multiple casting operations led to halving the processing
 time (to 77 seconds) and reducing the necessary memory to about 25%
 (approx. 1.8G). The resulting advbwdist.csv are identical.

 In future it will be necessary to split the aggregating process into
 years.  Or, to store data for years that won't have changes anymore and
 combine existing with freshly computed data.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

[tor-bugs] #25259 [Metrics/Statistics]: Tune advbwdist module of metrics-web

[Tor Bug Tracker & Wiki]

#25259: Tune advbwdist module of metrics-web
+--
 Reporter:  iwakeh  |  Owner:  metrics-team
 Type:  enhancement | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Metrics/Statistics  |Version:
 Severity:  Normal  |   Keywords:
Actual Points:  |  Parent ID:
   Points:  |   Reviewer:
  Sponsor:  |
+--
 A first step to improve performance mostly in memory usage, because the
 current input of 350M advbwdist-validafter.csv uses up to 7G already.

 (I'll post a branch once there is a ticket number.)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs