Re: [tor-bugs] #25100 [Metrics/CollecTor]: Make CollecTor's webstats module use less RAM and wall time

[Tor Bug Tracker & Wiki]

#25100: Make CollecTor's webstats module use less RAM and wall time
---+
 Reporter:  karsten|  Owner:  iwakeh
 Type:  enhancement| Status:  closed
 Priority:  High   |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:  fixed
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+
Changes (by iwakeh):

 * status:  merge_ready => closed
 * resolution:   => fixed


Comment:

 If anything new comes up, it should be a new ticket.

 Closing.  Thanks!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25100 [Metrics/CollecTor]: Make CollecTor's webstats module use less RAM and wall time

[Tor Bug Tracker & Wiki]

#25100: Make CollecTor's webstats module use less RAM and wall time
---+-
 Reporter:  karsten|  Owner:  iwakeh
 Type:  enhancement| Status:  merge_ready
 Priority:  High   |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+-
Changes (by karsten):

 * status:  needs_review => merge_ready


Comment:

 Sounds good! Pushed to master! We can leave this open for additional tests
 or close it, up to you. Thanks!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25100 [Metrics/CollecTor]: Make CollecTor's webstats module use less RAM and wall time

[Tor Bug Tracker & Wiki]

#25100: Make CollecTor's webstats module use less RAM and wall time
---+--
 Reporter:  karsten|  Owner:  iwakeh
 Type:  enhancement| Status:  needs_review
 Priority:  High   |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--

Comment (by iwakeh):

 I didn't find any more bugs processing the following benchmarks for
 determining how to approach the import using the two latest branches of
 this and the child ticket.

 The second quarter 2017 of logs from weschniakowii amounts to 32M
 (compressed) and can be processed in 36min using 8G.  The entire year
 won't work with just 8G.

 85min and 16G are needed for the entire available archives of meronense
 and weschniakowii together (compressed 59M).  The used heap median usage
 is 8.5G and the max 15.8G.

 So, depending on the hardware a conservative import strategy might be to
 import slices of quarters with subsequent CollecTor runs.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25100 [Metrics/CollecTor]: Make CollecTor's webstats module use less RAM and wall time

[Tor Bug Tracker & Wiki]

#25100: Make CollecTor's webstats module use less RAM and wall time
---+--
 Reporter:  karsten|  Owner:  iwakeh
 Type:  enhancement| Status:  needs_review
 Priority:  High   |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--

Comment (by karsten):

 Alright, I'll run more tests on Monday, or maybe over the weekend as time
 permits.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25100 [Metrics/CollecTor]: Make CollecTor's webstats module use less RAM and wall time

[Tor Bug Tracker & Wiki]

#25100: Make CollecTor's webstats module use less RAM and wall time
---+--
 Reporter:  karsten|  Owner:  iwakeh
 Type:  enhancement| Status:  needs_review
 Priority:  High   |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--

Comment (by iwakeh):

 Replying to [comment:15 karsten]:
 > My Java 8 is not good enough to say much about commit 6264477, but at
 least it doesn't look wrong to me. And commit ae12c2e looks obviously
 correct. I say if it compiles and passes tests, let's do it.

 Thanks for the quick review!

 >
 > Thanks for putting in this effort! I'm positive that it will be worth
 it.
 >
 > Let me know when I should do something. Otherwise I'll just wait for
 more commits or green light to merge.

 I didn't only mean review, but also testing or whatever you did to cause
 this ticket :-)

 Unless any bugs show up I don't intend more commits.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25100 [Metrics/CollecTor]: Make CollecTor's webstats module use less RAM and wall time

[Tor Bug Tracker & Wiki]

#25100: Make CollecTor's webstats module use less RAM and wall time
---+--
 Reporter:  karsten|  Owner:  iwakeh
 Type:  enhancement| Status:  needs_review
 Priority:  High   |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--

Comment (by karsten):

 My Java 8 is not good enough to say much about commit 6264477, but at
 least it doesn't look wrong to me. And commit ae12c2e looks obviously
 correct. I say if it compiles and passes tests, let's do it.

 Thanks for putting in this effort! I'm positive that it will be worth it.

 Let me know when I should do something. Otherwise I'll just wait for more
 commits or green light to merge.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25100 [Metrics/CollecTor]: Make CollecTor's webstats module use less RAM and wall time

[Tor Bug Tracker & Wiki]

#25100: Make CollecTor's webstats module use less RAM and wall time
---+--
 Reporter:  karsten|  Owner:  iwakeh
 Type:  enhancement| Status:  needs_review
 Priority:  High   |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--
Changes (by iwakeh):

 * status:  needs_revision => needs_review


Comment:

 Please review
 [https://gitweb.torproject.org/user/iwakeh/collector.git/log/?h=task-25100
 two small commits] applying parallelization tweaks and making use of the
 metrics-lib changes.

 With [https://gitweb.torproject.org/user/iwakeh/metrics-
 lib.git/log/?h=task-25103 metrics-lib task-25103 branch] meronense's 516
 logs can be processed using 2G (heap usage throughout was mostly below
 1.5G and with three peaks slightly above 1.5G) in 2.5 min.

 Test runs processing of larger imports using 8G are still on the way.  So
 far, the memory handling looks fine; mostly well below 4G, only the
 processing of aus1.tp.o logs, which are quite large needed the 8G to work.
 Considering that these are the logs for a year (327 days) this should be
 ok and CollecTor is surely able to handle the regular daily imports and
 occasionally imports of a few months now with modest RAM and time demand.

 I test further and post results, but I think the changes here and those in
 metrics-lib can be reviewed now.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25100 [Metrics/CollecTor]: Make CollecTor's webstats module use less RAM and wall time (was: Make CollecTor's webstats module use less RAM and CPU time)

[Tor Bug Tracker & Wiki]

#25100: Make CollecTor's webstats module use less RAM and wall time
---+
 Reporter:  karsten|  Owner:  iwakeh
 Type:  enhancement| Status:  needs_revision
 Priority:  High   |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs