#25100: Make CollecTor's webstats module use less RAM and CPU time
-------------------------------+--------------------------------
Reporter: karsten | Owner: iwakeh
Type: enhancement | Status: needs_revision
Priority: High | Milestone:
Component: Metrics/CollecTor | Version:
Severity: Normal | Resolution:
Keywords: | Actual Points:
Parent ID: | Points:
Reviewer: | Sponsor:
-------------------------------+--------------------------------
Changes (by karsten):
* status: accepted => needs_revision
Comment:
Commits a5f3d6a and 1873f12 look fine.
But these are all just tweaks that save some memory here and there. Should
we also think about taking a different approach that scales better by
design? Even if we can limit memory usage to 6G, that's far too much.
Ideally, we'd keep the `-Xmx2g` setting for all of CollecTor, or maybe
`-Xmx4g`. But imagine what we'd have to set when bulk-importing logs in
2019 or 2022.
How about we sanitize logs in two steps: in the first step we scan all
input files just for contained dates, and in the second step we iterate
over input files in an order that lets us just keep a sliding window of
log lines that we need to write output files.
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25100#comment:6>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs
