#20335: ReferenceChecker causes OOM -------------------------------+--------------------- Reporter: iwakeh | Owner: Type: defect | Status: new Priority: High | Milestone: Component: Metrics/CollecTor | Version: Severity: Normal | Resolution: Keywords: | Actual Points: Parent ID: | Points: Reviewer: | Sponsor: -------------------------------+---------------------
Comment (by iwakeh): Replying to [comment:10 karsten]: > I ''believe'' that's a smart thing to do. I'm not sure if that would break something. Unfortunately, you're right and this was not the reason for the oom :-/ > > But I think it doesn't solve the issue here. Or, I ran out of memory with a similar patch applied. And I think I know why: > > {{{ > -rw-r--r-- 1 karsten staff 2.1G Oct 18 11:52 /Users/karsten/src/collector/recent/relay- descriptors/votes/2016-10-18-09-22-46-votes > }}} > > With the recent change to append all votes to a single file, we're creating a 2.1G large file when doing the first synchronization run. However, metrics-lib wants to read that entire file to memory which probably doubles in size when it's parsed. Ugh! > > Note that the CollecTor's reference checker is just one application that breaks when input files are too large. Other applications using metrics- lib would similarly be affected. > > A longer-term fix might be to change metrics-lib to read through large files and pause when it has parsed enough descriptors that the application still has to process. Did you verify that metrics-lib is the culprit? Like, reading the 2G file for parsing? > > A short-term fix would be to stick with the current naming scheme for files in the `recent/` directory, as that would have avoided this issue. > The commit for the review in #18910 is easily adaptable, if necessary. > Fun times! Wouldn't it be boring without the unforeseen ;-) -- Ticket URL: <https://trac.torproject.org/projects/tor/ticket/20335#comment:11> Tor Bug Tracker & Wiki <https://trac.torproject.org/> The Tor Project: anonymity online _______________________________________________ tor-bugs mailing list tor-bugs@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs