Re: [tor-bugs] #25161 [Metrics/CollecTor]: Fix another memory problem with the webstats bulk import

[Tor Bug Tracker & Wiki]

#25161: Fix another memory problem with the webstats bulk import
---+---
 Reporter:  karsten|  Owner:  iwakeh
 Type:  defect | Status:  needs_information
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+---
Changes (by iwakeh):

 * status:  assigned => needs_information


Comment:

 Thanks, for identifying the new issue!

 I just created a new ticket for this large log issue, #25317.

 This ticket here should rather handle the import of a number of lesser
 sized logs.

 In order to see what other work is necessary regarding the import:
 Could you run the import again without the problematic files and report
 the memory use throughout the import?
 (That can be achieved by using jconsole and saving the values 'behind' the
 memory graph.)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25161 [Metrics/CollecTor]: Fix another memory problem with the webstats bulk import

[Tor Bug Tracker & Wiki]

#25161: Fix another memory problem with the webstats bulk import
---+--
 Reporter:  karsten|  Owner:  iwakeh
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--

Comment (by karsten):

 Looking at the stack trace and the input log files, I noticed that two log
 files are larger than 2G when decompressed:

 {{{
 3.2G in/webstats/archeotrichon.torproject.org/dist.torproject.org-
 access.log-20160531
 584K in/webstats/archeotrichon.torproject.org/dist.torproject.org-
 access.log-20160531.xz
 2.1G in/webstats/archeotrichon.torproject.org/dist.torproject.org-
 access.log-20160601
 404K in/webstats/archeotrichon.torproject.org/dist.torproject.org-
 access.log-20160601.xz
 }}}

 I just ran another bulk import with just those two files as import and ran
 into the same exception.

 It seems like we shouldn't attempt to decompress these files into a
 `byte[]` in `FileType.decompress`, because Java can only handle arrays
 with up to 2 billion elements:
 https://en.wikipedia.org/wiki/Criticism_of_Java#Large_arrays . Maybe we
 should work with streams there, not `byte[]`.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25161 [Metrics/CollecTor]: Fix another memory problem with the webstats bulk import

[Tor Bug Tracker & Wiki]

#25161: Fix another memory problem with the webstats bulk import
---+--
 Reporter:  karsten|  Owner:  iwakeh
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--
Changes (by karsten):

 * owner:  karsten => iwakeh


Comment:

 So, even with 64G RAM I'm running into the very same issue:

 {{{
 2018-02-20 16:40:46,425 INFO o.t.c.w.SanitizeWeblogs:108 Processing logs
 for dist.torproject.org on archeotrichon.torproject.org.
 2018-02-20 16:54:39,815 ERROR o.t.c.c.CollecTorMain:71 The webstats module
 failed: null
 java.lang.OutOfMemoryError: null
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
 Method)
 at
 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at
 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at
 java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
 at
 java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
 at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
 at
 java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
 at
 
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
 at
 java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
 at
 java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
 at
 java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:496)
 at
 
org.torproject.collector.webstats.SanitizeWeblogs.findCleanWrite(SanitizeWeblogs.java:113)
 at
 
org.torproject.collector.webstats.SanitizeWeblogs.startProcessing(SanitizeWeblogs.java:90)
 at
 org.torproject.collector.cron.CollecTorMain.run(CollecTorMain.java:67)
 at
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at
 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
 at
 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
 Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM
 limit
 at java.util.Arrays.copyOf(Arrays.java:3236)
 at
 java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
 at
 java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
 at
 java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:135)
 at
 org.torproject.descriptor.internal.FileType.decompress(FileType.java:109)
 at
 
org.torproject.collector.webstats.SanitizeWeblogs.lineStream(SanitizeWeblogs.java:190)
 at
 
org.torproject.collector.webstats.SanitizeWeblogs.lambda$findCleanWrite$1(SanitizeWeblogs.java:111)
 at
 
org.torproject.collector.webstats.SanitizeWeblogs$$Lambda$15/894365800.apply(Unknown
 Source)
 at
 java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
 at
 java.util.TreeMap$ValueSpliterator.forEachRemaining(TreeMap.java:2897)
 at
 java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
 at
 java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
 at
 java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
 at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
 at
 java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
 at
 java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
 at
 java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
 2018-02-20 16:54:39,917 INFO o.t.c.c.ShutdownHook:23 Shutdown in progress
 ...
 2018-02-20 16:54:39,917 INFO o.t.c.cron.Scheduler:127 Waiting at most 10
 minutes for termination of running tasks ...
 2018-02-20 16:54:39,917 INFO o.t.c.cron.Scheduler:132 Shutdown of all
 scheduled tasks completed 

Re: [tor-bugs] #25161 [Metrics/CollecTor]: Fix another memory problem with the webstats bulk import

[Tor Bug Tracker & Wiki]

#25161: Fix another memory problem with the webstats bulk import
---+--
 Reporter:  karsten|  Owner:  karsten
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--

Comment (by karsten):

 Great! I'll start processing files next Tuesday or Wednesday.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25161 [Metrics/CollecTor]: Fix another memory problem with the webstats bulk import

[Tor Bug Tracker & Wiki]

#25161: Fix another memory problem with the webstats bulk import
---+--
 Reporter:  karsten|  Owner:  karsten
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--
Changes (by iwakeh):

 * owner:  iwakeh => karsten
 * status:  accepted => assigned


Comment:

 Providing plenty of RAM for the import shortens the processing time quite
 a bit due to less GC time.  The 85min using 16G for the entire available
 archives of meronense and weschniakowii together (reported
 [https://trac.torproject.org/projects/tor/ticket/25100#comment:18 here])
 reduce to just 65 min with 30G (of which only 22G were actually used at
 peak time, 10G most of the time).  Of course, timing depends highly on
 available cores (here only four were available) and lesser the type of
 cpu.

 If a machine with 64G is available for import it can just be run on the
 entire 'out' folder of webstats.tp.o and should be fine with 48-56G
 (assuming that weschniakowii represents one of the hosts with the heavier
 log load).
 In case the import gets interrupted the logs will clearly indicate which
 hosts were processed successfully.  This should be used to move the
 already completed imports out of the import directory to save processing
 time.  No problem if that is forgotten, CollecTor won't re-add or
 overwrite anything, but the additional scanning might take longer than
 without.

 Collector properties should be set to single run and have limits turned
 off for importing the already existing sanitized logs.

 I used metrics-lib commit 9f2db9a19 and collector commit 06d1a81d4 and
 performed some manual checks that the resulting sanitized logs stay the
 same except for the intended changes (e.g. removal of '?' etc.).  All
 seemed fine.

 Assigning to 'karsten' as the import seems ready to go.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25161 [Metrics/CollecTor]: Fix another memory problem with the webstats bulk import

[Tor Bug Tracker & Wiki]

#25161: Fix another memory problem with the webstats bulk import
---+--
 Reporter:  karsten|  Owner:  iwakeh
 Type:  defect | Status:  accepted
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--

Comment (by iwakeh):

 All aiming at 64G RAM.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25161 [Metrics/CollecTor]: Fix another memory problem with the webstats bulk import

[Tor Bug Tracker & Wiki]

#25161: Fix another memory problem with the webstats bulk import
---+--
 Reporter:  karsten|  Owner:  iwakeh
 Type:  defect | Status:  accepted
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--

Comment (by karsten):

 Sounds good!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25161 [Metrics/CollecTor]: Fix another memory problem with the webstats bulk import

[Tor Bug Tracker & Wiki]

#25161: Fix another memory problem with the webstats bulk import
---+--
 Reporter:  karsten|  Owner:  iwakeh
 Type:  defect | Status:  accepted
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--
Changes (by iwakeh):

 * status:  new => accepted
 * owner:  metrics-team => iwakeh


Comment:

 I can device the shell script for slicing the imports also making sure the
 memory consumption of the single runs stays in reasonable limits.
 All based on the assumption that the folder structure of the files to be
 imported will be the same as below 'webstats.tp.o/out'.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25161 [Metrics/CollecTor]: Fix another memory problem with the webstats bulk import

[Tor Bug Tracker & Wiki]

#25161: Fix another memory problem with the webstats bulk import
---+--
 Reporter:  karsten|  Owner:  metrics-team
 Type:  defect | Status:  new
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--

Comment (by karsten):

 Replying to [comment:4 iwakeh]:
 > Is it really necessary to do the slicing using java?  Assuming such
 imports are limited in number (once a year?) the slices could simply be
 provided by moving the logs to be imported in appropriate folder
 structures and run CollecTor for importing these.

 You mean we should prepare slices manually? How many slices would we need?
 Just one per year?

 > As this is still set to 'new', are you working on this while importing?

 I'm not working on this. I'm waiting for new hardware to arrive that has
 more than 16G RAM. I hope to have that available by next week. Feel free
 to grab the ticket in the meantime!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25161 [Metrics/CollecTor]: Fix another memory problem with the webstats bulk import

[Tor Bug Tracker & Wiki]

#25161: Fix another memory problem with the webstats bulk import
---+--
 Reporter:  karsten|  Owner:  metrics-team
 Type:  defect | Status:  new
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--

Comment (by iwakeh):

 Is it really necessary to do the slicing using java?  Assuming such
 imports are limited in number (once a year?) the slices could simply be
 provided by moving the logs to be imported in appropriate folder
 structures and run CollecTor for importing these.

 As this is still set to 'new', are you working on this while importing?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25161 [Metrics/CollecTor]: Fix another memory problem with the webstats bulk import

[Tor Bug Tracker & Wiki]

#25161: Fix another memory problem with the webstats bulk import
---+--
 Reporter:  karsten|  Owner:  metrics-team
 Type:  defect | Status:  new
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--

Comment (by karsten):

 Replying to [comment:1 iwakeh]:
 > I didn't take a close look at the proposed code changes/additions, yet.
 In #25100 I suggested to partition the import by date, b/c that gives the
 reduction of heap usage and also is fine for the bulk import where we know
 one-file-one-date is true.

 Yes, my code implements such a partition by date. Though you have a point
 there with one-file-one-date. I could simplify the code a lot. Let me do
 that. (I think it won't affect the memory issue, though.)

 > > Or is this just the operation where we happen to run out of memory
 from accumulating stuff over time, and where fixing this issue would just
 mean that we're failing somewhere else, shortly after?
 >
 > I think this is the case.  We're also - opposed to all other CollecTor
 modules - compressing before writing.

 Okay. So maybe there's room for improvement here.

 > What heap setting is used, 8G?  How many cores are available?

 `-Xmx16g` with 16 GiB RAM available. 4 cores.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25161 [Metrics/CollecTor]: Fix another memory problem with the webstats bulk import

[Tor Bug Tracker & Wiki]

#25161: Fix another memory problem with the webstats bulk import
---+--
 Reporter:  karsten|  Owner:  metrics-team
 Type:  defect | Status:  new
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--

Comment (by iwakeh):

 Quote from #25100 comment 16:

  The second quarter 2017 of logs from weschniakowii amounts to 32M
 (compressed) and can be processed in 36min using 8G. The entire year won't
 work with just 8G.

  85min and 16G are needed for the entire available archives of
 meronense and weschniakowii together (compressed 59M). The used heap
 median usage is 8.5G and the max 15.8G.

  So, depending on the hardware a conservative import strategy might be
 to import slices of quarters with subsequent CollecTor runs.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #25161 [Metrics/CollecTor]: Fix another memory problem with the webstats bulk import

[Tor Bug Tracker & Wiki]

#25161: Fix another memory problem with the webstats bulk import
---+--
 Reporter:  karsten|  Owner:  metrics-team
 Type:  defect | Status:  new
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--

Comment (by iwakeh):

 I didn't take a close look at the proposed code changes/additions, yet.
 In #25100 I suggested to partition the import by date, b/c that gives the
 reduction of heap usage and also is fine for the bulk import where we know
 one-file-one-date is true.

 > Or is this just the operation where we happen to run out of memory from
 accumulating stuff over time, and where fixing this issue would just mean
 that we're failing somewhere else, shortly after?

 I think this is the case.  We're also opposed to all other CollecTor
 modules compressing before writing.

 What heap setting is used, 8G?  How many cores are available?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

[tor-bugs] #25161 [Metrics/CollecTor]: Fix another memory problem with the webstats bulk import

[Tor Bug Tracker & Wiki]

#25161: Fix another memory problem with the webstats bulk import
---+--
 Reporter:  karsten|  Owner:  metrics-team
 Type:  defect | Status:  new
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal |   Keywords:
Actual Points: |  Parent ID:
   Points: |   Reviewer:
  Sponsor: |
---+--
 I'm running a modified CollecTor that sanitizes webstats with some tweaks
 towards bulk importing existing webstats. In particular, it reads files in
 slices of 10 MiB plus another 2 MiB that overlap with the next slice. I
 just pushed the changes [https://gitweb.torproject.org/karsten/metrics-
 db.git/commit/?h=webstats-
 slices=6c1a266d726ceedf4518e7521a6ed58a6764aca4 here].

 First of all, the runtime is okay. Not great, but okay. It takes 36
 minutes to sanitize 10 MiB. We have 927 MiB of files, so 93 slices, which
 is going to take ~2.5 days.

 However, I ran into an out-of-memory problem at the 6th slice:

 {{{
 2018-02-06 13:30:36,499 INFO o.t.c.w.SanitizeWeblogs:116 Processing 20
 logs for dist.torproject.org on archeotrichon.torproject.org.
 2018-02-06 13:40:28,968 ERROR o.t.c.c.CollecTorMain:71 The webstats module
 failed: null
 java.lang.OutOfMemoryError: null
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
 Method)
 at
 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at
 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at
 java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
 at
 java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
 at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
 at
 java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
 at
 
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
 at
 java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
 at
 java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
 at
 
org.torproject.collector.webstats.SanitizeWeblogs.findCleanWrite(SanitizeWeblogs.java:127)
 at
 
org.torproject.collector.webstats.SanitizeWeblogs.startProcessing(SanitizeWeblogs.java:91)
 at
 org.torproject.collector.cron.CollecTorMain.run(CollecTorMain.java:67)
 at
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at
 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
 at
 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
 Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM
 limit
 at
 java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
 at java.lang.StringCoding.encode(StringCoding.java:344)
 at java.lang.StringCoding.encode(StringCoding.java:387)
 at java.lang.String.getBytes(String.java:958)
 at
 
org.torproject.descriptor.log.LogDescriptorImpl.collectionToBytes(LogDescriptorImpl.java:119)
 at
 
org.torproject.descriptor.log.WebServerAccessLogImpl.(WebServerAccessLogImpl.java:72)
 at
 
org.torproject.collector.webstats.SanitizeWeblogs.storeSanitized(SanitizeWeblogs.java:147)
 at
 
org.torproject.collector.webstats.SanitizeWeblogs.lambda$findCleanWrite$3(SanitizeWeblogs.java:127)
 at
 
org.torproject.collector.webstats.SanitizeWeblogs$$Lambda$38/1233367077.accept(Unknown
 Source)
 at
 java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
 at
 java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
 at
 
java.util.concurrent.ConcurrentHashMap$EntrySpliterator.forEachRemaining(ConcurrentHashMap.java:3606)
 at
 java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
 at
 java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
 at
 java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
 at