[ https://issues.apache.org/jira/browse/LUCENE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17201010#comment-17201010 ]
ASF subversion and git services commented on LUCENE-9535: --------------------------------------------------------- Commit d226abd4481a5bd837264a7c53d1b13f417842ad in lucene-solr's branch refs/heads/master from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=d226abd ] LUCENE-9535: Make ByteBuffersDataOutput#ramBytesUsed run in constant-time. (#1917) > Investigate recent indexing slowdown for wikimedium documents > ------------------------------------------------------------- > > Key: LUCENE-9535 > URL: https://issues.apache.org/jira/browse/LUCENE-9535 > Project: Lucene - Core > Issue Type: Task > Reporter: Adrien Grand > Priority: Minor > Attachments: cpu_profile.svg > > Time Spent: 0.5h > Remaining Estimate: 0h > > Nightly benchmarks report a ~10% slowdown for 1kB documents as of September > 9th: [http://people.apache.org/~mikemccand/lucenebench/indexing.html]. > On that day, we added stored fields in DWPT accounting (LUCENE-9511), so I > first thought this could be due to smaller flushed segments and more merging, > but I still wonder whether there's something else. The benchmark runs with > 8GB of heap, 2GB of RAM buffer and 36 indexing threads. So it's about 2GB/36 > = 57MB of RAM buffer per thread in the worst-case scenario that all DWPTs get > full at the same time. Stored fields account for about 0.7MB of memory, or 1% > of the indexing buffer size. How can a 1% reduction of buffering capacity > explain a 10% indexing slowdown? I looked into this further by running > indexing benchmarks locally with 8 indexing threads and 128MB of indexing > buffer memory, which would make this issue even more apparent if the smaller > RAM buffer was the cause, but I'm not seeing a regression and actually I'm > seeing similar number of flushes when I disabled memory accounting for stored > fields. > I ran indexing under a profiler to see whether something else could cause > this slowdown, e.g. slow implementations of ramBytesUsed on stored fields > writers, but nothing surprising showed up and the profile looked just like I > would have expected. > Another question I have is why the 4kB benchmark is not affected at all. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org