[
https://issues.apache.org/jira/browse/LUCENE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200698#comment-17200698
]
Adrien Grand commented on LUCENE-9535:
--------------------------------------
I tried to reproduce the slowdown locally but results do not look significant.
Since I don't have as many cores as Mike's beast, only 24, I ran with half the
index buffer size and the number of threads, ie. 1024MB of index buffer and 18
threads on the wikimediumall corpus.
Baseline (master):
- 247GB/h 224 flushes
- 259GB/h 225 flushes
- 248GB/h 226 flushes
- 262GB/h 224 flushes
Patch (stored fields ignored in IndexingChain memory accounting):
- 256GB/h 224 flushes
- 258GB/h 223 flushes
While the nightly benchmarks are seeing a ~10% slowdown, I'm not seeing a
significant change. I'm running out of ideas so I will decrease the block size
of stored fields later today to see whether that makes a difference for nightly
benchmarks, which might help confirm whether stored fields are actually the
problem or whether it's something else.
> Investigate recent indexing slowdown for wikimedium documents
> -------------------------------------------------------------
>
> Key: LUCENE-9535
> URL: https://issues.apache.org/jira/browse/LUCENE-9535
> Project: Lucene - Core
> Issue Type: Task
> Reporter: Adrien Grand
> Priority: Minor
> Attachments: cpu_profile.svg
>
>
> Nightly benchmarks report a ~10% slowdown for 1kB documents as of September
> 9th: [http://people.apache.org/~mikemccand/lucenebench/indexing.html].
> On that day, we added stored fields in DWPT accounting (LUCENE-9511), so I
> first thought this could be due to smaller flushed segments and more merging,
> but I still wonder whether there's something else. The benchmark runs with
> 8GB of heap, 2GB of RAM buffer and 36 indexing threads. So it's about 2GB/36
> = 57MB of RAM buffer per thread in the worst-case scenario that all DWPTs get
> full at the same time. Stored fields account for about 0.7MB of memory, or 1%
> of the indexing buffer size. How can a 1% reduction of buffering capacity
> explain a 10% indexing slowdown? I looked into this further by running
> indexing benchmarks locally with 8 indexing threads and 128MB of indexing
> buffer memory, which would make this issue even more apparent if the smaller
> RAM buffer was the cause, but I'm not seeing a regression and actually I'm
> seeing similar number of flushes when I disabled memory accounting for stored
> fields.
> I ran indexing under a profiler to see whether something else could cause
> this slowdown, e.g. slow implementations of ramBytesUsed on stored fields
> writers, but nothing surprising showed up and the profile looked just like I
> would have expected.
> Another question I have is why the 4kB benchmark is not affected at all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
