[
https://issues.apache.org/jira/browse/SOLR-17539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18066181#comment-18066181
]
Anshum Gupta commented on SOLR-17539:
-------------------------------------
The related issue also potentially causes an OOM in
{{DocValuesIteratorCache.getDocValues() }}method.
> Increased memory usage after upgrading to Solr 9.6.0 - DocValuesIteratorCache
> -----------------------------------------------------------------------------
>
> Key: SOLR-17539
> URL: https://issues.apache.org/jira/browse/SOLR-17539
> Project: Solr
> Issue Type: Bug
> Components: search
> Affects Versions: 9.4, 9.6, 9.7
> Reporter: Soham Gandhi
> Priority: Major
> Labels: docValues, memory, retrieval
>
> We recently upgraded from Solr 8.11.1 to Solr 9.6.0 and noticed an increase
> of in memory usage in our testing.
> Here's a typical test environment setup -
> * 2 mil documents in the index taking up ~3 GB disk size
> * Test suite executes 100 unique queries every other minute for an hour
> * Metrics like query response times, CPU and memory usage are captured and
> compared against a baseline benchmark.
> The upgrade did not include any change to schema or solrconfig as compared to
> Solr 8.11.1, The test results reported an increase in memory usage from 4 GB
> to 6 GB.
> Here are the memory usage snapshots from our test runs for Solr 8.11.1 (left)
> and Solr 9.6.0
> !https://mail.google.com/mail/u/0?ui=2&ik=8e78b77afe&attid=0.1&permmsgid=msg-a:r-8567447807355761378&th=1924eb8757fe54af&view=fimg&fur=ip&sz=s0-l75-ft&attbid=ANGjdJ8l_61BAciOjcTBSQgWAezcaZAc6wDNI7FzU8EXloaLbQbQUrqFhl8s0BKIUSwdacQJSO44H39Mu6u4fQfTH7o11M5pY_9XDvzvn1zOeMrZu2kqkho29mcoZO8&disp=emb&realattid=ii_m1s92j1g5|width=916,height=261!
> Here's a snippet of memory profiling during test runs for Solr 8.11.1 (left)
> and Solr 9.6.0
> !https://mail.google.com/mail/u/0?ui=2&ik=8e78b77afe&attid=0.2&permmsgid=msg-a:r-8567447807355761378&th=1924eb8757fe54af&view=fimg&fur=ip&sz=s0-l75-ft&attbid=ANGjdJ_snNNurQP7b4OrCQ53e_VylKmRw-pmgot0S9rCoCmLTOhAPdtVzkFEjOevL-Cm2pyHc6kz9RYJAQL-c3IhX2YZyLqN8E5UwhUX1K8Y7d-TClyWMt8Pw5WimgE&disp=emb&realattid=ii_m1s966xw6!
> From our investigation we found this behaviour to be linked to the
> {{DocValuesIteratorCache}} introduced in Solr 9.4.0 as part of SOLR-16989
> Optimize and consolidate reuse of DocValues iterators for value retrieval.
> The improvement in value retrieval is not seen in our runs as the queries
> were already fast (~100ms). Query response times improve if the page size is
> increased by a factor of 10 (50 -> 500); each query retrieves the first 50
> results.
> The schema in our test environment contains ~800 fields, with {{docValues}}
> enabled on 94 (44 predefined fields and 50 dynamic fields) where
> {{useDocValuesAsStored=true}} is set.
> Each query retrieves the first 50 documents, with 5 to 7 fields populated in
> each.
> The workaround suggested on the forum where we do not retrieve any
> {{docValues}} (with {{{}useDocValuesAsStored=true){}}}may not be trivial as
> our queries use a glob pattern ({{{}fl=id, field_prefix_*{}}}), where the
> {{field_prefix}} matches most of the fields.
>
> We would like to have a configuration to disable the feature introduced in
> SOLR-16989.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
