[
https://issues.apache.org/jira/browse/SOLR-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13603683#comment-13603683
]
Uwe Schindler commented on SOLR-4589:
-------------------------------------
bq. I wonder if this could be related to index compression (and maybe the same
block being repeatedly decompressed for each lazy field being accessed?)
This also happens in Solr 4.0, which had no compression.
The reason here might be the changes in stored fields altogether. Lucene
natively no longer has support for lazy field loading, but there is a
"backwards layer" just for Solr in modules/misc (LazyDocument.java). The
document does not use maps to lookup, if you have many fields its always a scan
through the ArrayList of all fields in the document. The lazyness in
LazyDocument is only that the *whole* document is loaded delayed, but no longer
single fields.
> 4.x + enableLazyFieldLoading + large nultivalued fields + varying fl =
> pathalogical CPU load & response time
> ------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-4589
> URL: https://issues.apache.org/jira/browse/SOLR-4589
> Project: Solr
> Issue Type: Bug
> Affects Versions: 4.0, 4.1, 4.2
> Reporter: Hoss Man
> Attachments: test-just-queries.out__4.0.0_mmap_lazy_using36index.txt,
> test-just-queries.sh, test.out__3.6.1_mmap_lazy.txt,
> test.out__3.6.1_mmap_nolazy.txt, test.out__3.6.1_nio_lazy.txt,
> test.out__3.6.1_nio_nolazy.txt, test.out__4.0.0_mmap_lazy.txt,
> test.out__4.0.0_mmap_nolazy.txt, test.out__4.0.0_nio_lazy.txt,
> test.out__4.0.0_nio_nolazy.txt, test.out__4.2.0_mmap_lazy.txt,
> test.out__4.2.0_mmap_nolazy.txt, test.out__4.2.0_nio_lazy.txt,
> test.out__4.2.0_nio_nolazy.txt, test.sh
>
>
> Following up on a [user report of exterme CPU usage in
> 4.1|http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201302.mbox/%[email protected]%3E],
> I've discovered that the following combination of factors can result in
> extreme CPU usage and excessively HTTP response times...
> * Solr 4.x (tested 3.6.1, 4.0.0, and 4.2.0)
> * enableLazyFieldLoading == true (included in example solrconfig.xml)
> * documents with a large number of values in multivalued fields (eg: tested
> ~10-15K values)
> * multiple requests returning the same doc with different "fl" lists
> I haven't dug into the route cause yet, but the essential observations is: if
> lazyloading is used in 4.x, then once a document has been fetched with an
> initial fl list X, subsequent requests for that document using a differnet fl
> list Y can be many orders of magnitute slower (while pegging the CPU) -- even
> if those same requests using fl Y uncached (or w/o lazy laoding) would be
> extremely fast.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
