[
https://issues.apache.org/jira/browse/SOLR-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-4589:
---------------------------
Attachment: SOLR-4589.patch
Digression...
{panel}
It really feals like the whole LazyDocument API and usage pattern --
particularly the semantics in which "LazyDocument.getField(FieldInfo)" can be
used -- is really brittle and confusiong.
It assumes it will be called exactly once for each StorableField that exists
for that field name, in order because the internal counter is assumed to
correspond exactly wit the array index of the corresponding array of values.
(In short: it's really only useful when called from StoredFieldVisitor)
If, for example, a client app tries to use the same LazyDocument instance to
generate LazyField instances for the same field twice -- it will most likeley
get an ArrayIndexOutOfBoundsError. Likewise a user could read the jacadocs for
LazyDocument.getField and assume that it works simlar to
StorableDocument.getField expecting it to return a LazyField of the _first_
field value for the specified field. They could wind up calling the method
twice on the same field and getting lazy fields pointing at two differnet
values (the first and second StorableField instances for that field name)
none of which directly relates to this particular bug, but since this public
class seems to have been added solely for solr, it may be worth considering
deprecating it, and moving the logic solr needs into protected/private classes.
{panel}
...as for the matter at hand...
I reworked the internals of hte class a bit so that:
* LazyField instances cache their underlying values similar to how lazy loading
worked in 3.x
* Asking for the value of a LazyField instance causes all LazyField's (produced
by the same LazyDocument) with the same field name to be "actualized" in a
single pass over the underlying document
The changes eliminate the pathalogical case of iterating over many LazyFields,
and shouldn't add much even in the edge case of only carrying about a single
field value when there are many -- it's still a single loop in the underlying
Document of the fields, there's just no more short circut for hte single value
case.
In order for these changes to work, the LazDocument has to now keep a reference
to every LazyField it generates so that they can all be repopulated when one of
their filed name brothers is repopulated. The memory footprint of this
shouldn't be too bad in the common case (afterall: the LazyDocument was already
fetching the entire underlying Document keeping the whole thing around as long
as there was a single LazyField in the heap) but just to be safe i made it
track the list of LazyFields using weak refrences -- so even if someone asks
for a bunch of LazyField instance for a field named "foo", and then throws away
all but a few of them, they can be garbage collected regardless of how many
other "foo" LazyFields are still arround or when/if their real values are
loaded.
Patch includes a few test updates in solr to sanity check that lazy loading
value caching works, but i definitely want to add some true LazyDocument unit
tests to the module (there don't seem to be any at all at the moment) to
exercise the "all lazy fields with the same name get populated at the same
time" and "what if a weakref LazyField goes away" logic before i'd consider
committing.
----
Feedback would be greatly appreciated.
> 4.x + enableLazyFieldLoading + large nultivalued fields + varying fl =
> pathalogical CPU load & response time
> ------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-4589
> URL: https://issues.apache.org/jira/browse/SOLR-4589
> Project: Solr
> Issue Type: Bug
> Affects Versions: 4.0, 4.1, 4.2
> Reporter: Hoss Man
> Attachments: SOLR-4589.patch,
> test-just-queries.out__4.0.0_mmap_lazy_using36index.txt,
> test-just-queries.sh, test.out__3.6.1_mmap_lazy.txt,
> test.out__3.6.1_mmap_nolazy.txt, test.out__3.6.1_nio_lazy.txt,
> test.out__3.6.1_nio_nolazy.txt, test.out__4.0.0_mmap_lazy.txt,
> test.out__4.0.0_mmap_nolazy.txt, test.out__4.0.0_nio_lazy.txt,
> test.out__4.0.0_nio_nolazy.txt, test.out__4.2.0_mmap_lazy.txt,
> test.out__4.2.0_mmap_nolazy.txt, test.out__4.2.0_nio_lazy.txt,
> test.out__4.2.0_nio_nolazy.txt, test.sh
>
>
> Following up on a [user report of exterme CPU usage in
> 4.1|http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201302.mbox/%[email protected]%3E],
> I've discovered that the following combination of factors can result in
> extreme CPU usage and excessively HTTP response times...
> * Solr 4.x (tested 3.6.1, 4.0.0, and 4.2.0)
> * enableLazyFieldLoading == true (included in example solrconfig.xml)
> * documents with a large number of values in multivalued fields (eg: tested
> ~10-15K values)
> * multiple requests returning the same doc with different "fl" lists
> I haven't dug into the route cause yet, but the essential observations is: if
> lazyloading is used in 4.x, then once a document has been fetched with an
> initial fl list X, subsequent requests for that document using a differnet fl
> list Y can be many orders of magnitute slower (while pegging the CPU) -- even
> if those same requests using fl Y uncached (or w/o lazy laoding) would be
> extremely fast.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
