Michael Gibney created SOLR-17063:
-------------------------------------
Summary: Do not buffer raw log params in LogWatcher
Key: SOLR-17063
URL: https://issues.apache.org/jira/browse/SOLR-17063
Project: Solr
Issue Type: Improvement
Affects Versions: 9.4, main (10.0)
Reporter: Michael Gibney
As [~mkhl] [suggested on
SOLR-13315|https://issues.apache.org/jira/browse/SOLR-13315?focusedCommentId=16790317#comment-16790317],
it would be possible for {{LogWatcher}}, rather than buffering all the raw
parameters of a log message, to pre-compute the derivative {{SolrDocument}}s
and buffer those instead.
SOLR-13315 ended up being addressed in a different way, but "memory-leak-like"
behavior is still possible and I think we should follow [~mkhl]'s suggestion.
The problem we ran into recently was a huge distributed query, dispersed to
lots of shards. One of the ShardRequests failed, causing the failed
ShardResponse to be logged at warn level in SearchHandler:
{code:java}
log.warn("Shard request failed : {}", srsp);
{code}
The logged ShardResponse indirectly holds a reference (via ShardRequest ref) to
all the other ShardResponses, some of which succeeded and were quite large.
These references then just sit in LogWatcher's {{history}} CircularBuffer until
they are overwritten by subsequent log messages (default WARN/ERROR level). So
as long as you have fewer than 50 (default config) subsequent WARN/ERROR level
messages, these references will be held (in principle forever). In our case it
consumed a substantial portion of the heap.
This issue proposes that LogWatcher should process LogEvents into SolrDocuments
(the form they take upon retrieval) when they are initially added to the
{{history}} buffer. The downside is some minimal amount of extra processing in
the case where the events are never retrieved from the buffer; the upside is
that the params don't needlessly retain heap.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
