[
https://issues.apache.org/jira/browse/SOLR-17063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794865#comment-17794865
]
ASF subversion and git services commented on SOLR-17063:
--------------------------------------------------------
Commit 9ad46fb21a1cf072e29ceb638f80ff187b020164 in solr's branch
refs/heads/branch_9x from Michael Gibney
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=9ad46fb21a1 ]
SOLR-17063: Do not retain log param references in LogWatcher (#2058)
(cherry picked from commit 6b74f86fc46cbac8e5e4278597de9e3a7334849c)
> Do not buffer raw log params in LogWatcher
> ------------------------------------------
>
> Key: SOLR-17063
> URL: https://issues.apache.org/jira/browse/SOLR-17063
> Project: Solr
> Issue Type: Improvement
> Affects Versions: main (10.0), 9.4
> Reporter: Michael Gibney
> Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> As [~mkhl] [suggested on
> SOLR-13315|https://issues.apache.org/jira/browse/SOLR-13315?focusedCommentId=16790317#comment-16790317],
> it would be possible for {{LogWatcher}}, rather than buffering all the raw
> parameters of a log message, to pre-compute the derivative {{SolrDocument}}s
> and buffer those instead.
> SOLR-13315 ended up being addressed in a different way, but
> "memory-leak-like" behavior is still possible and I think we should follow
> [~mkhl]'s suggestion. The problem we ran into recently was a huge distributed
> query, dispersed to lots of shards. One of the ShardRequests failed, causing
> the failed ShardResponse to be logged at warn level in SearchHandler:
> {code:java}
> log.warn("Shard request failed : {}", srsp);
> {code}
> The logged ShardResponse indirectly holds a reference (via ShardRequest ref)
> to all the other ShardResponses, some of which succeeded and were quite
> large. These references then just sit in LogWatcher's {{history}}
> CircularBuffer until they are overwritten by subsequent log messages (default
> WARN/ERROR level). So as long as you have fewer than 50 (default config)
> subsequent WARN/ERROR level messages, these references will be held (in
> principle forever). In our case it consumed a substantial portion of the heap.
> This issue proposes that LogWatcher should process LogEvents into
> SolrDocuments (the form they take upon retrieval) when they are initially
> added to the {{history}} buffer. The downside is some minimal amount of extra
> processing in the case where the events are never retrieved from the buffer;
> the upside is that the params don't needlessly retain heap.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
