[jira] [Comment Edited] (SOLR-13315) Possible SolrIndexSearcher leak through LogWatcher and FunctionScoreQuery
[
https://issues.apache.org/jira/browse/SOLR-13315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16845815#comment-16845815
]
Yury Pakhomov edited comment on SOLR-13315 at 5/23/19 11:32 AM:
I thought that disabling Log4j2Watcher in solr.xml
can be used as a temporary solution in case if you have no time to build solr
from source code with patches.
But this fix only solves searcher leak though Log4j2Watcher.
Not long ago I have found another leak with FunctionScoreQuery.
It looks pretty simple but it relatively hard to find.
If FunctionScoreQuery will be cached in any org.apache.solr.search.SolrCache
then when new searcher will be opened and old SolrIndexSearcher will try to
warm caches of newly opened searcher calling method
org.apache.solr.search.SolrIndexSearcher#warm()
As a result reference to old searcher will be cached in new searcher.
Also it seems that this warmed query will have not valid results as
FunctionScoreQuery will be partially executed with new searcher and partially
with old Searcher.
So it seems that if you are using 7.4 solr and FunctionScoreQuery you have to
apply patch created by Alan Woodward. I have checked it and this patch prevents
this leak through cache regeneration.
There are multiple *ValueSource classes
and some of them have method
public abstract *ValuesSource rewrite(IndexSearcher reader) throws IOException;
Reference for passed IndexSearcher should be used with care to avoid such
situations.
Currently only
org.apache.lucene.queries.function.ValueSource.WrappedDoubleValuesSource
was affected but it was fixed. Maybe some java doc should be added to avoid
such problems in future implementations.
Scroll to the right to see full picture.
!old_searcher_leak_through_cache_regeneration.png!
was (Author: ypakhomov):
I thought that disabling Log4j2Watcher in solr.xml
can be used as a temporary solution in case if you have no time to build solr
from source code with patches.
But this fix only solves searcher leak though Log4j2Watcher.
Not long ago I have found another leak with FunctionScoreQuery.
It looks pretty simple but it relatively hard to find.
If FunctionScoreQuery will be cached in any org.apache.solr.search.SolrCache
then when new searcher will opened SolrIndexSearcher will try to warm caches of
newly opened searcher calling method
org.apache.solr.search.SolrIndexSearcher#warm()
As a result reference to old searcher will be cached in new searcher.
Also it seems that this warmed query will have not valid results as
FunctionScoreQuery will be partially executed with new searcher and partially
with old Searcher.
So it seems that if you are using 7.4 solr and FunctionScoreQuery you have to
apply patch created by Alan Woodward. I have checked it and this patch prevents
this leak through cache regeneration.
There are multiple *ValueSource classes
and some of them have method
public abstract *ValuesSource rewrite(IndexSearcher reader) throws IOException;
Reference for passed IndexSearcher should be used with care to avoid such
situations.
Currently only
org.apache.lucene.queries.function.ValueSource.WrappedDoubleValuesSource
was affected but it was fixed. Maybe some java doc should be added to avoid
such problems in future implementations.
Scroll to the right to see full picture.
!old_searcher_leak_through_cache_regeneration.png!
> Possible SolrIndexSearcher leak through LogWatcher and FunctionScoreQuery
> -
>
> Key: SOLR-13315
> URL: https://issues.apache.org/jira/browse/SOLR-13315
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 7.5
>Reporter: Yury Pakhomov
>Assignee: Alan Woodward
>Priority: Major
> Fix For: 8.1
>
> Attachments: SOLR-13315.patch,
> old_searcher_leak_through_cache_regeneration.png,
> path_to_gc_root_from_heap_dump.png
>
>
> Here is possible leak of SolrIndexSearcher. Which prevents unused searchers
> to be reclaimed by gc.
> This problem was found after analyzing heap dump which was created before
> Full GC.
> 1) Where unused ref to SolrIndexSearcher is stored.
> Log4j2Watcher implements LogWatcher
> and has CircularList history inherited from LogWatcher
> In history we can store Log4jLogEvent which can hold ref to
> ParameterizedMessage
> and ParameterizedMessage stores refs to all arguments of event log. (here we
> can store objects which are no longer in use directly or indirectly)
> 2) How SolrIndexSearcher can be indirectly reached through this log buffer.
> If during FunctionScoreQuery execution ExitingReaderException("The request
> took too long to iterate over terms. Timeout: " ..) will be thrown this query
> will be
[jira] [Comment Edited] (SOLR-13315) Possible SolrIndexSearcher leak through LogWatcher and FunctionScoreQuery
[
https://issues.apache.org/jira/browse/SOLR-13315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16845815#comment-16845815
]
Yury Pakhomov edited comment on SOLR-13315 at 5/22/19 12:33 PM:
I thought that disabling Log4j2Watcher in solr.xml
can be used as a temporary solution in case if you have no time to build solr
from source code with patches.
But this fix only solves searcher leak though Log4j2Watcher.
Not long ago I have found another leak with FunctionScoreQuery.
It looks pretty simple but it relatively hard to find.
If FunctionScoreQuery will be cached in any org.apache.solr.search.SolrCache
then when new searcher will opened SolrIndexSearcher will try to warm caches of
newly opened searcher calling method
org.apache.solr.search.SolrIndexSearcher#warm()
As a result reference to old searcher will be cached in new searcher.
Also it seems that this warmed query will have not valid results as
FunctionScoreQuery will be partially executed with new searcher and partially
with old Searcher.
So it seems that if you are using 7.4 solr and FunctionScoreQuery you have to
apply patch created by Alan Woodward. I have checked it and this patch prevents
this leak through cache regeneration.
There are multiple *ValueSource classes
and some of them have method
public abstract *ValuesSource rewrite(IndexSearcher reader) throws IOException;
Reference for passed IndexSearcher should be used with care to avoid such
situations.
Currently only
org.apache.lucene.queries.function.ValueSource.WrappedDoubleValuesSource
was affected but it was fixed. Maybe some java doc should be added to avoid
such problems in future implementations.
Scroll to the right to see full picture.
!old_searcher_leak_through_cache_regeneration.png!
was (Author: ypakhomov):
I thought that disabling Log4j2Watcher in solr.xml
can be used as a temporary solution in case if you have no time to build solr
from source code with patches.
But this fix only solves searcher leak though Log4j2Watcher.
Not long ago I have found another leak with FunctionScoreQuery.
It looks pretty simple but it relatively hard to find.
If FunctionScoreQuery will be cached in any org.apache.solr.search.SolrCache
then when new searcher will opened SolrIndexSearcher will try to warm caches of
newly opened searcher calling method
org.apache.solr.search.SolrIndexSearcher#warm()
As a result reference to old searcher will be cached in new searcher.
Also it seems that this warmed query will have not valid results as
FunctionScoreQuery will be partially executed with new searcher and partially
with old Searcher.
So it seems that if you are using 7.4 solr and FunctionScoreQuery you have to
apply patch created by Alan Woodward. I have checked it and this patch prevents
this leak through cache regeneration.
There are multiple *ValueSource classes
and some of them have method
public abstract *ValuesSource rewrite(IndexSearcher reader) throws IOException;
Reference for passed IndexSearcher should be used with care to avoid such
situations.
Currently only
org.apache.lucene.queries.function.ValueSource.WrappedDoubleValuesSource
was affected but it was fixed. Maybe some java doc should be added to avoid
such problems in future implementations.
!old_searcher_leak_through_cache_regeneration.png!
> Possible SolrIndexSearcher leak through LogWatcher and FunctionScoreQuery
> -
>
> Key: SOLR-13315
> URL: https://issues.apache.org/jira/browse/SOLR-13315
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 7.5
>Reporter: Yury Pakhomov
>Assignee: Alan Woodward
>Priority: Major
> Fix For: 8.1
>
> Attachments: SOLR-13315.patch,
> old_searcher_leak_through_cache_regeneration.png,
> path_to_gc_root_from_heap_dump.png
>
>
> Here is possible leak of SolrIndexSearcher. Which prevents unused searchers
> to be reclaimed by gc.
> This problem was found after analyzing heap dump which was created before
> Full GC.
> 1) Where unused ref to SolrIndexSearcher is stored.
> Log4j2Watcher implements LogWatcher
> and has CircularList history inherited from LogWatcher
> In history we can store Log4jLogEvent which can hold ref to
> ParameterizedMessage
> and ParameterizedMessage stores refs to all arguments of event log. (here we
> can store objects which are no longer in use directly or indirectly)
> 2) How SolrIndexSearcher can be indirectly reached through this log buffer.
> If during FunctionScoreQuery execution ExitingReaderException("The request
> took too long to iterate over terms. Timeout: " ..) will be thrown this query
> will be logged with warn level and it ref will be store in
[jira] [Comment Edited] (SOLR-13315) Possible SolrIndexSearcher leak through LogWatcher and FunctionScoreQuery
[
https://issues.apache.org/jira/browse/SOLR-13315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16845815#comment-16845815
]
Yury Pakhomov edited comment on SOLR-13315 at 5/22/19 12:30 PM:
I thought that disabling Log4j2Watcher in solr.xml
can be used as a temporary solution in case if you have no time to build solr
from source code with patches.
But this fix only solves searcher leak though Log4j2Watcher.
Not long ago I have found another leak with FunctionScoreQuery.
It looks pretty simple but it relatively hard to find.
If FunctionScoreQuery will be cached in any org.apache.solr.search.SolrCache
then when new searcher will opened SolrIndexSearcher will try to warm caches of
newly opened searcher calling method
org.apache.solr.search.SolrIndexSearcher#warm()
As a result reference to old searcher will be cached in new searcher.
Also it seems that this warmed query will have not valid results as
FunctionScoreQuery will be partially executed with new searcher and partially
with old Searcher.
So it seems that if you are using 7.4 solr and FunctionScoreQuery you have to
apply patch created by Alan Woodward. I have checked it and this patch prevents
this leak through cache regeneration.
There are multiple *ValueSource classes
and some of them have method
public abstract *ValuesSource rewrite(IndexSearcher reader) throws IOException;
Reference for passed IndexSearcher should be used with care to avoid such
situations.
Currently only
org.apache.lucene.queries.function.ValueSource.WrappedDoubleValuesSource
was affected but it was fixed. Maybe some java doc should be added to avoid
such problems in future implementations.
!old_searcher_leak_through_cache_regeneration.png!
was (Author: ypakhomov):
I thought that disabling Log4j2Watcher in solr.xml
can be used as a temporary solution in case if you have no time to build solr
from source code with patches.
But this fix only solves searcher leak though Log4j2Watcher.
Not long ago I have found another leak with FunctionScoreQuery.
It looks pretty simple but it relatively hard to find.
If FunctionScoreQuery will be cached in any org.apache.solr.search.SolrCache
then when new searcher will opened SolrIndexSearcher will try to warm caches of
newly opened searcher calling method
org.apache.solr.search.SolrIndexSearcher#warm()
As a result reference to old searcher will be cached in new searcher.
Also it seems that this warmed query will have not valid results as
FunctionScoreQuery will be partially executed with new searcher and partially
with old Searcher.
So it seems that if you are using 7.4 solr and FunctionScoreQuery you have to
apply patch created by Alan Woodward. I have checked it and this patch prevents
this leak through cache regeneration.
There are multiple *ValueSource classes
and some of them have method
public abstract *ValuesSource rewrite(IndexSearcher reader) throws IOException;
Reference for passed IndexSearcher should be used with care to avoid such
situations.
Currently only
org.apache.lucene.queries.function.ValueSource.WrappedDoubleValuesSource
was affected but it was fixed. Maybe some java doc should be added to avoid
such problems in future implementations.
> Possible SolrIndexSearcher leak through LogWatcher and FunctionScoreQuery
> -
>
> Key: SOLR-13315
> URL: https://issues.apache.org/jira/browse/SOLR-13315
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 7.5
>Reporter: Yury Pakhomov
>Assignee: Alan Woodward
>Priority: Major
> Fix For: 8.1
>
> Attachments: SOLR-13315.patch,
> old_searcher_leak_through_cache_regeneration.png,
> path_to_gc_root_from_heap_dump.png
>
>
> Here is possible leak of SolrIndexSearcher. Which prevents unused searchers
> to be reclaimed by gc.
> This problem was found after analyzing heap dump which was created before
> Full GC.
> 1) Where unused ref to SolrIndexSearcher is stored.
> Log4j2Watcher implements LogWatcher
> and has CircularList history inherited from LogWatcher
> In history we can store Log4jLogEvent which can hold ref to
> ParameterizedMessage
> and ParameterizedMessage stores refs to all arguments of event log. (here we
> can store objects which are no longer in use directly or indirectly)
> 2) How SolrIndexSearcher can be indirectly reached through this log buffer.
> If during FunctionScoreQuery execution ExitingReaderException("The request
> took too long to iterate over terms. Timeout: " ..) will be thrown this query
> will be logged with warn level and it ref will be store in Log4j2Watcher.
> (Here can be any exception which will log this query to Log4j2Watcher)
> In general it
