[
https://issues.apache.org/jira/browse/SOLR-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated SOLR-16555:
--------------------------------
Description:
SolrIndexSearcher takes the bitset from the result and tries to combine it with
all the cached filter queries. Currently this duplicates the bitset multiple
times based on the number of filter queries. It looks like this isn't necessary
and instead could just operate on the bitset itself or a single mutable copy of
the bitset.
Lines 1219 to 1225
https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java#L1219
----
I've been using async profiler
(https://github.com/jvm-profiling-tools/async-profiler) to look at some
performance stuff with Solr for a client. Originally I looked at CPU in the
profile and found that I could also capture and look at memory allocations
during the same run. This led me to finding this crazy amount of memory
allocation over a short period of time.
Async profiler is being run with the following parameters which captures cpu,
memory, and lock information for a 300 second period on some pid.
{code:java}
/opt/async-profiler/profiler.sh -a -d 300 -o jfr -e cpu,alloc,lock -f
/tmp/profile.jfr PID_GOES_HERE
{code}
The resulting JFR for this is ~100-200MB usually and so not going to attach it
here since it has some client specific methods in some calls in it.
However screenshots of the findings from loading the jfr in both IntelliJ and
Java Mission Control you can see some of the findings:
!Screenshot 2022-11-16 at 14.52.37.png|width=750!
This shows that in 5 minutes ~1.06 terabytes of memory allocations for
SolrIndexSearcher#getProcessedFilter
!Screenshot 2022-11-16 at 14.53.23.png!
was:
SolrIndexSearcher takes the bitset from the result and tries to combine it with
all the cached filter queries. Currently this duplicates the bitset multiple
times based on the number of filter queries. It looks like this isn't necessary
and instead could just operate on the bitset itself or a single mutable copy of
the bitset.
Lines 1219 to 1225
https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java#L1219
----
I've been using async profiler
(https://github.com/jvm-profiling-tools/async-profiler) to look at some
performance stuff with Solr for a client. Originally I looked at CPU in the
profile and found that I could also capture and look at memory allocations
during the same run. This led me to finding this crazy amount of memory
allocation over a short period of time.
Async profiler is being run with the following parameters which captures cpu,
memory, and lock information for a 300 second period on some pid.
{code:java}
/opt/async-profiler/profiler.sh -a -d 300 -o jfr -e cpu,alloc,lock -f
/tmp/profile.jfr PID_GOES_HERE
{code}
The resulting JFR for this is ~100-200MB usually and so not going to attach it
here since it has some client specific methods in some calls in it.
However screenshots of the findings from loading the jfr in both IntelliJ and
Java Mission Control you can see some of the findings:
!Screenshot 2022-11-16 at 14.52.37.png!
> SolrIndexSearcher - FilterCache intersections/andNot should not clone bitsets
> repeatedly
> ----------------------------------------------------------------------------------------
>
> Key: SOLR-16555
> URL: https://issues.apache.org/jira/browse/SOLR-16555
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query
> Reporter: Kevin Risden
> Priority: Major
> Attachments: Screenshot 2022-11-16 at 14.52.37.png, Screenshot
> 2022-11-16 at 14.53.23.png, Screenshot 2022-11-16 at 14.53.35.png, Screenshot
> 2022-11-17 at 13.03.21.png
>
>
> SolrIndexSearcher takes the bitset from the result and tries to combine it
> with all the cached filter queries. Currently this duplicates the bitset
> multiple times based on the number of filter queries. It looks like this
> isn't necessary and instead could just operate on the bitset itself or a
> single mutable copy of the bitset.
> Lines 1219 to 1225
> https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java#L1219
> ----
> I've been using async profiler
> (https://github.com/jvm-profiling-tools/async-profiler) to look at some
> performance stuff with Solr for a client. Originally I looked at CPU in the
> profile and found that I could also capture and look at memory allocations
> during the same run. This led me to finding this crazy amount of memory
> allocation over a short period of time.
> Async profiler is being run with the following parameters which captures cpu,
> memory, and lock information for a 300 second period on some pid.
> {code:java}
> /opt/async-profiler/profiler.sh -a -d 300 -o jfr -e cpu,alloc,lock -f
> /tmp/profile.jfr PID_GOES_HERE
> {code}
> The resulting JFR for this is ~100-200MB usually and so not going to attach
> it here since it has some client specific methods in some calls in it.
> However screenshots of the findings from loading the jfr in both IntelliJ and
> Java Mission Control you can see some of the findings:
> !Screenshot 2022-11-16 at 14.52.37.png|width=750!
> This shows that in 5 minutes ~1.06 terabytes of memory allocations for
> SolrIndexSearcher#getProcessedFilter
> !Screenshot 2022-11-16 at 14.53.23.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
