On 10/18/2020 3:22 AM, Dominique Bejean wrote:
A few months ago, I reported an issue with Solr nodes crashing due to the
old generation heap growing suddenly and generating OOM. This problem
occurred again this week. I have threads dumps for each minute during the 3
minutes the problem occured. I am using fastthread.io in order to analyse
these dumps.
<snip>
* The Log4j issue starts (
https://blog.fastthread.io/2020/01/24/log4j-bug-slows-down-your-app/)
If the log4j bug is the root cause here, then the only way you can fix
this is to upgrade to at least Solr 7.4. That is the Solr version where
we first upgraded from log4j 1.2.x to log4j2. You cannot upgrade log4j
in Solr 6.6.2 without changing Solr code. The code changes required
were extensive. Note that I did not do anything to confirm whether the
log4j bug is responsible here. You seem pretty confident that this is
the case.
Note that if you upgrade to 8.x, you will need to reindex from scratch.
Upgrading an existing index is possible with one major version bump, but
if your index has ever been touched by a release that's two major
versions back, it won't work. In 8.x, that is enforced -- 8.x will not
even try to read an old index touched by 6.x or earlier.
In the following wiki page, I provided instructions for getting a
screenshot of the process listing.
https://cwiki.apache.org/confluence/display/solr/SolrPerformanceProblems
In addition to that screenshot, I would like to know the on-disk size of
all the cores running on the problem node, along with a document count
from those cores. It might be possible to work around the OOM just by
increasing the size of the heap. That won't do anything about problems
with log4j.
Thanks,
Shawn
