Very strange things start to happen when GC becomes unstable. The first and
simplest thing to do would be to bump up your heap, say to 20g (note, try to
stay under 32G or be prepared to jump significantly higher. At 32G long
pointers have to be used and you actually have less memory available than you
think).
The first three warnings indicate that you have both managed-schema and
schema.xml in your configset _and_ are using the managed schema (enabled in
solrconfig.xml). Which also suggests you’re upgrading from a previous version.
This is printed out in as a courtesy notification that schema.xml is no longer
being used so you should delete it to avoid confusion. NOTE: if you want to use
schema.xml like you have before, see the reference guide.
The fourth warning suggests that you have killed Solr without committing and
it’s replaying the transaction log. For instance, “kill -9” or other will do
it. If you do something like that before a commit completes, updates are
replayed from the tlog in order to preserve data.
Which leads to your second issue. I’d guess either you’re not committing after
your updates (and, BTW, please just let your autocommit settings handle that),
and forcefully killing Solr (e.g. kill -9). That can happen even if you use the
“bin/solr stop” command if it takes too long (3 minutes by default last I
knew). A “normal” shutdown that succeeds (i.e. bin/solr stop that doesn’t print
a message about forcefully killing Solr” will commit on shutdown BTW. Taking
over 3 minutes may be a symptom of GC going crazy.
You should to try to figure out why you have this kind of memory spike,
returning a zillion documents is one possible cause (i.e. rows=1000000 or
something). All the docs have to be assembled in memory, so if you need to
return lots of rows, use streaming or cursorMark.
So what I’d do:
1> bump up your heap
2> insure that you shut Solr down gracefully
3> see if any particular query triggers this memory spike and if you’re using
an anti-pattern.
Best,
Erick
> On Oct 2, 2020, at 7:10 PM, Training By Coding <trainingbycod...@gmail.com>
> wrote:
>
> Events:
> • GC logs showing continuous Full GC events. Log report attached.
> • Core filling failed , showing less data( Num Docs) than expected.
> • following warnings showing on dashboard before error.
>
> Level Logger Message
> WARN false ManagedIndexSchemaFactory The schema has been upgraded to
> managed, but the non-managed schema schema.xml is still loadable. PLEASE
> REMOVE THIS FILE.
> WARN false ManagedIndexSchemaFactory The schema has been upgraded to
> managed, but the non-managed schema schema.xml is still loadable. PLEASE
> REMOVE THIS FILE.
> WARN false SolrResourceLoader Solr loaded a deprecated
> plugin/analysis class [solr.TrieDateField]. Please consult documentation how
> to replace it accordingly.
> WARN false ManagedIndexSchemaFactory The schema has been upgraded to
> managed, but the non-managed schema schema.xml is still loadable. PLEASE
> REMOVE THIS FILE.
> WARN false UpdateLog Starting log replay
> tlog{file=........\data\tlog\tlog.0000000000000445482 refcount=2}
> active=false starting pos=0 inSortedOrder=false
> • Total data in all cores around 8 GB
> • Other Configurations:
> • -XX:+UseG1GC
> • -XX:+UseStringDeduplication
> • -XX:MaxGCPauseMillis=500
> • -Xms15g
> • -Xmx15g
> • -Xss256k
> • OS Environment :
> • Windows 10,
> • Filling cores by calling SQL query using jtds-1.3.1 library.
> • Solr Version 7.5
> • Runtime: Oracle Corporation OpenJDK 64-Bit Server VM 11.0.2
> 11.0.2+9
> • Processors : 48
> • System Physical Memory : 128 GB
> • Swap Space : 256GB
> • solr-spec7.5.0
> • solr-impl7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> jimczi - 2018-09-18 13:07:55
> • lucene-spec7.5.0
> • lucene-impl7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> jimczi - 2018-09-18 13:01:1
> Error Message :
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.JavaBinCodec.readStr(JavaBinCodec.java:863)
> at org.apache.solr.common.util.JavaBinCodec.readStr(JavaBinCodec.java:857)
> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:266)
> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
> at
> org.apache.solr.common.util.JavaBinCodec.readSolrInputDocument(JavaBinCodec.java:603)
> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:315)
> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
> at org.apache.solr.common.util.JavaBinCodec.readArray(JavaBinCodec.java:747)
> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:272)
> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
> at
> org.apache.solr.update.TransactionLog$LogReader.next(TransactionLog.java:673)
> at org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1832)
> at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1747)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
