Hi experts: After I sent out previous email, I issued commit on that replica core and observed the same "ClosedChannelException", please refer to below under "issuing core commit" section
Then I issued a core reload, and I see the timestamp of the latest tlog file changed, please refer to "files under tlog directory " section below. Not sure those information is useful or not. Thank you! --Michael Hu --- beginning for issuing core commit --- $ curl 'http://localhost:8983/solr/myconection_myshard_replica_t7/update?commit=true' { "responseHeader":{ "status":500, "QTime":71}, "error":{ "metadata":[ "error-class","org.apache.solr.common.SolrException", "root-error-class","java.nio.channels.ClosedChannelException"], "msg":"java.nio.channels.ClosedChannelException", "trace":"org.apache.solr.common.SolrException: --- end for issuing core commit --- --- beginning for files under tlog directory --- before core reload: -rw-r--r-- 1 solr solr 47527321 Mar 4 20:14 tlog.0000000000000000877 -rw-r--r-- 1 solr solr 42614907 Mar 4 20:14 tlog.0000000000000000878 -rw-r--r-- 1 solr solr 37524663 Mar 4 20:14 tlog.0000000000000000879 -rw-r--r-- 1 solr solr 44067997 Mar 4 20:14 tlog.0000000000000000880 -rw-r--r-- 1 solr solr 33209784 Mar 4 20:15 tlog.0000000000000000881 -rw-r--r-- 1 solr solr 55435186 Mar 4 20:15 tlog.0000000000000000882 -rw-r--r-- 1 solr solr 2179991713 Mar 4 20:29 tlog.0000000000000000883 after core reload: -rw-r--r-- 1 solr solr 47527321 Mar 4 20:14 tlog.0000000000000000877 -rw-r--r-- 1 solr solr 42614907 Mar 4 20:14 tlog.0000000000000000878 -rw-r--r-- 1 solr solr 37524663 Mar 4 20:14 tlog.0000000000000000879 -rw-r--r-- 1 solr solr 44067997 Mar 4 20:14 tlog.0000000000000000880 -rw-r--r-- 1 solr solr 33209784 Mar 4 20:15 tlog.0000000000000000881 -rw-r--r-- 1 solr solr 55435186 Mar 4 20:15 tlog.0000000000000000882 -rw-r--r-- 1 solr solr 2179991717 Mar 4 22:23 tlog.0000000000000000883 --- end for files under tlog directory --- ________________________________ From: Michael Hu <m...@vmware.com> Sent: Thursday, March 4, 2021 1:58 PM To: solr-user@lucene.apache.org <solr-user@lucene.apache.org> Subject: new tlog files are not created per commit but adding into latest existing tlog file after replica reload Hi experts: Need some help and suggestion about an issue I am facing Solr info: - Solr 8.7 - Solr cloud with tlog replica; replica size is 3 for my Solr collection Issue: - before issuing collection reload; I observed a new tlog file are created after every commit; and those tlog files are deleted after a while (may be after index are merged?) - then I issued a collection reload using collection API on my collection at 20:15 - after leader replica is reloaded; no new tlog file are created; instead latest tlog file is growing, and no tlog file is deleted after reload. Below under "files under tlog directory" section is a snapshot of the tlog files under tlog directory of the leader replica. Again, I issued collection reload at 20:15, and after that tlog.0000000000000000883 is growing - I looked into log file and find error log entries below under "log entries" section, and the log entry repeats continuously for every auto commit after reload. I hope this log entry can provide some information for the issue. Please help and suggestion what I may do incorrectly. Or this is a known issue, is there a way I can fix or work-around it? Thank you so much! --Michael Hu --- beginning for files under tlog directory --- -rw-r--r-- 1 solr solr 47527321 Mar 4 20:14 tlog.0000000000000000877 -rw-r--r-- 1 solr solr 42614907 Mar 4 20:14 tlog.0000000000000000878 -rw-r--r-- 1 solr solr 37524663 Mar 4 20:14 tlog.0000000000000000879 -rw-r--r-- 1 solr solr 44067997 Mar 4 20:14 tlog.0000000000000000880 -rw-r--r-- 1 solr solr 33209784 Mar 4 20:15 tlog.0000000000000000881 -rw-r--r-- 1 solr solr 55435186 Mar 4 20:15 tlog.0000000000000000882 -rw-r--r-- 1 solr solr 2179991713 Mar 4 20:29 tlog.0000000000000000883 --- end for files under tlog directory --- --- beginning for log entries --- 2021-03-04 20:15:38.251 ERROR (commitScheduler-4327-thread-1) [c:mycollection s:myshard r:core_node10 x:mycolletion_myshard_replica_t7] o.a.s.u.CommitTracker auto commit error...: org.apache.solr.common.SolrException: java.nio.channels.ClosedChannelException at org.apache.solr.update.TransactionLog.writeCommit(TransactionLog.java:503) at org.apache.solr.update.UpdateLog.postCommit(UpdateLog.java:835) at org.apache.solr.update.UpdateLog.preCommit(UpdateLog.java:819) at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:673) at org.apache.solr.update.CommitTracker.run(CommitTracker.java:273) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.nio.channels.ClosedChannelException at java.base/sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:150) at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:266) at java.base/java.nio.channels.Channels.writeFullyImpl(Channels.java:74) at java.base/java.nio.channels.Channels.writeFully(Channels.java:97) at java.base/java.nio.channels.Channels$1.write(Channels.java:172) at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216) at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209) at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:193) at org.apache.solr.update.TransactionLog.writeCommit(TransactionLog.java:498) ... 10 more --- end for log entries ---
