Hello, Disk space was WARN level. It seems only stack traces of ERROR level messages are visible via the GUI, and that is where the 'No space left' was hiding. Without logging in and inspecting the logs manually, you will never notice that message.
Regards, Markus 2018-04-24 12:23:44.215 WARN (recoveryExecutor-4-thread-1-processing-n:idx3.gr.nl.openindex.io:8983_solr x:search_shard1_replica2 s:shard1 c:search r:core_node3) [c:search s:shard1 r:core_ node3 x:search_shard1_replica2] o.a.s.h.IndexFetcher Error in fetching file: _l5d.cfs (downloaded 48234496 of 218562805 bytes) java.io.IOException: No space left on device at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65) at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) at java.nio.channels.Channels.writeFully(Channels.java:101) at java.nio.channels.Channels.access$000(Channels.java:61) at java.nio.channels.Channels$1.write(Channels.java:174) at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:419) at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122) at org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:53) at org.apache.solr.handler.IndexFetcher$DirectoryFile.write(IndexFetcher.java:1749) -----Original message----- > From:Shawn Heisey <elyog...@elyograg.org> > Sent: Tuesday 24th April 2018 17:39 > To: solr-user@lucene.apache.org > Subject: Re: IndexFetcher cannot download index file > > On 4/24/2018 6:52 AM, Markus Jelsma wrote: > > Forget about it, recovery got a java.io.IOException: No space left on > > device but it wasn't clear until i inspected the real logs. > > > > The logs in de web admin didn't show the disk space exception, even when i > > expand the log line. Maybe that could be changed. > > What was the severity of the log entry showing the disk space > exception? Can you share the whole message/stacktrace? > > If it doesn't show up in the admin UI logging tab, that would suggest > that it was an INFO level log, when it should probably be ERROR. > > Thanks, > Shawn > >