Re: Solr collection indexed to pdf in hdfs throws error during solr restart
On 3/14/2019 1:13 AM, VAIBHAV SHUKLA shuklavaibha...@yahoo.in wrote: When I restart Solr it throws the following error. Solr collection indexed to pdf in hdfs throws error during solr restart. Error Caused by: org.apache.lucene.store.LockObtainFailedException: Index dir 'hdfs://192.168.1.16:8020/PDFIndex/data/index/' of core 'PDFIndex' is already locked. The most likely cause is another Solr server (or another solr core in this server) also configured to use this directory; other possible causes may be specific to lockType: hdfs Solr has been shut down forcefully, so the lockfile is remaining in the core's directory (which in your case is in HDFS). A graceful shutdown would have deleted the lockfile. What version of Solr, and what OS do you have it running on? For a while now, on non-windows operating systems, the "stop" action in the bin/solr script has waited up to 3 minutes for Solr to gracefully shut down before forcefully killing it. This has eliminated most of these problems when running on one of those operating systems. On Windows, the bin\solr script is only waiting 5 seconds before forcefully killing Solr, which used to happen on all operating systems. This is extremely likely to cause problems like this. Fixing this on Windows is on the radar, but in general we lack adept skill with Windows, so it's not proceeding quickly. I'm having trouble locating the issue for fixing the problem on Windows. To fix it, find the "write.lock" file in your core's HDFS storage location and delete it. Thanks, Shawn
Re: Solr collection indexed to pdf in hdfs throws error during solr restart
> When I restart Solr How exactly are you restarting Solr? Are you running a "bin/solr restart"? Or is Solr already shut down and you're just starting it back up with a "bin/solr start "? Depending on how Solr was shut down, you might be running into a bit of a known-issue with Solr's HDFS support. Solr creates lock files for each index, to restrict who can write to that index in the interest of avoiding race conditions and protecting against file corruption. Often when Solr crashes or is shut down abruptly (via a "kill -9") it doesn't have time to clean up these lock files and it fails to start up the next time because it is still locked out from touching that index. This might be what you're running in to. In which case you could carefully make sure that no Solr nodes are using the index in question, delete the lock file manually out of HDFS, and try starting Solr again. The advice above is what we usually tell people with write.lock issues on HDFS...though some elements of the stack trace you provided make me wonder whether you're seeing the same exact problem. Your stack trace has a NullPointerException, and a "Filesystem Closed" error (typically seen when a Java object gets closed too early and may indicate a bug). I'm not used to seeing either of these associated with the "standard" write.lock issues. What version of Solr are you seeing this on? Best regards, Jason On Thu, Mar 14, 2019 at 5:28 AM VAIBHAV SHUKLA shuklavaibha...@yahoo.in wrote: > > When I restart Solr it throws the following error. Solr collection indexed to > pdf in hdfs throws error during solr restart. > > > > Error > > java.util.concurrent.ExecutionException: > org.apache.solr.common.SolrException: Unable to create core [PDFIndex] > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.solr.core.CoreContainer.lambda$load$6(CoreContainer.java:594) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.solr.common.SolrException: Unable to create core > [PDFIndex] > at > org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:966) > at > org.apache.solr.core.CoreContainer.lambda$load$5(CoreContainer.java:565) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) > ... 5 more > Caused by: org.apache.solr.common.SolrException: Index dir > 'hdfs://192.168.1.16:8020/PDFIndex/data/index/' of core 'PDFIndex' is already > locked. The most likely cause is another Solr server (or another solr core in > this server) also configured to use this directory; other possible causes may > be specific to lockType: hdfs > at org.apache.solr.core.SolrCore.(SolrCore.java:977) > at org.apache.solr.core.SolrCore.(SolrCore.java:830) > at > org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:950) > ... 7 more > Caused by: org.apache.lucene.store.LockObtainFailedException: Index dir > 'hdfs://192.168.1.16:8020/PDFIndex/data/index/' of core 'PDFIndex' is already > locked. The most likely cause is another Solr server (or another solr core in > this server) also configured to use this directory; other possible causes may > be specific to lockType: hdfs > at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:712) > at org.apache.solr.core.SolrCore.(SolrCore.java:923) > ... 9 more > 2018-12-22 07:55:13.431 ERROR > (OldIndexDirectoryCleanupThreadForCore-PDFIndex) [ x:PDFIndex] > o.a.s.c.HdfsDirectoryFactory Error checking for old index directories to > clean-up. > java.io.IOException: Filesystem closed > at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) > at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2083) > at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2069) > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:791) > at > org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106) > at >
Solr collection indexed to pdf in hdfs throws error during solr restart
When I restart Solr it throws the following error. Solr collection indexed to pdf in hdfs throws error during solr restart. Error java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException: Unable to create core [PDFIndex] at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.solr.core.CoreContainer.lambda$load$6(CoreContainer.java:594) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.solr.common.SolrException: Unable to create core [PDFIndex] at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:966) at org.apache.solr.core.CoreContainer.lambda$load$5(CoreContainer.java:565) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) ... 5 more Caused by: org.apache.solr.common.SolrException: Index dir 'hdfs://192.168.1.16:8020/PDFIndex/data/index/' of core 'PDFIndex' is already locked. The most likely cause is another Solr server (or another solr core in this server) also configured to use this directory; other possible causes may be specific to lockType: hdfs at org.apache.solr.core.SolrCore.(SolrCore.java:977) at org.apache.solr.core.SolrCore.(SolrCore.java:830) at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:950) ... 7 more Caused by: org.apache.lucene.store.LockObtainFailedException: Index dir 'hdfs://192.168.1.16:8020/PDFIndex/data/index/' of core 'PDFIndex' is already locked. The most likely cause is another Solr server (or another solr core in this server) also configured to use this directory; other possible causes may be specific to lockType: hdfs at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:712) at org.apache.solr.core.SolrCore.(SolrCore.java:923) ... 9 more 2018-12-22 07:55:13.431 ERROR (OldIndexDirectoryCleanupThreadForCore-PDFIndex) [ x:PDFIndex] o.a.s.c.HdfsDirectoryFactory Error checking for old index directories to clean-up. java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2083) at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2069) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:791) at org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:853) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:849) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:860) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1557) at org.apache.solr.core.HdfsDirectoryFactory.cleanupOldIndexDirectories(HdfsDirectoryFactory.java:546) at org.apache.solr.core.SolrCore.lambda$cleanupOldIndexDirectories$19(SolrCore.java:3050) at java.lang.Thread.run(Thread.java:748) 2018-12-22 07:55:13.433 ERROR (OldIndexDirectoryCleanupThreadForCore-PDFIndex) [ x:PDFIndex] o.a.s.c.SolrCore Failed to cleanup old index directories for core PDFIndex java.lang.NullPointerException at org.apache.solr.core.HdfsDirectoryFactory.cleanupOldIndexDirectories(HdfsDirectoryFactory.java:564) at org.apache.solr.core.SolrCore.lambda$cleanupOldIndexDirectories$19(SolrCore.java:3050) at java.lang.Thread.run(Thread.java:748) I have created the collection in Solr which will index the pdf files and this collection is indexing all the pdf in HDFS. Thanks & Regards Vaibhav Shukla Sent from Mail for Windows 10
Solr collection indexed to pdf in hdfs throws error during solr restart
When I restart Solr it throws the following error. Solr collection indexed to pdf in hdfs throws error during solr restart. Error java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException: Unable to create core [PDFIndex] at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.solr.core.CoreContainer.lambda$load$6(CoreContainer.java:594) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.solr.common.SolrException: Unable to create core [PDFIndex] at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:966) at org.apache.solr.core.CoreContainer.lambda$load$5(CoreContainer.java:565) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) ... 5 more Caused by: org.apache.solr.common.SolrException: Index dir 'hdfs://192.168.1.16:8020/PDFIndex/data/index/' of core 'PDFIndex' is already locked. The most likely cause is another Solr server (or another solr core in this server) also configured to use this directory; other possible causes may be specific to lockType: hdfs at org.apache.solr.core.SolrCore.(SolrCore.java:977) at org.apache.solr.core.SolrCore.(SolrCore.java:830) at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:950) ... 7 more Caused by: org.apache.lucene.store.LockObtainFailedException: Index dir 'hdfs://192.168.1.16:8020/PDFIndex/data/index/' of core 'PDFIndex' is already locked. The most likely cause is another Solr server (or another solr core in this server) also configured to use this directory; other possible causes may be specific to lockType: hdfs at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:712) at org.apache.solr.core.SolrCore.(SolrCore.java:923) ... 9 more 2018-12-22 07:55:13.431 ERROR (OldIndexDirectoryCleanupThreadForCore-PDFIndex) [ x:PDFIndex] o.a.s.c.HdfsDirectoryFactory Error checking for old index directories to clean-up. java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2083) at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2069) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:791) at org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:853) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:849) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:860) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1557) at org.apache.solr.core.HdfsDirectoryFactory.cleanupOldIndexDirectories(HdfsDirectoryFactory.java:546) at org.apache.solr.core.SolrCore.lambda$cleanupOldIndexDirectories$19(SolrCore.java:3050) at java.lang.Thread.run(Thread.java:748) 2018-12-22 07:55:13.433 ERROR (OldIndexDirectoryCleanupThreadForCore-PDFIndex) [ x:PDFIndex] o.a.s.c.SolrCore Failed to cleanup old index directories for core PDFIndex java.lang.NullPointerException at org.apache.solr.core.HdfsDirectoryFactory.cleanupOldIndexDirectories(HdfsDirectoryFactory.java:564) at org.apache.solr.core.SolrCore.lambda$cleanupOldIndexDirectories$19(SolrCore.java:3050) at java.lang.Thread.run(Thread.java:748) I have created the collection in Solr which will index the pdf files and this collection is indexing all the pdf in HDFS. Thanks & Regards Vaibhav Shukla Sent from Mail for Windows 10
