[
https://issues.apache.org/jira/browse/SOLR-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229730#comment-15229730
]
Robert Muir commented on SOLR-8630:
-----------------------------------
Stuff is broken if NRTCachingDirectory thinks a file is both cached in ram and
on disk, the file is "in two places": The safety check is correct, it was no
mistake or "WIP". The issue about "mkdirs has not yet been called" becomes
impossible due to the changes on that issue so there was no reason to have this
as only an assert.
> IllegalStateException in NRTCachingDirectory.listAll
> ----------------------------------------------------
>
> Key: SOLR-8630
> URL: https://issues.apache.org/jira/browse/SOLR-8630
> Project: Solr
> Issue Type: Bug
> Affects Versions: 5.2.1
> Environment: Production, QA
> Reporter: Semion Mc Alice
>
> Hey,
> we are getting IllegalStateException in 2 different circumstances. The first
> one is on Status calls:
> {noformat}
> ERROR - 2016-02-01 22:32:43.164; [ ] org.apache.solr.common.SolrException;
> org.apache.solr.common.SolrException: Error handling 'status' action
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleStatusAction(CoreAdminHandler.java:748)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:228)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:193)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:660)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:431)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at
> org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1129)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:497)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.IllegalStateException: file:
> MMapDirectory@D:\Solr\server\solr\Prod_Core1_shard1_replica2\data\index
> lockFactory=org.apache.lucene.store.NativeFSLockFactory@65d307e5 appears both
> in delegate and in cache: cache=[_a0.fdt, _9t_7.liv, _a0.fdx,
> _a0_Lucene50_0.tip, _a0.nvm, _a0_Lucene50_0.doc, _a0_Lucene50_0.tim, _a0.fnm,
> _a0_Lucene50_0.pos, _a0.si],delegate=[pending_segments_93, segments_92,
> write.lock, _9t.fdt, _9t.fdx, _9t.fnm, _9t.nvd, _9t.nvm, _9t.si, _9t_6.liv,
> _9t_Lucene50_0.doc, _9t_Lucene50_0.pos, _9t_Lucene50_0.tim,
> _9t_Lucene50_0.tip, _9u.fdt, _9u.fdx, _9u.fnm, _9u.nvd, _9u.nvm, _9u.si,
> _9u_Lucene50_0.doc, _9u_Lucene50_0.pos, _9u_Lucene50_0.tim,
> _9u_Lucene50_0.tip, _9v.fdt, _9v.fdx, _9v.fnm, _9v.nvd, _9v.nvm, _9v.si,
> _9v_Lucene50_0.doc, _9v_Lucene50_0.pos, _9v_Lucene50_0.tim,
> _9v_Lucene50_0.tip, _9w.fdt, _9w.fdx, _9w.fnm, _9w.nvd, _9w.nvm, _9w.si,
> _9w_Lucene50_0.doc, _9w_Lucene50_0.pos, _9w_Lucene50_0.tim,
> _9w_Lucene50_0.tip, _9x.fdt, _9x.fdx, _9x.fnm, _9x.nvd, _9x.nvm, _9x.si,
> _9x_Lucene50_0.doc, _9x_Lucene50_0.pos, _9x_Lucene50_0.tim,
> _9x_Lucene50_0.tip, _9y.fdt, _9y.fdx, _9y.fnm, _9y.nvd, _9y.nvm, _9y.si,
> _9y_Lucene50_0.doc, _9y_Lucene50_0.pos, _9y_Lucene50_0.tim,
> _9y_Lucene50_0.tip, _9z.fdt, _9z.fdx, _9z.fnm, _9z.nvd, _9z.nvm, _9z.si,
> _9z_Lucene50_0.doc, _9z_Lucene50_0.pos, _9z_Lucene50_0.tim,
> _9z_Lucene50_0.tip, _a0.nvd, _a0_Lucene50_0.pos]
> at
> org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:103)
> at
> org.apache.solr.core.DirectoryFactory.sizeOfDirectory(DirectoryFactory.java:208)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.getIndexSize(CoreAdminHandler.java:1195)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.getCoreStatus(CoreAdminHandler.java:1173)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleStatusAction(CoreAdminHandler.java:736)
> ... 30 more
> {noformat}
> and the second one is on some kind of replication related request:
> {noformat}
> null:java.lang.IllegalStateException: file:
> MMapDirectory@D:\Solr\server\solr\Prod_Core1_shard1_replica3\data\index
> lockFactory=org.apache.lucene.store.NativeFSLockFactory@65d307e5 appears both
> in delegate and in cache: cache=[_g3x.si, _g3x.fdx,
> _g3x.fdt],delegate=[pending_segments_eia, segments_ei9, write.lock, _g3q.fdt,
> _g3q.fdx, _g3q.fnm, _g3q.nvd, _g3q.nvm, _g3q.si, _g3q_6.liv, _g3q_7.liv,
> _g3q_Lucene50_0.doc, _g3q_Lucene50_0.pos, _g3q_Lucene50_0.tim,
> _g3q_Lucene50_0.tip, _g3r.fdt, _g3r.fdx, _g3r.fnm, _g3r.nvd, _g3r.nvm,
> _g3r.si, _g3r_Lucene50_0.doc, _g3r_Lucene50_0.pos, _g3r_Lucene50_0.tim,
> _g3r_Lucene50_0.tip, _g3s.fdt, _g3s.fdx, _g3s.fnm, _g3s.nvd, _g3s.nvm,
> _g3s.si, _g3s_Lucene50_0.doc, _g3s_Lucene50_0.pos, _g3s_Lucene50_0.tim,
> _g3s_Lucene50_0.tip, _g3t.fdt, _g3t.fdx, _g3t.fnm, _g3t.nvd, _g3t.nvm,
> _g3t.si, _g3t_Lucene50_0.doc, _g3t_Lucene50_0.pos, _g3t_Lucene50_0.tim,
> _g3t_Lucene50_0.tip, _g3u.fdt, _g3u.fdx, _g3u.fnm, _g3u.nvd, _g3u.nvm,
> _g3u.si, _g3u_Lucene50_0.doc, _g3u_Lucene50_0.pos, _g3u_Lucene50_0.tim,
> _g3u_Lucene50_0.tip, _g3v.fdt, _g3v.fdx, _g3v.fnm, _g3v.nvd, _g3v.nvm,
> _g3v.si, _g3v_Lucene50_0.doc, _g3v_Lucene50_0.pos, _g3v_Lucene50_0.tim,
> _g3v_Lucene50_0.tip, _g3w.fdt, _g3w.fdx, _g3w.fnm, _g3w.nvd, _g3w.nvm,
> _g3w.si, _g3w_Lucene50_0.doc, _g3w_Lucene50_0.pos, _g3w_Lucene50_0.tim,
> _g3w_Lucene50_0.tip, _g3x.fnm, _g3x.nvd, _g3x.nvm, _g3x.si,
> _g3x_Lucene50_0.doc, _g3x_Lucene50_0.pos, _g3x_Lucene50_0.tim,
> _g3x_Lucene50_0.tip]
> at
> org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:103)
> at
> org.apache.solr.core.DirectoryFactory.sizeOfDirectory(DirectoryFactory.java:208)
> at
> org.apache.solr.handler.ReplicationHandler.getIndexSize(ReplicationHandler.java:705)
> at
> org.apache.solr.handler.ReplicationHandler.getStatistics(ReplicationHandler.java:741)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:165)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:135)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:66)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at
> org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1129)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:497)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Unknown Source)
> {noformat}
> The Environment is a 3 node SOLR Cloud setup running SOLR 5.2.1.
> when looking at the code:
> {code}
> public synchronized String[] listAll() throws IOException {
> final Set<String> files = new HashSet<>();
> for(String f : cache.listAll()) {
> files.add(f);
> }
> for(String f : in.listAll()) {
> if (!files.add(f)) {
> throw new IllegalStateException("file: " + in + " appears both in
> delegate and in cache: " +
> "cache=" +
> Arrays.toString(cache.listAll()) + ",delegate=" +
> Arrays.toString(in.listAll()));
> }
> }
> return files.toArray(new String[files.size()]);
> }
> {code}
> i can see that the exception is thrown because the file exists in both the
> cache and the file system.
> this however doesn't make sense to me, because a file that is in cache will
> always be in the file system as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
