[jira] [Commented] (SOLR-8291) NPE calling export handler when useFilterForSortedQuery=true
[
https://issues.apache.org/jira/browse/SOLR-8291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831022#comment-16831022
]
Gus Heck commented on SOLR-8291:
Just hit this recently. [~ronnn]'s explanation seems to jibe with what I
was finding as I prowled the code. He found the spot where the bitsets were
left null which I hadn't found yet, so kudos and thanks to him there :). It
looks like there have been no major changes to the logic of the code in
sortDocSet since LUCENE-2831 in 2011, and the export handler code seems to only
date back to 2014, so the assumptions in ExportQParserPlugin don't seem to
match the previously established behavior of not calling getLeafCollector after
the last matching doc. Given the history, I'd say that the assumption by
ExportQParserPlugin that it can create the array of bit sets in advance based
on the leaf count is the underlying flaw.
{code:java}
public TopDocsCollector getTopDocsCollector(int len,
QueryCommand cmd,
IndexSearcher searcher) throws
IOException {
int leafCount = searcher.getTopReaderContext().leaves().size();
FixedBitSet[] sets = new FixedBitSet[leafCount];
return new ExportCollector(sets);
} {code}
This makes the problem specific to the Export handler code path, and general
search path stuff like sortDocSet() probably should be left alone. A quick
browse into half a dozen other TopDocsCollectors don't show any use of the leaf
count similar to this, but I did not do an exhaustive search.
Two fixes come to mind within the export handler code, the patch supplied by
Ron, tweaked to add the null check to the loop condition, or an else block
break to avoid useless iterations. The second is for ExportQParserPlugin to
populate with empty bitsets, which is unattractive for memory reasons, but
solves the problem at the source.
The first solution (a version of Ron's patch) amounts to ExportWriter being
more tolerant in what it accepts and short circuiting earlier, so I think
that's the preferable path vs creating more objects for GC. I'll be testing
such a patch on the system that exposed this for me soon.
> NPE calling export handler when useFilterForSortedQuery=true
>
>
> Key: SOLR-8291
> URL: https://issues.apache.org/jira/browse/SOLR-8291
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Affects Versions: 5.2.1
>Reporter: Ray
>Priority: Major
> Attachments: SOLR-8291.patch, solr.log
>
>
> *Updated*: The stacktrace below was created when the solrconfig.xml has the
> following element:
> {code}
> true
> {code}
> It was determined that useFilterForSortedQuery is incompatible with the
> /export handler.
> See the comments near the end of the ticket for a potential work around if
> this flag needs to be set.
> Get NPE during calling export handler, here is the stack trace:
> at org.apache.lucene.util.BitSetIterator.(BitSetIterator.java:58)
> at
> org.apache.solr.response.SortingResponseWriter.write(SortingResponseWriter.java:138)
> at
> org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:53)
> at
> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:727)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:459)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
> at
> org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:181)
> at
> org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.event(CatalinaContext.java:285)
> at
> org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.invoke(CatalinaContext.java:261)
> at
> org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:88)
> at
> org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:100)
>
[jira] [Commented] (SOLR-8291) NPE calling export handler when useFilterForSortedQuery=true
[
https://issues.apache.org/jira/browse/SOLR-8291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744483#comment-16744483
]
Rahul Goswami commented on SOLR-8291:
-
Facing the same issue on 7.2.1 while using /export handler... Is Ron's patch
the way to go?
> NPE calling export handler when useFilterForSortedQuery=true
>
>
> Key: SOLR-8291
> URL: https://issues.apache.org/jira/browse/SOLR-8291
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Affects Versions: 5.2.1
>Reporter: Ray
>Priority: Major
> Attachments: SOLR-8291.patch, solr.log
>
>
> *Updated*: The stacktrace below was created when the solrconfig.xml has the
> following element:
> {code}
> true
> {code}
> It was determined that useFilterForSortedQuery is incompatible with the
> /export handler.
> See the comments near the end of the ticket for a potential work around if
> this flag needs to be set.
> Get NPE during calling export handler, here is the stack trace:
> at org.apache.lucene.util.BitSetIterator.(BitSetIterator.java:58)
> at
> org.apache.solr.response.SortingResponseWriter.write(SortingResponseWriter.java:138)
> at
> org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:53)
> at
> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:727)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:459)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
> at
> org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:181)
> at
> org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.event(CatalinaContext.java:285)
> at
> org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.invoke(CatalinaContext.java:261)
> at
> org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:88)
> at
> org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:100)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:159)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:53)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362)
> at
> org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:489)
> at
> org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:452)
> at
> org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2019)
> at java.lang.Thread.run(Thread.java:745)
> It seems there are some FixedBitSet was set to null
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8291) NPE calling export handler when useFilterForSortedQuery=true
[
https://issues.apache.org/jira/browse/SOLR-8291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438876#comment-16438876
]
Ron Ben-Yosef commented on SOLR-8291:
-
[^SOLR-8291.patch]
^The patch implements the first option - adding a null check for the bitsets in
ExportWriter::writeDocs. Didn't want to do the second option for now, just
being careful in case it has undesirable effects elsewhere that I might have
missed (other collector types, other flows, etc.)^
^First time preparing and submitting a git patch, so I hope I'm doing it
right...^
> NPE calling export handler when useFilterForSortedQuery=true
>
>
> Key: SOLR-8291
> URL: https://issues.apache.org/jira/browse/SOLR-8291
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Affects Versions: 5.2.1
>Reporter: Ray
>Priority: Major
> Attachments: SOLR-8291.patch, solr.log
>
>
> *Updated*: The stacktrace below was created when the solrconfig.xml has the
> following element:
> {code}
> true
> {code}
> It was determined that useFilterForSortedQuery is incompatible with the
> /export handler.
> See the comments near the end of the ticket for a potential work around if
> this flag needs to be set.
> Get NPE during calling export handler, here is the stack trace:
> at org.apache.lucene.util.BitSetIterator.(BitSetIterator.java:58)
> at
> org.apache.solr.response.SortingResponseWriter.write(SortingResponseWriter.java:138)
> at
> org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:53)
> at
> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:727)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:459)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
> at
> org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:181)
> at
> org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.event(CatalinaContext.java:285)
> at
> org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.invoke(CatalinaContext.java:261)
> at
> org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:88)
> at
> org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:100)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:159)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:53)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362)
> at
> org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:489)
> at
> org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:452)
> at
> org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2019)
> at java.lang.Thread.run(Thread.java:745)
> It seems there are some FixedBitSet was set to null
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8291) NPE calling export handler when useFilterForSortedQuery=true
[ https://issues.apache.org/jira/browse/SOLR-8291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16378465#comment-16378465 ] Ron Ben-Yosef commented on SOLR-8291: - Hi, I've run into the same issue, or at least what I think is the same issue (I'm on 7.2.1, and the call stack leading up to the exception looks somewhat different, but reading everything written here I'm fairly sure it's the same issue. If it's not - let me know and I can open a separate bug for it). The scenario is similar - querying the /export handler with the "useFilterForSortedQuery" configuration option enabled. Here's the call stack leading up to the exception: o.a.s.s.HttpSolrCall null:java.lang.NullPointerException at org.apache.lucene.util.BitSetIterator.(BitSetIterator.java:61) at org.apache.solr.handler.ExportWriter.writeDocs(ExportWriter.java:243) at org.apache.solr.handler.ExportWriter.lambda$null$1(ExportWriter.java:222) at org.apache.solr.response.JSONWriter.writeIterator(JSONResponseWriter.java:523) at org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:180) at org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter.java:559) at org.apache.solr.handler.ExportWriter.lambda$null$2(ExportWriter.java:222) at org.apache.solr.response.JSONWriter.writeMap(JSONResponseWriter.java:547) at org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:198) at org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter.java:559) at org.apache.solr.handler.ExportWriter.lambda$write$3(ExportWriter.java:220) at org.apache.solr.response.JSONWriter.writeMap(JSONResponseWriter.java:547) at org.apache.solr.handler.ExportWriter.write(ExportWriter.java:218) at org.apache.solr.core.SolrCore$3.write(SolrCore.java:2627) at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:49) at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:788) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:525) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:534) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) Debugging the issue, it seems what's happening is that when there's more than one leaf (=segment?) then the per-leaf bitsets are being created only upto the highest leaf with matching documents, and if that's not the last leaf then you're going to have null bitsets for all the leaves higher than the highest one containing a matching document. This happens in SolrIndexSearcher.java in the functi
