[
https://issues.apache.org/jira/browse/SOLR-7954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14710272#comment-14710272
]
Hoss Man commented on SOLR-7954:
--------------------------------
bq. I tested following schema with the same data in field and field2. Both
reproduced the problem.
Ok good -- that means the problem is _not_ actually dependent on docValues or
not -- which was the most confusing and suprising part of your initial bug
report.
bq. Then I tried to find if it is value in cardinality which is causing the
issue. I tried with 100000 to 120000 document and both the field returned
cardinality but after increasing it to around 150000 it caused the exception.
ok, so somewhere arround 150K docs is the sweetspot.
----
Reviewing the code you posted, i noticed a few things:
1) every doc gets a unique value in the field you are computing stats on
2) your query matches all docs
3) because of how your uniqueKey is defined using composite routing keys ("!")
every doc will wind up in the same shard.
the combination of all of these means that ulitmately what's causing problems
is:
* building an HLL data struc using the max possible log2m & regwidth opts
(that's what cardinality=1.0 does)
* adding ~150K unique(ish) hash values to the HLL
* serializing the HLL to bytes (which is what happens in a distributed query to
coordinate)
based on that, i was able to create a unit test that demonstrates the same
underlying ArrayIndexOutOfBoundsException which i'll attach shortly -- still
haven't dug in enough to udnerstand hte cause.
(NOTE: since Solr 5.2.1, we've forked the HLL and imported it directly into the
org.apache.solr.util.hll package, but the basic structure/functionality of the
various classes is still the same)
> Exception while using {!cardinality=1.0}.
> -----------------------------------------
>
> Key: SOLR-7954
> URL: https://issues.apache.org/jira/browse/SOLR-7954
> Project: Solr
> Issue Type: Bug
> Affects Versions: 5.2.1
> Environment: SolrCloud 4 node cluster.
> Ubuntu 12.04
> OS Type 64 bit
> Reporter: Modassar Ather
> Assignee: Hoss Man
>
> Following exception is thrown for the query :
> bq. q=field1:*&stats=true&stats.field={!cardinality=1.0}field.
> The exception is not seen once the cardinality is set to 0.9 or less.
> The field is docValues enabled and indexed=false. The same exception I tried
> to reproduce on non docValues field but could not.
> ERROR - 2015-08-11 12:24:00.222; [core] org.apache.solr.common.SolrException;
> null:java.lang.ArrayIndexOutOfBoundsException: 3
> at
> net.agkn.hll.serialization.BigEndianAscendingWordSerializer.writeWord(BigEndianAscendingWordSerializer.java:152)
> at net.agkn.hll.util.BitVector.getRegisterContents(BitVector.java:247)
> at net.agkn.hll.HLL.toBytes(HLL.java:917)
> at net.agkn.hll.HLL.toBytes(HLL.java:869)
> at
> org.apache.solr.handler.component.AbstractStatsValues.getStatsValues(StatsValuesFactory.java:348)
> at
> org.apache.solr.handler.component.StatsComponent.convertToResponse(StatsComponent.java:151)
> at
> org.apache.solr.handler.component.StatsComponent.process(StatsComponent.java:62)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:255)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:497)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
