[jira] [Commented] (SOLR-7954) Exception while using {!cardinality=1.0}.
[
https://issues.apache.org/jira/browse/SOLR-7954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708909#comment-14708909
]
Modassar Ather commented on SOLR-7954:
--
I tested following schema with the same data in field and field2. Both
reproduced the problem. Then I tried to find if it is value in cardinality
which is causing the issue. I tried with 10 to 12 document and both the
field returned cardinality but after increasing it to around 15 it caused
the exception.
{noformat}
?xml version=1.0 encoding=UTF-8 ?
schema name=collection version=1.5
types
fieldType name=string class=solr.StrField sortMissingLast=true
stored=false omitNorms=true/
fieldType name=string_dv class=solr.StrField sortMissingLast=true
stored=false indexed=false docValues=true/
fieldType name=long class=solr.TrieLongField precisionStep=0
omitNorms=true positionIncrementGap=0 stored=false/
/types
fields
field name=field type=string_dv multiValued=true /
field name=field1 type=string stored=true /
field name=field2 type=string multiValued=true /
field name=versiontype=long stored=true /
field name=colidtype=string stored=true /
/fields
uniqueKeycolid/uniqueKey
/schema
{noformat}
Following is the method to index data.
{noformat}
public void index() throws SolrServerException, IOException {
CloudSolrClient s = new CloudSolrClient(ZKHOST:ZKPORT);
int count = 0;
s.setDefaultCollection(COLECTION);
ListSolrInputDocument documents = new ArrayList();
for (int i = 1; i = 100; i++) {
SolrInputDocument doc = new SolrInputDocument();
doc.addField(field1, i);
doc.addField(colid, val!+i+!-+ref+i);
doc.addField(field, DATA+(12345+i));
doc.addField(field2, DATA+(12345+i));
documents.add(doc);
if((documents.size() % 1) == 0){
count = count + 1;
s.add(documents);
System.out.println(System.currentTimeMillis() + - Indexed
document # + NumberFormat.getInstance().format(count));
documents = new ArrayList();
}
}
System.out.println(Comitting.);
s.commit(true, true);
System.out.println(Optimizing.);
s.optimize(true, true, 1);
s.close();
System.out.println(Done.);
}
{noformat}
Exception while using {!cardinality=1.0}.
-
Key: SOLR-7954
URL: https://issues.apache.org/jira/browse/SOLR-7954
Project: Solr
Issue Type: Bug
Affects Versions: 5.2.1
Environment: SolrCloud 4 node cluster.
Ubuntu 12.04
OS Type 64 bit
Reporter: Modassar Ather
Assignee: Hoss Man
Following exception is thrown for the query :
bq. q=field1:*stats=truestats.field={!cardinality=1.0}field.
The exception is not seen once the cardinality is set to 0.9 or less.
The field is docValues enabled and indexed=false. The same exception I tried
to reproduce on non docValues field but could not.
ERROR - 2015-08-11 12:24:00.222; [core] org.apache.solr.common.SolrException;
null:java.lang.ArrayIndexOutOfBoundsException: 3
at
net.agkn.hll.serialization.BigEndianAscendingWordSerializer.writeWord(BigEndianAscendingWordSerializer.java:152)
at net.agkn.hll.util.BitVector.getRegisterContents(BitVector.java:247)
at net.agkn.hll.HLL.toBytes(HLL.java:917)
at net.agkn.hll.HLL.toBytes(HLL.java:869)
at
org.apache.solr.handler.component.AbstractStatsValues.getStatsValues(StatsValuesFactory.java:348)
at
org.apache.solr.handler.component.StatsComponent.convertToResponse(StatsComponent.java:151)
at
org.apache.solr.handler.component.StatsComponent.process(StatsComponent.java:62)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:255)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
[jira] [Commented] (SOLR-7954) Exception while using {!cardinality=1.0}.
[
https://issues.apache.org/jira/browse/SOLR-7954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710272#comment-14710272
]
Hoss Man commented on SOLR-7954:
bq. I tested following schema with the same data in field and field2. Both
reproduced the problem.
Ok good -- that means the problem is _not_ actually dependent on docValues or
not -- which was the most confusing and suprising part of your initial bug
report.
bq. Then I tried to find if it is value in cardinality which is causing the
issue. I tried with 10 to 12 document and both the field returned
cardinality but after increasing it to around 15 it caused the exception.
ok, so somewhere arround 150K docs is the sweetspot.
Reviewing the code you posted, i noticed a few things:
1) every doc gets a unique value in the field you are computing stats on
2) your query matches all docs
3) because of how your uniqueKey is defined using composite routing keys (!)
every doc will wind up in the same shard.
the combination of all of these means that ulitmately what's causing problems
is:
* building an HLL data struc using the max possible log2m regwidth opts
(that's what cardinality=1.0 does)
* adding ~150K unique(ish) hash values to the HLL
* serializing the HLL to bytes (which is what happens in a distributed query to
coordinate)
based on that, i was able to create a unit test that demonstrates the same
underlying ArrayIndexOutOfBoundsException which i'll attach shortly -- still
haven't dug in enough to udnerstand hte cause.
(NOTE: since Solr 5.2.1, we've forked the HLL and imported it directly into the
org.apache.solr.util.hll package, but the basic structure/functionality of the
various classes is still the same)
Exception while using {!cardinality=1.0}.
-
Key: SOLR-7954
URL: https://issues.apache.org/jira/browse/SOLR-7954
Project: Solr
Issue Type: Bug
Affects Versions: 5.2.1
Environment: SolrCloud 4 node cluster.
Ubuntu 12.04
OS Type 64 bit
Reporter: Modassar Ather
Assignee: Hoss Man
Following exception is thrown for the query :
bq. q=field1:*stats=truestats.field={!cardinality=1.0}field.
The exception is not seen once the cardinality is set to 0.9 or less.
The field is docValues enabled and indexed=false. The same exception I tried
to reproduce on non docValues field but could not.
ERROR - 2015-08-11 12:24:00.222; [core] org.apache.solr.common.SolrException;
null:java.lang.ArrayIndexOutOfBoundsException: 3
at
net.agkn.hll.serialization.BigEndianAscendingWordSerializer.writeWord(BigEndianAscendingWordSerializer.java:152)
at net.agkn.hll.util.BitVector.getRegisterContents(BitVector.java:247)
at net.agkn.hll.HLL.toBytes(HLL.java:917)
at net.agkn.hll.HLL.toBytes(HLL.java:869)
at
org.apache.solr.handler.component.AbstractStatsValues.getStatsValues(StatsValuesFactory.java:348)
at
org.apache.solr.handler.component.StatsComponent.convertToResponse(StatsComponent.java:151)
at
org.apache.solr.handler.component.StatsComponent.process(StatsComponent.java:62)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:255)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at
[jira] [Commented] (SOLR-7954) Exception while using {!cardinality=1.0}.
[
https://issues.apache.org/jira/browse/SOLR-7954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706525#comment-14706525
]
Modassar Ather commented on SOLR-7954:
--
Following test method can be used to add data using which the exception can be
reproduced. Please do the necessary changes.
Change ZKHOST:ZKPORT to point to zkhost available and COLECTION to the
available collection.
{noformat}
public void index() throws SolrServerException, IOException {
CloudSolrClient s = new CloudSolrClient(ZKHOST:ZKPORT);
int count = 0;
s.setDefaultCollection(COLECTION);
ListSolrInputDocument documents = new ArrayList();
for (int i = 1; i = 100; i++) {
SolrInputDocument doc = new SolrInputDocument();
doc.addField(field1, i);
doc.addField(colid, val!+i+!-+ref+i);
doc.addField(field, DATA+(12345+i));
documents.add(doc);
if((documents.size() % 1) == 0){
count = count + 1;
s.add(documents);
System.out.println(System.currentTimeMillis() + - Indexed
document # + NumberFormat.getInstance().format(count));
documents = new ArrayList();
}
}
System.out.println(Comitting.);
s.commit(true, true);
System.out.println(Optimizing.);
s.close();
System.out.println(Done.);
}
{noformat}
Exception while using {!cardinality=1.0}.
-
Key: SOLR-7954
URL: https://issues.apache.org/jira/browse/SOLR-7954
Project: Solr
Issue Type: Bug
Affects Versions: 5.2.1
Environment: SolrCloud 4 node cluster.
Ubuntu 12.04
OS Type 64 bit
Reporter: Modassar Ather
Following exception is thrown for the query :
bq. q=field:querystats=truestats.field={!cardinality=1.0}field.
The exception is not seen once the cardinality is set to 0.9 or less.
The field is docValues enabled and indexed=false. The same exception I tried
to reproduce on non docValues field but could not.
ERROR - 2015-08-11 12:24:00.222; [core] org.apache.solr.common.SolrException;
null:java.lang.ArrayIndexOutOfBoundsException: 3
at
net.agkn.hll.serialization.BigEndianAscendingWordSerializer.writeWord(BigEndianAscendingWordSerializer.java:152)
at net.agkn.hll.util.BitVector.getRegisterContents(BitVector.java:247)
at net.agkn.hll.HLL.toBytes(HLL.java:917)
at net.agkn.hll.HLL.toBytes(HLL.java:869)
at
org.apache.solr.handler.component.AbstractStatsValues.getStatsValues(StatsValuesFactory.java:348)
at
org.apache.solr.handler.component.StatsComponent.convertToResponse(StatsComponent.java:151)
at
org.apache.solr.handler.component.StatsComponent.process(StatsComponent.java:62)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:255)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at
[jira] [Commented] (SOLR-7954) Exception while using {!cardinality=1.0}.
[
https://issues.apache.org/jira/browse/SOLR-7954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706280#comment-14706280
]
Modassar Ather commented on SOLR-7954:
--
The schema used is as follows.
?xml version=1.0 encoding=UTF-8 ?
schema name=collection version=1.5
types
fieldType name=string class=solr.StrField sortMissingLast=true
stored=false omitNorms=true/
fieldType name=string_dv class=solr.StrField
sortMissingLast=true stored=false indexed=false docValues=true/
fieldType name=long class=solr.TrieLongField precisionStep=0
omitNorms=true positionIncrementGap=0 stored=false/
/types
fields
field name=field1 type=string stored=true /
field name=field type=string_dv multiValued=true /
field name=_version_ type=long stored=true /
field name=colid type=string stored=true /
/fields
uniqueKeycolid/uniqueKey
/schema
Exception while using {!cardinality=1.0}.
-
Key: SOLR-7954
URL: https://issues.apache.org/jira/browse/SOLR-7954
Project: Solr
Issue Type: Bug
Affects Versions: 5.2.1
Reporter: Modassar Ather
Following exception is thrown for the query :
bq. q=field:querystats=truestats.field={!cardinality=1.0}field.
The exception is not seen once the cardinality is set to 0.9 or less.
The field is docValues enabled and indexed=false. The same exception I tried
to reproduce on non docValues field but could not.
ERROR - 2015-08-11 12:24:00.222; [core] org.apache.solr.common.SolrException;
null:java.lang.ArrayIndexOutOfBoundsException: 3
at
net.agkn.hll.serialization.BigEndianAscendingWordSerializer.writeWord(BigEndianAscendingWordSerializer.java:152)
at net.agkn.hll.util.BitVector.getRegisterContents(BitVector.java:247)
at net.agkn.hll.HLL.toBytes(HLL.java:917)
at net.agkn.hll.HLL.toBytes(HLL.java:869)
at
org.apache.solr.handler.component.AbstractStatsValues.getStatsValues(StatsValuesFactory.java:348)
at
org.apache.solr.handler.component.StatsComponent.convertToResponse(StatsComponent.java:151)
at
org.apache.solr.handler.component.StatsComponent.process(StatsComponent.java:62)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:255)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To
