Issue with indexing document - solr 7.5
Hi All,
Am indexing json documents with parent child relationship's and have the
following schema on Solr 7.5
id
Document -
{
"id": "Year-2019",
"year": 2019,
"_childDocuments_": [
{
"id": "5-2019",
"month": 5
}
]
}
and the error am getting is -
'_childDocuments_.id_str' is not an explicit field and doesn't match a
dynamicField."
Not using any dynamic fields in my schema.
Have anyone faced such issue ?
--
*Regards,R K*
Re: Facet on multicore search when one field exists only in one of cores
Hi Shawn,Thank you very much for your quick answer.The use of dynamic field "ignored" works, but it does not seem to be the correct way to solve the problem.We will try to equalize the cores. Em sexta-feira, 14 de junho de 2019 11:18:40 BRT, Shawn Heisey escreveu: On 6/14/2019 7:54 AM, Claudio R wrote: > When I try this request to get facet of fields: fieldA, fieldB and fieldC on > multicore search, I get error: > > http://localhost:8983/solr/core1/select?q=*:*=localhost:8983/solr/core1,localhost:8983/solr/core2=*,[shard]=true=fieldA=fieldB=fieldC > > Error from server at http://localhost:8983/solr/core2: > undefined field: "fieldB" > 400 > > Is there any config / parameter in Solr to avoid this throw exception on > facet of multicore when one field not exists in a core? Distributed queries must have compatible schemas in all the shards referenced. If all fields referenced in the query are not covered by every one of those schemas, you're going to get an error. This is just basic error-checking. The only fix would be to correct the schemas. You could add the missing field, or if you want all invalid fields ignored, simply set up a dynamicField named "*" that connects to a type that's ignored. This should work: I haven't actually tried this so I can't be SURE it will work, but I think it would. A note for devs: In the 8.1.0 _default configset, the "ignored" type does not have docValues="false" ... and I think docValues defaults to true on the StrField class. I think that MIGHT be a problem. Worth an issue? Thanks, Shawn
Re: Facing issue ith MinMaxNormalizer
Hi Christine,
Thanks for the reply.
I had identified the same issue. So, applied the below fix in the custom
jar.
this.max = Float.parseFloat(max.replace("\"", ""));
this.min = Float.parseFloat(min.replace("\"", ""));
And, it worked. Thanks for confirming that this is an issue in solr 6.6.2
Regards
Kamal Kishore.
On Fri, Jun 14, 2019 at 2:17 PM Christine Poerschke (BLOOMBERG/ LONDON) <
cpoersc...@bloomberg.net> wrote:
> Hello Kamal Kishore,
>
> Thanks for including the Solr version alongside your question! What you
> describe sounds like the https://issues.apache.org/jira/browse/SOLR-11163
> issue which is fixed in 7.0.0 but not 6.6.2 release. The fix is a simple
> two line change to MinMaxNormalizer and perhaps one workaround could be for
> you to build a custom MinMaxNormalizer locally for use with your 6.6.2
> setup.
>
> Hope that helps.
>
> Regards,
>
> Christine
>
> From: solr-user@lucene.apache.org At: 06/12/19 12:41:51To:
> solr-user@lucene.apache.org
> Subject: Facing issue ith MinMaxNormalizer
>
> Hi All,
>
> Appreciate if someone can help.
> I am using LTR with MinMaxNormalizer in solr 6.6.2.
>
> Model.json
>
> "class": "org.apache.solr.ltr.model.MultipleAdditiveTreesModel",
> "name": "XGBOOST-BBB-LTR-Model",
> "store":"BBB-Feature-Model",
> "features": [
> {
> "name": "TFIDF",
> "norm":{
> "class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
> "params" : { "min":"0.0", "max":"1.0"}
> }
> },
> {
> "name": "p_ratings_f",
> "norm":{
> "class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
> "params" : { "min":"1.0", "max":"5.0"}
> }
> },
> {
> "name": "p_instore_trans_cnt_f",
> "norm":{
> "class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
> "params" : { "min":"1.0", "max":"209561.0" }
> }
> },
> {
> "name": "p_reviews_f",
> "norm":{
> "class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
> "params" : { "min":"0.0", "max":"58375.0"}
> }
> }
>
> The model got uploaded successfully, but when I reloaded the collection,
> it failed and below error is observed:
>
> Caused by: java.lang.RuntimeException: Error invoking setter setMin on
> class : org.apache.solr.ltr.norm.MinMaxNormalizer
> at
> org.apache.solr.util.SolrPluginUtils.invokeSetters(SolrPluginUtils.java:1084)
> at
> org.apache.solr.ltr.norm.Normalizer.getInstance(Normalizer.java:49)
> at
> org.apache.solr.ltr.store.rest.ManagedModelStore.fromNormalizerMap(ManagedModelStore.java:293)
> at
> org.apache.solr.ltr.store.rest.ManagedModelStore.createNormalizerFromFeatureMap(ManagedModelStore.java:276)
> at
> org.apache.solr.ltr.store.rest.ManagedModelStore.fromLTRScoringModelMap(ManagedModelStore.java:230)
> at
> org.apache.solr.ltr.store.rest.ManagedModelStore.addModelFromMap(ManagedModelStore.java:133)
> at
> org.apache.solr.ltr.store.rest.ManagedModelStore.loadStoredModels(ManagedModelStore.java:126)
> at
> org.apache.solr.ltr.search.LTRQParserPlugin.onManagedResourceInitialized(LTRQParserPlugin.java:133)
> at
> org.apache.solr.rest.ManagedResource.notifyObserversDuringInit(ManagedResource.java:115)
> at
> org.apache.solr.rest.ManagedResource.loadManagedDataAndNotify(ManagedResource.java:91)
> at
> org.apache.solr.rest.RestManager.createManagedResource(RestManager.java:694)
> ... 41 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.solr.util.SolrPluginUtils.invokeSetters(SolrPluginUtils.java:1082)
> ... 51 more
> Caused by: java.lang.NumberFormatException: For input string: ""0.0""
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.solr.ltr.norm.MinMaxNormalizer.setMin(MinMaxNormalizer.java:58)
>
> I tried uploading the model without double quotes in param value for min
> and max, it also got failed with below error.
>
> "java.lang.IllegalArgumentException: argument type mismatch\n\tat
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
>
Re: Facet on multicore search when one field exists only in one of cores
On 6/14/2019 7:54 AM, Claudio R wrote: When I try this request to get facet of fields: fieldA, fieldB and fieldC on multicore search, I get error: http://localhost:8983/solr/core1/select?q=*:*=localhost:8983/solr/core1,localhost:8983/solr/core2=*,[shard]=true=fieldA=fieldB=fieldC Error from server at http://localhost:8983/solr/core2: undefined field: "fieldB" 400 Is there any config / parameter in Solr to avoid this throw exception on facet of multicore when one field not exists in a core? Distributed queries must have compatible schemas in all the shards referenced. If all fields referenced in the query are not covered by every one of those schemas, you're going to get an error. This is just basic error-checking. The only fix would be to correct the schemas. You could add the missing field, or if you want all invalid fields ignored, simply set up a dynamicField named "*" that connects to a type that's ignored. This should work: docValues="false" multiValued="true" class="solr.StrField" /> I haven't actually tried this so I can't be SURE it will work, but I think it would. A note for devs: In the 8.1.0 _default configset, the "ignored" type does not have docValues="false" ... and I think docValues defaults to true on the StrField class. I think that MIGHT be a problem. Worth an issue? Thanks, Shawn
Facet on multicore search when one field exists only in one of cores
Hi, I am using Solr 6.6.0 in mode standalone with 2 cores. The first core has the schema: id fieldA fieldB The second core has the schema: id fieldA fieldC When I try this request to get facet of fields: fieldA, fieldB and fieldC on multicore search, I get error: http://localhost:8983/solr/core1/select?q=*:*=localhost:8983/solr/core1,localhost:8983/solr/core2=*,[shard]=true=fieldA=fieldB=fieldC Error from server at http://localhost:8983/solr/core2: undefined field: "fieldB" 400 Is there any config / parameter in Solr to avoid this throw exception on facet of multicore when one field not exists in a core? If I have this documents on cores: core1: id: "10" fieldA: "productA" fieldB: "value1" core2: id: "23" fieldA: "productA" fieldC: "value2" I wish get a response like this: Facet fieldA productA (2) fieldB value1 (1) fieldC value2 (1)
Solr Sudden I/O spike
Hi, Any help would be appreciated, I am using solr 7.1.0, Suddenly we got a high I/O even with a very low request rate and the core went down. Did anybody experience the same or root cause of this. Below are the log error msg that we got from solr.log 2019-06-06 10:37:14.490 INFO (qtp761960786-8618) [ ] o.a.s.s.HttpSolrCall Unable to write response, client closed connection or we are shutting down org.eclipse.jetty.io.EofException at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:199) at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:420) at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:375) at org.eclipse.jetty.io.SelectChannelEndPoint$3.run(SelectChannelEndPoint.java:107) at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:193) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:283) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceExecuteConsume(ExecuteProduceConsume.java:169) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:145) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:51) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:177) ... 12 more Thanks, Sripradeep P
Re: SOLR JOIN
They are logically different that is why kept in different collections. Collection1: The purpose is to do Global Text Search for all objects through multiple document types Collection3: The purpose is to have search for some other type of objects whose documents are in Millions and takes GBs size. On very specific purpose we need to go to collection3 and do JOIN with collection1 to fetch some data and facets Then go to collection1 and JOIN with collection3 to get facets of collection1 but no data. -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re:Facing issue ith MinMaxNormalizer
Hello Kamal Kishore,
Thanks for including the Solr version alongside your question! What you
describe sounds like the https://issues.apache.org/jira/browse/SOLR-11163 issue
which is fixed in 7.0.0 but not 6.6.2 release. The fix is a simple two line
change to MinMaxNormalizer and perhaps one workaround could be for you to build
a custom MinMaxNormalizer locally for use with your 6.6.2 setup.
Hope that helps.
Regards,
Christine
From: solr-user@lucene.apache.org At: 06/12/19 12:41:51To:
solr-user@lucene.apache.org
Subject: Facing issue ith MinMaxNormalizer
Hi All,
Appreciate if someone can help.
I am using LTR with MinMaxNormalizer in solr 6.6.2.
Model.json
"class": "org.apache.solr.ltr.model.MultipleAdditiveTreesModel",
"name": "XGBOOST-BBB-LTR-Model",
"store":"BBB-Feature-Model",
"features": [
{
"name": "TFIDF",
"norm":{
"class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
"params" : { "min":"0.0", "max":"1.0"}
}
},
{
"name": "p_ratings_f",
"norm":{
"class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
"params" : { "min":"1.0", "max":"5.0"}
}
},
{
"name": "p_instore_trans_cnt_f",
"norm":{
"class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
"params" : { "min":"1.0", "max":"209561.0" }
}
},
{
"name": "p_reviews_f",
"norm":{
"class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
"params" : { "min":"0.0", "max":"58375.0"}
}
}
The model got uploaded successfully, but when I reloaded the collection, it
failed and below error is observed:
Caused by: java.lang.RuntimeException: Error invoking setter setMin on class :
org.apache.solr.ltr.norm.MinMaxNormalizer
at
org.apache.solr.util.SolrPluginUtils.invokeSetters(SolrPluginUtils.java:1084)
at org.apache.solr.ltr.norm.Normalizer.getInstance(Normalizer.java:49)
at
org.apache.solr.ltr.store.rest.ManagedModelStore.fromNormalizerMap(ManagedModelStore.java:293)
at
org.apache.solr.ltr.store.rest.ManagedModelStore.createNormalizerFromFeatureMap(ManagedModelStore.java:276)
at
org.apache.solr.ltr.store.rest.ManagedModelStore.fromLTRScoringModelMap(ManagedModelStore.java:230)
at
org.apache.solr.ltr.store.rest.ManagedModelStore.addModelFromMap(ManagedModelStore.java:133)
at
org.apache.solr.ltr.store.rest.ManagedModelStore.loadStoredModels(ManagedModelStore.java:126)
at
org.apache.solr.ltr.search.LTRQParserPlugin.onManagedResourceInitialized(LTRQParserPlugin.java:133)
at
org.apache.solr.rest.ManagedResource.notifyObserversDuringInit(ManagedResource.java:115)
at
org.apache.solr.rest.ManagedResource.loadManagedDataAndNotify(ManagedResource.java:91)
at
org.apache.solr.rest.RestManager.createManagedResource(RestManager.java:694)
... 41 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.solr.util.SolrPluginUtils.invokeSetters(SolrPluginUtils.java:1082)
... 51 more
Caused by: java.lang.NumberFormatException: For input string: ""0.0""
at
sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
at java.lang.Float.parseFloat(Float.java:451)
at
org.apache.solr.ltr.norm.MinMaxNormalizer.setMin(MinMaxNormalizer.java:58)
I tried uploading the model without double quotes in param value for min and
max, it also got failed with below error.
"java.lang.IllegalArgumentException: argument type mismatch\n\tat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
java.lang.reflect.Method.invoke(Method.java:498)\n\tat
org.apache.solr.util.SolrPluginUtils.invokeSetters(SolrPluginUtils.java:1082)\n\tat
org.apache.solr.ltr.norm.Normalizer.getInstance(Normalizer.java:49)\n\tat
org.apache.solr.ltr.store.rest.ManagedModelStore.fromNormalizerMap(ManagedModelStore.java:293)\n\tat
org.apache.solr.ltr.store.rest.ManagedModelStore.createNormalizerFromFeatureMap(ManagedModelStore.java:276)\n\tat
org.apache.solr.ltr.store.rest.ManagedModelStore.fromLTRScoringModelMap(ManagedModelStore.java:230)\n\tat
Re: SOLR JOIN
Put them in one collection > Am 14.06.2019 um 07:17 schrieb Paresh : > > I am actually able to make this work by doing multiple JOINs - > 1. Do query on Collection1 > 2. JOIN with Collection3 (document type1 field) > 3. JOIN with Collection3 (document type2 field) > 4. Do not fetch any data except faceting information > > But the other problem is, it is very slow. > > Are there any other solution when we have two collections and want data that > satisfy fields from both collections and also get facets from both > collection? > > > > -- > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: SOLR JOIN
I am actually able to make this work by doing multiple JOINs - 1. Do query on Collection1 2. JOIN with Collection3 (document type1 field) 3. JOIN with Collection3 (document type2 field) 4. Do not fetch any data except faceting information But the other problem is, it is very slow. Are there any other solution when we have two collections and want data that satisfy fields from both collections and also get facets from both collection? -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
