Re: Elevation with distributed search causes NPE

[Erick Erickson]

I "enabled patch review”… Thanks!

> On Aug 17, 2020, at 9:43 AM, smk  wrote:
> 
> H Erick,
> 
> I am a colleague of Marc and have attached a patch to his ticket
> https://issues.apache.org/jira/browse/SOLR-14662
> Can you set the ticket to "Patch Available"? For me there is no such option.
> 
> Thank you very much, Thomas
> 
> 
> 
> 
> 
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Elevation with distributed search causes NPE

[smk]

H Erick,

I am a colleague of Marc and have attached a patch to his ticket
https://issues.apache.org/jira/browse/SOLR-14662
Can you set the ticket to "Patch Available"? For me there is no such option.

Thank you very much, Thomas





--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Elevation with distributed search causes NPE

[Erick Erickson]

https://cwiki.apache.org/confluence/display/solr/HowToContribute#HowToContribute-JIRAtips(ourissue/bugtracker)

> On Jul 22, 2020, at 5:30 AM, Marc Linden  
> wrote:
> 
> Hi Erick,
> 
> could you tell me or point me to the documentation on how the process around 
> raised JIRA issues usually works. I could not find anything that helped me.
> 
> Thanks in advance.
> 
> Best,
> Marc
> 
> -Ursprüngliche Nachricht-
> Von: Marc Linden 
> Gesendet: Freitag, 17. Juli 2020 11:35
> An: solr-user@lucene.apache.org
> Betreff: AW: Elevation with distributed search causes NPE
> 
> There it is: https://issues.apache.org/jira/browse/SOLR-14662. I'd love to 
> dig deeper into that and provide a patch for it, but I'm fully occupied with 
> our own project.
> 
> Thanks and best regards,
> Marc
> 
> -Ursprüngliche Nachricht-
> Von: Erick Erickson 
> Gesendet: Donnerstag, 16. Juli 2020 15:06
> An: solr-user@lucene.apache.org
> Betreff: Re: Elevation with distributed search causes NPE
> 
> Can you raise a JIRA? If you’re ambitious, you can add a patch too ;)
> 
>> On Jul 16, 2020, at 2:52 AM, Marc Linden  
>> wrote:
>> 
>> Thanks Erick for your fast response.
>> 
>> I've checked out adding the sort param and yes that vanished the problem but 
>> it also disables elevation if I'm not mistaken. So after adding 
>> forceElevation=true to the query then a ClassCastException was thrown:
>> 
>> http://localhost:9983/solr/core1/select?q=elevatedTerm
>> ors=false=text_en=edismax=lang:en=localhost:9983/
>> solr/core1,localhost:9983/solr/core2=[elevated],[shard],area,id
>> s=10=0=area%20asc=true
>> 
>> java.lang.ClassCastException: java.lang.Integer cannot be cast to
>> java.lang.String  at
>> org.apache.solr.schema.FieldType.unmarshalStringSortValue(FieldType.ja
>> va:1229)  at
>> org.apache.solr.schema.StrField.unmarshalSortValue(StrField.java:122)
>> at
>> org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(Q
>> ueryComponent.java:1092)  at
>> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryCompone
>> nt.java:917)  at
>> org.apache.solr.handler.component.QueryComponent.handleRegularResponse
>> s(QueryComponent.java:613)  at
>> org.apache.solr.handler.component.QueryComponent.handleResponses(Query
>> Component.java:592)  at
>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(Sear
>> chHandler.java:431)  at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandle
>> rBase.java:199)  at
>> org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)
>> at
>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:780)
>> ...
>> 
>> Best regards,
>> Marc
>> 
>> -Ursprüngliche Nachricht-
>> Von: Erick Erickson 
>> Gesendet: Mittwoch, 15. Juli 2020 14:32
>> An: solr-user@lucene.apache.org
>> Betreff: Re: Elevation with distributed search causes NPE
>> 
>> Hmmm, looking at the code that line looks like this:
>> 
>> sortSpec.getSort().getSort();
>> 
>> I’m curious what happens if you specify a sort on the query? If that makes 
>> the problem go away, it’s a smoking gun.
>> 
>> Whether or not adding sorting makes the problem go away, this looks like 
>> something that’s a legitimate JIRA, please go ahead and raise one.
>> 
>> Best,
>> Erick
>> 
>>> On Jul 15, 2020, at 4:34 AM, Marc Linden  
>>> wrote:
>>> 
>>> Hi all,
>>> 
>>> I'm facing the problem that Solr is throwing a NullPointerException when 
>>> performing a distributed search with multiple shards having elevation 
>>> configured where one or more shards do have elevated results but others do 
>>> not.
>>> 
>>> We are using Solr 8.2 and have the QueryElevationComponent configured with 
>>> "last-components" of the default search handler "/select". But the problem 
>>> also occurs when using the explicit "/elevate" search handler.
>>>  ...
>>> 
>>> elevator
>>> 
>>> 
>>> ...
>>> >>> 
>>>  >> name="queryFieldType">string
>>> elevate.xml
>>> 
>>> 
>>> ### Steps to reproduce:
>>> 
>>> (1) Add entries to the elevate.xml of each core to elevate a specific 
>>> document for the text "searchTerm":
>>> 
>>> core1:
>>> 
>>> ...
>>>   
>>> core2:
>>> 
>>> ...
>>>

AW: Elevation with distributed search causes NPE

[Marc Linden]

Hi Erick,

could you tell me or point me to the documentation on how the process around 
raised JIRA issues usually works. I could not find anything that helped me.

Thanks in advance.

Best,
Marc

-Ursprüngliche Nachricht-
Von: Marc Linden 
Gesendet: Freitag, 17. Juli 2020 11:35
An: solr-user@lucene.apache.org
Betreff: AW: Elevation with distributed search causes NPE

There it is: https://issues.apache.org/jira/browse/SOLR-14662. I'd love to dig 
deeper into that and provide a patch for it, but I'm fully occupied with our 
own project.

Thanks and best regards,
Marc

-Ursprüngliche Nachricht-
Von: Erick Erickson 
Gesendet: Donnerstag, 16. Juli 2020 15:06
An: solr-user@lucene.apache.org
Betreff: Re: Elevation with distributed search causes NPE

Can you raise a JIRA? If you’re ambitious, you can add a patch too ;)

> On Jul 16, 2020, at 2:52 AM, Marc Linden  
> wrote:
>
> Thanks Erick for your fast response.
>
> I've checked out adding the sort param and yes that vanished the problem but 
> it also disables elevation if I'm not mistaken. So after adding 
> forceElevation=true to the query then a ClassCastException was thrown:
>
> http://localhost:9983/solr/core1/select?q=elevatedTerm
> ors=false=text_en=edismax=lang:en=localhost:9983/
> solr/core1,localhost:9983/solr/core2=[elevated],[shard],area,id
> s=10=0=area%20asc=true
>
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.String  at
> org.apache.solr.schema.FieldType.unmarshalStringSortValue(FieldType.ja
> va:1229)  at
> org.apache.solr.schema.StrField.unmarshalSortValue(StrField.java:122)
>  at
> org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(Q
> ueryComponent.java:1092)  at
> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryCompone
> nt.java:917)  at
> org.apache.solr.handler.component.QueryComponent.handleRegularResponse
> s(QueryComponent.java:613)  at
> org.apache.solr.handler.component.QueryComponent.handleResponses(Query
> Component.java:592)  at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(Sear
> chHandler.java:431)  at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandle
> rBase.java:199)  at
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)
>  at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:780)
>  ...
>
> Best regards,
> Marc
>
> -Ursprüngliche Nachricht-
> Von: Erick Erickson 
> Gesendet: Mittwoch, 15. Juli 2020 14:32
> An: solr-user@lucene.apache.org
> Betreff: Re: Elevation with distributed search causes NPE
>
> Hmmm, looking at the code that line looks like this:
>
> sortSpec.getSort().getSort();
>
> I’m curious what happens if you specify a sort on the query? If that makes 
> the problem go away, it’s a smoking gun.
>
> Whether or not adding sorting makes the problem go away, this looks like 
> something that’s a legitimate JIRA, please go ahead and raise one.
>
> Best,
> Erick
>
>> On Jul 15, 2020, at 4:34 AM, Marc Linden  
>> wrote:
>>
>> Hi all,
>>
>> I'm facing the problem that Solr is throwing a NullPointerException when 
>> performing a distributed search with multiple shards having elevation 
>> configured where one or more shards do have elevated results but others do 
>> not.
>>
>> We are using Solr 8.2 and have the QueryElevationComponent configured with 
>> "last-components" of the default search handler "/select". But the problem 
>> also occurs when using the explicit "/elevate" search handler.
>>  ...
>> 
>> elevator
>> 
>> 
>> ...
>> >>
>>  > name="queryFieldType">string
>> elevate.xml
>> 
>>
>> ### Steps to reproduce:
>>
>> (1) Add entries to the elevate.xml of each core to elevate a specific 
>> document for the text "searchTerm":
>>
>>  core1:
>>  
>> ...
>>   
>>  core2:
>>  
>> ...
>>   
>>
>> (2) Execute query (we use port 9983):
>> http://localhost:9983/solr/web/select?q=elevatedTerm
>> r
>> s=false=text_en=edismax=lang:en=localhost:9983/s
>> o
>> lr/core1,localhost:9983/solr/core2=[elevated],[shard],area,id
>> =
>> 10=0
>>
>> Now as both shards have elevated documents for the requested "searchTerm" 
>> the search results are as expected:
>>
>> response: {
>> numFound: 5192,
>> start: 0,
>> maxScore: 1.9032197,
>> docs: [{
>> area: "press",
>> id: "core1docId1",
>> [elevated]: true,
>>

AW: Elevation with distributed search causes NPE

[Marc Linden]

There it is: https://issues.apache.org/jira/browse/SOLR-14662. I'd love to dig 
deeper into that and provide a patch for it, but I'm fully occupied with our 
own project.

Thanks and best regards,
Marc

-Ursprüngliche Nachricht-
Von: Erick Erickson 
Gesendet: Donnerstag, 16. Juli 2020 15:06
An: solr-user@lucene.apache.org
Betreff: Re: Elevation with distributed search causes NPE

Can you raise a JIRA? If you’re ambitious, you can add a patch too ;)

> On Jul 16, 2020, at 2:52 AM, Marc Linden  
> wrote:
>
> Thanks Erick for your fast response.
>
> I've checked out adding the sort param and yes that vanished the problem but 
> it also disables elevation if I'm not mistaken. So after adding 
> forceElevation=true to the query then a ClassCastException was thrown:
>
> http://localhost:9983/solr/core1/select?q=elevatedTerm
> ors=false=text_en=edismax=lang:en=localhost:9983/
> solr/core1,localhost:9983/solr/core2=[elevated],[shard],area,id
> s=10=0=area%20asc=true
>
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.String  at
> org.apache.solr.schema.FieldType.unmarshalStringSortValue(FieldType.ja
> va:1229)  at
> org.apache.solr.schema.StrField.unmarshalSortValue(StrField.java:122)
>  at
> org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(Q
> ueryComponent.java:1092)  at
> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryCompone
> nt.java:917)  at
> org.apache.solr.handler.component.QueryComponent.handleRegularResponse
> s(QueryComponent.java:613)  at
> org.apache.solr.handler.component.QueryComponent.handleResponses(Query
> Component.java:592)  at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(Sear
> chHandler.java:431)  at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandle
> rBase.java:199)  at
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)
>  at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:780)
>  ...
>
> Best regards,
> Marc
>
> -Ursprüngliche Nachricht-
> Von: Erick Erickson 
> Gesendet: Mittwoch, 15. Juli 2020 14:32
> An: solr-user@lucene.apache.org
> Betreff: Re: Elevation with distributed search causes NPE
>
> Hmmm, looking at the code that line looks like this:
>
> sortSpec.getSort().getSort();
>
> I’m curious what happens if you specify a sort on the query? If that makes 
> the problem go away, it’s a smoking gun.
>
> Whether or not adding sorting makes the problem go away, this looks like 
> something that’s a legitimate JIRA, please go ahead and raise one.
>
> Best,
> Erick
>
>> On Jul 15, 2020, at 4:34 AM, Marc Linden  
>> wrote:
>>
>> Hi all,
>>
>> I'm facing the problem that Solr is throwing a NullPointerException when 
>> performing a distributed search with multiple shards having elevation 
>> configured where one or more shards do have elevated results but others do 
>> not.
>>
>> We are using Solr 8.2 and have the QueryElevationComponent configured with 
>> "last-components" of the default search handler "/select". But the problem 
>> also occurs when using the explicit "/elevate" search handler.
>>  ...
>> 
>> elevator
>> 
>> 
>> ...
>> >>
>>  > name="queryFieldType">string
>> elevate.xml
>> 
>>
>> ### Steps to reproduce:
>>
>> (1) Add entries to the elevate.xml of each core to elevate a specific 
>> document for the text "searchTerm":
>>
>>  core1:
>>  
>> ...
>>   
>>  core2:
>>  
>> ...
>>   
>>
>> (2) Execute query (we use port 9983):
>> http://localhost:9983/solr/web/select?q=elevatedTerm
>> r
>> s=false=text_en=edismax=lang:en=localhost:9983/s
>> o
>> lr/core1,localhost:9983/solr/core2=[elevated],[shard],area,id
>> =
>> 10=0
>>
>> Now as both shards have elevated documents for the requested "searchTerm" 
>> the search results are as expected:
>>
>> response: {
>> numFound: 5192,
>> start: 0,
>> maxScore: 1.9032197,
>> docs: [{
>> area: "press",
>> id: "core1docId1",
>> [elevated]: true,
>> [shard]: "localhost:9983/solr/core1"
>> }, {
>> area: "products",
>> id: "core2docId1",
>> [elevated]: true,
>> [shard]: "localhost:9983/solr/core2"
>> }, {
>> area: "press",
>> id: "core1docId2",
>> [elevated]: false,
>> [shard]: "localhost:9983/solr/core1"
>> },
>> ...
>>
>> (3) Rem

Re: Elevation with distributed search causes NPE

[Erick Erickson]

Can you raise a JIRA? If you’re ambitious, you can add a patch too ;)

> On Jul 16, 2020, at 2:52 AM, Marc Linden  
> wrote:
> 
> Thanks Erick for your fast response.
> 
> I've checked out adding the sort param and yes that vanished the problem but 
> it also disables elevation if I'm not mistaken. So after adding 
> forceElevation=true to the query then a ClassCastException was thrown:
> 
> http://localhost:9983/solr/core1/select?q=elevatedTerm=false=text_en=edismax=lang:en=localhost:9983/solr/core1,localhost:9983/solr/core2=[elevated],[shard],area,id=10=0=area%20asc=true
> 
> java.lang.ClassCastException: java.lang.Integer cannot be cast to 
> java.lang.String
>  at 
> org.apache.solr.schema.FieldType.unmarshalStringSortValue(FieldType.java:1229)
>  at org.apache.solr.schema.StrField.unmarshalSortValue(StrField.java:122)
>  at 
> org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1092)
>  at 
> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:917)
>  at 
> org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:613)
>  at 
> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:592)
>  at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:431)
>  at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
>  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)
>  at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:780)
>  ...
> 
> Best regards,
> Marc
> 
> -Ursprüngliche Nachricht-
> Von: Erick Erickson 
> Gesendet: Mittwoch, 15. Juli 2020 14:32
> An: solr-user@lucene.apache.org
> Betreff: Re: Elevation with distributed search causes NPE
> 
> Hmmm, looking at the code that line looks like this:
> 
> sortSpec.getSort().getSort();
> 
> I’m curious what happens if you specify a sort on the query? If that makes 
> the problem go away, it’s a smoking gun.
> 
> Whether or not adding sorting makes the problem go away, this looks like 
> something that’s a legitimate JIRA, please go ahead and raise one.
> 
> Best,
> Erick
> 
>> On Jul 15, 2020, at 4:34 AM, Marc Linden  
>> wrote:
>> 
>> Hi all,
>> 
>> I'm facing the problem that Solr is throwing a NullPointerException when 
>> performing a distributed search with multiple shards having elevation 
>> configured where one or more shards do have elevated results but others do 
>> not.
>> 
>> We are using Solr 8.2 and have the QueryElevationComponent configured with 
>> "last-components" of the default search handler "/select". But the problem 
>> also occurs when using the explicit "/elevate" search handler.
>>  ...
>> 
>> elevator
>> 
>> 
>> ...
>> >> 
>>  > name="queryFieldType">string
>> elevate.xml
>> 
>> 
>> ### Steps to reproduce:
>> 
>> (1) Add entries to the elevate.xml of each core to elevate a specific 
>> document for the text "searchTerm":
>> 
>>  core1:
>>  
>> ...
>> 
>>  
>>  core2:
>>  
>> ...
>> 
>>  
>> 
>> (2) Execute query (we use port 9983):
>> http://localhost:9983/solr/web/select?q=elevatedTerm
>> s=false=text_en=edismax=lang:en=localhost:9983/so
>> lr/core1,localhost:9983/solr/core2=[elevated],[shard],area,id=
>> 10=0
>> 
>> Now as both shards have elevated documents for the requested "searchTerm" 
>> the search results are as expected:
>> 
>> response: {
>> numFound: 5192,
>> start: 0,
>> maxScore: 1.9032197,
>> docs: [{
>> area: "press",
>> id: "core1docId1",
>> [elevated]: true,
>> [shard]: "localhost:9983/solr/core1"
>> }, {
>> area: "products",
>> id: "core2docId1",
>> [elevated]: true,
>> [shard]: "localhost:9983/solr/core2"
>> }, {
>> area: "press",
>> id: "core1docId2",
>> [elevated]: false,
>> [shard]: "localhost:9983/solr/core1"
>> },
>> ...
>> 
>> (3) Remove the elevation entry for that "searchTerm" from one of the
>> cores, e.g. via comment
>> 
>>  core2:
>>  
>> ...
>> 
>>  
>> 
>> 
>> (4) Reload the modified core:
>> http://localhost:9983/solr/admin/cores?action=RELOAD=core2
>> 
>> (5) Request same query again and you get the NPE:
>> 
>> error: {

AW: Elevation with distributed search causes NPE

[Marc Linden]

Thanks Erick for your fast response.

I've checked out adding the sort param and yes that vanished the problem but it 
also disables elevation if I'm not mistaken. So after adding 
forceElevation=true to the query then a ClassCastException was thrown:

http://localhost:9983/solr/core1/select?q=elevatedTerm=false=text_en=edismax=lang:en=localhost:9983/solr/core1,localhost:9983/solr/core2=[elevated],[shard],area,id=10=0=area%20asc=true

java.lang.ClassCastException: java.lang.Integer cannot be cast to 
java.lang.String
  at 
org.apache.solr.schema.FieldType.unmarshalStringSortValue(FieldType.java:1229)
  at org.apache.solr.schema.StrField.unmarshalSortValue(StrField.java:122)
  at 
org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1092)
  at 
org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:917)
  at 
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:613)
  at 
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:592)
  at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:431)
  at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)
  at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:780)
  ...

Best regards,
Marc

-Ursprüngliche Nachricht-
Von: Erick Erickson 
Gesendet: Mittwoch, 15. Juli 2020 14:32
An: solr-user@lucene.apache.org
Betreff: Re: Elevation with distributed search causes NPE

Hmmm, looking at the code that line looks like this:

sortSpec.getSort().getSort();

I’m curious what happens if you specify a sort on the query? If that makes the 
problem go away, it’s a smoking gun.

Whether or not adding sorting makes the problem go away, this looks like 
something that’s a legitimate JIRA, please go ahead and raise one.

Best,
Erick

> On Jul 15, 2020, at 4:34 AM, Marc Linden  
> wrote:
>
> Hi all,
>
> I'm facing the problem that Solr is throwing a NullPointerException when 
> performing a distributed search with multiple shards having elevation 
> configured where one or more shards do have elevated results but others do 
> not.
>
> We are using Solr 8.2 and have the QueryElevationComponent configured with 
> "last-components" of the default search handler "/select". But the problem 
> also occurs when using the explicit "/elevate" search handler.
>   ...
> 
>  elevator
> 
>  
>  ...
>   >
>   name="queryFieldType">string
> elevate.xml
>  
>
> ### Steps to reproduce:
>
> (1) Add entries to the elevate.xml of each core to elevate a specific 
> document for the text "searchTerm":
>
>   core1:
>   
> ...
> 
>   
>   core2:
>   
> ...
> 
>   
>
> (2) Execute query (we use port 9983):
> http://localhost:9983/solr/web/select?q=elevatedTerm
> s=false=text_en=edismax=lang:en=localhost:9983/so
> lr/core1,localhost:9983/solr/core2=[elevated],[shard],area,id=
> 10=0
>
> Now as both shards have elevated documents for the requested "searchTerm" the 
> search results are as expected:
>
> response: {
> numFound: 5192,
> start: 0,
> maxScore: 1.9032197,
> docs: [{
> area: "press",
> id: "core1docId1",
> [elevated]: true,
> [shard]: "localhost:9983/solr/core1"
> }, {
> area: "products",
> id: "core2docId1",
> [elevated]: true,
> [shard]: "localhost:9983/solr/core2"
> }, {
> area: "press",
> id: "core1docId2",
> [elevated]: false,
> [shard]: "localhost:9983/solr/core1"
> },
> ...
>
> (3) Remove the elevation entry for that "searchTerm" from one of the
> cores, e.g. via comment
>
>   core2:
>   
> ...
> 
>   
>
>
> (4) Reload the modified core:
> http://localhost:9983/solr/admin/cores?action=RELOAD=core2
>
> (5) Request same query again and you get the NPE:
>
> error: {
> trace: "java.lang.NullPointerException
>   at 
> org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1068)
>   at 
> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:917)
>   at 
> org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:613)
>   at 
> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:592)
>   at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:431)
>   at 
> org.apache.solr.handler.RequestHandlerBase.handleR

Re: Elevation with distributed search causes NPE

[Erick Erickson]

Hmmm, looking at the code that line looks like this:

sortSpec.getSort().getSort();

I’m curious what happens if you specify a sort on the query? If that makes the 
problem go away, 
it’s a smoking gun.

Whether or not adding sorting makes the problem go away, this looks like 
something
that’s a legitimate JIRA, please go ahead and raise one.

Best,
Erick

> On Jul 15, 2020, at 4:34 AM, Marc Linden  
> wrote:
> 
> Hi all,
> 
> I'm facing the problem that Solr is throwing a NullPointerException when 
> performing a distributed search with multiple shards having elevation 
> configured where one or more shards do have elevated results but others do 
> not.
> 
> We are using Solr 8.2 and have the QueryElevationComponent configured with 
> "last-components" of the default search handler "/select". But the problem 
> also occurs when using the explicit "/elevate" search handler.
>  
> ...
> 
>  elevator
> 
>  
>  ...
>  
> 
> string
> elevate.xml
>  
> 
> ### Steps to reproduce:
> 
> (1) Add entries to the elevate.xml of each core to elevate a specific 
> document for the text "searchTerm":
> 
>   core1:
>   
> ...
> 
>   
>   core2:
>   
> ...
> 
>   
> 
> (2) Execute query (we use port 9983): 
> http://localhost:9983/solr/web/select?q=elevatedTerm=false=text_en=edismax=lang:en=localhost:9983/solr/core1,localhost:9983/solr/core2=[elevated],[shard],area,id=10=0
> 
> Now as both shards have elevated documents for the requested "searchTerm" the 
> search results are as expected:
> 
> response: {
> numFound: 5192,
> start: 0,
> maxScore: 1.9032197,
> docs: [{
> area: "press",
> id: "core1docId1",
> [elevated]: true,
> [shard]: "localhost:9983/solr/core1"
> }, {
> area: "products",
> id: "core2docId1",
> [elevated]: true,
> [shard]: "localhost:9983/solr/core2"
> }, {
> area: "press",
> id: "core1docId2",
> [elevated]: false,
> [shard]: "localhost:9983/solr/core1"
> },
> ...
> 
> (3) Remove the elevation entry for that "searchTerm" from one of the cores, 
> e.g. via comment
> 
>   core2:
>   
> ...
> 
>   
> 
> 
> (4) Reload the modified core: 
> http://localhost:9983/solr/admin/cores?action=RELOAD=core2
> 
> (5) Request same query again and you get the NPE:
> 
> error: {
> trace: "java.lang.NullPointerException
>   at 
> org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1068)
>   at 
> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:917)
>   at 
> org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:613)
>   at 
> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:592)
>   at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:431)
>   at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)
>   at 
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:780)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:566)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
>   at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>   ...
> 
> 
> I want to ask the community if I'm missing something or if this really is a 
> bug.
> 
> Thanks in advance,
> Marc
> Marc Linden
> Developer
> 
> Virtual Identity AG
> Grünwälderstraße 10-14
> 79098 Freiburg
> 
> +49 761 20758-422
> --
> 
> marc.lin...@virtual-identity.com
> http://www.virtual-identity.com
> 
> Freiburg | München | Wien | Porto
> 
> 
> 
> Virtual Identity AG
> Grünwälderstraße 10-14
> 79098 Freiburg
> Amtsgericht Freiburg, HRB 6218
> Vorstand: Ralf Heller
> Vorsitzende des Aufsichtsrates: Kirsten Heller
> Umsatzsteuer-ID: DE208002218

Elevation with distributed search causes NPE

[Marc Linden]

Hi all,

I'm facing the problem that Solr is throwing a NullPointerException when 
performing a distributed search with multiple shards having elevation 
configured where one or more shards do have elevated results but others do not.

We are using Solr 8.2 and have the QueryElevationComponent configured with 
"last-components" of the default search handler "/select". But the problem also 
occurs when using the explicit "/elevate" search handler.
  
...

  elevator

  
  ...
  

string
elevate.xml
  

### Steps to reproduce:

(1) Add entries to the elevate.xml of each core to elevate a specific document 
for the text "searchTerm":

   core1:
   
 ...
 
   
   core2:
   
 ...
 
   

(2) Execute query (we use port 9983): 
http://localhost:9983/solr/web/select?q=elevatedTerm=false=text_en=edismax=lang:en=localhost:9983/solr/core1,localhost:9983/solr/core2=[elevated],[shard],area,id=10=0

Now as both shards have elevated documents for the requested "searchTerm" the 
search results are as expected:

response: {
numFound: 5192,
start: 0,
maxScore: 1.9032197,
docs: [{
area: "press",
id: "core1docId1",
[elevated]: true,
[shard]: "localhost:9983/solr/core1"
}, {
area: "products",
id: "core2docId1",
[elevated]: true,
[shard]: "localhost:9983/solr/core2"
}, {
area: "press",
id: "core1docId2",
[elevated]: false,
[shard]: "localhost:9983/solr/core1"
},
...

(3) Remove the elevation entry for that "searchTerm" from one of the cores, 
e.g. via comment

   core2:
   
 ...
 
   


(4) Reload the modified core: 
http://localhost:9983/solr/admin/cores?action=RELOAD=core2

(5) Request same query again and you get the NPE:

error: {
trace: "java.lang.NullPointerException
   at 
org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1068)
   at 
org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:917)
   at 
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:613)
   at 
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:592)
   at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:431)
   at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)
   at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:780)
   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:566)
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
   at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
   at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
   ...


I want to ask the community if I'm missing something or if this really is a bug.

Thanks in advance,
Marc
Marc Linden
Developer

Virtual Identity AG
Grünwälderstraße 10-14
79098 Freiburg

+49 761 20758-422
--

marc.lin...@virtual-identity.com
http://www.virtual-identity.com

Freiburg | München | Wien | Porto



Virtual Identity AG
Grünwälderstraße 10-14
79098 Freiburg
Amtsgericht Freiburg, HRB 6218
Vorstand: Ralf Heller
Vorsitzende des Aufsichtsrates: Kirsten Heller
Umsatzsteuer-ID: DE208002218

Re: Federated / Distributed search using Solr

[Erick Erickson]

bq. I have 3 indexes and all three use different schema.

I assume that the user can't indicate a field to search against
or this is a total non-starter.

Even if you are just sending bare search terms to the different
collections, how are you going to compare returns from them?
Scores aren't comparable since the term stats will be different. You
can't sort across the different collections unless the sort fields you
use are common. Etc.

You have to define precisely what you mean by "federated search".
Return the top N/3 docs from each collection by the score each returns?
Sort by a common field? Other???

Best,
Erick

On Sat, Dec 29, 2018 at 2:24 PM Gus Heck  wrote:
>
> If the indexes are on the same cluster, you can use an alias for querying.
> Updating via aliases doesn't work well (updates go to the first collection
> listed only) unless it's a time routed alias.
>
> http://lucene.apache.org/solr/guide/7_6/collections-api.html#createalias
>
>
>
> On Sat, Dec 29, 2018 at 4:35 PM Steven White  wrote:
>
> > Hi everyone,
> >
> > Does Solr support federated / distributed search when the schema of the 2
> > or more indexes are different?
> >
> > I have 3 indexes and all three use different schema.  However, the unique
> > key field name is the same across all 3 indexes.  I need to search on all 3
> > indexes and return results as if the user searched on one index.
> >
> > If Solr supports federated search, what are the limitations if any?
> >
> > Thanks,
> >
> > Steve
> >
>
>
> --
> http://www.the111shift.com

Re: Federated / Distributed search using Solr

[Gus Heck]

If the indexes are on the same cluster, you can use an alias for querying.
Updating via aliases doesn't work well (updates go to the first collection
listed only) unless it's a time routed alias.

http://lucene.apache.org/solr/guide/7_6/collections-api.html#createalias



On Sat, Dec 29, 2018 at 4:35 PM Steven White  wrote:

> Hi everyone,
>
> Does Solr support federated / distributed search when the schema of the 2
> or more indexes are different?
>
> I have 3 indexes and all three use different schema.  However, the unique
> key field name is the same across all 3 indexes.  I need to search on all 3
> indexes and return results as if the user searched on one index.
>
> If Solr supports federated search, what are the limitations if any?
>
> Thanks,
>
> Steve
>


-- 
http://www.the111shift.com

Federated / Distributed search using Solr

[Steven White]

Hi everyone,

Does Solr support federated / distributed search when the schema of the 2
or more indexes are different?

I have 3 indexes and all three use different schema.  However, the unique
key field name is the same across all 3 indexes.  I need to search on all 3
indexes and return results as if the user searched on one index.

If Solr supports federated search, what are the limitations if any?

Thanks,

Steve

RE: QueryElevator prepare() in in distributed search

[Markus Jelsma]

Anything on this one to share?

Thanks,
Markus


-Original message-
> From:Markus Jelsma <markus.jel...@openindex.io>
> Sent: Friday 16th March 2018 18:13
> To: Solr-user <solr-user@lucene.apache.org>
> Subject: QueryElevator prepare() in in distributed search
> 
> Hello,
> 
> QueryElevator.prepare() runs five times for a single query in distributed 
> search, this is probably not how it should be, but in what phase of 
> distributed search is it supposed to actually run?
> 
> Many thanks,
> Markus
> 
> 

QueryElevator prepare() in in distributed search

[Markus Jelsma]

Hello,

QueryElevator.prepare() runs five times for a single query in distributed 
search, this is probably not how it should be, but in what phase of distributed 
search is it supposed to actually run?

Many thanks,
Markus

Authentication and distributed search in 7.2.1

[Peter Sturge]

Hi,
In 7.2.1 there's the authentication module and associated security.json
file, which works well for single cores. (Note: standalone mode, no
SolrCloud)
It doesn't appear to work with distributed searches, including multi-shard
local searches .
  e.g. shards=localhost:8983/solr/core1,localhost:8983/solr/core2

Even when shards is just a single core  - shards=localhost:8983/solr/core1,
if the base search is to a different core (e.g.
http://localhost:8983/solr/somecore/select?
shards=localhost:8983/solr/core1.. , no error and no results are returned
status=0 numfound=0.

Can anyone please confirm if Solr 7 authentication does/doesn't support
distributed/sharded searches?

Many thanks,
Peter

Re: Distributed search cross cluster

[Jan Høydahl]

Erick:

> ...one for each cluster and just merged the docs when it got them back


This would be the logical way. I'm afraid that "just merged the docs" is the 
crux here, that would
make this an expensive task. You'd have to merge docs, facets, highlights etc, 
handle the
different search phases (ID fetch, doc fetch, potentially global idf fetch?) 
etc.
It may be that the code necessary to do the merge already exists in the 
project, haven't looked...

Charlie:

Yes it should "just" work. Until someone upgrades the schema in one cloud and 
not the others
of course :) and we still need to handle failure cases such as high latency or 
one cluster down…

Besides, we'll have SSL certs with client auth and probably some sort of 
auth in place in
all clouds, and we'd of course need to make sure that the user exists in all 
clusters and that
cross cluster traffic is allowed in everywhere. PKI auth is not really intended 
for accepting
requests from a foreign node that is not in its ZK etc.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 31. jan. 2018 kl. 10:06 skrev Charlie Hull :
> 
> On 30/01/2018 16:09, Jan Høydahl wrote:
>> Hi,
>> A customer has 10 separate SolrCloud clusters, with same schema across all, 
>> but different content.
>> Now they want users in each location to be able to federate a search across 
>> all locations.
>> Each location is 100% independent, with separate ZK etc. Bandwidth and 
>> latency between the
>> clusters is not an issue, they are actually in the same physical datacenter.
>> Now my first thought was using a custom  parameter, and let the 
>> receiving node fan
>> out to all shards of all clusters. We’d need to contact the ZK for each 
>> environment and find
>> all shards and replicas participating in the collection and then construct 
>> the shards=A1|A2,B1|B2…
>> sting which would be quite big, but if we get it right, it should “just 
>> work".
>> Now, my question is whether there are other smarter ways that would leave it 
>> up to existing Solr
>> logic to select shards and load balance, that would also take into account 
>> any shard.keys/_route_
>> info etc. I thought of these
>>   * =collA,collB  — but it only supports collections local to one 
>> cloud
>>   * Create a collection ALIAS to point to all 10 — but same here, only local 
>> to one cluster
>>   * Streaming expression top(merge(search(q=,zkHost=blabla))) — but we want 
>> it with pure search API
>>   * Write a custom ShardHandler plugin that knows about all clusters — but 
>> this is complex stuff :)
>>   * Write a custom SearchComponent plugin that knows about all clusters and 
>> adds the = param
>> Another approach would be for the originating cluster to fan out just ONE 
>> request to each of the other
>> clusters and then write some SearchComponent to merge those responses. That 
>> would let us query
>> the other clusters using one LB IP address instead of requiring full 
>> visibility to all solr nodes
>> of all clusters, but if we don’t need that isolation, that extra merge code 
>> seems fairly complex.
>> So far I opt for the custom SearchComponent and = param approach. Any 
>> useful input from
>> someone who tried a similar approach would be priceless!
> 
> Hi Jan,
> 
> We actually looked at this for the BioSolr project - a SolrCloud of 
> SolrClouds. Unfortunately the funding didn't appear for the project so we 
> didn't take it any further than some rough ideas - as you say, if you get it 
> right it should 'just work'. We had some extra complications in terms of 
> shared partial schemas...
> 
> Cheers
> 
> Charlie
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com 
> 
> 
> -- 
> Charlie Hull
> Flax - Open Source Enterprise Search
> 
> tel/fax: +44 (0)8700 118334
> mobile:  +44 (0)7767 825828
> web: www.flax.co.uk 

Re: Distributed search cross cluster

[Jan Høydahl]

Hi,

I am an ex FAST employee and actually used Unity a lot myself, even hacking the 
code
writing custom mixers etc :)

That is all cool, if you want to write a generic federation layer. In our case 
we only ever
need to talk to Solr instances with exactly the same schema and doument types,
compatible scores etc. So that’s why I figure it is out of scope to write 
custom merge
code. It would also be less efficient since you’d get, say 10 hits from 10 
clusters = 100 hits
while if you just let the original node talk to all the shards then you only 
fetch the top docs
across all clusters.

I see many many open OLD JIRAs for federated features, which never got anywhere,
so I take that also as a hint that this is either not needed or very complex :)

Takling about FAST ESP, the "fsearch" process responsible for merging results 
from 
underlying indices was actually used at multiple levels, so to federate two 
FAST clusters
all you had to do was put a top level fsearch process above all of them and 
point it to
the right host:port list, then a QRServer on top of that fsearch again. Those 
were the days.

If there was some class that would delegate an incoming search request to sub 
shards
in a generic way, without writing all the merge and two-phase stuff over again, 
then
that would be ideal.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 31. jan. 2018 kl. 10:41 skrev Bernd Fehling :
> 
> Many years ago, in a different universe, when Federated Search was a buzzword 
> we
> used Unity from FAST FDS (which is now MS ESP). It worked pretty well across
> many systems like FAST FDS, Google, Gigablast, ...
> Very flexible with different mixers, parsers, query transformers.
> Was written in Python and used pylib.medusa.
> Search for "unity federated search", there is a book at Google about this, 
> just
> to get an idea.
> 
> Regards, Bernd
> 
> 
> Am 30.01.2018 um 17:09 schrieb Jan Høydahl:
>> Hi,
>> 
>> A customer has 10 separate SolrCloud clusters, with same schema across all, 
>> but different content.
>> Now they want users in each location to be able to federate a search across 
>> all locations.
>> Each location is 100% independent, with separate ZK etc. Bandwidth and 
>> latency between the
>> clusters is not an issue, they are actually in the same physical datacenter.
>> 
>> Now my first thought was using a custom  parameter, and let the 
>> receiving node fan
>> out to all shards of all clusters. We’d need to contact the ZK for each 
>> environment and find
>> all shards and replicas participating in the collection and then construct 
>> the shards=A1|A2,B1|B2…
>> sting which would be quite big, but if we get it right, it should “just 
>> work".
>> 
>> Now, my question is whether there are other smarter ways that would leave it 
>> up to existing Solr
>> logic to select shards and load balance, that would also take into account 
>> any shard.keys/_route_
>> info etc. I thought of these
>>  * =collA,collB  — but it only supports collections local to one 
>> cloud
>>  * Create a collection ALIAS to point to all 10 — but same here, only local 
>> to one cluster
>>  * Streaming expression top(merge(search(q=,zkHost=blabla))) — but we want 
>> it with pure search API
>>  * Write a custom ShardHandler plugin that knows about all clusters — but 
>> this is complex stuff :)
>>  * Write a custom SearchComponent plugin that knows about all clusters and 
>> adds the = param
>> 
>> Another approach would be for the originating cluster to fan out just ONE 
>> request to each of the other
>> clusters and then write some SearchComponent to merge those responses. That 
>> would let us query
>> the other clusters using one LB IP address instead of requiring full 
>> visibility to all solr nodes
>> of all clusters, but if we don’t need that isolation, that extra merge code 
>> seems fairly complex.
>> 
>> So far I opt for the custom SearchComponent and = param approach. Any 
>> useful input from
>> someone who tried a similar approach would be priceless!
>> 
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>> 

Re: Distributed search cross cluster

[Bernd Fehling]

Many years ago, in a different universe, when Federated Search was a buzzword we
used Unity from FAST FDS (which is now MS ESP). It worked pretty well across
many systems like FAST FDS, Google, Gigablast, ...
Very flexible with different mixers, parsers, query transformers.
Was written in Python and used pylib.medusa.
Search for "unity federated search", there is a book at Google about this, just
to get an idea.

Regards, Bernd


Am 30.01.2018 um 17:09 schrieb Jan Høydahl:
> Hi,
> 
> A customer has 10 separate SolrCloud clusters, with same schema across all, 
> but different content.
> Now they want users in each location to be able to federate a search across 
> all locations.
> Each location is 100% independent, with separate ZK etc. Bandwidth and 
> latency between the
> clusters is not an issue, they are actually in the same physical datacenter.
> 
> Now my first thought was using a custom  parameter, and let the 
> receiving node fan
> out to all shards of all clusters. We’d need to contact the ZK for each 
> environment and find
> all shards and replicas participating in the collection and then construct 
> the shards=A1|A2,B1|B2…
> sting which would be quite big, but if we get it right, it should “just work".
> 
> Now, my question is whether there are other smarter ways that would leave it 
> up to existing Solr
> logic to select shards and load balance, that would also take into account 
> any shard.keys/_route_
> info etc. I thought of these
>   * =collA,collB  — but it only supports collections local to one 
> cloud
>   * Create a collection ALIAS to point to all 10 — but same here, only local 
> to one cluster
>   * Streaming expression top(merge(search(q=,zkHost=blabla))) — but we want 
> it with pure search API
>   * Write a custom ShardHandler plugin that knows about all clusters — but 
> this is complex stuff :)
>   * Write a custom SearchComponent plugin that knows about all clusters and 
> adds the = param
> 
> Another approach would be for the originating cluster to fan out just ONE 
> request to each of the other
> clusters and then write some SearchComponent to merge those responses. That 
> would let us query
> the other clusters using one LB IP address instead of requiring full 
> visibility to all solr nodes
> of all clusters, but if we don’t need that isolation, that extra merge code 
> seems fairly complex.
> 
> So far I opt for the custom SearchComponent and = param approach. Any 
> useful input from
> someone who tried a similar approach would be priceless!
> 
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
> 

Re: Distributed search cross cluster

[Charlie Hull]

On 30/01/2018 16:09, Jan Høydahl wrote:

Hi,

A customer has 10 separate SolrCloud clusters, with same schema across all, but 
different content.
Now they want users in each location to be able to federate a search across all 
locations.
Each location is 100% independent, with separate ZK etc. Bandwidth and latency 
between the
clusters is not an issue, they are actually in the same physical datacenter.

Now my first thought was using a custom  parameter, and let the 
receiving node fan
out to all shards of all clusters. We’d need to contact the ZK for each 
environment and find
all shards and replicas participating in the collection and then construct the 
shards=A1|A2,B1|B2…
sting which would be quite big, but if we get it right, it should “just work".

Now, my question is whether there are other smarter ways that would leave it up 
to existing Solr
logic to select shards and load balance, that would also take into account any 
shard.keys/_route_
info etc. I thought of these
   * =collA,collB  — but it only supports collections local to one 
cloud
   * Create a collection ALIAS to point to all 10 — but same here, only local 
to one cluster
   * Streaming expression top(merge(search(q=,zkHost=blabla))) — but we want it 
with pure search API
   * Write a custom ShardHandler plugin that knows about all clusters — but 
this is complex stuff :)
   * Write a custom SearchComponent plugin that knows about all clusters and adds 
the = param

Another approach would be for the originating cluster to fan out just ONE 
request to each of the other
clusters and then write some SearchComponent to merge those responses. That 
would let us query
the other clusters using one LB IP address instead of requiring full visibility 
to all solr nodes
of all clusters, but if we don’t need that isolation, that extra merge code 
seems fairly complex.

So far I opt for the custom SearchComponent and = param approach. Any 
useful input from
someone who tried a similar approach would be priceless!


Hi Jan,

We actually looked at this for the BioSolr project - a SolrCloud of 
SolrClouds. Unfortunately the funding didn't appear for the project so 
we didn't take it any further than some rough ideas - as you say, if you 
get it right it should 'just work'. We had some extra complications in 
terms of shared partial schemas...


Cheers

Charlie


--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com




--
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk

Re: Distributed search cross cluster

[Erick Erickson]

Jan:

Hmmm, must Solr do the work? On some level it seems easier if your
middle layer (behind your single IP) has 10 CloudSolrClient thread
pools, one for each cluster and just merged the docs when it got them
back. That would take care of all of the goodness of internal LBs and
all that.

Somewhere you have to know about all 10 ZK ensembles (or the IP
address of one of your Solr instances in each cluster or). You're
talking about building that into a SearchComponent, but would a simple
client work as well? It wouldn't even have to be on a node hosting
Solr.

Streaming doesn't really seem to fit the bill, I don't think. It's
built to handle large result sets and it doesn't sound like this is
that. It _used_ to read to end of stream even when closed, although
that's been fixed but check your version (don't have the JIRA number
offhand).

If you use a "shards" approach, don't you then have a single point of
failure if it's going to just "some Solr node" in each of the other
collections?

I admit I just scanned your post and haven't thought about it very deeply

Erick

On Tue, Jan 30, 2018 at 8:09 AM, Jan Høydahl  wrote:
> Hi,
>
> A customer has 10 separate SolrCloud clusters, with same schema across all, 
> but different content.
> Now they want users in each location to be able to federate a search across 
> all locations.
> Each location is 100% independent, with separate ZK etc. Bandwidth and 
> latency between the
> clusters is not an issue, they are actually in the same physical datacenter.
>
> Now my first thought was using a custom  parameter, and let the 
> receiving node fan
> out to all shards of all clusters. We’d need to contact the ZK for each 
> environment and find
> all shards and replicas participating in the collection and then construct 
> the shards=A1|A2,B1|B2…
> sting which would be quite big, but if we get it right, it should “just work".
>
> Now, my question is whether there are other smarter ways that would leave it 
> up to existing Solr
> logic to select shards and load balance, that would also take into account 
> any shard.keys/_route_
> info etc. I thought of these
>   * =collA,collB  — but it only supports collections local to one 
> cloud
>   * Create a collection ALIAS to point to all 10 — but same here, only local 
> to one cluster
>   * Streaming expression top(merge(search(q=,zkHost=blabla))) — but we want 
> it with pure search API
>   * Write a custom ShardHandler plugin that knows about all clusters — but 
> this is complex stuff :)
>   * Write a custom SearchComponent plugin that knows about all clusters and 
> adds the = param
>
> Another approach would be for the originating cluster to fan out just ONE 
> request to each of the other
> clusters and then write some SearchComponent to merge those responses. That 
> would let us query
> the other clusters using one LB IP address instead of requiring full 
> visibility to all solr nodes
> of all clusters, but if we don’t need that isolation, that extra merge code 
> seems fairly complex.
>
> So far I opt for the custom SearchComponent and = param approach. Any 
> useful input from
> someone who tried a similar approach would be priceless!
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>

Distributed search cross cluster

[Jan Høydahl]

Hi,

A customer has 10 separate SolrCloud clusters, with same schema across all, but 
different content.
Now they want users in each location to be able to federate a search across all 
locations.
Each location is 100% independent, with separate ZK etc. Bandwidth and latency 
between the
clusters is not an issue, they are actually in the same physical datacenter.

Now my first thought was using a custom  parameter, and let the 
receiving node fan
out to all shards of all clusters. We’d need to contact the ZK for each 
environment and find
all shards and replicas participating in the collection and then construct the 
shards=A1|A2,B1|B2…
sting which would be quite big, but if we get it right, it should “just work".

Now, my question is whether there are other smarter ways that would leave it up 
to existing Solr
logic to select shards and load balance, that would also take into account any 
shard.keys/_route_
info etc. I thought of these
  * =collA,collB  — but it only supports collections local to one 
cloud
  * Create a collection ALIAS to point to all 10 — but same here, only local to 
one cluster
  * Streaming expression top(merge(search(q=,zkHost=blabla))) — but we want it 
with pure search API
  * Write a custom ShardHandler plugin that knows about all clusters — but this 
is complex stuff :)
  * Write a custom SearchComponent plugin that knows about all clusters and 
adds the = param

Another approach would be for the originating cluster to fan out just ONE 
request to each of the other
clusters and then write some SearchComponent to merge those responses. That 
would let us query
the other clusters using one LB IP address instead of requiring full visibility 
to all solr nodes
of all clusters, but if we don’t need that isolation, that extra merge code 
seems fairly complex.

So far I opt for the custom SearchComponent and = param approach. Any 
useful input from
someone who tried a similar approach would be priceless!

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

score field returns NaN with distributed Search after Upgrade from 4.5 to 4.10

[Natarajan, Rajeswari]

Hi  All,

We are upgrading from Solr/Lucene 4.5.1 to 4.10.4. When testing we found the 
below issue.

The score field in  the query response for a distributed query results in NaN. 
This happens if the indexes were created in 4.5 and the query is received in 
4.10.1.

Also the score in the explain statement with the debugQuery on is same with 4.5 
or 4.10.  

This issue does not happen for queries  which involves single shard. Has anyone 
faced this issue or know the cause/remedy of this behavior.


Thank you,
Rajeswari 


Query is just Q=*:*=score ,with the correct URL

Part of the Output


NaN
  


LuceneQParser
  

0.003922721 = (MATCH) product of:
  0.07845442 = (MATCH) sum of:
0.07845442 = (MATCH) 
weight(arches_id:PR^test.abcindexadapter^en_US^4^13^ABC-02-docid01-0-ABC-02-docid01-0-ABC02-docid01-0
 in 0) [ABCSimilarity], result of:
  0.07845442 = score(doc=0,freq=1.0 = termFreq=1.0
), product of:
0.22360682 = queryWeight, product of:
  4.912023 = idf(docFreq=1, maxDocs=100)
  0.045522347 = queryNorm
0.35085878 = fieldWeight in 0, product of:
  0.07142857 = (tf = 1.0, Boost = 0, Boost (Binary) = 0, normValue = 
0.0, bmf = 50)
1.0 = termFreq=1.0
  4.912023 = idf(docFreq=1, maxDocs=100)
  0.05 = coord(1/20)

RE: How to avoid unnecessary query parsing on distributed search in QueryComponent.prepare()?

[Markus Jelsma]

I've asked myself this question too some times. In this case extending MLT 
QParser. So far, i've not found a simple means to propagate a parsed top-level 
Lucene query object over the wire. 

But, since there is a clear toString for that Query object, if we could 
retranslate that String to a real object, could that be lighter than parsing 
the incoming query text?

Markus
 
-Original message-
> From:Mikhail Khludnev <m...@apache.org>
> Sent: Thursday 11th May 2017 10:43
> To: solr-user <solr-user@lucene.apache.org>
> Subject: How to avoid unnecessary query parsing on distributed search in 
> QueryComponent.prepare()?
> 
> Hello,
> When the distributed search is requested (SolrCloud), the query component
> invokes prepare() where a query is parsed. But then it's just ignored, I
> suppose because all work is done by subordinate shards' requests.
> It's fine the most of the times because query parsing is cheap. Until we
> have a heavy wildcarded complexphrase query, where the expensive expansion
> is done during parsing.
> How to avoid this waste of resources? Just bypass parsing is rb.isDistrib?
> or refactor complexphrase parser moving expansion downstream to rewrite or
> createWeight?
> --
> Mikhail
> 

d: org.apache.http.ParseException: Invalid content type: - solr distributed search 4.10.4

[Natarajan, Rajeswari]

Hi,

When doing a distributed query from solr 4.10.4 ,getting below exception

org.apache.solr.common.SolrException: org.apache.http.ParseException: Invalid 
content type:

org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)

org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)

ariba.arches.search.ArchesSearcher.invokeSearch(ArchesSearcher.java:306)

ariba.arches.search.ArchesSearcher.search(ArchesSearcher.java:169)

ariba.arches.search.SearchManagerServlet.handleSelect(SearchManagerServlet.java:651)

ariba.arches.search.SearchManagerServlet.service(SearchManagerServlet.java:146)


javax.servlet.http.HttpServlet.service(HttpServlet.java:848)


query is below:

http://:20042/ 
/search/select?q=(*:*)=xml=5=SupplierID,MarketPrice=:20042
 /search/select/executeS2-63,:20022/ search/select/execute/S1-69


In the code

Below method n SolrCore is used to execute the query.
execute(SolrRequestHandler handler, SolrQueryRequest req, SolrQueryResponse 
rsp) {


Saw ssame issue in https://lists.gt.net/lucene/java-dev/242650.

If we test the distributed query in a stand alone solr  as below it works

http://localhost:8983/solr/select?shards=localhost:8983/solr,localhost:8984/solr=true=ipod+solr


Any pointers  to resolve this issue please.


Thank you,
Raji

How to avoid unnecessary query parsing on distributed search in QueryComponent.prepare()?

[Mikhail Khludnev]

Hello,
When the distributed search is requested (SolrCloud), the query component
invokes prepare() where a query is parsed. But then it's just ignored, I
suppose because all work is done by subordinate shards' requests.
It's fine the most of the times because query parsing is cheap. Until we
have a heavy wildcarded complexphrase query, where the expensive expansion
is done during parsing.
How to avoid this waste of resources? Just bypass parsing is rb.isDistrib?
or refactor complexphrase parser moving expansion downstream to rewrite or
createWeight?
--
Mikhail

honouring terms.limit parameter in a distributed search for /terms api

[radha krishnan]

Hi,

in the TermsComponent.java's, createShardQuery, the motive to specify
terms.limit to -1 is clearly specified in java comment

but we have a usecase where have thousands  of terms and we want each core
to return only the value specfied by terms.limit.

can we have two flavours of TermsComponent where in
1. we specify terms.limit = -1 (which is the current one)
2. we honour terms.limit passed by the caller. something like below


public class LimitBasedTermsComponent extends TermsComponent {

@Override
public int distributedProcess(ResponseBuilder rb) throws IOException {
if (!rb.doTerms) {
return ResponseBuilder.STAGE_DONE;
}

if (rb.stage == ResponseBuilder.STAGE_EXECUTE_QUERY) {
TermsHelper th = rb._termsHelper;
if (th == null) {
th = rb._termsHelper = new TermsHelper();
th.init(rb.req.getParams());
}
ShardRequest sreq = createShardQuery(rb.req.getParams());
rb.addRequest(this, sreq);
}

if (rb.stage < ResponseBuilder.STAGE_EXECUTE_QUERY) {
return ResponseBuilder.STAGE_EXECUTE_QUERY;
} else {
return ResponseBuilder.STAGE_DONE;
}
}


private ShardRequest createShardQuery(SolrParams params) {
ShardRequest sreq = new ShardRequest();
sreq.purpose = ShardRequest.PURPOSE_GET_TERMS;

// base shard request on original parameters
sreq.params = new ModifiableSolrParams(params);

// if TermsParams.TERMS_LIMIT is present in the params, use
the value, else default it to 10
sreq.params.remove(TermsParams.TERMS_MAXCOUNT);
sreq.params.remove(TermsParams.TERMS_MINCOUNT);
sreq.params.set(TermsParams.TERMS_LIMIT,
params.getInt(TermsParams.TERMS_LIMIT, 10));
sreq.params.set(TermsParams.TERMS_SORT, TermsParams.TERMS_SORT_INDEX);

return sreq;
}



Thanks,
Radhakrishnan D

Re: Distributed Search: Wrong count?

[Kelly, Frank]

Quick extra clarification – the documents in question we are searching for are 
child documents we are searching direct (no parent/child in the query)

-Frank

From: Frank J Kelly <frank.ke...@here.com<mailto:frank.ke...@here.com>>
Reply-To: "solr-user@lucene.apache.org<mailto:solr-user@lucene.apache.org>" 
<solr-user@lucene.apache.org<mailto:solr-user@lucene.apache.org>>
Date: Wednesday, March 1, 2017 at 2:04 PM
To: "solr-user@lucene.apache.org<mailto:solr-user@lucene.apache.org>" 
<solr-user@lucene.apache.org<mailto:solr-user@lucene.apache.org>>
Subject: Distributed Search: Wrong count?

Environment: SolrCloud 5.3
Collection has 12.3m docs split across 3 shards and 3 replicas

In the query below I get one document ID returned but a numFound of 365

{
  "responseHeader":{
"status":0,
"QTime":47,
"params":{
  "q":"haUserId:  AND haAccountType:google  AND 
type:userLinkedAccount\n",
  "indent":"true",
  "rows":"10",
  "wt":"json"}},
  "response":{"numFound":365,"start":0,"maxScore":16.051413,"docs":[
  {
"id":"userLinkedAccount/260daf67a93036ec7ee4ed401f14242718a4d6cf",
"indexTime":1487187266965}]
  }}


If I change the count parameter on my query to 100 I now get a numFound of 185


And if I change the count parameter to 1000 I now get a numFound of 1


I am seeing no API Error responses from Solr when posting.


Any thoughts on what is happening?


-Frank
[Description: Macintosh 
HD:Users:jerchow:Downloads:Asset_Package_01_160721:HERE_Logo_2016:sRGB:PDF:HERE_Logo_2016_POS_sRGB.pdf]



Frank Kelly

Principal Software Engineer

Identity Profile Team (SCBE, Traces, CDA)


HERE

5 Wayside Rd, Burlington, MA 01803, USA

42° 29' 7" N 71° 11' 32" W

[Description: 
/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_360.gif]<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F360.here.com%2F=01%7C01%7C%7C301c055b871041d8893508d460d5e75f%7C6d4034cd72254f72b85391feaea64919%7C1=u0%2Fdtg%2FF0dfTVrjtkcoXB0dXZlUR4X7ZAxeBkvN5rlc%3D=0>
[Description: 
/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_Twitter.gif]
 
<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.twitter.com%2Fhere=01%7C01%7C%7C301c055b871041d8893508d460d5e75f%7C6d4034cd72254f72b85391feaea64919%7C1=a7fLyKul%2FW%2FGklSCGcJgHkcv2dP3CvdecWiXkH8fdI0%3D=0>
[Description: 
/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_FB.gif]
 
<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2Fhere=01%7C01%7C%7C301c055b871041d8893508d460d5e75f%7C6d4034cd72254f72b85391feaea64919%7C1=AbHNVYzxQKHnREwxxuBvgjkxi6D1JUmxeVh6yhyxiOU%3D=0>
 [Description: 
/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_IN.gif]
 
<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fheremaps=01%7C01%7C%7C301c055b871041d8893508d460d5e75f%7C6d4034cd72254f72b85391feaea64919%7C1=Rq2c0IQMYsc%2FYpazo3PtMWXdJqxVGUnij4jF%2BEwVeis%3D=0>
 [Description: 
/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_Insta.gif]
 
<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.instagram.com%2Fhere%2F=01%7C01%7C%7C301c055b871041d8893508d460d5e75f%7C6d4034cd72254f72b85391feaea64919%7C1=OjvAeHGId5%2FcmcYZ0xUfCYWegKShpE1PQeMlHYWQ%2BcE%3D=0>

Distributed Search: Wrong count?

[Kelly, Frank]

Environment: SolrCloud 5.3
Collection has 12.3m docs split across 3 shards and 3 replicas

In the query below I get one document ID returned but a numFound of 365

{
  "responseHeader":{
"status":0,
"QTime":47,
"params":{
  "q":"haUserId:  AND haAccountType:google  AND 
type:userLinkedAccount\n",
  "indent":"true",
  "rows":"10",
  "wt":"json"}},
  "response":{"numFound":365,"start":0,"maxScore":16.051413,"docs":[
  {
"id":"userLinkedAccount/260daf67a93036ec7ee4ed401f14242718a4d6cf",
"indexTime":1487187266965}]
  }}


If I change the count parameter on my query to 100 I now get a numFound of 185


And if I change the count parameter to 1000 I now get a numFound of 1


I am seeing no API Error responses from Solr when posting.


Any thoughts on what is happening?


-Frank
[Description: Macintosh 
HD:Users:jerchow:Downloads:Asset_Package_01_160721:HERE_Logo_2016:sRGB:PDF:HERE_Logo_2016_POS_sRGB.pdf]



Frank Kelly

Principal Software Engineer

Identity Profile Team (SCBE, Traces, CDA)


HERE

5 Wayside Rd, Burlington, MA 01803, USA

42° 29' 7" N 71° 11' 32" W

[Description: 
/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_360.gif]
[Description: 
/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_Twitter.gif]
 [Description: 
/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_FB.gif]
  [Description: 
/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_IN.gif]
  [Description: 
/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_Insta.gif]
 

Re: Distributed Search (across collections) + partial Filter query

[alessandro.benedetti]

Hi all,
thanks to Andrea Gazzarini suggestion I solved it using local params ( which
is different from macro expansion even if conceptually similar).
Local params were available in Solr 4.10.x

I appended this filter query in the request handler of interest:


  {!lucene df=filterField v=$allowed_values}
  *


This will appear in all the collections request handler that want to use the
filter.

Cheers



-
---
Alessandro Benedetti
Search Consultant, R Software Engineer, Director
Sease Ltd. - www.sease.io
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Distributed-Search-across-collections-partial-Filter-query-tp4319166p4319292.html
Sent from the Solr - User mailing list archive at Nabble.com.

Distributed Search (across collections) + partial Filter query

[alessandro.benedetti]

Hi all,
*Scenario*
I run searches across different collections.
e.g.
we have 2 collections, 1 with a *field1* defined and 1 not.

*Use Case*
I would like to send a distributed request with a parameter that will filter
the results on field1 on the collection1 but do nothing with the
collection2.
Basically getting back :
Partial results from collection1(fq applied) and total results from
collection2 (no fq applied)

*Version*
4.10

out of the box collection2 will not be happy and raise a "field not defined"
exception.

If I was in Solr >5.1 I would have used macro expansion, passing a new
request parameter :
1) collection1 request handler would have used it to append a new filter
query
2) collection2 request handler would have discarded it doing nothing

Unfortunately in the Solr version I am , macro expansion was not there
yet.[1]
I will continue thinking on it but if you have any idea, let me know!

Cheers

[1] http://yonik.com/solr-query-parameter-substitution/





-
---
Alessandro Benedetti
Search Consultant, R Software Engineer, Director
Sease Ltd. - www.sease.io
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Distributed-Search-across-collections-partial-Filter-query-tp4319166.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: [Rerank Query] Distributed search + pagination

[Alessandro Benedetti]

Perfect Joel,
keep me updated !

Cheers

On Mon, Sep 19, 2016 at 10:26 PM, Joel Bernstein  wrote:

> Alessandro, I'll be doing some testing with the re-ranker as part of
> SOLR-9403 for Solr 6.3. I'll see if I can better understand the issue
> you're bringing up during the testing. I'll report back to this thread
> after I've done some testing.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Fri, Sep 16, 2016 at 11:17 AM, Alessandro Benedetti <
> abenede...@apache.org> wrote:
>
> > In addition to that, I think the only way to solve this is to rely on the
> > aggregator node to actually re-rank after having aggregated.
> >
> > Cheer
> >
> > On Fri, Sep 9, 2016 at 11:48 PM, Alessandro Benedetti <
> > abenede...@apache.org
> > > wrote:
> >
> > > Let me explain further,
> > > let's assume a simple case when we have 2 shards.
> > > ReRankDocs =10 , rows=10 .
> > >
> > > Correct me if I am wrong Joel,
> > > What we would like :
> > > 1 page : top 10 re-scored
> > > 2 page: remaining 10 re-scored
> > > From page 3 the original scored docs.
> > > This is what is happening in a single sol instance if we put reRankDocs
> > to
> > > 20.
> > >
> > > Let's see with sharing :
> > > To get the first page we get top 10 ( re-scored) from shard1 and top 10
> > > reranked for shard 2.
> > > Then the merged top 10 ( re-scored) will be calculated, and that is the
> > > page 1.
> > >
> > > But when we require the page 2, which means we additionally ask now :
> > > 20 docs to shard1, 10 re-scored and 10 not.
> > > 20 docs to shard2, 10 re-scored and 10 not.
> > > At this point we have 40 docs to merge and rank..
> > > The docs with the original score can go at any position ( not
> necessarily
> > > the last 20)
> > > In the page 2 we can find potentially docs with the original score.
> > > This is even more likely if the scores are on differente scales (e.g.
> the
> > > re-scored 0100 ) .
> > >
> > > Am I right ?
> > > Did I make any wrong assumption so far ?
> > >
> > > Cheers
> > >
> > >
> > > On Fri, Sep 9, 2016 at 7:47 PM, Joel Bernstein 
> > wrote:
> > >
> > >> I'm not understanding where the inconsistency comes into play.
> > >>
> > >> The re-ranking occurs on the shards. The aggregator node will be sent
> > some
> > >> docs that have been re-scored and others that are not. But the sorting
> > >> should be the same as someone pages through the result set.
> > >>
> > >>
> > >>
> > >> Joel Bernstein
> > >> http://joelsolr.blogspot.com/
> > >>
> > >> On Fri, Sep 9, 2016 at 9:28 AM, Alessandro Benedetti <
> > >> abenede...@apache.org>
> > >> wrote:
> > >>
> > >> > Hi guys,
> > >> > was just experimenting some reranker with really low number of
> rerank
> > >> docs
> > >> > ( 10= pageSize) .
> > >> > Let's focus on the distributed enviroment and  the manual sharding
> > >> > approach.
> > >> >
> > >> > Currently what happens is that the reranking task is delivered by
> the
> > >> > shards, they rescore the docs and then send them back to the
> > aggregator
> > >> > node.
> > >> >
> > >> > If you want to rerank only few docs ( leaving the others with the
> > >> original
> > >> > score following), this can be done in a single Solr instance ( the
> > >> howmany
> > >> > logic manages that in the reranker) .
> > >> >
> > >> > What happens when you move to a distributed environment ?
> > >> > The aggregator will aggregate both rescored and original scored
> > >> documents,
> > >> > making the final ranking inconsistent.
> > >> > In the other hand if we make the rarankingDocs threshold dynamic (
> to
> > >> adapt
> > >> > to start+rows) we can incur in the very annoying issue of having a
> > >> document
> > >> > sliding through the pages ( visible in the first page , then
> appearing
> > >> > again in the third ect ect).
> > >> >
> > >> > Any thought ?
> > >> >
> > >> > Cheers
> > >> >
> > >> > --
> > >> > --
> > >> >
> > >> > Benedetti Alessandro
> > >> > Visiting card : http://about.me/alessandro_benedetti
> > >> >
> > >> > "Tyger, tyger burning bright
> > >> > In the forests of the night,
> > >> > What immortal hand or eye
> > >> > Could frame thy fearful symmetry?"
> > >> >
> > >> > William Blake - Songs of Experience -1794 England
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > --
> > >
> > > Benedetti Alessandro
> > > Visiting card : http://about.me/alessandro_benedetti
> > >
> > > "Tyger, tyger burning bright
> > > In the forests of the night,
> > > What immortal hand or eye
> > > Could frame thy fearful symmetry?"
> > >
> > > William Blake - Songs of Experience -1794 England
> > >
> >
> >
> >
> > --
> > --
> >
> > Benedetti Alessandro
> > Visiting card : http://about.me/alessandro_benedetti
> >
> > "Tyger, tyger burning bright
> > In the forests of the night,
> > What immortal hand or eye
> > Could frame thy fearful symmetry?"
> >
> > William Blake - Songs of Experience -1794 England

Re: [Rerank Query] Distributed search + pagination

[Joel Bernstein]

Alessandro, I'll be doing some testing with the re-ranker as part of
SOLR-9403 for Solr 6.3. I'll see if I can better understand the issue
you're bringing up during the testing. I'll report back to this thread
after I've done some testing.

Joel Bernstein
http://joelsolr.blogspot.com/

On Fri, Sep 16, 2016 at 11:17 AM, Alessandro Benedetti <
abenede...@apache.org> wrote:

> In addition to that, I think the only way to solve this is to rely on the
> aggregator node to actually re-rank after having aggregated.
>
> Cheer
>
> On Fri, Sep 9, 2016 at 11:48 PM, Alessandro Benedetti <
> abenede...@apache.org
> > wrote:
>
> > Let me explain further,
> > let's assume a simple case when we have 2 shards.
> > ReRankDocs =10 , rows=10 .
> >
> > Correct me if I am wrong Joel,
> > What we would like :
> > 1 page : top 10 re-scored
> > 2 page: remaining 10 re-scored
> > From page 3 the original scored docs.
> > This is what is happening in a single sol instance if we put reRankDocs
> to
> > 20.
> >
> > Let's see with sharing :
> > To get the first page we get top 10 ( re-scored) from shard1 and top 10
> > reranked for shard 2.
> > Then the merged top 10 ( re-scored) will be calculated, and that is the
> > page 1.
> >
> > But when we require the page 2, which means we additionally ask now :
> > 20 docs to shard1, 10 re-scored and 10 not.
> > 20 docs to shard2, 10 re-scored and 10 not.
> > At this point we have 40 docs to merge and rank..
> > The docs with the original score can go at any position ( not necessarily
> > the last 20)
> > In the page 2 we can find potentially docs with the original score.
> > This is even more likely if the scores are on differente scales (e.g. the
> > re-scored 0100 ) .
> >
> > Am I right ?
> > Did I make any wrong assumption so far ?
> >
> > Cheers
> >
> >
> > On Fri, Sep 9, 2016 at 7:47 PM, Joel Bernstein 
> wrote:
> >
> >> I'm not understanding where the inconsistency comes into play.
> >>
> >> The re-ranking occurs on the shards. The aggregator node will be sent
> some
> >> docs that have been re-scored and others that are not. But the sorting
> >> should be the same as someone pages through the result set.
> >>
> >>
> >>
> >> Joel Bernstein
> >> http://joelsolr.blogspot.com/
> >>
> >> On Fri, Sep 9, 2016 at 9:28 AM, Alessandro Benedetti <
> >> abenede...@apache.org>
> >> wrote:
> >>
> >> > Hi guys,
> >> > was just experimenting some reranker with really low number of rerank
> >> docs
> >> > ( 10= pageSize) .
> >> > Let's focus on the distributed enviroment and  the manual sharding
> >> > approach.
> >> >
> >> > Currently what happens is that the reranking task is delivered by the
> >> > shards, they rescore the docs and then send them back to the
> aggregator
> >> > node.
> >> >
> >> > If you want to rerank only few docs ( leaving the others with the
> >> original
> >> > score following), this can be done in a single Solr instance ( the
> >> howmany
> >> > logic manages that in the reranker) .
> >> >
> >> > What happens when you move to a distributed environment ?
> >> > The aggregator will aggregate both rescored and original scored
> >> documents,
> >> > making the final ranking inconsistent.
> >> > In the other hand if we make the rarankingDocs threshold dynamic ( to
> >> adapt
> >> > to start+rows) we can incur in the very annoying issue of having a
> >> document
> >> > sliding through the pages ( visible in the first page , then appearing
> >> > again in the third ect ect).
> >> >
> >> > Any thought ?
> >> >
> >> > Cheers
> >> >
> >> > --
> >> > --
> >> >
> >> > Benedetti Alessandro
> >> > Visiting card : http://about.me/alessandro_benedetti
> >> >
> >> > "Tyger, tyger burning bright
> >> > In the forests of the night,
> >> > What immortal hand or eye
> >> > Could frame thy fearful symmetry?"
> >> >
> >> > William Blake - Songs of Experience -1794 England
> >> >
> >>
> >
> >
> >
> > --
> > --
> >
> > Benedetti Alessandro
> > Visiting card : http://about.me/alessandro_benedetti
> >
> > "Tyger, tyger burning bright
> > In the forests of the night,
> > What immortal hand or eye
> > Could frame thy fearful symmetry?"
> >
> > William Blake - Songs of Experience -1794 England
> >
>
>
>
> --
> --
>
> Benedetti Alessandro
> Visiting card : http://about.me/alessandro_benedetti
>
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
>
> William Blake - Songs of Experience -1794 England
>

Re: [Rerank Query] Distributed search + pagination

[Alessandro Benedetti]

In addition to that, I think the only way to solve this is to rely on the
aggregator node to actually re-rank after having aggregated.

Cheer

On Fri, Sep 9, 2016 at 11:48 PM, Alessandro Benedetti  wrote:

> Let me explain further,
> let's assume a simple case when we have 2 shards.
> ReRankDocs =10 , rows=10 .
>
> Correct me if I am wrong Joel,
> What we would like :
> 1 page : top 10 re-scored
> 2 page: remaining 10 re-scored
> From page 3 the original scored docs.
> This is what is happening in a single sol instance if we put reRankDocs to
> 20.
>
> Let's see with sharing :
> To get the first page we get top 10 ( re-scored) from shard1 and top 10
> reranked for shard 2.
> Then the merged top 10 ( re-scored) will be calculated, and that is the
> page 1.
>
> But when we require the page 2, which means we additionally ask now :
> 20 docs to shard1, 10 re-scored and 10 not.
> 20 docs to shard2, 10 re-scored and 10 not.
> At this point we have 40 docs to merge and rank..
> The docs with the original score can go at any position ( not necessarily
> the last 20)
> In the page 2 we can find potentially docs with the original score.
> This is even more likely if the scores are on differente scales (e.g. the
> re-scored 0100 ) .
>
> Am I right ?
> Did I make any wrong assumption so far ?
>
> Cheers
>
>
> On Fri, Sep 9, 2016 at 7:47 PM, Joel Bernstein  wrote:
>
>> I'm not understanding where the inconsistency comes into play.
>>
>> The re-ranking occurs on the shards. The aggregator node will be sent some
>> docs that have been re-scored and others that are not. But the sorting
>> should be the same as someone pages through the result set.
>>
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>> On Fri, Sep 9, 2016 at 9:28 AM, Alessandro Benedetti <
>> abenede...@apache.org>
>> wrote:
>>
>> > Hi guys,
>> > was just experimenting some reranker with really low number of rerank
>> docs
>> > ( 10= pageSize) .
>> > Let's focus on the distributed enviroment and  the manual sharding
>> > approach.
>> >
>> > Currently what happens is that the reranking task is delivered by the
>> > shards, they rescore the docs and then send them back to the aggregator
>> > node.
>> >
>> > If you want to rerank only few docs ( leaving the others with the
>> original
>> > score following), this can be done in a single Solr instance ( the
>> howmany
>> > logic manages that in the reranker) .
>> >
>> > What happens when you move to a distributed environment ?
>> > The aggregator will aggregate both rescored and original scored
>> documents,
>> > making the final ranking inconsistent.
>> > In the other hand if we make the rarankingDocs threshold dynamic ( to
>> adapt
>> > to start+rows) we can incur in the very annoying issue of having a
>> document
>> > sliding through the pages ( visible in the first page , then appearing
>> > again in the third ect ect).
>> >
>> > Any thought ?
>> >
>> > Cheers
>> >
>> > --
>> > --
>> >
>> > Benedetti Alessandro
>> > Visiting card : http://about.me/alessandro_benedetti
>> >
>> > "Tyger, tyger burning bright
>> > In the forests of the night,
>> > What immortal hand or eye
>> > Could frame thy fearful symmetry?"
>> >
>> > William Blake - Songs of Experience -1794 England
>> >
>>
>
>
>
> --
> --
>
> Benedetti Alessandro
> Visiting card : http://about.me/alessandro_benedetti
>
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
>
> William Blake - Songs of Experience -1794 England
>



-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Re: [Rerank Query] Distributed search + pagination

[Alessandro Benedetti]

Let me explain further,
let's assume a simple case when we have 2 shards.
ReRankDocs =10 , rows=10 .

Correct me if I am wrong Joel,
What we would like :
1 page : top 10 re-scored
2 page: remaining 10 re-scored
>From page 3 the original scored docs.
This is what is happening in a single sol instance if we put reRankDocs to
20.

Let's see with sharing :
To get the first page we get top 10 ( re-scored) from shard1 and top 10
reranked for shard 2.
Then the merged top 10 ( re-scored) will be calculated, and that is the
page 1.

But when we require the page 2, which means we additionally ask now :
20 docs to shard1, 10 re-scored and 10 not.
20 docs to shard2, 10 re-scored and 10 not.
At this point we have 40 docs to merge and rank..
The docs with the original score can go at any position ( not necessarily
the last 20)
In the page 2 we can find potentially docs with the original score.
This is even more likely if the scores are on differente scales (e.g. the
re-scored 0100 ) .

Am I right ?
Did I make any wrong assumption so far ?

Cheers


On Fri, Sep 9, 2016 at 7:47 PM, Joel Bernstein  wrote:

> I'm not understanding where the inconsistency comes into play.
>
> The re-ranking occurs on the shards. The aggregator node will be sent some
> docs that have been re-scored and others that are not. But the sorting
> should be the same as someone pages through the result set.
>
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Fri, Sep 9, 2016 at 9:28 AM, Alessandro Benedetti <
> abenede...@apache.org>
> wrote:
>
> > Hi guys,
> > was just experimenting some reranker with really low number of rerank
> docs
> > ( 10= pageSize) .
> > Let's focus on the distributed enviroment and  the manual sharding
> > approach.
> >
> > Currently what happens is that the reranking task is delivered by the
> > shards, they rescore the docs and then send them back to the aggregator
> > node.
> >
> > If you want to rerank only few docs ( leaving the others with the
> original
> > score following), this can be done in a single Solr instance ( the
> howmany
> > logic manages that in the reranker) .
> >
> > What happens when you move to a distributed environment ?
> > The aggregator will aggregate both rescored and original scored
> documents,
> > making the final ranking inconsistent.
> > In the other hand if we make the rarankingDocs threshold dynamic ( to
> adapt
> > to start+rows) we can incur in the very annoying issue of having a
> document
> > sliding through the pages ( visible in the first page , then appearing
> > again in the third ect ect).
> >
> > Any thought ?
> >
> > Cheers
> >
> > --
> > --
> >
> > Benedetti Alessandro
> > Visiting card : http://about.me/alessandro_benedetti
> >
> > "Tyger, tyger burning bright
> > In the forests of the night,
> > What immortal hand or eye
> > Could frame thy fearful symmetry?"
> >
> > William Blake - Songs of Experience -1794 England
> >
>



-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Re: [Rerank Query] Distributed search + pagination

[Joel Bernstein]

I'm not understanding where the inconsistency comes into play.

The re-ranking occurs on the shards. The aggregator node will be sent some
docs that have been re-scored and others that are not. But the sorting
should be the same as someone pages through the result set.



Joel Bernstein
http://joelsolr.blogspot.com/

On Fri, Sep 9, 2016 at 9:28 AM, Alessandro Benedetti 
wrote:

> Hi guys,
> was just experimenting some reranker with really low number of rerank docs
> ( 10= pageSize) .
> Let's focus on the distributed enviroment and  the manual sharding
> approach.
>
> Currently what happens is that the reranking task is delivered by the
> shards, they rescore the docs and then send them back to the aggregator
> node.
>
> If you want to rerank only few docs ( leaving the others with the original
> score following), this can be done in a single Solr instance ( the howmany
> logic manages that in the reranker) .
>
> What happens when you move to a distributed environment ?
> The aggregator will aggregate both rescored and original scored documents,
> making the final ranking inconsistent.
> In the other hand if we make the rarankingDocs threshold dynamic ( to adapt
> to start+rows) we can incur in the very annoying issue of having a document
> sliding through the pages ( visible in the first page , then appearing
> again in the third ect ect).
>
> Any thought ?
>
> Cheers
>
> --
> --
>
> Benedetti Alessandro
> Visiting card : http://about.me/alessandro_benedetti
>
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
>
> William Blake - Songs of Experience -1794 England
>

[Rerank Query] Distributed search + pagination

[Alessandro Benedetti]

Hi guys,
was just experimenting some reranker with really low number of rerank docs
( 10= pageSize) .
Let's focus on the distributed enviroment and  the manual sharding approach.

Currently what happens is that the reranking task is delivered by the
shards, they rescore the docs and then send them back to the aggregator
node.

If you want to rerank only few docs ( leaving the others with the original
score following), this can be done in a single Solr instance ( the howmany
logic manages that in the reranker) .

What happens when you move to a distributed environment ?
The aggregator will aggregate both rescored and original scored documents,
making the final ranking inconsistent.
In the other hand if we make the rarankingDocs threshold dynamic ( to adapt
to start+rows) we can incur in the very annoying issue of having a document
sliding through the pages ( visible in the first page , then appearing
again in the third ect ect).

Any thought ?

Cheers

-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Re: Solr Cloud - how to implement local indexing without SSL and distributed search with SSL

[Shawn Heisey]

On 7/17/2016 8:13 AM, Sarit Weber wrote:
> We noticed that indexing is much faster without SSL, but we can not
> remove it from distributed search. 

Solr doesn't handle the networking.  That's Jetty.  Jetty sets up one
listening port, and that port either uses SSL or it doesn't.  All
requests for Solr are handled by that network setup, which is out of
Solr's control.

To use SSL for one part of the app but not for another part would
require that somebody split Solr into *two* applications, and have jetty
handle each application with a different TCP port.  Then the two
applications would need a way to talk to each other.  I can almost
guarantee that the Solr developers are not going to implement that.

You might want to remove SSL from Solr and have your query clients
access it via a proxy/loadbalancer that handles the SSL.  Because you're
looking for this kind of solution, I imagine that the clients that will
be indexing are very different clients than the ones that are doing the
queries, and that maybe the clients doing the queries are in a network
that you do not trust.

One final word -- Solr should not be accessible from any network
location that you cannot completely trust, especially the Internet.

Thanks,
Shawn

Re: Solr Cloud - how to implement local indexing without SSL and distributed search with SSL

[Sarit Weber]

Hi again,

Do you think it's possible to do that with server that will be dedicate to 
indexing and server that will be dedicate to search but will work on the 
same collections?

Thanks,
Sarit Weber 
Guardium Software Developer
IBM Israel Software Labs, Jerusalem
Phone: +972-2-649-1712
email:  sar...@il.ibm.com 





From:   Sarit Weber/Haifa/IBM@IBMIL
To: solr-user@lucene.apache.org
Date:   17/07/2016 05:21 PM
Subject:Solr Cloud - how to implement local indexing without SSL 
and distributed search with SSL



Hi all,
 
We are currently using Solr 6.0.0, with Solr Cloud and SSL 
where:
1. Collections are defined with router.name=implicit 
2. Collections are built of shard per machine 
3. Data from a specific machine is indexed on that shard and documents 
always keep on coming. 
4. Search is distributed - from each machine we want to search for data on 

the entire collection - using the Solr Cloud API. 
 
We noticed that indexing is much faster without SSL, but we can not remove 

it from distributed search. 
 
Because we force the index to be local (each machine index its own data) 
- 
We were wondering if there is an option to remove SSL from indexing and 
keep using it for Searching.
The solution will have to require the indexing to be done locally, not 
calling the remote zookeeper. 
 
Is there any way to achieve this with Solr Cloud?

Thanks,
Sarit Weber 

Solr Cloud - how to implement local indexing without SSL and distributed search with SSL

[Sarit Weber]

Hi all,
 
We are currently using Solr 6.0.0, with Solr Cloud and SSL 
where:
1. Collections are defined with router.name=implicit 
2. Collections are built of shard per machine 
3. Data from a specific machine is indexed on that shard and documents 
always keep on coming.  
4. Search is distributed - from each machine we want to search for data on 
the entire collection - using the Solr Cloud API. 
 
We noticed that indexing is much faster without SSL, but we can not remove 
it from distributed search. 
 
Because we force the index to be local (each machine index its own data) 
- 
We were wondering if there is an option to remove SSL from indexing and 
keep using it for Searching.
The solution will have to require the indexing to be done locally, not 
calling the remote zookeeper.  
 
Is there any way to achieve this with Solr Cloud?

Thanks,
Sarit Weber 

Solr Cloud - how to implement local indexing without SSL and distributed search with SSL

[Sarit Weber]

Hi all,
 
We are currently using Solr 6.0.0, with Solr Cloud and SSL 
where:
1. Collections are defined with router.name=implicit 
2. Collections are built of shard per machine 
3. Data from a specific machine is indexed on that shard and documents 
always keep on coming.  
4. Search is distributed - from each machine we want to search for data on 
the entire collection - using the Solr Cloud API. 
 
We noticed that indexing is much faster without SSL, but we can not remove 
it from distributed search. 
 
Because we force the index to be local (each machine index its own data) 
- 
We were wondering if there is an option to remove SSL from indexing and 
keep using it for Searching.
The solution will have to require the indexing to be done locally, not 
calling the remote zookeeper.  
 
Is there any way to achieve this with Solr Cloud?


Thanks,
Sarit Weber 

Re: Pivot facets - distributed search - request

[Yonik Seeley]

On Tue, Apr 12, 2016 at 8:47 AM, Pablo  wrote:
> Hi,
> Is there any way of requesting limit 10 order by a stat within facet pivot?

No.

> I know that the "json facet" component can do this and it has a very
> comphrehensive api, but it has a problem of consistency (refinement) when
> querying across multiple shards.

I know a lot of people have been looking for refinements, I'll try to
get to this relatively soon!
In the meantime, one can probably minimize the error somewhat by
requesting a larger limit.

-Yonik

Pivot facets - distributed search - request

[Pablo]

Hi,
Is there any way of requesting limit 10 order by a stat within facet pivot?
I know that the "json facet" component can do this and it has a very
comphrehensive api, but it has a problem of consistency (refinement) when
querying across multiple shards. 
And given that pivot facets supports distributed searching, I tried to make
a similar request, but couldn't find how to do it.
Thanks in advance!



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Pivot-facets-distributed-search-request-tp4269570.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Null Pointer Exception on distributed search

[Shawn Heisey]

On 2/24/2016 9:58 AM, Lokesh Chhaparwal wrote:
> Can someone please update on this exception trace while we are using
> distributed search using shards parameter (solr-master-slave).

The line of code where the NPE happened (from the 4.7.2 source) is in
XMLWriter.java, at line 190:

for (String fname : doc.getFieldNames()) {

This means that "doc" is null.  This variable is pulled out of a
SolrDocumentList object, which means that whatever created that
SolrDocumentList managed to add a null entry.

I ran into a nearly identical stacktrace last year:

https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201509.mbox/%3c55f9e5df.1050...@elyograg.org%3E

I never did get a response to that message on the mailing list, and I
can no longer recall exactly what I did to fix it.  What little I do
remember suggests that this was caused by a situation where the field I
was grouping on was not present in at least one document.  This
situation should have been impossible in our data set, but occasionally
a badly formatted document will result in bad data in the database used
to populate Solr, which causes unexpected behavior.

Is this perhaps a query that uses grouping?  A similar problem might
happen with facets, but I am unsure about that.

Thanks,
Shawn

Re: Null Pointer Exception on distributed search

[Lokesh Chhaparwal]

Hi,

Can someone please update on this exception trace while we are using
distributed search using shards parameter (solr-master-slave).

Thanks,
Lokesh


On Wed, Feb 17, 2016 at 5:33 PM, Lokesh Chhaparwal <xyzlu...@gmail.com>
wrote:

> Hi,
>
> We are facing NPE while using distributed search (Solr version 4.7.2)
> (using *shards* parameter in solr query)
>
> Exception Trace:
> ERROR - 2016-02-17 16:44:26.616; org.apache.solr.common.SolrException;
> null:java.lang.NullPointerException
> at org.apache.solr.response.XMLWriter.writeSolrDocument(XMLWriter.java:190)
> at
> org.apache.solr.response.TextResponseWriter.writeSolrDocumentList(TextResponseWriter.java:222)
> at
> org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:184)
> at org.apache.solr.response.XMLWriter.writeNamedList(XMLWriter.java:227)
> at
> org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:188)
> at org.apache.solr.response.XMLWriter.writeArray(XMLWriter.java:273)
> at
> org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:190)
> at org.apache.solr.response.XMLWriter.writeNamedList(XMLWriter.java:227)
> at
> org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:188)
> at org.apache.solr.response.XMLWriter.writeNamedList(XMLWriter.java:227)
> at
> org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:188)
> at org.apache.solr.response.XMLWriter.writeResponse(XMLWriter.java:111)
> at
> org.apache.solr.response.XMLResponseWriter.write(XMLResponseWriter.java:40)
> at
> org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:756)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:428)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:205)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
> at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Can somebody help us in finding the root cause of this exception?
>
> FYI, we have documents split across 8 shards (num docs ~ 20 million) with
> index size ~ 4 GB per node. We are using c3.2xlarge amazon ec2 machines
> with solr running in apache tomcat (memory config 8 to 10 gb). Request
> count ~ 200/sec.
>
> Thanks,
> Lokesh
>
>

Null Pointer Exception on distributed search

[Lokesh Chhaparwal]

Hi,

We are facing NPE while using distributed search (Solr version 4.7.2)
(using *shards* parameter in solr query)

Exception Trace:
ERROR - 2016-02-17 16:44:26.616; org.apache.solr.common.SolrException;
null:java.lang.NullPointerException
at org.apache.solr.response.XMLWriter.writeSolrDocument(XMLWriter.java:190)
at
org.apache.solr.response.TextResponseWriter.writeSolrDocumentList(TextResponseWriter.java:222)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:184)
at org.apache.solr.response.XMLWriter.writeNamedList(XMLWriter.java:227)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:188)
at org.apache.solr.response.XMLWriter.writeArray(XMLWriter.java:273)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:190)
at org.apache.solr.response.XMLWriter.writeNamedList(XMLWriter.java:227)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:188)
at org.apache.solr.response.XMLWriter.writeNamedList(XMLWriter.java:227)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:188)
at org.apache.solr.response.XMLWriter.writeResponse(XMLWriter.java:111)
at
org.apache.solr.response.XMLResponseWriter.write(XMLResponseWriter.java:40)
at
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:756)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:428)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:205)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)


Can somebody help us in finding the root cause of this exception?

FYI, we have documents split across 8 shards (num docs ~ 20 million) with
index size ~ 4 GB per node. We are using c3.2xlarge amazon ec2 machines
with solr running in apache tomcat (memory config 8 to 10 gb). Request
count ~ 200/sec.

Thanks,
Lokesh

RE: How for distributed search only log collective search response

[Koorosh Vakhshoori]

It turns out there is a better way to do this. It does not require code change 
in Solr, if you are using log4j. However, you need to migrate to log4j.xml file 
format. The solution is to use the filter feature. Here is what my console 
appender looks like with the filter:











Regards,

Koorosh

How for distributed search only log collective search response

[Koorosh Vakhshoori]

  In my use case, I have a number of shards where a query would run as 
distributed search.  I am not using Solr Cloud, I have just a Solr server. Now, 
when the search runs, I see one entry for each shard query as well as the 
finally collective search query response. As the results, I am ending up with a 
very noisy log. I don't care about individual shard queries, just the aggregate 
result. Is there a way to configure Solr so it would only log the final 
collective response? I believe this use case also applies to Solr Cloud.

  Looking at the Solr code, class SolrCore, I see the following lines is 
performing the logging:

if (rsp.getToLog().size() > 0) {
  if (requestLog.isInfoEnabled()) {
requestLog.info(rsp.getToLogAsString(logid));
  }

  I was thinking of adding a flag that filter the distributed logs by looking 
at the 'params' and check for 'isShard=true' and if present don't log it.

  Any suggestion or comment? Is this something people would be interested in?

Regards,

Koorosh

Re: Merging documents from a distributed search

[tedsolr]

Joel,

It needs to perform. Typically users will have 1 - 5 million rows in a
query, returning 10 - 15 fields. Grouping reduces the return by 50% or more
normally. Responses tend be less than a half second.

It sounds like the manipulation of docs at the collector level has been left
to the single solr node implementations, and that your streaming API is the
way forward for cloud implementations. Even if it does have some performance
drawbacks. I can bear slower searches as long as they are not seconds
slower.

I could implement some business strategy that forks searching to either the
AnalyticsQuery or the streaming API based on the shard count in the
collection. Most of my customers will have single shard collections. A goal
of mine is to keep each collection whole as long as possible. If one gets
too big for the pond I'll move it to a bigger pond, until some heap limit is
reached when it will have to be split. 



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Merging-documents-from-a-distributed-search-tp4226802p4227595.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Merging documents from a distributed search

[Joel Bernstein]

It's possible that the ReducerStream's buffer can grow too large if
document groups are very large. But the ReducerStream only needs to hold
one group at a time in memory. The RollupStream, in trunk, has a grouping
implementation that doesn't hang on to all the Tuples from a group. You
could also implement a custom stream that does exactly what you need.

The AnalyicsQuery is much more efficient because the data is left in place.
The Streaming API has streaming overhead which is considerable. But it's
the Stream "shuffling" that gives you the power to do things like fully
distributed grouping.

How many records are processed in a typical query and what type of response
time do you need?

Joel Bernstein
http://joelsolr.blogspot.com/

On Thu, Sep 3, 2015 at 3:25 PM, tedsolr <tsm...@sciquest.com> wrote:

> Thanks Joel, that link looks promising. The CloudSolrStream bypasses my
> issue
> of multiple shards. Perhaps the ReducerStream would provide what I need. At
> first glance I worry that the the buffer would grow too large - if its
> really holding the values for all the fields in each document
> (Tuple.getMaps()). I use a Map in my DelegatingCollector to store the
> "unique" docs, but I only keep the docId, my stats, and the ordinals for
> each field. Would you expect the new streams API to perform as well as my
> implementation of an AnalyticsQuery and a DelegatingCollector?
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Merging-documents-from-a-distributed-search-tp4226802p4227034.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Merging documents from a distributed search

[tedsolr]

Upayavira ,

The docs are all unique. In my example the two docs are considered to be
dupes because the requested fields all have the same values.
fields   AB   C   D E
Doc 1: apple, 10, 15, bye, yellow
Doc 2: apple, 12, 15, by, green

The two docs are certainly unique. Say they are on different shards in the
same collection. If the search request has fl:A,C then the two are dupes and
the user wants to see them collapsed. If the search request has fl:A,B,C
then the two are unique from the user's perspective and display separately.

Each doc typically has a couple hundred fields. When viewed through the lens
of just 3 or 4 fields, lots of docs, sometimes 1000s will be rolled up and
I'll compute some stats on that group. Bringing all those docs back to the
calling app for processing is too slow. The AnalyticsQuery does a great job
of filtering out the dupes, but it looks like I need another solution for
multi shard collections.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Merging-documents-from-a-distributed-search-tp4226802p4227261.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Merging documents from a distributed search

[Markus Jelsma]

Hello - another current topic is also covering this issue, you may want to 
check it out:
http://lucene.472066.n3.nabble.com/Merging-documents-from-a-distributed-search-td4226802.html

 
 
-Original message-
> From:Markus Jelsma <markus.jel...@openindex.io>
> Sent: Thursday 3rd September 2015 10:27
> To: solr-user@lucene.apache.org
> Subject: RE: Merging documents from a distributed search
> 
> Hello - We're doing something similar ended up overriding QueryComponent 
> (https://issues.apache.org/jira/browse/SOLR-7968) which needs protected 
> members instead of private members first. We could do a RankQuery and use its 
> cool MergeStrategy, but we would also ened RankQuery to provide an entry 
> point for QueryComponent.createMainQuery(). That would be ideal because we 
> can then use the Collector there for local deduplication, and a combination 
> of createMainQuery and mergeIds to do the distributed deduplication.
> 
> Markus
>  
> -Original message-
> > From:Joel Bernstein <joels...@gmail.com>
> > Sent: Wednesday 2nd September 2015 23:46
> > To: solr-user@lucene.apache.org
> > Subject: Re: Merging documents from a distributed search
> > 
> > The merge strategy probably won't work for the type of distributed collapse
> > you're describing.
> > 
> > You may want to begin exploring the Streaming API which supports real-time
> > map/reduce operations,
> > 
> > http://joelsolr.blogspot.com/2015/03/parallel-computing-with-solrcloud.html
> > 
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> > 
> > On Wed, Sep 2, 2015 at 5:12 PM, tedsolr <tsm...@sciquest.com> wrote:
> > 
> > > I've read from  http://heliosearch.org/solrs-mergestrategy/
> > > <http://heliosearch.org/solrs-mergestrategy/>   that the AnalyticsQuery
> > > component only works for a single instance of Solr. I'm planning to
> > > "migrate" to the SolrCloud soon and I have a custom AnalyticsQuery module
> > > that collapses what I consider to be duplicate documents, keeping stats
> > > like
> > > a "count" of the dupes. For my purposes "dupes" are determined at run time
> > > and vary by the search request. Once a collection has multiple shards I
> > > will
> > > not be able to prevent "dupes" from appearing across those shards. A 
> > > custom
> > > merge strategy should allow me to merge my stats, but I don't see how I 
> > > can
> > > drop duplicate docs at that point.
> > >
> > > If shard1 returns docs A & B and shard2 returns docs B & C (letters
> > > denoting
> > > what I consider to be unique docs), can my implementation of a merge
> > > strategy return only docs A, B, & C, rather than A, B, B, & C?
> > >
> > > thanks!
> > > solr 5.2.1
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > > http://lucene.472066.n3.nabble.com/Merging-documents-from-a-distributed-search-tp4226802.html
> > > Sent from the Solr - User mailing list archive at Nabble.com.
> > >
> > 
> 

RE: Merging documents from a distributed search

[tedsolr]

Markus, did you mistakingly post a link to this same thread?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Merging-documents-from-a-distributed-search-tp4226802p4227035.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Merging documents from a distributed search

[tedsolr]

Thanks Joel, that link looks promising. The CloudSolrStream bypasses my issue
of multiple shards. Perhaps the ReducerStream would provide what I need. At
first glance I worry that the the buffer would grow too large - if its
really holding the values for all the fields in each document
(Tuple.getMaps()). I use a Map in my DelegatingCollector to store the
"unique" docs, but I only keep the docId, my stats, and the ordinals for
each field. Would you expect the new streams API to perform as well as my
implementation of an AnalyticsQuery and a DelegatingCollector?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Merging-documents-from-a-distributed-search-tp4226802p4227034.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Merging documents from a distributed search

[Upayavira]

On Wed, Sep 2, 2015, at 10:12 PM, tedsolr wrote:
> I've read from  http://heliosearch.org/solrs-mergestrategy/
>    that the AnalyticsQuery
> component only works for a single instance of Solr. I'm planning to
> "migrate" to the SolrCloud soon and I have a custom AnalyticsQuery module
> that collapses what I consider to be duplicate documents, keeping stats
> like
> a "count" of the dupes. For my purposes "dupes" are determined at run
> time
> and vary by the search request. Once a collection has multiple shards I
> will
> not be able to prevent "dupes" from appearing across those shards. A
> custom
> merge strategy should allow me to merge my stats, but I don't see how I
> can
> drop duplicate docs at that point.
> 
> If shard1 returns docs A & B and shard2 returns docs B & C (letters
> denoting
> what I consider to be unique docs), can my implementation of a merge
> strategy return only docs A, B, & C, rather than A, B, B, & C?

How did you end up with document B in both shard1 and shard2? Can't you
prevent that from happening, and thus not have this issue?

Upayavira

RE: Merging documents from a distributed search

[Markus Jelsma]

Hello - We're doing something similar ended up overriding QueryComponent 
(https://issues.apache.org/jira/browse/SOLR-7968) which needs protected members 
instead of private members first. We could do a RankQuery and use its cool 
MergeStrategy, but we would also ened RankQuery to provide an entry point for 
QueryComponent.createMainQuery(). That would be ideal because we can then use 
the Collector there for local deduplication, and a combination of 
createMainQuery and mergeIds to do the distributed deduplication.

Markus
 
-Original message-
> From:Joel Bernstein <joels...@gmail.com>
> Sent: Wednesday 2nd September 2015 23:46
> To: solr-user@lucene.apache.org
> Subject: Re: Merging documents from a distributed search
> 
> The merge strategy probably won't work for the type of distributed collapse
> you're describing.
> 
> You may want to begin exploring the Streaming API which supports real-time
> map/reduce operations,
> 
> http://joelsolr.blogspot.com/2015/03/parallel-computing-with-solrcloud.html
> 
> Joel Bernstein
> http://joelsolr.blogspot.com/
> 
> On Wed, Sep 2, 2015 at 5:12 PM, tedsolr <tsm...@sciquest.com> wrote:
> 
> > I've read from  http://heliosearch.org/solrs-mergestrategy/
> > <http://heliosearch.org/solrs-mergestrategy/>   that the AnalyticsQuery
> > component only works for a single instance of Solr. I'm planning to
> > "migrate" to the SolrCloud soon and I have a custom AnalyticsQuery module
> > that collapses what I consider to be duplicate documents, keeping stats
> > like
> > a "count" of the dupes. For my purposes "dupes" are determined at run time
> > and vary by the search request. Once a collection has multiple shards I
> > will
> > not be able to prevent "dupes" from appearing across those shards. A custom
> > merge strategy should allow me to merge my stats, but I don't see how I can
> > drop duplicate docs at that point.
> >
> > If shard1 returns docs A & B and shard2 returns docs B & C (letters
> > denoting
> > what I consider to be unique docs), can my implementation of a merge
> > strategy return only docs A, B, & C, rather than A, B, B, & C?
> >
> > thanks!
> > solr 5.2.1
> >
> >
> >
> > --
> > View this message in context:
> > http://lucene.472066.n3.nabble.com/Merging-documents-from-a-distributed-search-tp4226802.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
> 

RE: Merging documents from a distributed search

[Markus Jelsma]

It seems so indeed. Please look up the thread titled "Custom merge logic in 
SolrCloud."   

 
 
-Original message-
> From:tedsolr <tsm...@sciquest.com>
> Sent: Thursday 3rd September 2015 21:28
> To: solr-user@lucene.apache.org
> Subject: RE: Merging documents from a distributed search
> 
> Markus, did you mistakingly post a link to this same thread?
> 
> 
> 
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Merging-documents-from-a-distributed-search-tp4226802p4227035.html
> Sent from the Solr - User mailing list archive at Nabble.com.
> 

Merging documents from a distributed search

[tedsolr]

I've read from  http://heliosearch.org/solrs-mergestrategy/
<http://heliosearch.org/solrs-mergestrategy/>   that the AnalyticsQuery
component only works for a single instance of Solr. I'm planning to
"migrate" to the SolrCloud soon and I have a custom AnalyticsQuery module
that collapses what I consider to be duplicate documents, keeping stats like
a "count" of the dupes. For my purposes "dupes" are determined at run time
and vary by the search request. Once a collection has multiple shards I will
not be able to prevent "dupes" from appearing across those shards. A custom
merge strategy should allow me to merge my stats, but I don't see how I can
drop duplicate docs at that point.

If shard1 returns docs A & B and shard2 returns docs B & C (letters denoting
what I consider to be unique docs), can my implementation of a merge
strategy return only docs A, B, & C, rather than A, B, B, & C?

thanks! 
solr 5.2.1



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Merging-documents-from-a-distributed-search-tp4226802.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Merging documents from a distributed search

[Joel Bernstein]

The merge strategy probably won't work for the type of distributed collapse
you're describing.

You may want to begin exploring the Streaming API which supports real-time
map/reduce operations,

http://joelsolr.blogspot.com/2015/03/parallel-computing-with-solrcloud.html

Joel Bernstein
http://joelsolr.blogspot.com/

On Wed, Sep 2, 2015 at 5:12 PM, tedsolr <tsm...@sciquest.com> wrote:

> I've read from  http://heliosearch.org/solrs-mergestrategy/
> <http://heliosearch.org/solrs-mergestrategy/>   that the AnalyticsQuery
> component only works for a single instance of Solr. I'm planning to
> "migrate" to the SolrCloud soon and I have a custom AnalyticsQuery module
> that collapses what I consider to be duplicate documents, keeping stats
> like
> a "count" of the dupes. For my purposes "dupes" are determined at run time
> and vary by the search request. Once a collection has multiple shards I
> will
> not be able to prevent "dupes" from appearing across those shards. A custom
> merge strategy should allow me to merge my stats, but I don't see how I can
> drop duplicate docs at that point.
>
> If shard1 returns docs A & B and shard2 returns docs B & C (letters
> denoting
> what I consider to be unique docs), can my implementation of a merge
> strategy return only docs A, B, & C, rather than A, B, B, & C?
>
> thanks!
> solr 5.2.1
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Merging-documents-from-a-distributed-search-tp4226802.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

RE: Problem with distributed search using grouping and highlighting

[Cario, Elaine]

Rich,

I've run into various problems with group.query and highlighting.  You noted 
one below (SOLR-5046), and there is also SOLR-6712, which might be related to 
what you are experiencing.  Still waiting for that patch to be reviewed...

-Original Message-
From: Rich Hume [mailto:rh...@identifix.com] 
Sent: Monday, June 08, 2015 2:23 PM
To: solr-user@lucene.apache.org
Subject: Problem with distributed search using grouping and highlighting

I am currently using Solr 4.5.1.  In the hopes of seeing better query 
performance, I have sharded an index and I am trying to use the shards 
parameter along with grouping and highlighting.  I am not currently using Solr 
cloud.

I got past an earlier problem by adding a second sort parameter (as described 
in JIRA Solr-5046).  Unfortunately, I have found nothing related to my latest 
index out of bounds problem.  I do not believe that JIRA Solr-5709 is related 
since my unique keys are in fact unique across the shards.

If anyone can point out something that I am doing wrong it would be greatly 
appreciated.

Thanks,
Rich

I am seeing the following error, the parameters I am passing are below the 
stack trace.

null:java.lang.ArrayIndexOutOfBoundsException: 35
 at 
org.apache.solr.handler.component.HighlightComponent.finishStage(HighlightComponent.java:185)
 at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:317)
 at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
 at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
 at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703)
 at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:406)
 at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)


Here are the parameters I am passing:

group=truegroup.offset=0group.limit=10group.field=DeDup
group.query=DocumentTypes:34group.query=DocumentTypes:35group.query=DocumentTypes:32
shards=localhost:8983/solr/IX1,localhost:8983/solr/IX2
fq=+DocumentTypes:(34 35 32)
defType=edismaxqf=csTitle^100 csContent q=any matching search string
start=0rows=10
fl=PageNumber,FilePath,DocumentGUID,ResultDisplayContent,DocumentTypes
sort=score desc,DocumentGUID asc
hl=on
hl.fl=csTitle,csContent

Distributed Search component question

[Mihran Shahinian]

Hi all,
I have the following search components that I don't have a solution at the
moment to get them working in distributed mode on solr 4.10.4.

[standard query component]
[search component-1] (StageID - 2500):
 handleResponses: get few values from docs and populate parameters for
stats component and set some metadata in the ResponseBuilder
  rb.rsp.add(metadata, NamedList...)

distributedProcess:
   rb.doFacets=false;
   if (rb.stage  StageID)
  if( null == rb.rsp[metadata] ) {
   return StageID;
   }
return component-2.StageID

[search component-2] (StageID - 2800):
distributedProcess:
   rb.doFacets=true;
   formatAndSet some facetParams based on rb.rsp[metadata]
   return ResponseBuilder.STAGE_GET_FIELDS

[standard facet component]:


Things seem to work fine between component-1 and component-2, I just can't
prevent facets from running
until component-2 sets proper facet params. And than facet component sets
the rb._facetInfo to null. Should I move my logic in component-2 from
distributeProcess to handleResponses and modify ShardRequest and set
rb.addRequest?

Any hints are much appreciated.
Mihran

Problem with distributed search using grouping and highlighting

[Rich Hume]

I am currently using Solr 4.5.1.  In the hopes of seeing better query 
performance, I have sharded an index and I am trying to use the shards 
parameter along with grouping and highlighting.  I am not currently using Solr 
cloud.

I got past an earlier problem by adding a second sort parameter (as described 
in JIRA Solr-5046).  Unfortunately, I have found nothing related to my latest 
index out of bounds problem.  I do not believe that JIRA Solr-5709 is related 
since my unique keys are in fact unique across the shards.

If anyone can point out something that I am doing wrong it would be greatly 
appreciated.

Thanks,
Rich

I am seeing the following error, the parameters I am passing are below the 
stack trace.

null:java.lang.ArrayIndexOutOfBoundsException: 35
 at 
org.apache.solr.handler.component.HighlightComponent.finishStage(HighlightComponent.java:185)
 at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:317)
 at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
 at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
 at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703)
 at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:406)
 at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)


Here are the parameters I am passing:

group=truegroup.offset=0group.limit=10group.field=DeDup
group.query=DocumentTypes:34group.query=DocumentTypes:35group.query=DocumentTypes:32
shards=localhost:8983/solr/IX1,localhost:8983/solr/IX2
fq=+DocumentTypes:(34 35 32)
defType=edismaxqf=csTitle^100 csContent
q=any matching search string
start=0rows=10
fl=PageNumber,FilePath,DocumentGUID,ResultDisplayContent,DocumentTypes
sort=score desc,DocumentGUID asc
hl=on
hl.fl=csTitle,csContent

Re: distributed search limitations via SolrCloud

[Erick Erickson]

5.x will still build a war file that you an deploy on Tomcat. But
support for that is going away eventually, certainly by 6.0. But you
do have to make the decision sometime before 6.0 at least.

Best,
Erick

On Wed, May 27, 2015 at 1:24 PM, Vishal Swaroop vishal@gmail.com wrote:
 Thanks a lot Erick... great inputs...

 Currently our deployment is on Tomcat 7 and I think SOLR 5.x does not
 support Tomcat but runs on its own Jetty server, right ?
 I will discuss this with the team.

 Thanks again.

 Regards
 Vishal

 On Wed, May 27, 2015 at 4:16 PM, Erick Erickson erickerick...@gmail.com
 wrote:

 I'd move to Solr 4.10.3 at least, but preferably Solr 5.x. Solr 5.2 is
 being readied for release as we speak, it'll probably be available in
 a week or so barring unforeseen problems and that's the one I'd go
 with by preference.

 Do be aware, though, that the 5.x Solr world deprecates using a war
 file. It's still actually produced, but Solr is moving towards start
 scripts instead. This is something new to get used to. See:
 https://wiki.apache.org/solr/WhyNoWar

 Best,
 Erick

 On Wed, May 27, 2015 at 12:51 PM, Vishal Swaroop vishal@gmail.com
 wrote:
  Thanks a lot Erick... You are right we should not delay moving to
  sharding/SolrCloud process.
 
  As you all are expert... currently we are using SOLR 4.7.. Do you suggest
  we should move to latest SOLR release 5.1.0 ? or we can manage the above
  issue using SOLR 4.7
 
  Regards
  Vishal
 
  On Wed, May 27, 2015 at 2:21 PM, Erick Erickson erickerick...@gmail.com
 
  wrote:
 
  Hard to say. I've seen 20M doc be the place you need to consider
  sharding/SolrCloud. I've seen 300M docs be the place you need to start
  sharding. That said I'm quite sure you'll need to shard before you get
  to 2B. There's no good reason to delay that process.
 
  You'll have to do something about the join issue though, that's the
  problem you might want to solve first. The new streaming aggregation
  stuff might help there, you'll have to figure that out.
 
  The first thing I'd explore is whether you can denormlized your way
  out of the need to join. Or whether you can use block joins instead.
 
  Best,
  Erick
 
  On Wed, May 27, 2015 at 11:15 AM, Vishal Swaroop vishal@gmail.com
  wrote:
   Currently, we have SOLR configured on single linux server (24 GB
 physical
   memory) with multiple cores.
   We are using SOLR joins (https://wiki.apache.org/solr/Join) across
  cores on
   this single server.
  
   But, as data will grow to ~2 billion we need to assess whether we’ll
 need
   to run SolrCloud as In a DistributedSearch environment, you can not
 Join
   across cores on multiple nodes
  
   Please suggest at what point or index size should we start
 considering to
   run SolrCloud ?
  
   Regards
 

distributed search limitations via SolrCloud

[Vishal Swaroop]

Currently, we have SOLR configured on single linux server (24 GB physical
memory) with multiple cores.
We are using SOLR joins (https://wiki.apache.org/solr/Join) across cores on
this single server.

But, as data will grow to ~2 billion we need to assess whether we’ll need
to run SolrCloud as In a DistributedSearch environment, you can not Join
across cores on multiple nodes

Please suggest at what point or index size should we start considering to
run SolrCloud ?

Regards

Re: distributed search limitations via SolrCloud

[Erick Erickson]

Hard to say. I've seen 20M doc be the place you need to consider
sharding/SolrCloud. I've seen 300M docs be the place you need to start
sharding. That said I'm quite sure you'll need to shard before you get
to 2B. There's no good reason to delay that process.

You'll have to do something about the join issue though, that's the
problem you might want to solve first. The new streaming aggregation
stuff might help there, you'll have to figure that out.

The first thing I'd explore is whether you can denormlized your way
out of the need to join. Or whether you can use block joins instead.

Best,
Erick

On Wed, May 27, 2015 at 11:15 AM, Vishal Swaroop vishal@gmail.com wrote:
 Currently, we have SOLR configured on single linux server (24 GB physical
 memory) with multiple cores.
 We are using SOLR joins (https://wiki.apache.org/solr/Join) across cores on
 this single server.

 But, as data will grow to ~2 billion we need to assess whether we’ll need
 to run SolrCloud as In a DistributedSearch environment, you can not Join
 across cores on multiple nodes

 Please suggest at what point or index size should we start considering to
 run SolrCloud ?

 Regards

Re: distributed search limitations via SolrCloud

[Vishal Swaroop]

Thanks a lot Erick... You are right we should not delay moving to
sharding/SolrCloud process.

As you all are expert... currently we are using SOLR 4.7.. Do you suggest
we should move to latest SOLR release 5.1.0 ? or we can manage the above
issue using SOLR 4.7

Regards
Vishal

On Wed, May 27, 2015 at 2:21 PM, Erick Erickson erickerick...@gmail.com
wrote:

 Hard to say. I've seen 20M doc be the place you need to consider
 sharding/SolrCloud. I've seen 300M docs be the place you need to start
 sharding. That said I'm quite sure you'll need to shard before you get
 to 2B. There's no good reason to delay that process.

 You'll have to do something about the join issue though, that's the
 problem you might want to solve first. The new streaming aggregation
 stuff might help there, you'll have to figure that out.

 The first thing I'd explore is whether you can denormlized your way
 out of the need to join. Or whether you can use block joins instead.

 Best,
 Erick

 On Wed, May 27, 2015 at 11:15 AM, Vishal Swaroop vishal@gmail.com
 wrote:
  Currently, we have SOLR configured on single linux server (24 GB physical
  memory) with multiple cores.
  We are using SOLR joins (https://wiki.apache.org/solr/Join) across
 cores on
  this single server.
 
  But, as data will grow to ~2 billion we need to assess whether we’ll need
  to run SolrCloud as In a DistributedSearch environment, you can not Join
  across cores on multiple nodes
 
  Please suggest at what point or index size should we start considering to
  run SolrCloud ?
 
  Regards

Re: distributed search limitations via SolrCloud

[Erick Erickson]

I'd move to Solr 4.10.3 at least, but preferably Solr 5.x. Solr 5.2 is
being readied for release as we speak, it'll probably be available in
a week or so barring unforeseen problems and that's the one I'd go
with by preference.

Do be aware, though, that the 5.x Solr world deprecates using a war
file. It's still actually produced, but Solr is moving towards start
scripts instead. This is something new to get used to. See:
https://wiki.apache.org/solr/WhyNoWar

Best,
Erick

On Wed, May 27, 2015 at 12:51 PM, Vishal Swaroop vishal@gmail.com wrote:
 Thanks a lot Erick... You are right we should not delay moving to
 sharding/SolrCloud process.

 As you all are expert... currently we are using SOLR 4.7.. Do you suggest
 we should move to latest SOLR release 5.1.0 ? or we can manage the above
 issue using SOLR 4.7

 Regards
 Vishal

 On Wed, May 27, 2015 at 2:21 PM, Erick Erickson erickerick...@gmail.com
 wrote:

 Hard to say. I've seen 20M doc be the place you need to consider
 sharding/SolrCloud. I've seen 300M docs be the place you need to start
 sharding. That said I'm quite sure you'll need to shard before you get
 to 2B. There's no good reason to delay that process.

 You'll have to do something about the join issue though, that's the
 problem you might want to solve first. The new streaming aggregation
 stuff might help there, you'll have to figure that out.

 The first thing I'd explore is whether you can denormlized your way
 out of the need to join. Or whether you can use block joins instead.

 Best,
 Erick

 On Wed, May 27, 2015 at 11:15 AM, Vishal Swaroop vishal@gmail.com
 wrote:
  Currently, we have SOLR configured on single linux server (24 GB physical
  memory) with multiple cores.
  We are using SOLR joins (https://wiki.apache.org/solr/Join) across
 cores on
  this single server.
 
  But, as data will grow to ~2 billion we need to assess whether we’ll need
  to run SolrCloud as In a DistributedSearch environment, you can not Join
  across cores on multiple nodes
 
  Please suggest at what point or index size should we start considering to
  run SolrCloud ?
 
  Regards

Re: distributed search on tables

[avinash09]

thanks Erick



--
View this message in context: 
http://lucene.472066.n3.nabble.com/distributed-search-on-tables-tp4197456p4198285.html
Sent from the Solr - User mailing list archive at Nabble.com.

distributed search on tables

[avinash09]

Hi,

I have a use case search all the name=*test* from two tables (product and
department)
i need distributed result 5 result from product and 5 from department

but i am getting first all result from which is 434 and then department as
shared below

http://localhost:8983/solr/test_core/select?q=name:*test*wt=jsonfacet=truefacet.field=table_name


{
responseHeader: {
status: 0,
QTime: 19
},
response: {
numFound: 569,
start: 0,
docs: [
{
name: test 6576,
table_name: product
},
{
name: test 6578,
table_name: product
},
{
name: test 65760,
table_name: product
},
{
name: test 657699,
table_name: product
},
{
name: test 657699,
table_name: product
},
{
name: test 657666,
table_name: product
},
{
name: test 657689,
table_name: product
},
{
name: test 6576yu,
table_name: product
},
{
name: test 657607,
table_name: product
},
{
name: test 657687,
table_name: product
}
]
},
facet_counts: {
facet_queries: { },
facet_fields: {
table_name: [
product,
434,
dealer,
135
]
},
facet_dates: { },
facet_ranges: { }
}
}

please help me



--
View this message in context: 
http://lucene.472066.n3.nabble.com/distributed-search-on-tables-tp4197456.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: distributed search on tables

[Erick Erickson]

You can do what you want either by using two queries or using
grouping/field collapsing.

Best,
Erick

On Fri, Apr 3, 2015 at 8:03 AM, avinash09 avinash.i...@gmail.com wrote:
 Hi,

 I have a use case search all the name=*test* from two tables (product and
 department)
 i need distributed result 5 result from product and 5 from department

 but i am getting first all result from which is 434 and then department as
 shared below

 http://localhost:8983/solr/test_core/select?q=name:*test*wt=jsonfacet=truefacet.field=table_name


 {
 responseHeader: {
 status: 0,
 QTime: 19
 },
 response: {
 numFound: 569,
 start: 0,
 docs: [
 {
 name: test 6576,
 table_name: product
 },
 {
 name: test 6578,
 table_name: product
 },
 {
 name: test 65760,
 table_name: product
 },
 {
 name: test 657699,
 table_name: product
 },
 {
 name: test 657699,
 table_name: product
 },
 {
 name: test 657666,
 table_name: product
 },
 {
 name: test 657689,
 table_name: product
 },
 {
 name: test 6576yu,
 table_name: product
 },
 {
 name: test 657607,
 table_name: product
 },
 {
 name: test 657687,
 table_name: product
 }
 ]
 },
 facet_counts: {
 facet_queries: { },
 facet_fields: {
 table_name: [
 product,
 434,
 dealer,
 135
 ]
 },
 facet_dates: { },
 facet_ranges: { }
 }
 }

 please help me



 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/distributed-search-on-tables-tp4197456.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Distributed Search returns Empty document list

[Jaikit Savla]

Hello,
I am running Solr (4.10) in cloud mode by configuring multiple collections (1 
for each day). Structure is as shown below. I can fetch documents for given 
query, if I query individual collection. However when I send distributed 
request to multiple shards, I only see numFound and no documents returned. 
Appreciate any pointers on setup. 

--- directory structure:solrcollection1   //(does 
not have any index)collection_20150112collection_20150113
Command to run Solr:sh bin/solr restart -d example -cloud -p  -noprompt

Set up RequestHandler called alias in solrconfig.xml of collecton1
  requestHandler name=/alias class=solr.StandardRequestHandler 
default=true
 lst name=defaults
   str name=echoParamsexplicit/str
   str name=wtjson/str
   str name=indenttrue/str
   str name=dftext/str
   str name=flscore,*/str
   str 
name=shardshttp://localhost:/solr/collection_20150113,http://localhost:/solr/collection_20150112/str
 /lst


http://localhost:/solr/collection1/alias?q=domain:comdebug=falseshard.info=true
e.g{   
   - responseHeader: {  
  - status: 0,
  - QTime: 19,
  - params: { 
 - q: domain:com,
 - debug: false,
 - shard.info: true
}
},
   - response: {  
  - numFound: 11696,
  - start: 0,
  - maxScore: 1.3015664,
  - docs: [ ]
}
}

Thanks

Re: Distributed search across Solr cores in a collection - NPE

[Jaikit Savla]

It was because I did not have unique id's in my index. I added that and it 
worked. Also it is mentioned as one of the requirement for Distributed Search.
Thanks,Jaikit

 

 On Wednesday, January 14, 2015 1:53 AM, Jaikit Savla 
jaikit.sa...@yahoo.com wrote:
   

 Folks,
I have set up 3 cores in a single collection and they all have same schema but 
different index. I have set unique Id required field to false.field name=id 
type=string indexed=true stored=true required=false/
When I run query against single core, it works fine. But when I add the shard 
param and point to different core than request fails with NPE. I looked up on 
the source code for QueryComponent and line 1043 
isresultIds.put(shardDoc.id.toString(), shardDoc);
looks like the the shardDoc id.toString() is throwing 
NPE.http://grepcode.com/file/repo1.maven.org/maven2/org.apache.solr/solr-core/4.10.1/org/apache/solr/handler/component/QueryComponent.java#QueryComponent.mergeIds%28org.apache.solr.handler.component.ResponseBuilder%2Corg.apache.solr.handler.component.ShardRequest%29
Any clue on if my set up is incorrect ?

 
http://localhost:/solr/core0/select?shards=localhost:/solr/core1q=title:amazonfl=*rows=10wt=json

RESPONSE:   
{responseHeader:{status:500,QTime:11,params:{fl:*,shards:localhost:/solr/core1,q:domain:amazon,wt:json,rows:10}},error:{trace:java.lang.NullPointerException\n\tat
 
org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:1043)\n\tat
 
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:716)\n\tat
 
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:695)\n\tat
 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:324)\n\tat
 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)\n\tat
 org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)\n\tat 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)\n\tat
 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)\n\tat
 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)\n\tat
 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)\n\tat
 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)\n\tat
 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)\n\tat
 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)\n\tat
 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)\n\tat
 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)\n\tat
 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)\n\tat 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)\n\tat
 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)\n\tat
 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)\n\tat
 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)\n\tat
 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)\n\tat
 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)\n\tat
 org.eclipse.jetty.server.Server.handle(Server.java:368)\n\tat 
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)\n\tat
 
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)\n\tat
 
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)\n\tat
 
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)\n\tat
 org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)\n\tat 
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)\n\tat 
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)\n\tat
 
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)\n\tat
 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)\n\tat
 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)\n\tat
 java.lang.Thread.run(Thread.java:745)\n,code:500}}
Appreciate any pointers.
Thanks,Jaikit


   

Distributed search across Solr cores in a collection - NPE

[Jaikit Savla]

Folks,
I have set up 3 cores in a single collection and they all have same schema but 
different index. I have set unique Id required field to false.field name=id 
type=string indexed=true stored=true required=false/
When I run query against single core, it works fine. But when I add the shard 
param and point to different core than request fails with NPE. I looked up on 
the source code for QueryComponent and line 1043 
isresultIds.put(shardDoc.id.toString(), shardDoc);
looks like the the shardDoc id.toString() is throwing 
NPE.http://grepcode.com/file/repo1.maven.org/maven2/org.apache.solr/solr-core/4.10.1/org/apache/solr/handler/component/QueryComponent.java#QueryComponent.mergeIds%28org.apache.solr.handler.component.ResponseBuilder%2Corg.apache.solr.handler.component.ShardRequest%29
Any clue on if my set up is incorrect ?

 
http://localhost:/solr/core0/select?shards=localhost:/solr/core1q=title:amazonfl=*rows=10wt=json

RESPONSE:   
{responseHeader:{status:500,QTime:11,params:{fl:*,shards:localhost:/solr/core1,q:domain:amazon,wt:json,rows:10}},error:{trace:java.lang.NullPointerException\n\tat
 
org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:1043)\n\tat
 
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:716)\n\tat
 
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:695)\n\tat
 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:324)\n\tat
 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)\n\tat
 org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)\n\tat 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)\n\tat
 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)\n\tat
 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)\n\tat
 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)\n\tat
 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)\n\tat
 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)\n\tat
 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)\n\tat
 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)\n\tat
 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)\n\tat
 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)\n\tat 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)\n\tat
 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)\n\tat
 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)\n\tat
 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)\n\tat
 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)\n\tat
 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)\n\tat
 org.eclipse.jetty.server.Server.handle(Server.java:368)\n\tat 
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)\n\tat
 
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)\n\tat
 
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)\n\tat
 
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)\n\tat
 org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)\n\tat 
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)\n\tat 
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)\n\tat
 
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)\n\tat
 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)\n\tat
 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)\n\tat
 java.lang.Thread.run(Thread.java:745)\n,code:500}}
Appreciate any pointers.
Thanks,Jaikit

Re: Distributed search across Solr cores in a collection - NPE

[Mikhail Khludnev]

Jaikit,
uniq key is mandatory for distributed search. if most of your docs have ids
assigned, you can drop remaining ones by adding something like ..fq=id:[*
TO *]

On Wed, Jan 14, 2015 at 12:53 PM, Jaikit Savla 
jaikit.sa...@yahoo.com.invalid wrote:

 Folks,
 I have set up 3 cores in a single collection and they all have same schema
 but different index. I have set unique Id required field to false.field
 name=id type=string indexed=true stored=true required=false/
 When I run query against single core, it works fine. But when I add the
 shard param and point to different core than request fails with NPE. I
 looked up on the source code for QueryComponent and line 1043
 isresultIds.put(shardDoc.id.toString(), shardDoc);
 looks like the the shardDoc id.toString() is throwing NPE.
 http://grepcode.com/file/repo1.maven.org/maven2/org.apache.solr/solr-core/4.10.1/org/apache/solr/handler/component/QueryComponent.java#QueryComponent.mergeIds%28org.apache.solr.handler.component.ResponseBuilder%2Corg.apache.solr.handler.component.ShardRequest%29
 Any clue on if my set up is incorrect ?


 http://localhost:/solr/core0/select?shards=localhost:/solr/core1q=title:amazonfl=*rows=10wt=json

 RESPONSE:
  
 {responseHeader:{status:500,QTime:11,params:{fl:*,shards:localhost:/solr/core1,q:domain:amazon,wt:json,rows:10}},error:{trace:java.lang.NullPointerException\n\tat
 org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:1043)\n\tat
 org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:716)\n\tat
 org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:695)\n\tat
 org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:324)\n\tat
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)\n\tat
 org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)\n\tat
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)\n\tat
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)\n\tat
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)\n\tat
 org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)\n\tat
 org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)\n\tat
 org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)\n\tat
 org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)\n\tat
 org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)\n\tat
 org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)\n\tat
 org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)\n\tat
 org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)\n\tat
 org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)\n\tat
 org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)\n\tat
 org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)\n\tat
 org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)\n\tat
 org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)\n\tat
 org.eclipse.jetty.server.Server.handle(Server.java:368)\n\tat
 org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)\n\tat
 org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)\n\tat
 org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)\n\tat
 org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)\n\tat
 org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)\n\tat
 org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)\n\tat
 org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)\n\tat
 org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)\n\tat
 org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)\n\tat
 org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)\n\tat
 java.lang.Thread.run(Thread.java:745)\n,code:500}}
 Appreciate any pointers.
 Thanks,Jaikit




-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

http://www.griddynamics.com
mkhlud...@griddynamics.com

Re: Fwd: Issue with SOLR Distributed Search

[Shawn Heisey]

On 12/18/2014 12:35 AM, rashi gandhi wrote:
 Also, as per our investigation currently there is work ongoing in SOLR
 community to support this concept of distributed/Global IDF. But, I wanted
 to know if there is any solution possible right now to manage/control the
 score of the documents during distributed search, so that the results seem
 more relevant.

SOLR-1632 covers the distributed IDF issue.  Plans right now are to
include this in Solr 5.0 when it is released.

https://issues.apache.org/jira/browse/SOLR-1632

The only way to have a reasonably accurate distributed score currently
is to load your shards as evenly as possible.  A good way to do this is
to use the hash value of the uniqueKey field as the deciding factor for
which shard gets the request.  This is what SolrCloud does if you let it
handle the routing.

Thanks,
Shawn

Fwd: Issue with SOLR Distributed Search

[rashi gandhi]

Hi,

This is regarding the issue that we are facing with SOLR distributed search.
In our application, we are managing multiple shards at SOLR server to
manage the load. But there is a problem with the order of results that we
going to return to client during the search.

For Example: Currently there are two shards on which data is randomly
distributed.
When I search something, it was observerd that the results from one shard
appear first and then results from other shard.

Moreover, we are ordering results by applying two levels of sorting
(configurable as per user also):
1. Score
2. Modified Time

I did investigations for the above scenario and found that it is not
necessary that documents coming from one shard will always have the same
score as documents coming from other shard, even if they are identical.
I also went through the various SOLR documentations and links, and found
that currently there is a limitation to distributed search in SOLR that
Inverse-document frequency (IDF) calculations cannot be distributed and
TF/IDF computations are per shard.

This issue is particularly visible when there is significant difference
between the number of documents indexed in each shard. (For Ex: first shard
has 15000 docs and second shard has 5000).

Please review and let me know whether our findings for the above scenario
are appropriate or not.

Also, as per our investigation currently there is work ongoing in SOLR
community to support this concept of distributed/Global IDF. But, I wanted
to know if there is any solution possible right now to manage/control the
score of the documents during distributed search, so that the results seem
more relevant.

Thanks
Rashi

Re: Exception in unit tests for distributed search component

[Shalin Shekhar Mangar]

Is that the complete stack trace? There are multiple indexDoc methods in
that class. Some of them assert that the response from control collection
and the default collection are the same. However, in this case, it seems
that an AssertionError is being sent from the server itself as a
RemoteSolrException.

Without more details about the test case and the server response, I can't
say much. Maybe you should try printing out the response from the server to
see what is being returned.

On Wed, Nov 26, 2014 at 5:11 AM, Suchi Amalapurapu su...@bloomreach.com
wrote:

 Hi
 I am trying to test a custom distributed component with solr 4.6.1 which
 extends
 BaseDistributedSearchTestCase but end up with the following error.

 There are lot of tests in the solr code base which extend
 BaseDistributedSearchTestCase. Not sure what is wrong here.
 Suchi

 testDistribSearch(com.test.DistributedTest)  Time elapsed: 2.288 sec  
 ERROR!

 org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
 java.lang.AssertionError

 at __randomizedtesting.SeedInfo.seed([EB2AD095C59CFFE7:6ACC5E8DB2C39FDB]:0)

 at

 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)

 at

 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)

 at

 org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)

 at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)

 at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)

 at

 org.apache.solr.BaseDistributedSearchTestCase.indexDoc(BaseDistributedSearchTestCase.java:436)




-- 
Regards,
Shalin Shekhar Mangar.

Exception in unit tests for distributed search component

[Suchi Amalapurapu]

Hi
I am trying to test a custom distributed component with solr 4.6.1 which
extends
BaseDistributedSearchTestCase but end up with the following error.

There are lot of tests in the solr code base which extend
BaseDistributedSearchTestCase. Not sure what is wrong here.
Suchi

testDistribSearch(com.test.DistributedTest)  Time elapsed: 2.288 sec  
ERROR!

org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
java.lang.AssertionError

at __randomizedtesting.SeedInfo.seed([EB2AD095C59CFFE7:6ACC5E8DB2C39FDB]:0)

at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)

at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)

at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)

at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)

at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)

at
org.apache.solr.BaseDistributedSearchTestCase.indexDoc(BaseDistributedSearchTestCase.java:436)

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[S.L]

Shawn ,

Just wanted to follow up , I still face this issue of inconsistent search
results on Solr Cloud 4.1.0.1 , upon further looking into logs , I found
out a few exceptions , what was obvious was zkConnection time out issues
and other exceptions , please take a look .

*Logs*

/opt/tomcat1/logs/catalina.out:103651230 [http-bio-8081-exec-206] WARN
org.apache.solr.handler.ReplicationHandler  – Exception while writing
response for params:
file=_68v.fnmcommand=filecontentchecksum=truewt=filestreamqt=/replicationgeneration=2410
/opt/tomcat1/logs/catalina.out:java.nio.file.NoSuchFileException:
/opt/solr/home1/dyCollection1_shard2_replica1/data/index/_68v.fnm
/opt/tomcat1/logs/catalina.out: at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
/opt/tomcat1/logs/catalina.out: at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
/opt/tomcat1/logs/catalina.out: at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
/opt/tomcat1/logs/catalina.out:103651579 [http-bio-8081-exec-206] WARN
org.apache.solr.handler.ReplicationHandler  – Exception while writing
response for params:
file=_68v.fnmcommand=filecontentchecksum=truewt=filestreamqt=/replicationgeneration=2410
/opt/tomcat1/logs/catalina.out:java.nio.file.NoSuchFileException:
/opt/solr/home1/dyCollection1_shard2_replica1/data/index/_68v.fnm
/opt/tomcat1/logs/catalina.out: at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
/opt/tomcat1/logs/catalina.out: at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
/opt/tomcat1/logs/catalina.out: at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
/opt/tomcat1/logs/catalina.out:103651586 [http-bio-8081-exec-206] WARN
org.apache.solr.handler.ReplicationHandler  – Exception while writing
response for params:
file=_68v.fnmcommand=filecontentchecksum=truewt=filestreamqt=/replicationgeneration=2410
/opt/tomcat1/logs/catalina.out:java.nio.file.NoSuchFileException:
/opt/solr/home1/dyCollection1_shard2_replica1/data/index/_68v.fnm
/opt/tomcat1/logs/catalina.out: at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
/opt/tomcat1/logs/catalina.out: at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
/opt/tomcat1/logs/catalina.out: at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
/opt/tomcat1/logs/catalina.out:103651592 [http-bio-8081-exec-206] WARN
org.apache.solr.handler.ReplicationHandler  – Exception while writing
response for params:
file=_68v.fnmcommand=filecontentchecksum=truewt=filestreamqt=/replicationgeneration=2410
/opt/tomcat1/logs/catalina.out:java.nio.file.NoSuchFileException:
/opt/solr/home1/dyCollection1_shard2_replica1/data/index/_68v.fnm
/opt/tomcat1/logs/catalina.out: at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
/opt/tomcat1/logs/catalina.out: at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
/opt/tomcat1/logs/catalina.out: at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
/opt/tomcat1/logs/catalina.out:103651600 [http-bio-8081-exec-206] WARN
org.apache.solr.handler.ReplicationHandler  – Exception while writing
response for params:
file=_68v.fnmcommand=filecontentchecksum=truewt=filestreamqt=/replicationgeneration=2410
/opt/tomcat1/logs/catalina.out:java.nio.file.NoSuchFileException:
/opt/solr/home1/dyCollection1_shard2_replica1/data/index/_68v.fnm
/opt/tomcat1/logs/catalina.out: at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
/opt/tomcat1/logs/catalina.out: at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
/opt/tomcat1/logs/catalina.out: at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
/opt/tomcat1/logs/catalina.out:103651611 [http-bio-8081-exec-203] WARN
org.apache.solr.handler.ReplicationHandler  – Exception while writing
response for params:
file=_68v.fnmcommand=filecontentchecksum=truewt=filestreamqt=/replicationgeneration=2410
/opt/tomcat1/logs/catalina.out:java.nio.file.NoSuchFileException:
/opt/solr/home1/dyCollection1_shard2_replica1/data/index/_68v.fnm
/opt/tomcat1/logs/catalina.out: at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
471640118 [localhost-startStop-1-EventThread] INFO
org.apache.solr.common.cloud.ConnectionManager  – Watcher
org.apache.solr.common.cloud.ConnectionManager@2a7dcd74
name:ZooKeeperConnection Watcher:server1.mydomain.com:2181,
server2.mydomain.com:2181,server3.mydomain.com:2181 got event WatchedEvent
state:Disconnected type:None path:null path:null type:None
471640120 [localhost-startStop-1-EventThread] INFO
org.apache.solr.common.cloud.ConnectionManager  – zkClient has disconnected
471642457 [zkCallback-2-thread-8] INFO
org.apache.solr.cloud.DistributedQueue  – LatchChildWatcher fired on path:
null state: Expired type None
471642458 [localhost-startStop-1-EventThread] INFO
org.apache.solr.common.cloud.ConnectionManager  – 

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[S.L]

Shawn,

Just wondering if you have any other suggestions on what the next steps
whould be ? Thanks.

On Thu, Oct 16, 2014 at 11:12 PM, S.L simpleliving...@gmail.com wrote:

 Shawn ,


1. I will upgrade to 67 JVM  shortly .
2. This is  a new collection as , I was facing a similar issue in 4.7
and based on Erick's recommendation I updated to 4.10.1 and created a new
collection.
3. Yes, I am hitting the replicas of the same shard and I see the
lists are completely non overlapping.I am using CloudSolrServer to add the
documents.
4. I have a 3 physical node cluster , with each having 16GB in memory.
5. I also have a custom request handler defined in my solrconfig.xml
as below , however I am not using that and I am only using the default
select handler, but my MyCustomHandler class has been been added to the
source and included in the build , but not being used for any requests yet.

   requestHandler name=/mycustomselect class=solr.MyCustomHandler
 startup=lazy
 lst name=defaults
   str name=dfsuggestAggregate/str

   str name=spellcheck.dictionarydirect/str
   !--str name=spellcheck.dictionarywordbreak/str--
   str name=spellcheckon/str
   str name=spellcheck.extendedResultstrue/str
   str name=spellcheck.count10/str
   str name=spellcheck.alternativeTermCount5/str
   str name=spellcheck.maxResultsForSuggest5/str
   str name=spellcheck.collatetrue/str
   str name=spellcheck.collateExtendedResultstrue/str
   str name=spellcheck.maxCollationTries10/str
   str name=spellcheck.maxCollations5/str
 /lst
 arr name=last-components
   strspellcheck/str
 /arr
   /requestHandler


 5. The clusterstate.json is copied below

 {dyCollection1:{
 shards:{
   shard1:{
 range:8000-d554,
 state:active,
 replicas:{
   core_node3:{
 state:active,
 core:dyCollection1_shard1_replica1,
 node_name:server3.mydomain.com:8082_solr,
 base_url:http://server3.mydomain.com:8082/solr},
   core_node4:{
 state:active,
 core:dyCollection1_shard1_replica2,
 node_name:server2.mydomain.com:8081_solr,
 base_url:http://server2.mydomain.com:8081/solr;,
 leader:true}}},
   shard2:{
 range:d555-2aa9,
 state:active,
 replicas:{
   core_node1:{
 state:active,
 core:dyCollection1_shard2_replica1,
 node_name:server1.mydomain.com:8081_solr,
 base_url:http://server1.mydomain.com:8081/solr;,
 leader:true},
   core_node6:{
 state:active,
 core:dyCollection1_shard2_replica2,
 node_name:server3.mydomain.com:8081_solr,
 base_url:http://server3.mydomain.com:8081/solr}}},
   shard3:{
 range:2aaa-7fff,
 state:active,
 replicas:{
   core_node2:{
 state:active,
 core:dyCollection1_shard3_replica2,
 node_name:server1.mydomain.com:8082_solr,
 base_url:http://server1.mydomain.com:8082/solr;,
 leader:true},
   core_node5:{
 state:active,
 core:dyCollection1_shard3_replica1,
 node_name:server2.mydomain.com:8082_solr,
 base_url:http://server2.mydomain.com:8082/solr,
 maxShardsPerNode:1,
 router:{name:compositeId},
 replicationFactor:2,
 autoAddReplicas:false}}

   Thanks!

 On Thu, Oct 16, 2014 at 9:02 PM, Shawn Heisey apa...@elyograg.org wrote:

 On 10/16/2014 6:27 PM, S.L wrote:

 1. Java Version :java version 1.7.0_51
 Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
 Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)


 I believe that build 51 is one of those that is known to have bugs
 related to Lucene.  If you can upgrade this to 67, that would be good, but
 I don't know that it's a pressing matter.  It looks like the Oracle JVM,
 which is good.

  2.OS
 CentOS Linux release 7.0.1406 (Core)

 3. Everything is 64 bit , OS , Java , and CPU.

 4. Java Args.
  -Djava.io.tmpdir=/opt/tomcat1/temp
  -Dcatalina.home=/opt/tomcat1
  -Dcatalina.base=/opt/tomcat1
  -Djava.endorsed.dirs=/opt/tomcat1/endorsed
  -DzkHost=server1.mydomain.com:2181,server2.mydomain.com:2181,
 server3.mydomain.com:2181
  -DzkClientTimeout=2
  -DhostContext=solr
  -Dport=8081
  -Dhost=server1.mydomain.com
  -Dsolr.solr.home=/opt/solr/home1
  -Dfile.encoding=UTF8
  -Duser.timezone=UTC
  -XX:+UseG1GC
  -XX:MaxPermSize=128m
  -XX:PermSize=64m
  -Xmx2048m
  -Xms128m
  -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
  -Djava.util.logging.config.file=/opt/tomcat1/conf/
 logging.properties


 I would not use the G1 collector myself, but with the heap at 

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[S.L]

Shawn,

Please find the answers to your questions.

1. Java Version :java version 1.7.0_51
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

2.OS
CentOS Linux release 7.0.1406 (Core)

3. Everything is 64 bit , OS , Java , and CPU.

4. Java Args.
-Djava.io.tmpdir=/opt/tomcat1/temp
-Dcatalina.home=/opt/tomcat1
-Dcatalina.base=/opt/tomcat1
-Djava.endorsed.dirs=/opt/tomcat1/endorsed
-DzkHost=server1.mydomain.com:2181,server2.mydomain.com:2181,
server3.mydomain.com:2181
-DzkClientTimeout=2
-DhostContext=solr
-Dport=8081
-Dhost=server1.mydomain.com
-Dsolr.solr.home=/opt/solr/home1
-Dfile.encoding=UTF8
-Duser.timezone=UTC
-XX:+UseG1GC
-XX:MaxPermSize=128m
-XX:PermSize=64m
-Xmx2048m
-Xms128m
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.util.logging.config.file=/opt/tomcat1/conf/logging.properties

5. Zookeeper ensemble has 3 zookeeper instances , which are external and
are not embedded.


6. Container : I am using Tomcat Apache Tomcat Version 7.0.42

*Additional Observations:*

I queries all docs on both replicas with distrib=falsefl=idsort=id+asc,
then compared the two lists, I could see by eyeballing the first few lines
of ids in both the lists ,I could say that even though each list has equal
number of documents i.e 96309 each , but the document ids in them seem to
be *mutually exclusive* ,  , I did not find even a single  common id in
those lists , I tried at least 15 manually ,it looks like to me that the
replicas are disjoint sets.

Thanks.



On Thu, Oct 16, 2014 at 1:41 AM, Shawn Heisey apa...@elyograg.org wrote:

 On 10/15/2014 10:24 PM, S.L wrote:

 Yes , I tried those two queries with distrib=false , I get 0 results for
 first and 1 result  for the second query( (i.e. server 3 shard 2 replica
 2)  consistently.

 However if I run the same second query (i.e. server 3 shard 2 replica 2)
 with distrib=true, I sometimes get a result and sometimes not , should'nt
 this query always return a result when its pointing to a core that seems
 to
 have that document regardless of distrib=true or false ?

 Unfortunately I dont see anything particular in the logs to point to any
 information.

 BTW you asked me to replace the request handler , I use the select request
 handler ,so I cannot replace it with anything else , is that  a problem ?


 If you send the query with distrib=true (which is the default value in
 SolrCloud), then it treats it just as if you had sent it to
 /solr/collection instead of /solr/collection_shardN_replicaN, so it's a
 full distributed query. The distrib=false is required to turn that behavior
 off and ONLY query the index on the actual core where you sent it.

 I only said to replace those things as appropriate.  Since you are using
 /select, it's no problem that you left it that way. If I were to assume
 that you used /select, but you didn't, the URLs as I wrote them might not
 have worked.

 As discussed, this means that your replicas are truly out of sync.  It's
 difficult to know what caused it, especially if you can't see anything in
 the log when you indexed the missing documents.

 We know you're on Solr 4.10.1.  This means that your Java is a 1.7
 version, since Java7 is required.

 Here's where I ask a whole lot of questions about your setup. What is the
 precise Java version, and which vendor's Java are you using?  What
 operating system is it on?  Is everything 64-bit, or is any piece (CPU, OS,
 Java) 32-bit?  On the Solr admin UI dashboard, it lists all parameters used
 when starting Java, labelled as Args.  Can you include those?  Is
 zookeeper external, or embedded in Solr?  Is it a 3-server (or more)
 ensemble?  Are you using the example jetty, or did you provide your own
 servlet container?

 We recommend 64-bit Oracle Java, the latest 1.7 version.  OpenJDK (since
 version 1.7.x) should be pretty safe as well, but IBM's Java should be
 avoided.  IBM does very aggressive runtime optimizations.  These can make
 programs run faster, but they are known to negatively affect Lucene/Solr.

 Thanks,
 Shawn

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[Shawn Heisey]

On 10/16/2014 6:27 PM, S.L wrote:

1. Java Version :java version 1.7.0_51
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)


I believe that build 51 is one of those that is known to have bugs 
related to Lucene.  If you can upgrade this to 67, that would be good, 
but I don't know that it's a pressing matter.  It looks like the Oracle 
JVM, which is good.



2.OS
CentOS Linux release 7.0.1406 (Core)

3. Everything is 64 bit , OS , Java , and CPU.

4. Java Args.
 -Djava.io.tmpdir=/opt/tomcat1/temp
 -Dcatalina.home=/opt/tomcat1
 -Dcatalina.base=/opt/tomcat1
 -Djava.endorsed.dirs=/opt/tomcat1/endorsed
 -DzkHost=server1.mydomain.com:2181,server2.mydomain.com:2181,
server3.mydomain.com:2181
 -DzkClientTimeout=2
 -DhostContext=solr
 -Dport=8081
 -Dhost=server1.mydomain.com
 -Dsolr.solr.home=/opt/solr/home1
 -Dfile.encoding=UTF8
 -Duser.timezone=UTC
 -XX:+UseG1GC
 -XX:MaxPermSize=128m
 -XX:PermSize=64m
 -Xmx2048m
 -Xms128m
 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
 -Djava.util.logging.config.file=/opt/tomcat1/conf/logging.properties


I would not use the G1 collector myself, but with the heap at only 2GB, 
I don't know that it matters all that much.  Even a worst-case 
collection probably is not going to take more than a few seconds, and 
you've already increased the zookeeper client timeout.


http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning


5. Zookeeper ensemble has 3 zookeeper instances , which are external and
are not embedded.


6. Container : I am using Tomcat Apache Tomcat Version 7.0.42

*Additional Observations:*

I queries all docs on both replicas with distrib=falsefl=idsort=id+asc,
then compared the two lists, I could see by eyeballing the first few lines
of ids in both the lists ,I could say that even though each list has equal
number of documents i.e 96309 each , but the document ids in them seem to
be *mutually exclusive* ,  , I did not find even a single  common id in
those lists , I tried at least 15 manually ,it looks like to me that the
replicas are disjoint sets.


Are you sure you hit both replicas of the same shard number?  If you 
are, then it sounds like something is going wrong with your document 
routing, or maybe your clusterstate is really messed up.  Recreating the 
collection from scratch and doing a full reindex might be a good plan 
... assuming this is possible for you.  You could create a whole new 
collection, and then when you're ready to switch, delete the original 
collection and create an alias so your app can still use the old name.


How much total RAM do you have on these systems, and how large are those 
index shards?  With a shard having 96K documents, it sounds like your 
whole index is probably just shy of 300K documents.


Thanks,
Shawn

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[S.L]

Shawn ,


   1. I will upgrade to 67 JVM  shortly .
   2. This is  a new collection as , I was facing a similar issue in 4.7
   and based on Erick's recommendation I updated to 4.10.1 and created a new
   collection.
   3. Yes, I am hitting the replicas of the same shard and I see the lists
   are completely non overlapping.I am using CloudSolrServer to add the
   documents.
   4. I have a 3 physical node cluster , with each having 16GB in memory.
   5. I also have a custom request handler defined in my solrconfig.xml as
   below , however I am not using that and I am only using the default select
   handler, but my MyCustomHandler class has been been added to the source and
   included in the build , but not being used for any requests yet.

  requestHandler name=/mycustomselect class=solr.MyCustomHandler
startup=lazy
lst name=defaults
  str name=dfsuggestAggregate/str

  str name=spellcheck.dictionarydirect/str
  !--str name=spellcheck.dictionarywordbreak/str--
  str name=spellcheckon/str
  str name=spellcheck.extendedResultstrue/str
  str name=spellcheck.count10/str
  str name=spellcheck.alternativeTermCount5/str
  str name=spellcheck.maxResultsForSuggest5/str
  str name=spellcheck.collatetrue/str
  str name=spellcheck.collateExtendedResultstrue/str
  str name=spellcheck.maxCollationTries10/str
  str name=spellcheck.maxCollations5/str
/lst
arr name=last-components
  strspellcheck/str
/arr
  /requestHandler


5. The clusterstate.json is copied below

{dyCollection1:{
shards:{
  shard1:{
range:8000-d554,
state:active,
replicas:{
  core_node3:{
state:active,
core:dyCollection1_shard1_replica1,
node_name:server3.mydomain.com:8082_solr,
base_url:http://server3.mydomain.com:8082/solr},
  core_node4:{
state:active,
core:dyCollection1_shard1_replica2,
node_name:server2.mydomain.com:8081_solr,
base_url:http://server2.mydomain.com:8081/solr;,
leader:true}}},
  shard2:{
range:d555-2aa9,
state:active,
replicas:{
  core_node1:{
state:active,
core:dyCollection1_shard2_replica1,
node_name:server1.mydomain.com:8081_solr,
base_url:http://server1.mydomain.com:8081/solr;,
leader:true},
  core_node6:{
state:active,
core:dyCollection1_shard2_replica2,
node_name:server3.mydomain.com:8081_solr,
base_url:http://server3.mydomain.com:8081/solr}}},
  shard3:{
range:2aaa-7fff,
state:active,
replicas:{
  core_node2:{
state:active,
core:dyCollection1_shard3_replica2,
node_name:server1.mydomain.com:8082_solr,
base_url:http://server1.mydomain.com:8082/solr;,
leader:true},
  core_node5:{
state:active,
core:dyCollection1_shard3_replica1,
node_name:server2.mydomain.com:8082_solr,
base_url:http://server2.mydomain.com:8082/solr,
maxShardsPerNode:1,
router:{name:compositeId},
replicationFactor:2,
autoAddReplicas:false}}

  Thanks!

On Thu, Oct 16, 2014 at 9:02 PM, Shawn Heisey apa...@elyograg.org wrote:

 On 10/16/2014 6:27 PM, S.L wrote:

 1. Java Version :java version 1.7.0_51
 Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
 Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)


 I believe that build 51 is one of those that is known to have bugs related
 to Lucene.  If you can upgrade this to 67, that would be good, but I don't
 know that it's a pressing matter.  It looks like the Oracle JVM, which is
 good.

  2.OS
 CentOS Linux release 7.0.1406 (Core)

 3. Everything is 64 bit , OS , Java , and CPU.

 4. Java Args.
  -Djava.io.tmpdir=/opt/tomcat1/temp
  -Dcatalina.home=/opt/tomcat1
  -Dcatalina.base=/opt/tomcat1
  -Djava.endorsed.dirs=/opt/tomcat1/endorsed
  -DzkHost=server1.mydomain.com:2181,server2.mydomain.com:2181,
 server3.mydomain.com:2181
  -DzkClientTimeout=2
  -DhostContext=solr
  -Dport=8081
  -Dhost=server1.mydomain.com
  -Dsolr.solr.home=/opt/solr/home1
  -Dfile.encoding=UTF8
  -Duser.timezone=UTC
  -XX:+UseG1GC
  -XX:MaxPermSize=128m
  -XX:PermSize=64m
  -Xmx2048m
  -Xms128m
  -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
  -Djava.util.logging.config.file=/opt/tomcat1/conf/logging.properties


 I would not use the G1 collector myself, but with the heap at only 2GB, I
 don't know that it matters all that much.  Even a worst-case collection
 probably is not going to take more than a few seconds, and you've already
 increased the zookeeper client timeout.

 http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning

  5. 

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[S.L]

 surprised that this issue never got reported for 4.7 up
   until
now.
   
Thanks again for your help!
   
   
   
On Mon, Oct 6, 2014 at 10:52 PM, Erick Erickson 
  erickerick...@gmail.com
   
wrote:
   
I think there were some holes that would allow replicas and leaders
 to
be out of synch that have been patched up in the last 3 releases.
   
There shouldn't be anything you need to do to keep these in synch,
 so
if you can capture what happened when things got out of synch we'll
fix it. But a lot has changed in the last several months, so the
 first
thing I'd do if possible is to upgrade to 4.10.1.
   
   
Best,
Erick
   
On Mon, Oct 6, 2014 at 2:41 PM, S.L simpleliving...@gmail.com
  wrote:
 Hi Erick,

 Before I tried your suggestion of  issung a commit=true update, I
realized that for eaach shard there was atleast a node that had its
   index
directory named like index.timestamp.

 I went ahead and deleted index directory that restarted that core
  and
now the index directory got syched with the other node and is
 properly
named as 'index' without any timestamp attached to it.This is now
   giving me
consistent results for distrib=true using a load balancer.Also
distrib=false returns expexted results for a given shard.

 The underlying issue appears to be that in every shard the leader
  and
the replica(follower) were out of sych.

 How can I avoid this from happening again?

 Thanks for your help!

 Sent from my HTC

 - Reply message -
 From: Erick Erickson erickerick...@gmail.com
 To: solr-user@lucene.apache.org
 Subject: SolrCloud 4.7 not doing distributed search when querying
   from a
load balancer.
 Date: Fri, Oct 3, 2014 12:56 AM

 H. Assuming that you aren't re-indexing the doc you're
 searching
for...

 Try issuing http://blah
  blah:8983/solr/collection/update?commit=true.
 That'll force all the docs to be searchable. Does 1 still hold
 for
 the document in question? Because this is exactly backwards of
 what
 I'd expect. I'd expect, if anything, the replica (I'm trying to
 call
 it the follower when a distinction needs to be made since the
  leader
 is a replica too) would be out of sync. This is still a Bad
 Thing, but the leader gets first crack at indexing thing.

 bq: only the replica of the shard that has this key returns the
  result
 , and the leader does not ,

 Just to be sure we're talking about the same thing. When you say
 leader, you mean the shard leader, right? The filled-in circle
 on
 the graph view from the admin/cloud page.

 And let's see your soft and hard commit settings please.

 Best,
 Erick

 On Thu, Oct 2, 2014 at 9:48 PM, S.L simpleliving...@gmail.com
   wrote:
 Eirck,

 0 Load balancer is out of the picture
 .
 1When I query with *distrib=false* , I get consistent results as
expected
 for those shards that dont have the key i.e I dont get the
 results
   back
for
 those shards, however I just realized that while *distrib=false*
 is
present
 in the query for the shard that is supposed to contain the
 key,only
   the
 replica of the shard that has this key returns the result , and
 the
leader
 does not , looks like replica and the leader do not have the same
   data
and
 replica seems to contain the key in the query for that shard.

 2 By indexing I mean this collection is being populated by a web
crawler.

 So looks like 1 above  is pointing to leader and replica being
 out
   of
 synch for atleast one shard.



 On Thu, Oct 2, 2014 at 11:57 PM, Erick Erickson 
erickerick...@gmail.com
 wrote:

 bq: Also ,the collection is being actively indexed as I query
  this,
could
 that
 be an issue too ?

 Not if the documents you're searching aren't being added as you
   search
 (and all your autocommit intervals have expired).

 I would turn off indexing for testing, it's just one more
 variable
 that can get in the way of understanding this.

 Do note that if the problem were endemic to Solr, there would
   probably
 be a _lot_ more noise out there.

 So to recap:
 0 we can take the load balancer out of the picture all
 together.

 1 when you query each shard individually with distrib=true,
  every
 replica in a particular shard returns the same count.

 2 when you query without distrib=true you get varying counts.

 This is very strange and not at all expected. Let's try it again
 without indexing going on

 And what do you mean by indexing anyway? How are documents
 being
   fed
 to your system?

 Best,
 Erick@PuzzledAsWell

 On Thu, Oct 2, 2014 at 7:32 PM, S.L

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[S.L]

-bio-8081-exec-169] INFO  org.apache.solr.core.SolrCore  –
  [dyCollection1_shard2_replica1] webapp=/solr path=/select/
 
 
 params={q=*:*distrib=truewt=jsonfq=(id:220a8dce-3b31-4d46-8386-da8405595c47)}
  hits=1 status=0 QTime=7
 
 
  *Autocommit and Soft commit settings.*
 
   autoSoftCommit
 maxTime${solr.autoSoftCommit.maxTime:-1}/maxTime
   /autoSoftCommit
 
   autoCommit
 maxTime${solr.autoCommit.maxTime:15000}/maxTime
 
 openSearchertrue/openSearcher
   /autoCommit
 
 
 
  On Tue, Oct 7, 2014 at 12:22 AM, Erick Erickson 
 erickerick...@gmail.com
  wrote:
 
   Not, I'm not guaranteeing that it'll actually cure the problem, just
   that enough has changed since 4.7 that it'd be a good place to start.
  
   Things have been reported off and on, but they're often pesky race
   conditions or something else that takes a long time to track down, you
   just are lucky perhaps ;)...
  
   Erick
  
   On Mon, Oct 6, 2014 at 8:04 PM, S.L simpleliving...@gmail.com
 wrote:
Erick,
   
Thanks for the suggestion , I am not sure if I would be able to
 capture
what went wrong , so upgrading to 4.10 seems easier even though it
  means
   ,
a days work of effort :) . I will go ahead and upgrade and let me
 know
  ,
although I am surprised that this issue never got reported for 4.7
 up
   until
now.
   
Thanks again for your help!
   
   
   
On Mon, Oct 6, 2014 at 10:52 PM, Erick Erickson 
  erickerick...@gmail.com
   
wrote:
   
I think there were some holes that would allow replicas and
 leaders to
be out of synch that have been patched up in the last 3 releases.
   
There shouldn't be anything you need to do to keep these in synch,
 so
if you can capture what happened when things got out of synch we'll
fix it. But a lot has changed in the last several months, so the
 first
thing I'd do if possible is to upgrade to 4.10.1.
   
   
Best,
Erick
   
On Mon, Oct 6, 2014 at 2:41 PM, S.L simpleliving...@gmail.com
  wrote:
 Hi Erick,

 Before I tried your suggestion of  issung a commit=true update, I
realized that for eaach shard there was atleast a node that had its
   index
directory named like index.timestamp.

 I went ahead and deleted index directory that restarted that core
  and
now the index directory got syched with the other node and is
 properly
named as 'index' without any timestamp attached to it.This is now
   giving me
consistent results for distrib=true using a load balancer.Also
distrib=false returns expexted results for a given shard.

 The underlying issue appears to be that in every shard the leader
  and
the replica(follower) were out of sych.

 How can I avoid this from happening again?

 Thanks for your help!

 Sent from my HTC

 - Reply message -
 From: Erick Erickson erickerick...@gmail.com
 To: solr-user@lucene.apache.org
 Subject: SolrCloud 4.7 not doing distributed search when querying
   from a
load balancer.
 Date: Fri, Oct 3, 2014 12:56 AM

 H. Assuming that you aren't re-indexing the doc you're
 searching
for...

 Try issuing http://blah
  blah:8983/solr/collection/update?commit=true.
 That'll force all the docs to be searchable. Does 1 still hold
 for
 the document in question? Because this is exactly backwards of
 what
 I'd expect. I'd expect, if anything, the replica (I'm trying to
 call
 it the follower when a distinction needs to be made since the
  leader
 is a replica too) would be out of sync. This is still a Bad
 Thing, but the leader gets first crack at indexing thing.

 bq: only the replica of the shard that has this key returns the
  result
 , and the leader does not ,

 Just to be sure we're talking about the same thing. When you say
 leader, you mean the shard leader, right? The filled-in circle
 on
 the graph view from the admin/cloud page.

 And let's see your soft and hard commit settings please.

 Best,
 Erick

 On Thu, Oct 2, 2014 at 9:48 PM, S.L simpleliving...@gmail.com
   wrote:
 Eirck,

 0 Load balancer is out of the picture
 .
 1When I query with *distrib=false* , I get consistent results
 as
expected
 for those shards that dont have the key i.e I dont get the
 results
   back
for
 those shards, however I just realized that while
 *distrib=false* is
present
 in the query for the shard that is supposed to contain the
 key,only
   the
 replica of the shard that has this key returns the result , and
 the
leader
 does not , looks like replica and the leader do not have the
 same
   data
and
 replica seems to contain the key in the query for that shard.

 2 By indexing I mean this collection is being populated by a
 web
crawler.

 So looks like 1 above

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[Shawn Heisey]

On 10/15/2014 9:26 PM, S.L wrote:

Look at the logging information I provided below , looks like the results
are only being returned back for this solrCloud cluster  if the request
goes to one of the two replicas of a shard.

I have verified that numDocs in the replicas for a given shard is same but
there is difference in the maxDoc and deletedDocs, does this signal the
replicas being out of sync ?

Even if the numDocs are same , how do we guarantee that those docs are
identical and have the same uniquekeys , is there a way to verify this ? I
am suspecting that  as the numDocs is same across the replicas , and still
only when the request goes to one of  the  replicas of the shard that I get
a result back , the documents with in those replicas with in a shard are
not an exact replica set of each other.

I suspect the issue I am facing in 4.10.1 cloud is related to
https://issues.apache.org/jira/browse/SOLR-4924  .

Can anyone please let me know , how to solve this issue of intermittent no
results for a query ?


query with no results hits these cores:
server 2 shard 3 replica1
server 3 shard 1 replica 1
server 1 shard 2 replica 1

query with 1 result hits these cores:
server 2 shard 1 replica 2
server 3 shard 2 replica 2 (found 1)
server 1 shard 3 replica 2

Here's some URLs for some testing.  They are directed at specific shard 
replicas and are specifically NOT distributed queries:


http://server1.mydomain.com:8081/solr/dyCollection1_shard2_replica1/select?q=*:*fq=id:e8995da8-7d98-4010-93b4-8ff7dffb8bfbdistrib=false

http://server3.mydomain.com:8081/solr/dyCollection1_shard2_replica2/select?q=*:*fq=id:e8995da8-7d98-4010-93b4-8ff7dffb8bfbdistrib=false

If you run these queries (replacing server names and the /select request 
handler as appropriate), do you get 0 results on the first one and 1 
result on the second one?  If you do, then you've definitely got 
replicas out of sync.  If you get 1 result on both queries, then 
something else is breaking.  If by chance you have taken steps to fix 
this particular ID, pick another one that you know has a problem.


There is no automated way to detect replicas out of sync.  You could 
request all docs on both replicas with distrib=falsefl=idsort=id+asc, 
then compare the two lists.  Depending on how many docs you have, those 
queries could take a while to run.


If the replicas are out of sync, are there any ERROR entries in the Solr 
log, especially at the time that the problem docs were indexed?


Thanks,
Shawn

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[S.L]

Shawn,

Yes , I tried those two queries with distrib=false , I get 0 results for
first and 1 result  for the second query( (i.e. server 3 shard 2 replica
2)  consistently.

However if I run the same second query (i.e. server 3 shard 2 replica 2)
with distrib=true, I sometimes get a result and sometimes not , should'nt
this query always return a result when its pointing to a core that seems to
have that document regardless of distrib=true or false ?

Unfortunately I dont see anything particular in the logs to point to any
information.

BTW you asked me to replace the request handler , I use the select request
handler ,so I cannot replace it with anything else , is that  a problem ?

Thanks.

On Thu, Oct 16, 2014 at 12:05 AM, Shawn Heisey apa...@elyograg.org wrote:

 On 10/15/2014 9:26 PM, S.L wrote:

 Look at the logging information I provided below , looks like the results
 are only being returned back for this solrCloud cluster  if the request
 goes to one of the two replicas of a shard.

 I have verified that numDocs in the replicas for a given shard is same but
 there is difference in the maxDoc and deletedDocs, does this signal the
 replicas being out of sync ?

 Even if the numDocs are same , how do we guarantee that those docs are
 identical and have the same uniquekeys , is there a way to verify this ? I
 am suspecting that  as the numDocs is same across the replicas , and still
 only when the request goes to one of  the  replicas of the shard that I
 get
 a result back , the documents with in those replicas with in a shard are
 not an exact replica set of each other.

 I suspect the issue I am facing in 4.10.1 cloud is related to
 https://issues.apache.org/jira/browse/SOLR-4924  .

 Can anyone please let me know , how to solve this issue of intermittent no
 results for a query ?


 query with no results hits these cores:
 server 2 shard 3 replica1
 server 3 shard 1 replica 1
 server 1 shard 2 replica 1

 query with 1 result hits these cores:
 server 2 shard 1 replica 2
 server 3 shard 2 replica 2 (found 1)
 server 1 shard 3 replica 2

 Here's some URLs for some testing.  They are directed at specific shard
 replicas and are specifically NOT distributed queries:

 http://server1.mydomain.com:8081/solr/dyCollection1_
 shard2_replica1/select?q=*:*fq=id:e8995da8-7d98-4010-93b4-
 8ff7dffb8bfbdistrib=false

 http://server3.mydomain.com:8081/solr/dyCollection1_
 shard2_replica2/select?q=*:*fq=id:e8995da8-7d98-4010-93b4-
 8ff7dffb8bfbdistrib=false

 If you run these queries (replacing server names and the /select request
 handler as appropriate), do you get 0 results on the first one and 1 result
 on the second one?  If you do, then you've definitely got replicas out of
 sync.  If you get 1 result on both queries, then something else is
 breaking.  If by chance you have taken steps to fix this particular ID,
 pick another one that you know has a problem.

 There is no automated way to detect replicas out of sync.  You could
 request all docs on both replicas with distrib=falsefl=idsort=id+asc,
 then compare the two lists.  Depending on how many docs you have, those
 queries could take a while to run.

 If the replicas are out of sync, are there any ERROR entries in the Solr
 log, especially at the time that the problem docs were indexed?

 Thanks,
 Shawn

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[Shawn Heisey]

On 10/15/2014 10:24 PM, S.L wrote:

Yes , I tried those two queries with distrib=false , I get 0 results for
first and 1 result  for the second query( (i.e. server 3 shard 2 replica
2)  consistently.

However if I run the same second query (i.e. server 3 shard 2 replica 2)
with distrib=true, I sometimes get a result and sometimes not , should'nt
this query always return a result when its pointing to a core that seems to
have that document regardless of distrib=true or false ?

Unfortunately I dont see anything particular in the logs to point to any
information.

BTW you asked me to replace the request handler , I use the select request
handler ,so I cannot replace it with anything else , is that  a problem ?


If you send the query with distrib=true (which is the default value in 
SolrCloud), then it treats it just as if you had sent it to 
/solr/collection instead of /solr/collection_shardN_replicaN, so it's a 
full distributed query. The distrib=false is required to turn that 
behavior off and ONLY query the index on the actual core where you sent it.


I only said to replace those things as appropriate.  Since you are using 
/select, it's no problem that you left it that way. If I were to assume 
that you used /select, but you didn't, the URLs as I wrote them might 
not have worked.


As discussed, this means that your replicas are truly out of sync.  It's 
difficult to know what caused it, especially if you can't see anything 
in the log when you indexed the missing documents.


We know you're on Solr 4.10.1.  This means that your Java is a 1.7 
version, since Java7 is required.


Here's where I ask a whole lot of questions about your setup. What is 
the precise Java version, and which vendor's Java are you using?  What 
operating system is it on?  Is everything 64-bit, or is any piece (CPU, 
OS, Java) 32-bit?  On the Solr admin UI dashboard, it lists all 
parameters used when starting Java, labelled as Args.  Can you include 
those?  Is zookeeper external, or embedded in Solr?  Is it a 3-server 
(or more) ensemble?  Are you using the example jetty, or did you provide 
your own servlet container?


We recommend 64-bit Oracle Java, the latest 1.7 version.  OpenJDK (since 
version 1.7.x) should be pretty safe as well, but IBM's Java should be 
avoided.  IBM does very aggressive runtime optimizations.  These can 
make programs run faster, but they are known to negatively affect 
Lucene/Solr.


Thanks,
Shawn

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[Tim Potter]

 Erickson erickerick...@gmail.com
 wrote:

  Not, I'm not guaranteeing that it'll actually cure the problem, just
  that enough has changed since 4.7 that it'd be a good place to start.
 
  Things have been reported off and on, but they're often pesky race
  conditions or something else that takes a long time to track down, you
  just are lucky perhaps ;)...
 
  Erick
 
  On Mon, Oct 6, 2014 at 8:04 PM, S.L simpleliving...@gmail.com wrote:
   Erick,
  
   Thanks for the suggestion , I am not sure if I would be able to capture
   what went wrong , so upgrading to 4.10 seems easier even though it
 means
  ,
   a days work of effort :) . I will go ahead and upgrade and let me know
 ,
   although I am surprised that this issue never got reported for 4.7 up
  until
   now.
  
   Thanks again for your help!
  
  
  
   On Mon, Oct 6, 2014 at 10:52 PM, Erick Erickson 
 erickerick...@gmail.com
  
   wrote:
  
   I think there were some holes that would allow replicas and leaders to
   be out of synch that have been patched up in the last 3 releases.
  
   There shouldn't be anything you need to do to keep these in synch, so
   if you can capture what happened when things got out of synch we'll
   fix it. But a lot has changed in the last several months, so the first
   thing I'd do if possible is to upgrade to 4.10.1.
  
  
   Best,
   Erick
  
   On Mon, Oct 6, 2014 at 2:41 PM, S.L simpleliving...@gmail.com
 wrote:
Hi Erick,
   
Before I tried your suggestion of  issung a commit=true update, I
   realized that for eaach shard there was atleast a node that had its
  index
   directory named like index.timestamp.
   
I went ahead and deleted index directory that restarted that core
 and
   now the index directory got syched with the other node and is properly
   named as 'index' without any timestamp attached to it.This is now
  giving me
   consistent results for distrib=true using a load balancer.Also
   distrib=false returns expexted results for a given shard.
   
The underlying issue appears to be that in every shard the leader
 and
   the replica(follower) were out of sych.
   
How can I avoid this from happening again?
   
Thanks for your help!
   
Sent from my HTC
   
- Reply message -
From: Erick Erickson erickerick...@gmail.com
To: solr-user@lucene.apache.org
Subject: SolrCloud 4.7 not doing distributed search when querying
  from a
   load balancer.
Date: Fri, Oct 3, 2014 12:56 AM
   
H. Assuming that you aren't re-indexing the doc you're searching
   for...
   
Try issuing http://blah
 blah:8983/solr/collection/update?commit=true.
That'll force all the docs to be searchable. Does 1 still hold for
the document in question? Because this is exactly backwards of what
I'd expect. I'd expect, if anything, the replica (I'm trying to call
it the follower when a distinction needs to be made since the
 leader
is a replica too) would be out of sync. This is still a Bad
Thing, but the leader gets first crack at indexing thing.
   
bq: only the replica of the shard that has this key returns the
 result
, and the leader does not ,
   
Just to be sure we're talking about the same thing. When you say
leader, you mean the shard leader, right? The filled-in circle on
the graph view from the admin/cloud page.
   
And let's see your soft and hard commit settings please.
   
Best,
Erick
   
On Thu, Oct 2, 2014 at 9:48 PM, S.L simpleliving...@gmail.com
  wrote:
Eirck,
   
0 Load balancer is out of the picture
.
1When I query with *distrib=false* , I get consistent results as
   expected
for those shards that dont have the key i.e I dont get the results
  back
   for
those shards, however I just realized that while *distrib=false* is
   present
in the query for the shard that is supposed to contain the key,only
  the
replica of the shard that has this key returns the result , and the
   leader
does not , looks like replica and the leader do not have the same
  data
   and
replica seems to contain the key in the query for that shard.
   
2 By indexing I mean this collection is being populated by a web
   crawler.
   
So looks like 1 above  is pointing to leader and replica being out
  of
synch for atleast one shard.
   
   
   
On Thu, Oct 2, 2014 at 11:57 PM, Erick Erickson 
   erickerick...@gmail.com
wrote:
   
bq: Also ,the collection is being actively indexed as I query
 this,
   could
that
be an issue too ?
   
Not if the documents you're searching aren't being added as you
  search
(and all your autocommit intervals have expired).
   
I would turn off indexing for testing, it's just one more variable
that can get in the way of understanding this.
   
Do note that if the problem were endemic to Solr, there would
  probably
be a _lot_ more noise out there.
   
So to recap:
0 we can take the load

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[S.L]

 to track down, you
 just are lucky perhaps ;)...

 Erick

 On Mon, Oct 6, 2014 at 8:04 PM, S.L simpleliving...@gmail.com wrote:
  Erick,
 
  Thanks for the suggestion , I am not sure if I would be able to capture
  what went wrong , so upgrading to 4.10 seems easier even though it means
 ,
  a days work of effort :) . I will go ahead and upgrade and let me know ,
  although I am surprised that this issue never got reported for 4.7 up
 until
  now.
 
  Thanks again for your help!
 
 
 
  On Mon, Oct 6, 2014 at 10:52 PM, Erick Erickson erickerick...@gmail.com
 
  wrote:
 
  I think there were some holes that would allow replicas and leaders to
  be out of synch that have been patched up in the last 3 releases.
 
  There shouldn't be anything you need to do to keep these in synch, so
  if you can capture what happened when things got out of synch we'll
  fix it. But a lot has changed in the last several months, so the first
  thing I'd do if possible is to upgrade to 4.10.1.
 
 
  Best,
  Erick
 
  On Mon, Oct 6, 2014 at 2:41 PM, S.L simpleliving...@gmail.com wrote:
   Hi Erick,
  
   Before I tried your suggestion of  issung a commit=true update, I
  realized that for eaach shard there was atleast a node that had its
 index
  directory named like index.timestamp.
  
   I went ahead and deleted index directory that restarted that core and
  now the index directory got syched with the other node and is properly
  named as 'index' without any timestamp attached to it.This is now
 giving me
  consistent results for distrib=true using a load balancer.Also
  distrib=false returns expexted results for a given shard.
  
   The underlying issue appears to be that in every shard the leader and
  the replica(follower) were out of sych.
  
   How can I avoid this from happening again?
  
   Thanks for your help!
  
   Sent from my HTC
  
   - Reply message -
   From: Erick Erickson erickerick...@gmail.com
   To: solr-user@lucene.apache.org
   Subject: SolrCloud 4.7 not doing distributed search when querying
 from a
  load balancer.
   Date: Fri, Oct 3, 2014 12:56 AM
  
   H. Assuming that you aren't re-indexing the doc you're searching
  for...
  
   Try issuing http://blah blah:8983/solr/collection/update?commit=true.
   That'll force all the docs to be searchable. Does 1 still hold for
   the document in question? Because this is exactly backwards of what
   I'd expect. I'd expect, if anything, the replica (I'm trying to call
   it the follower when a distinction needs to be made since the leader
   is a replica too) would be out of sync. This is still a Bad
   Thing, but the leader gets first crack at indexing thing.
  
   bq: only the replica of the shard that has this key returns the result
   , and the leader does not ,
  
   Just to be sure we're talking about the same thing. When you say
   leader, you mean the shard leader, right? The filled-in circle on
   the graph view from the admin/cloud page.
  
   And let's see your soft and hard commit settings please.
  
   Best,
   Erick
  
   On Thu, Oct 2, 2014 at 9:48 PM, S.L simpleliving...@gmail.com
 wrote:
   Eirck,
  
   0 Load balancer is out of the picture
   .
   1When I query with *distrib=false* , I get consistent results as
  expected
   for those shards that dont have the key i.e I dont get the results
 back
  for
   those shards, however I just realized that while *distrib=false* is
  present
   in the query for the shard that is supposed to contain the key,only
 the
   replica of the shard that has this key returns the result , and the
  leader
   does not , looks like replica and the leader do not have the same
 data
  and
   replica seems to contain the key in the query for that shard.
  
   2 By indexing I mean this collection is being populated by a web
  crawler.
  
   So looks like 1 above  is pointing to leader and replica being out
 of
   synch for atleast one shard.
  
  
  
   On Thu, Oct 2, 2014 at 11:57 PM, Erick Erickson 
  erickerick...@gmail.com
   wrote:
  
   bq: Also ,the collection is being actively indexed as I query this,
  could
   that
   be an issue too ?
  
   Not if the documents you're searching aren't being added as you
 search
   (and all your autocommit intervals have expired).
  
   I would turn off indexing for testing, it's just one more variable
   that can get in the way of understanding this.
  
   Do note that if the problem were endemic to Solr, there would
 probably
   be a _lot_ more noise out there.
  
   So to recap:
   0 we can take the load balancer out of the picture all together.
  
   1 when you query each shard individually with distrib=true, every
   replica in a particular shard returns the same count.
  
   2 when you query without distrib=true you get varying counts.
  
   This is very strange and not at all expected. Let's try it again
   without indexing going on
  
   And what do you mean by indexing anyway? How are documents being
 fed
   to your system?
  
   Best,
   Erick

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[S.L]

Hi Erick,

Before I tried your suggestion of  issung a commit=true update, I realized that 
for eaach shard there was atleast a node that had its index directory named 
like index.timestamp.

I went ahead and deleted index directory that restarted that core and now the 
index directory got syched with the other node and is properly named as 'index' 
without any timestamp attached to it.This is now giving me consistent results 
for distrib=true using a load balancer.Also distrib=false returns expexted 
results for a given shard.

The underlying issue appears to be that in every shard the leader and the 
replica(follower) were out of sych.

How can I avoid this from happening again?

Thanks for your help!

Sent from my HTC

- Reply message -
From: Erick Erickson erickerick...@gmail.com
To: solr-user@lucene.apache.org
Subject: SolrCloud 4.7 not doing distributed search when querying from a load 
balancer.
Date: Fri, Oct 3, 2014 12:56 AM

H. Assuming that you aren't re-indexing the doc you're searching for...

Try issuing http://blah blah:8983/solr/collection/update?commit=true.
That'll force all the docs to be searchable. Does 1 still hold for
the document in question? Because this is exactly backwards of what
I'd expect. I'd expect, if anything, the replica (I'm trying to call
it the follower when a distinction needs to be made since the leader
is a replica too) would be out of sync. This is still a Bad
Thing, but the leader gets first crack at indexing thing.

bq: only the replica of the shard that has this key returns the result
, and the leader does not ,

Just to be sure we're talking about the same thing. When you say
leader, you mean the shard leader, right? The filled-in circle on
the graph view from the admin/cloud page.

And let's see your soft and hard commit settings please.

Best,
Erick

On Thu, Oct 2, 2014 at 9:48 PM, S.L simpleliving...@gmail.com wrote:
 Eirck,

 0 Load balancer is out of the picture
 .
 1When I query with *distrib=false* , I get consistent results as expected
 for those shards that dont have the key i.e I dont get the results back for
 those shards, however I just realized that while *distrib=false* is present
 in the query for the shard that is supposed to contain the key,only the
 replica of the shard that has this key returns the result , and the leader
 does not , looks like replica and the leader do not have the same data and
 replica seems to contain the key in the query for that shard.

 2 By indexing I mean this collection is being populated by a web crawler.

 So looks like 1 above  is pointing to leader and replica being out of
 synch for atleast one shard.



 On Thu, Oct 2, 2014 at 11:57 PM, Erick Erickson erickerick...@gmail.com
 wrote:

 bq: Also ,the collection is being actively indexed as I query this, could
 that
 be an issue too ?

 Not if the documents you're searching aren't being added as you search
 (and all your autocommit intervals have expired).

 I would turn off indexing for testing, it's just one more variable
 that can get in the way of understanding this.

 Do note that if the problem were endemic to Solr, there would probably
 be a _lot_ more noise out there.

 So to recap:
 0 we can take the load balancer out of the picture all together.

 1 when you query each shard individually with distrib=true, every
 replica in a particular shard returns the same count.

 2 when you query without distrib=true you get varying counts.

 This is very strange and not at all expected. Let's try it again
 without indexing going on

 And what do you mean by indexing anyway? How are documents being fed
 to your system?

 Best,
 Erick@PuzzledAsWell

 On Thu, Oct 2, 2014 at 7:32 PM, S.L simpleliving...@gmail.com wrote:
  Erick,
 
  I would like to add that the interesting behavior i.e point #2 that I
  mentioned in my earlier reply  happens in all the shards , if this were
 to
  be a distributed search issue this should have not manifested itself in
 the
  shard that contains the key that I am searching for , looks like the
 search
  is just failing as whole intermittently .
 
  Also ,the collection is being actively indexed as I query this, could
 that
  be an issue too ?
 
  Thanks.
 
  On Thu, Oct 2, 2014 at 10:24 PM, S.L simpleliving...@gmail.com wrote:
 
  Erick,
 
  Thanks for your reply, I tried your suggestions.
 
  1 . When not using loadbalancer if  *I have distrib=false* I get
  consistent results across the replicas.
 
  2. However here's the insteresting part , while not using load balancer
 if
  I *dont have distrib=false* , then when I query a particular node ,I get
  the same behaviour as if I were using a loadbalancer , meaning the
  distributed search from a node works intermittently .Does this give any
  clue ?
 
 
 
  On Thu, Oct 2, 2014 at 7:47 PM, Erick Erickson erickerick...@gmail.com
 
  wrote:
 
  Hmmm, nothing quite makes sense here
 
  Here are some experiments:
  1 avoid the load balancer and issue queries like

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[Erick Erickson]

I think there were some holes that would allow replicas and leaders to
be out of synch that have been patched up in the last 3 releases.

There shouldn't be anything you need to do to keep these in synch, so
if you can capture what happened when things got out of synch we'll
fix it. But a lot has changed in the last several months, so the first
thing I'd do if possible is to upgrade to 4.10.1.


Best,
Erick

On Mon, Oct 6, 2014 at 2:41 PM, S.L simpleliving...@gmail.com wrote:
 Hi Erick,

 Before I tried your suggestion of  issung a commit=true update, I realized 
 that for eaach shard there was atleast a node that had its index directory 
 named like index.timestamp.

 I went ahead and deleted index directory that restarted that core and now the 
 index directory got syched with the other node and is properly named as 
 'index' without any timestamp attached to it.This is now giving me consistent 
 results for distrib=true using a load balancer.Also distrib=false returns 
 expexted results for a given shard.

 The underlying issue appears to be that in every shard the leader and the 
 replica(follower) were out of sych.

 How can I avoid this from happening again?

 Thanks for your help!

 Sent from my HTC

 - Reply message -
 From: Erick Erickson erickerick...@gmail.com
 To: solr-user@lucene.apache.org
 Subject: SolrCloud 4.7 not doing distributed search when querying from a load 
 balancer.
 Date: Fri, Oct 3, 2014 12:56 AM

 H. Assuming that you aren't re-indexing the doc you're searching for...

 Try issuing http://blah blah:8983/solr/collection/update?commit=true.
 That'll force all the docs to be searchable. Does 1 still hold for
 the document in question? Because this is exactly backwards of what
 I'd expect. I'd expect, if anything, the replica (I'm trying to call
 it the follower when a distinction needs to be made since the leader
 is a replica too) would be out of sync. This is still a Bad
 Thing, but the leader gets first crack at indexing thing.

 bq: only the replica of the shard that has this key returns the result
 , and the leader does not ,

 Just to be sure we're talking about the same thing. When you say
 leader, you mean the shard leader, right? The filled-in circle on
 the graph view from the admin/cloud page.

 And let's see your soft and hard commit settings please.

 Best,
 Erick

 On Thu, Oct 2, 2014 at 9:48 PM, S.L simpleliving...@gmail.com wrote:
 Eirck,

 0 Load balancer is out of the picture
 .
 1When I query with *distrib=false* , I get consistent results as expected
 for those shards that dont have the key i.e I dont get the results back for
 those shards, however I just realized that while *distrib=false* is present
 in the query for the shard that is supposed to contain the key,only the
 replica of the shard that has this key returns the result , and the leader
 does not , looks like replica and the leader do not have the same data and
 replica seems to contain the key in the query for that shard.

 2 By indexing I mean this collection is being populated by a web crawler.

 So looks like 1 above  is pointing to leader and replica being out of
 synch for atleast one shard.



 On Thu, Oct 2, 2014 at 11:57 PM, Erick Erickson erickerick...@gmail.com
 wrote:

 bq: Also ,the collection is being actively indexed as I query this, could
 that
 be an issue too ?

 Not if the documents you're searching aren't being added as you search
 (and all your autocommit intervals have expired).

 I would turn off indexing for testing, it's just one more variable
 that can get in the way of understanding this.

 Do note that if the problem were endemic to Solr, there would probably
 be a _lot_ more noise out there.

 So to recap:
 0 we can take the load balancer out of the picture all together.

 1 when you query each shard individually with distrib=true, every
 replica in a particular shard returns the same count.

 2 when you query without distrib=true you get varying counts.

 This is very strange and not at all expected. Let's try it again
 without indexing going on

 And what do you mean by indexing anyway? How are documents being fed
 to your system?

 Best,
 Erick@PuzzledAsWell

 On Thu, Oct 2, 2014 at 7:32 PM, S.L simpleliving...@gmail.com wrote:
  Erick,
 
  I would like to add that the interesting behavior i.e point #2 that I
  mentioned in my earlier reply  happens in all the shards , if this were
 to
  be a distributed search issue this should have not manifested itself in
 the
  shard that contains the key that I am searching for , looks like the
 search
  is just failing as whole intermittently .
 
  Also ,the collection is being actively indexed as I query this, could
 that
  be an issue too ?
 
  Thanks.
 
  On Thu, Oct 2, 2014 at 10:24 PM, S.L simpleliving...@gmail.com wrote:
 
  Erick,
 
  Thanks for your reply, I tried your suggestions.
 
  1 . When not using loadbalancer if  *I have distrib=false* I get
  consistent results across the replicas.
 
  2

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[S.L]

Erick,

Thanks for the suggestion , I am not sure if I would be able to capture
what went wrong , so upgrading to 4.10 seems easier even though it means ,
a days work of effort :) . I will go ahead and upgrade and let me know ,
although I am surprised that this issue never got reported for 4.7 up until
now.

Thanks again for your help!



On Mon, Oct 6, 2014 at 10:52 PM, Erick Erickson erickerick...@gmail.com
wrote:

 I think there were some holes that would allow replicas and leaders to
 be out of synch that have been patched up in the last 3 releases.

 There shouldn't be anything you need to do to keep these in synch, so
 if you can capture what happened when things got out of synch we'll
 fix it. But a lot has changed in the last several months, so the first
 thing I'd do if possible is to upgrade to 4.10.1.


 Best,
 Erick

 On Mon, Oct 6, 2014 at 2:41 PM, S.L simpleliving...@gmail.com wrote:
  Hi Erick,
 
  Before I tried your suggestion of  issung a commit=true update, I
 realized that for eaach shard there was atleast a node that had its index
 directory named like index.timestamp.
 
  I went ahead and deleted index directory that restarted that core and
 now the index directory got syched with the other node and is properly
 named as 'index' without any timestamp attached to it.This is now giving me
 consistent results for distrib=true using a load balancer.Also
 distrib=false returns expexted results for a given shard.
 
  The underlying issue appears to be that in every shard the leader and
 the replica(follower) were out of sych.
 
  How can I avoid this from happening again?
 
  Thanks for your help!
 
  Sent from my HTC
 
  - Reply message -
  From: Erick Erickson erickerick...@gmail.com
  To: solr-user@lucene.apache.org
  Subject: SolrCloud 4.7 not doing distributed search when querying from a
 load balancer.
  Date: Fri, Oct 3, 2014 12:56 AM
 
  H. Assuming that you aren't re-indexing the doc you're searching
 for...
 
  Try issuing http://blah blah:8983/solr/collection/update?commit=true.
  That'll force all the docs to be searchable. Does 1 still hold for
  the document in question? Because this is exactly backwards of what
  I'd expect. I'd expect, if anything, the replica (I'm trying to call
  it the follower when a distinction needs to be made since the leader
  is a replica too) would be out of sync. This is still a Bad
  Thing, but the leader gets first crack at indexing thing.
 
  bq: only the replica of the shard that has this key returns the result
  , and the leader does not ,
 
  Just to be sure we're talking about the same thing. When you say
  leader, you mean the shard leader, right? The filled-in circle on
  the graph view from the admin/cloud page.
 
  And let's see your soft and hard commit settings please.
 
  Best,
  Erick
 
  On Thu, Oct 2, 2014 at 9:48 PM, S.L simpleliving...@gmail.com wrote:
  Eirck,
 
  0 Load balancer is out of the picture
  .
  1When I query with *distrib=false* , I get consistent results as
 expected
  for those shards that dont have the key i.e I dont get the results back
 for
  those shards, however I just realized that while *distrib=false* is
 present
  in the query for the shard that is supposed to contain the key,only the
  replica of the shard that has this key returns the result , and the
 leader
  does not , looks like replica and the leader do not have the same data
 and
  replica seems to contain the key in the query for that shard.
 
  2 By indexing I mean this collection is being populated by a web
 crawler.
 
  So looks like 1 above  is pointing to leader and replica being out of
  synch for atleast one shard.
 
 
 
  On Thu, Oct 2, 2014 at 11:57 PM, Erick Erickson 
 erickerick...@gmail.com
  wrote:
 
  bq: Also ,the collection is being actively indexed as I query this,
 could
  that
  be an issue too ?
 
  Not if the documents you're searching aren't being added as you search
  (and all your autocommit intervals have expired).
 
  I would turn off indexing for testing, it's just one more variable
  that can get in the way of understanding this.
 
  Do note that if the problem were endemic to Solr, there would probably
  be a _lot_ more noise out there.
 
  So to recap:
  0 we can take the load balancer out of the picture all together.
 
  1 when you query each shard individually with distrib=true, every
  replica in a particular shard returns the same count.
 
  2 when you query without distrib=true you get varying counts.
 
  This is very strange and not at all expected. Let's try it again
  without indexing going on
 
  And what do you mean by indexing anyway? How are documents being fed
  to your system?
 
  Best,
  Erick@PuzzledAsWell
 
  On Thu, Oct 2, 2014 at 7:32 PM, S.L simpleliving...@gmail.com wrote:
   Erick,
  
   I would like to add that the interesting behavior i.e point #2 that I
   mentioned in my earlier reply  happens in all the shards , if this
 were
  to
   be a distributed search issue

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[Erick Erickson]

Not, I'm not guaranteeing that it'll actually cure the problem, just
that enough has changed since 4.7 that it'd be a good place to start.

Things have been reported off and on, but they're often pesky race
conditions or something else that takes a long time to track down, you
just are lucky perhaps ;)...

Erick

On Mon, Oct 6, 2014 at 8:04 PM, S.L simpleliving...@gmail.com wrote:
 Erick,

 Thanks for the suggestion , I am not sure if I would be able to capture
 what went wrong , so upgrading to 4.10 seems easier even though it means ,
 a days work of effort :) . I will go ahead and upgrade and let me know ,
 although I am surprised that this issue never got reported for 4.7 up until
 now.

 Thanks again for your help!



 On Mon, Oct 6, 2014 at 10:52 PM, Erick Erickson erickerick...@gmail.com
 wrote:

 I think there were some holes that would allow replicas and leaders to
 be out of synch that have been patched up in the last 3 releases.

 There shouldn't be anything you need to do to keep these in synch, so
 if you can capture what happened when things got out of synch we'll
 fix it. But a lot has changed in the last several months, so the first
 thing I'd do if possible is to upgrade to 4.10.1.


 Best,
 Erick

 On Mon, Oct 6, 2014 at 2:41 PM, S.L simpleliving...@gmail.com wrote:
  Hi Erick,
 
  Before I tried your suggestion of  issung a commit=true update, I
 realized that for eaach shard there was atleast a node that had its index
 directory named like index.timestamp.
 
  I went ahead and deleted index directory that restarted that core and
 now the index directory got syched with the other node and is properly
 named as 'index' without any timestamp attached to it.This is now giving me
 consistent results for distrib=true using a load balancer.Also
 distrib=false returns expexted results for a given shard.
 
  The underlying issue appears to be that in every shard the leader and
 the replica(follower) were out of sych.
 
  How can I avoid this from happening again?
 
  Thanks for your help!
 
  Sent from my HTC
 
  - Reply message -
  From: Erick Erickson erickerick...@gmail.com
  To: solr-user@lucene.apache.org
  Subject: SolrCloud 4.7 not doing distributed search when querying from a
 load balancer.
  Date: Fri, Oct 3, 2014 12:56 AM
 
  H. Assuming that you aren't re-indexing the doc you're searching
 for...
 
  Try issuing http://blah blah:8983/solr/collection/update?commit=true.
  That'll force all the docs to be searchable. Does 1 still hold for
  the document in question? Because this is exactly backwards of what
  I'd expect. I'd expect, if anything, the replica (I'm trying to call
  it the follower when a distinction needs to be made since the leader
  is a replica too) would be out of sync. This is still a Bad
  Thing, but the leader gets first crack at indexing thing.
 
  bq: only the replica of the shard that has this key returns the result
  , and the leader does not ,
 
  Just to be sure we're talking about the same thing. When you say
  leader, you mean the shard leader, right? The filled-in circle on
  the graph view from the admin/cloud page.
 
  And let's see your soft and hard commit settings please.
 
  Best,
  Erick
 
  On Thu, Oct 2, 2014 at 9:48 PM, S.L simpleliving...@gmail.com wrote:
  Eirck,
 
  0 Load balancer is out of the picture
  .
  1When I query with *distrib=false* , I get consistent results as
 expected
  for those shards that dont have the key i.e I dont get the results back
 for
  those shards, however I just realized that while *distrib=false* is
 present
  in the query for the shard that is supposed to contain the key,only the
  replica of the shard that has this key returns the result , and the
 leader
  does not , looks like replica and the leader do not have the same data
 and
  replica seems to contain the key in the query for that shard.
 
  2 By indexing I mean this collection is being populated by a web
 crawler.
 
  So looks like 1 above  is pointing to leader and replica being out of
  synch for atleast one shard.
 
 
 
  On Thu, Oct 2, 2014 at 11:57 PM, Erick Erickson 
 erickerick...@gmail.com
  wrote:
 
  bq: Also ,the collection is being actively indexed as I query this,
 could
  that
  be an issue too ?
 
  Not if the documents you're searching aren't being added as you search
  (and all your autocommit intervals have expired).
 
  I would turn off indexing for testing, it's just one more variable
  that can get in the way of understanding this.
 
  Do note that if the problem were endemic to Solr, there would probably
  be a _lot_ more noise out there.
 
  So to recap:
  0 we can take the load balancer out of the picture all together.
 
  1 when you query each shard individually with distrib=true, every
  replica in a particular shard returns the same count.
 
  2 when you query without distrib=true you get varying counts.
 
  This is very strange and not at all expected. Let's try it again
  without indexing going

SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[S.L]

Hi All,

I am trying to query a 6 node Solr4.7  cluster with 3 shards and  a
replication factor of 2 .

I have fronted these 6 Solr nodes using a load balancer , what I notice is
that every time I do a search of the form
q=*:*fq=(id:9e78c064-919f-4ef3-b236-dc66351b4acf)  it gives me a result
only once in every 3 tries , telling me that the load balancer is
distributing the requests between the 3 shards and SolrCloud only returns a
result if the request goes to the core that as that id .

However if I do a simple search like q=*:* , I consistently get the right
aggregated results back of all the documents across all the shards for
every request from the load balancer. Can someone please let me know what
this is symptomatic of ?

Somehow Solr Cloud seems to be doing search query distribution and
aggregation for queries of type *:* only.

Thanks.

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[Erick Erickson]

Hmmm, nothing quite makes sense here

Here are some experiments:
1 avoid the load balancer and issue queries like
http://solr_server:8983/solr/collection/q=whateverdistrib=false

the distrib=false bit will cause keep SolrCloud from trying to send
the queries anywhere, they'll be served only from the node you address them to.
that'll help check whether the nodes are consistent. You should be
getting back the same results from each replica in a shard (i.e. 2 of
your 6 machines).

Next, try your failing query the same way.

Next, try your failing query from a browser, pointing it at successive
nodes.

Where is the first place problems show up?

My _guess_ is that your load balancer isn't quite doing what you think, or
your cluster isn't set up the way you think it is, but those are guesses.

Best,
Erick

On Thu, Oct 2, 2014 at 2:51 PM, S.L simpleliving...@gmail.com wrote:
 Hi All,

 I am trying to query a 6 node Solr4.7  cluster with 3 shards and  a
 replication factor of 2 .

 I have fronted these 6 Solr nodes using a load balancer , what I notice is
 that every time I do a search of the form
 q=*:*fq=(id:9e78c064-919f-4ef3-b236-dc66351b4acf)  it gives me a result
 only once in every 3 tries , telling me that the load balancer is
 distributing the requests between the 3 shards and SolrCloud only returns a
 result if the request goes to the core that as that id .

 However if I do a simple search like q=*:* , I consistently get the right
 aggregated results back of all the documents across all the shards for
 every request from the load balancer. Can someone please let me know what
 this is symptomatic of ?

 Somehow Solr Cloud seems to be doing search query distribution and
 aggregation for queries of type *:* only.

 Thanks.

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[S.L]

Erick,

Thanks for your reply, I tried your suggestions.

1 . When not using loadbalancer if  *I have distrib=false* I get consistent
results across the replicas.

2. However here's the insteresting part , while not using load balancer if
I *dont have distrib=false* , then when I query a particular node ,I get
the same behaviour as if I were using a loadbalancer , meaning the
distributed search from a node works intermittently .Does this give any
clue ?



On Thu, Oct 2, 2014 at 7:47 PM, Erick Erickson erickerick...@gmail.com
wrote:

 Hmmm, nothing quite makes sense here

 Here are some experiments:
 1 avoid the load balancer and issue queries like
 http://solr_server:8983/solr/collection/q=whateverdistrib=false

 the distrib=false bit will cause keep SolrCloud from trying to send
 the queries anywhere, they'll be served only from the node you address
 them to.
 that'll help check whether the nodes are consistent. You should be
 getting back the same results from each replica in a shard (i.e. 2 of
 your 6 machines).

 Next, try your failing query the same way.

 Next, try your failing query from a browser, pointing it at successive
 nodes.

 Where is the first place problems show up?

 My _guess_ is that your load balancer isn't quite doing what you think, or
 your cluster isn't set up the way you think it is, but those are guesses.

 Best,
 Erick

 On Thu, Oct 2, 2014 at 2:51 PM, S.L simpleliving...@gmail.com wrote:
  Hi All,
 
  I am trying to query a 6 node Solr4.7  cluster with 3 shards and  a
  replication factor of 2 .
 
  I have fronted these 6 Solr nodes using a load balancer , what I notice
 is
  that every time I do a search of the form
  q=*:*fq=(id:9e78c064-919f-4ef3-b236-dc66351b4acf)  it gives me a result
  only once in every 3 tries , telling me that the load balancer is
  distributing the requests between the 3 shards and SolrCloud only
 returns a
  result if the request goes to the core that as that id .
 
  However if I do a simple search like q=*:* , I consistently get the right
  aggregated results back of all the documents across all the shards for
  every request from the load balancer. Can someone please let me know what
  this is symptomatic of ?
 
  Somehow Solr Cloud seems to be doing search query distribution and
  aggregation for queries of type *:* only.
 
  Thanks.

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[S.L]

Erick,

I would like to add that the interesting behavior i.e point #2 that I
mentioned in my earlier reply  happens in all the shards , if this were to
be a distributed search issue this should have not manifested itself in the
shard that contains the key that I am searching for , looks like the search
is just failing as whole intermittently .

Also ,the collection is being actively indexed as I query this, could that
be an issue too ?

Thanks.

On Thu, Oct 2, 2014 at 10:24 PM, S.L simpleliving...@gmail.com wrote:

 Erick,

 Thanks for your reply, I tried your suggestions.

 1 . When not using loadbalancer if  *I have distrib=false* I get
 consistent results across the replicas.

 2. However here's the insteresting part , while not using load balancer if
 I *dont have distrib=false* , then when I query a particular node ,I get
 the same behaviour as if I were using a loadbalancer , meaning the
 distributed search from a node works intermittently .Does this give any
 clue ?



 On Thu, Oct 2, 2014 at 7:47 PM, Erick Erickson erickerick...@gmail.com
 wrote:

 Hmmm, nothing quite makes sense here

 Here are some experiments:
 1 avoid the load balancer and issue queries like
 http://solr_server:8983/solr/collection/q=whateverdistrib=false

 the distrib=false bit will cause keep SolrCloud from trying to send
 the queries anywhere, they'll be served only from the node you address
 them to.
 that'll help check whether the nodes are consistent. You should be
 getting back the same results from each replica in a shard (i.e. 2 of
 your 6 machines).

 Next, try your failing query the same way.

 Next, try your failing query from a browser, pointing it at successive
 nodes.

 Where is the first place problems show up?

 My _guess_ is that your load balancer isn't quite doing what you think, or
 your cluster isn't set up the way you think it is, but those are guesses.

 Best,
 Erick

 On Thu, Oct 2, 2014 at 2:51 PM, S.L simpleliving...@gmail.com wrote:
  Hi All,
 
  I am trying to query a 6 node Solr4.7  cluster with 3 shards and  a
  replication factor of 2 .
 
  I have fronted these 6 Solr nodes using a load balancer , what I notice
 is
  that every time I do a search of the form
  q=*:*fq=(id:9e78c064-919f-4ef3-b236-dc66351b4acf)  it gives me a result
  only once in every 3 tries , telling me that the load balancer is
  distributing the requests between the 3 shards and SolrCloud only
 returns a
  result if the request goes to the core that as that id .
 
  However if I do a simple search like q=*:* , I consistently get the
 right
  aggregated results back of all the documents across all the shards for
  every request from the load balancer. Can someone please let me know
 what
  this is symptomatic of ?
 
  Somehow Solr Cloud seems to be doing search query distribution and
  aggregation for queries of type *:* only.
 
  Thanks.

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[Erick Erickson]

bq: Also ,the collection is being actively indexed as I query this, could that
be an issue too ?

Not if the documents you're searching aren't being added as you search
(and all your autocommit intervals have expired).

I would turn off indexing for testing, it's just one more variable
that can get in the way of understanding this.

Do note that if the problem were endemic to Solr, there would probably
be a _lot_ more noise out there.

So to recap:
0 we can take the load balancer out of the picture all together.

1 when you query each shard individually with distrib=true, every
replica in a particular shard returns the same count.

2 when you query without distrib=true you get varying counts.

This is very strange and not at all expected. Let's try it again
without indexing going on

And what do you mean by indexing anyway? How are documents being fed
to your system?

Best,
Erick@PuzzledAsWell

On Thu, Oct 2, 2014 at 7:32 PM, S.L simpleliving...@gmail.com wrote:
 Erick,

 I would like to add that the interesting behavior i.e point #2 that I
 mentioned in my earlier reply  happens in all the shards , if this were to
 be a distributed search issue this should have not manifested itself in the
 shard that contains the key that I am searching for , looks like the search
 is just failing as whole intermittently .

 Also ,the collection is being actively indexed as I query this, could that
 be an issue too ?

 Thanks.

 On Thu, Oct 2, 2014 at 10:24 PM, S.L simpleliving...@gmail.com wrote:

 Erick,

 Thanks for your reply, I tried your suggestions.

 1 . When not using loadbalancer if  *I have distrib=false* I get
 consistent results across the replicas.

 2. However here's the insteresting part , while not using load balancer if
 I *dont have distrib=false* , then when I query a particular node ,I get
 the same behaviour as if I were using a loadbalancer , meaning the
 distributed search from a node works intermittently .Does this give any
 clue ?



 On Thu, Oct 2, 2014 at 7:47 PM, Erick Erickson erickerick...@gmail.com
 wrote:

 Hmmm, nothing quite makes sense here

 Here are some experiments:
 1 avoid the load balancer and issue queries like
 http://solr_server:8983/solr/collection/q=whateverdistrib=false

 the distrib=false bit will cause keep SolrCloud from trying to send
 the queries anywhere, they'll be served only from the node you address
 them to.
 that'll help check whether the nodes are consistent. You should be
 getting back the same results from each replica in a shard (i.e. 2 of
 your 6 machines).

 Next, try your failing query the same way.

 Next, try your failing query from a browser, pointing it at successive
 nodes.

 Where is the first place problems show up?

 My _guess_ is that your load balancer isn't quite doing what you think, or
 your cluster isn't set up the way you think it is, but those are guesses.

 Best,
 Erick

 On Thu, Oct 2, 2014 at 2:51 PM, S.L simpleliving...@gmail.com wrote:
  Hi All,
 
  I am trying to query a 6 node Solr4.7  cluster with 3 shards and  a
  replication factor of 2 .
 
  I have fronted these 6 Solr nodes using a load balancer , what I notice
 is
  that every time I do a search of the form
  q=*:*fq=(id:9e78c064-919f-4ef3-b236-dc66351b4acf)  it gives me a result
  only once in every 3 tries , telling me that the load balancer is
  distributing the requests between the 3 shards and SolrCloud only
 returns a
  result if the request goes to the core that as that id .
 
  However if I do a simple search like q=*:* , I consistently get the
 right
  aggregated results back of all the documents across all the shards for
  every request from the load balancer. Can someone please let me know
 what
  this is symptomatic of ?
 
  Somehow Solr Cloud seems to be doing search query distribution and
  aggregation for queries of type *:* only.
 
  Thanks.

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[S.L]

Eirck,

0 Load balancer is out of the picture
.
1When I query with *distrib=false* , I get consistent results as expected
for those shards that dont have the key i.e I dont get the results back for
those shards, however I just realized that while *distrib=false* is present
in the query for the shard that is supposed to contain the key,only the
replica of the shard that has this key returns the result , and the leader
does not , looks like replica and the leader do not have the same data and
replica seems to contain the key in the query for that shard.

2 By indexing I mean this collection is being populated by a web crawler.

So looks like 1 above  is pointing to leader and replica being out of
synch for atleast one shard.



On Thu, Oct 2, 2014 at 11:57 PM, Erick Erickson erickerick...@gmail.com
wrote:

 bq: Also ,the collection is being actively indexed as I query this, could
 that
 be an issue too ?

 Not if the documents you're searching aren't being added as you search
 (and all your autocommit intervals have expired).

 I would turn off indexing for testing, it's just one more variable
 that can get in the way of understanding this.

 Do note that if the problem were endemic to Solr, there would probably
 be a _lot_ more noise out there.

 So to recap:
 0 we can take the load balancer out of the picture all together.

 1 when you query each shard individually with distrib=true, every
 replica in a particular shard returns the same count.

 2 when you query without distrib=true you get varying counts.

 This is very strange and not at all expected. Let's try it again
 without indexing going on

 And what do you mean by indexing anyway? How are documents being fed
 to your system?

 Best,
 Erick@PuzzledAsWell

 On Thu, Oct 2, 2014 at 7:32 PM, S.L simpleliving...@gmail.com wrote:
  Erick,
 
  I would like to add that the interesting behavior i.e point #2 that I
  mentioned in my earlier reply  happens in all the shards , if this were
 to
  be a distributed search issue this should have not manifested itself in
 the
  shard that contains the key that I am searching for , looks like the
 search
  is just failing as whole intermittently .
 
  Also ,the collection is being actively indexed as I query this, could
 that
  be an issue too ?
 
  Thanks.
 
  On Thu, Oct 2, 2014 at 10:24 PM, S.L simpleliving...@gmail.com wrote:
 
  Erick,
 
  Thanks for your reply, I tried your suggestions.
 
  1 . When not using loadbalancer if  *I have distrib=false* I get
  consistent results across the replicas.
 
  2. However here's the insteresting part , while not using load balancer
 if
  I *dont have distrib=false* , then when I query a particular node ,I get
  the same behaviour as if I were using a loadbalancer , meaning the
  distributed search from a node works intermittently .Does this give any
  clue ?
 
 
 
  On Thu, Oct 2, 2014 at 7:47 PM, Erick Erickson erickerick...@gmail.com
 
  wrote:
 
  Hmmm, nothing quite makes sense here
 
  Here are some experiments:
  1 avoid the load balancer and issue queries like
  http://solr_server:8983/solr/collection/q=whateverdistrib=false
 
  the distrib=false bit will cause keep SolrCloud from trying to send
  the queries anywhere, they'll be served only from the node you address
  them to.
  that'll help check whether the nodes are consistent. You should be
  getting back the same results from each replica in a shard (i.e. 2 of
  your 6 machines).
 
  Next, try your failing query the same way.
 
  Next, try your failing query from a browser, pointing it at successive
  nodes.
 
  Where is the first place problems show up?
 
  My _guess_ is that your load balancer isn't quite doing what you
 think, or
  your cluster isn't set up the way you think it is, but those are
 guesses.
 
  Best,
  Erick
 
  On Thu, Oct 2, 2014 at 2:51 PM, S.L simpleliving...@gmail.com wrote:
   Hi All,
  
   I am trying to query a 6 node Solr4.7  cluster with 3 shards and  a
   replication factor of 2 .
  
   I have fronted these 6 Solr nodes using a load balancer , what I
 notice
  is
   that every time I do a search of the form
   q=*:*fq=(id:9e78c064-919f-4ef3-b236-dc66351b4acf)  it gives me a
 result
   only once in every 3 tries , telling me that the load balancer is
   distributing the requests between the 3 shards and SolrCloud only
  returns a
   result if the request goes to the core that as that id .
  
   However if I do a simple search like q=*:* , I consistently get the
  right
   aggregated results back of all the documents across all the shards
 for
   every request from the load balancer. Can someone please let me know
  what
   this is symptomatic of ?
  
   Somehow Solr Cloud seems to be doing search query distribution and
   aggregation for queries of type *:* only.
  
   Thanks.
 
 
 

Re: SolrCloud 4.7 not doing distributed search when querying from a load balancer.

[Erick Erickson]

H. Assuming that you aren't re-indexing the doc you're searching for...

Try issuing http://blah blah:8983/solr/collection/update?commit=true.
That'll force all the docs to be searchable. Does 1 still hold for
the document in question? Because this is exactly backwards of what
I'd expect. I'd expect, if anything, the replica (I'm trying to call
it the follower when a distinction needs to be made since the leader
is a replica too) would be out of sync. This is still a Bad
Thing, but the leader gets first crack at indexing thing.

bq: only the replica of the shard that has this key returns the result
, and the leader does not ,

Just to be sure we're talking about the same thing. When you say
leader, you mean the shard leader, right? The filled-in circle on
the graph view from the admin/cloud page.

And let's see your soft and hard commit settings please.

Best,
Erick

On Thu, Oct 2, 2014 at 9:48 PM, S.L simpleliving...@gmail.com wrote:
 Eirck,

 0 Load balancer is out of the picture
 .
 1When I query with *distrib=false* , I get consistent results as expected
 for those shards that dont have the key i.e I dont get the results back for
 those shards, however I just realized that while *distrib=false* is present
 in the query for the shard that is supposed to contain the key,only the
 replica of the shard that has this key returns the result , and the leader
 does not , looks like replica and the leader do not have the same data and
 replica seems to contain the key in the query for that shard.

 2 By indexing I mean this collection is being populated by a web crawler.

 So looks like 1 above  is pointing to leader and replica being out of
 synch for atleast one shard.



 On Thu, Oct 2, 2014 at 11:57 PM, Erick Erickson erickerick...@gmail.com
 wrote:

 bq: Also ,the collection is being actively indexed as I query this, could
 that
 be an issue too ?

 Not if the documents you're searching aren't being added as you search
 (and all your autocommit intervals have expired).

 I would turn off indexing for testing, it's just one more variable
 that can get in the way of understanding this.

 Do note that if the problem were endemic to Solr, there would probably
 be a _lot_ more noise out there.

 So to recap:
 0 we can take the load balancer out of the picture all together.

 1 when you query each shard individually with distrib=true, every
 replica in a particular shard returns the same count.

 2 when you query without distrib=true you get varying counts.

 This is very strange and not at all expected. Let's try it again
 without indexing going on

 And what do you mean by indexing anyway? How are documents being fed
 to your system?

 Best,
 Erick@PuzzledAsWell

 On Thu, Oct 2, 2014 at 7:32 PM, S.L simpleliving...@gmail.com wrote:
  Erick,
 
  I would like to add that the interesting behavior i.e point #2 that I
  mentioned in my earlier reply  happens in all the shards , if this were
 to
  be a distributed search issue this should have not manifested itself in
 the
  shard that contains the key that I am searching for , looks like the
 search
  is just failing as whole intermittently .
 
  Also ,the collection is being actively indexed as I query this, could
 that
  be an issue too ?
 
  Thanks.
 
  On Thu, Oct 2, 2014 at 10:24 PM, S.L simpleliving...@gmail.com wrote:
 
  Erick,
 
  Thanks for your reply, I tried your suggestions.
 
  1 . When not using loadbalancer if  *I have distrib=false* I get
  consistent results across the replicas.
 
  2. However here's the insteresting part , while not using load balancer
 if
  I *dont have distrib=false* , then when I query a particular node ,I get
  the same behaviour as if I were using a loadbalancer , meaning the
  distributed search from a node works intermittently .Does this give any
  clue ?
 
 
 
  On Thu, Oct 2, 2014 at 7:47 PM, Erick Erickson erickerick...@gmail.com
 
  wrote:
 
  Hmmm, nothing quite makes sense here
 
  Here are some experiments:
  1 avoid the load balancer and issue queries like
  http://solr_server:8983/solr/collection/q=whateverdistrib=false
 
  the distrib=false bit will cause keep SolrCloud from trying to send
  the queries anywhere, they'll be served only from the node you address
  them to.
  that'll help check whether the nodes are consistent. You should be
  getting back the same results from each replica in a shard (i.e. 2 of
  your 6 machines).
 
  Next, try your failing query the same way.
 
  Next, try your failing query from a browser, pointing it at successive
  nodes.
 
  Where is the first place problems show up?
 
  My _guess_ is that your load balancer isn't quite doing what you
 think, or
  your cluster isn't set up the way you think it is, but those are
 guesses.
 
  Best,
  Erick
 
  On Thu, Oct 2, 2014 at 2:51 PM, S.L simpleliving...@gmail.com wrote:
   Hi All,
  
   I am trying to query a 6 node Solr4.7  cluster with 3 shards and  a
   replication factor of 2 .
  
   I have fronted these 6 Solr