Re: Using parameter values in a sort

[Walter Underwood]

No, I tried that before adding the default.

But the solrconfig.xml is rejected before there is a request, so this is not 
about requests. I did try “scores” in the requests, but of course it didn’t 
work because the solrconfig.xml was not loaded. I did not turn off parameter 
substitutions. This is a pretty vanilla solrconfig.xml.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Feb 27, 2017, at 6:44 PM, Erik Hatcher  wrote:
> 
> `scores` (plural), you’ve got this below:   
> 
> Remove that, and like my previous e-mail, and use `scores` (plural) from the 
> request and _should_ work?
> 
>   Erik
> 
>> On Feb 27, 2017, at 9:42 PM, Walter Underwood  wrote:
>> 
>> I’ve passed in a score parameter, but the solrconfig.xml is rejected before 
>> any requests.
>> 
>> Pretty ready to give up. The documentation around function queries and 
>> params is not working for me, though I’ve been using Solr for ten years. I 
>> have figured out a lot of systems. This is impenetrable.
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>> 
>>> On Feb 27, 2017, at 6:35 PM, Erik Hatcher  wrote:
>>> 
>>> You have an empty “scores” parameter in there.  You’re not showing your 
>>> full search request, but did you provide that in the request?   Have you 
>>> perhaps turned off parameter substitutions?
>>> 
>>> Erik
>>> 
 On Feb 27, 2017, at 9:26 PM, Walter Underwood  
 wrote:
 
 With this in the config…
 
 
   
  
edismax
0
false
id,
image_thumb_large, image_thumb_medium, image_thumb_small,
image_thumb_xlarge, uri, user_id, last_name, first_name,
name, school, major, graduation_year, tutor_profile_id,
positive_reviews, negative_reviews, gender, about_experience,
about_extracurricular, time_approved
*:*
about_experience  about_extracurricular School  
 Major

>>> name="sort">sum(interaction_responsiveness_score,profile_completeness_score,school_score,us_tax_id_score,highlight_score,${scores})
  desc
log(sum(1,max(positive_reviews,0)))
0.1
  
 
 
 I see this… [Solr 6.3.0]
 
 org.apache.solr.common.SolrException: Unable to reload core 
 [tutors_shard1_replica11]
at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:950)
at 
 org.apache.solr.core.SolrCore.lambda$getConfListener$6(SolrCore.java:2708)
at 
 org.apache.solr.cloud.ZkController.lambda$fireEventListeners$4(ZkController.java:2448)
at java.lang.Thread.run(Thread.java:745)
 Caused by: org.apache.solr.common.SolrException: Could not load conf for 
 core tutors_shard1_replica11: Error loading solr config from solrconfig.xml
at 
 org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:85)
at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:942)
... 3 more
 Caused by: org.apache.solr.common.SolrException: Error loading solr config 
 from solrconfig.xml
at 
 org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:187)
at 
 org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:97)
at 
 org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:77)
... 4 more
 Caused by: org.apache.solr.common.SolrException: No system property or 
 default value specified for scores 
 value:sum(interaction_responsiveness_score,profile_completeness_score,school_score,us_tax_id_score,highlight_score,${scores})
  desc
at 
 org.apache.solr.util.PropertiesUtil.substituteProperty(PropertiesUtil.java:65)
at org.apache.solr.util.DOMUtil.substituteProperties(DOMUtil.java:298)
 
 wunder
 Walter Underwood
 wun...@wunderwood.org
 http://observer.wunderwood.org/  (my blog)
 
 
> On Feb 27, 2017, at 6:17 PM, Erik Hatcher  wrote:
> 
> Walter -
> 
> How about this, for the latter part of your request:
> 
> /handler?features=a,b,c
> 
> with =sum(${features}) desc
> 
> That ought to do the trick.   At first I thought the #foreach nature of 
> the list of features was prohibitive, but since you’re literally plugging 
> in the exact string value and it’s used as a comma-separated list then 
> this should work.
> 
> But with the just a list of subject ID’s I think you’re in custom 
> development now (or a JavaScript stage in a Fusion query pipeline ;) in 
> building a SearchComponent that takes a `features` parameter and builds 
> the sort param as needed from that.
> 
>   Erik
> 
> 
> 
>> On Feb 27, 2017, at 7:17 PM, Walter Underwood 

Re: Using parameter values in a sort

[Erik Hatcher]

It looks like you’ve got score/scores mismatching going on.   

I feel your frustration through e-mail.   It’s voodoo for sure, but hopefully 
it’s just typos that are getting  you now.  I’d simplify it down to an example 
like I provided without mixing in any other variables or params (even for us 
seasoned pros, simplification is often the root of problem solving in 
frustrating times; get back to what works) and let’s see the request (and 
echoParam=all output when the request is successful).

Erik


> On Feb 27, 2017, at 9:48 PM, Walter Underwood  wrote:
> 
> I added that line because I was getting an error about it being undefined.
> 
>   
> 
> At this point, I’m just doing random shit hoping it will work. There is not 
> enough documentation to use this.
> 
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
> 
> 
>> On Feb 27, 2017, at 6:44 PM, Erik Hatcher  wrote:
>> 
>> `scores` (plural), you’ve got this below:   
>> 
>> Remove that, and like my previous e-mail, and use `scores` (plural) from the 
>> request and _should_ work?
>> 
>>  Erik
>> 
>>> On Feb 27, 2017, at 9:42 PM, Walter Underwood  wrote:
>>> 
>>> I’ve passed in a score parameter, but the solrconfig.xml is rejected before 
>>> any requests.
>>> 
>>> Pretty ready to give up. The documentation around function queries and 
>>> params is not working for me, though I’ve been using Solr for ten years. I 
>>> have figured out a lot of systems. This is impenetrable.
>>> 
>>> wunder
>>> Walter Underwood
>>> wun...@wunderwood.org
>>> http://observer.wunderwood.org/  (my blog)
>>> 
>>> 
 On Feb 27, 2017, at 6:35 PM, Erik Hatcher  wrote:
 
 You have an empty “scores” parameter in there.  You’re not showing your 
 full search request, but did you provide that in the request?   Have you 
 perhaps turned off parameter substitutions?
 
Erik
 
> On Feb 27, 2017, at 9:26 PM, Walter Underwood  
> wrote:
> 
> With this in the config…
> 
> 
>  
> 
>   edismax
>   0
>   false
>   id,
>   image_thumb_large, image_thumb_medium, image_thumb_small,
>   image_thumb_xlarge, uri, user_id, last_name, first_name,
>   name, school, major, graduation_year, tutor_profile_id,
>   positive_reviews, negative_reviews, gender, about_experience,
>   about_extracurricular, time_approved
>   *:*
>   about_experience  about_extracurricular School  
> Major
>   
>    name="sort">sum(interaction_responsiveness_score,profile_completeness_score,school_score,us_tax_id_score,highlight_score,${scores})
>  desc
>   log(sum(1,max(positive_reviews,0)))
>   0.1
> 
> 
> 
> I see this… [Solr 6.3.0]
> 
> org.apache.solr.common.SolrException: Unable to reload core 
> [tutors_shard1_replica11]
>   at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:950)
>   at 
> org.apache.solr.core.SolrCore.lambda$getConfListener$6(SolrCore.java:2708)
>   at 
> org.apache.solr.cloud.ZkController.lambda$fireEventListeners$4(ZkController.java:2448)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.solr.common.SolrException: Could not load conf for 
> core tutors_shard1_replica11: Error loading solr config from 
> solrconfig.xml
>   at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:85)
>   at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:942)
>   ... 3 more
> Caused by: org.apache.solr.common.SolrException: Error loading solr 
> config from solrconfig.xml
>   at 
> org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:187)
>   at 
> org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:97)
>   at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:77)
>   ... 4 more
> Caused by: org.apache.solr.common.SolrException: No system property or 
> default value specified for scores 
> value:sum(interaction_responsiveness_score,profile_completeness_score,school_score,us_tax_id_score,highlight_score,${scores})
>  desc
>   at 
> org.apache.solr.util.PropertiesUtil.substituteProperty(PropertiesUtil.java:65)
>   at org.apache.solr.util.DOMUtil.substituteProperties(DOMUtil.java:298)
> 
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
> 
> 
>> On Feb 27, 2017, at 6:17 PM, Erik Hatcher  wrote:
>> 
>> Walter -
>> 
>> How about this, for the latter part of your request:
>> 
>> /handler?features=a,b,c
>> 
>> with =sum(${features}) desc
>> 
>> That ought to do the trick.   At first I thought the #foreach 

Re: Using parameter values in a sort

[Erik Hatcher]

> On Feb 27, 2017, at 9:42 PM, Walter Underwood  wrote:
> Pretty ready to give up. The documentation around function queries and params 
> is not working for me, though I’ve been using Solr for ten years. I have 
> figured out a lot of systems. This is impenetrable.

Here’s how I (try to) explain it in Solr training:

There’s plain uncurly bracketed parameter substitution.  This is where a 
feature (say local params, or function queries) is designed to indirect an 
exact parameter and pull it in:

/select?q=*:*={!terms f=id v=$id_list}_list=1,2,3

And then there’s macro substitution, with curly bracketed syntax, and this 
is an in place string substitution allowing gluing of things together:

  /select?q=*:*=${sort_field_name} desc_field_name=price

When in doubt, the curly bracketed syntax actually should do the trick, but 
there’s always the fun aspect of whitespace and escaping and quoting and such, 
so it is tricky business.   A browser, some trial-and-error, and hopefully some 
tips and tricks like these that emerge online will be helpful in demystifying 
this and making it more accessible and useful/usable.

Erik

Re: Using parameter values in a sort

[Walter Underwood]

I added that line because I was getting an error about it being undefined.

   

At this point, I’m just doing random shit hoping it will work. There is not 
enough documentation to use this.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Feb 27, 2017, at 6:44 PM, Erik Hatcher  wrote:
> 
> `scores` (plural), you’ve got this below:   
> 
> Remove that, and like my previous e-mail, and use `scores` (plural) from the 
> request and _should_ work?
> 
>   Erik
> 
>> On Feb 27, 2017, at 9:42 PM, Walter Underwood  wrote:
>> 
>> I’ve passed in a score parameter, but the solrconfig.xml is rejected before 
>> any requests.
>> 
>> Pretty ready to give up. The documentation around function queries and 
>> params is not working for me, though I’ve been using Solr for ten years. I 
>> have figured out a lot of systems. This is impenetrable.
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>> 
>>> On Feb 27, 2017, at 6:35 PM, Erik Hatcher  wrote:
>>> 
>>> You have an empty “scores” parameter in there.  You’re not showing your 
>>> full search request, but did you provide that in the request?   Have you 
>>> perhaps turned off parameter substitutions?
>>> 
>>> Erik
>>> 
 On Feb 27, 2017, at 9:26 PM, Walter Underwood  
 wrote:
 
 With this in the config…
 
 
   
  
edismax
0
false
id,
image_thumb_large, image_thumb_medium, image_thumb_small,
image_thumb_xlarge, uri, user_id, last_name, first_name,
name, school, major, graduation_year, tutor_profile_id,
positive_reviews, negative_reviews, gender, about_experience,
about_extracurricular, time_approved
*:*
about_experience  about_extracurricular School  
 Major

>>> name="sort">sum(interaction_responsiveness_score,profile_completeness_score,school_score,us_tax_id_score,highlight_score,${scores})
  desc
log(sum(1,max(positive_reviews,0)))
0.1
  
 
 
 I see this… [Solr 6.3.0]
 
 org.apache.solr.common.SolrException: Unable to reload core 
 [tutors_shard1_replica11]
at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:950)
at 
 org.apache.solr.core.SolrCore.lambda$getConfListener$6(SolrCore.java:2708)
at 
 org.apache.solr.cloud.ZkController.lambda$fireEventListeners$4(ZkController.java:2448)
at java.lang.Thread.run(Thread.java:745)
 Caused by: org.apache.solr.common.SolrException: Could not load conf for 
 core tutors_shard1_replica11: Error loading solr config from solrconfig.xml
at 
 org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:85)
at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:942)
... 3 more
 Caused by: org.apache.solr.common.SolrException: Error loading solr config 
 from solrconfig.xml
at 
 org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:187)
at 
 org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:97)
at 
 org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:77)
... 4 more
 Caused by: org.apache.solr.common.SolrException: No system property or 
 default value specified for scores 
 value:sum(interaction_responsiveness_score,profile_completeness_score,school_score,us_tax_id_score,highlight_score,${scores})
  desc
at 
 org.apache.solr.util.PropertiesUtil.substituteProperty(PropertiesUtil.java:65)
at org.apache.solr.util.DOMUtil.substituteProperties(DOMUtil.java:298)
 
 wunder
 Walter Underwood
 wun...@wunderwood.org
 http://observer.wunderwood.org/  (my blog)
 
 
> On Feb 27, 2017, at 6:17 PM, Erik Hatcher  wrote:
> 
> Walter -
> 
> How about this, for the latter part of your request:
> 
> /handler?features=a,b,c
> 
> with =sum(${features}) desc
> 
> That ought to do the trick.   At first I thought the #foreach nature of 
> the list of features was prohibitive, but since you’re literally plugging 
> in the exact string value and it’s used as a comma-separated list then 
> this should work.
> 
> But with the just a list of subject ID’s I think you’re in custom 
> development now (or a JavaScript stage in a Fusion query pipeline ;) in 
> building a SearchComponent that takes a `features` parameter and builds 
> the sort param as needed from that.
> 
>   Erik
> 
> 
> 
>> On Feb 27, 2017, at 7:17 PM, Walter Underwood  
>> wrote:
>> 
>> We have documents with parameterized features. For a school subject 
>> (calculus, 

Re: Using parameter values in a sort

[Erik Hatcher]

`scores` (plural), you’ve got this below:   

Remove that, and like my previous e-mail, and use `scores` (plural) from the 
request and _should_ work?

Erik

> On Feb 27, 2017, at 9:42 PM, Walter Underwood  wrote:
> 
> I’ve passed in a score parameter, but the solrconfig.xml is rejected before 
> any requests.
> 
> Pretty ready to give up. The documentation around function queries and params 
> is not working for me, though I’ve been using Solr for ten years. I have 
> figured out a lot of systems. This is impenetrable.
> 
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
> 
> 
>> On Feb 27, 2017, at 6:35 PM, Erik Hatcher  wrote:
>> 
>> You have an empty “scores” parameter in there.  You’re not showing your full 
>> search request, but did you provide that in the request?   Have you perhaps 
>> turned off parameter substitutions?
>> 
>>  Erik
>> 
>>> On Feb 27, 2017, at 9:26 PM, Walter Underwood  wrote:
>>> 
>>> With this in the config…
>>> 
>>> 
>>>
>>>   
>>> edismax
>>> 0
>>> false
>>> id,
>>> image_thumb_large, image_thumb_medium, image_thumb_small,
>>> image_thumb_xlarge, uri, user_id, last_name, first_name,
>>> name, school, major, graduation_year, tutor_profile_id,
>>> positive_reviews, negative_reviews, gender, about_experience,
>>> about_extracurricular, time_approved
>>> *:*
>>> about_experience  about_extracurricular School  
>>> Major
>>> 
>>> >> name="sort">sum(interaction_responsiveness_score,profile_completeness_score,school_score,us_tax_id_score,highlight_score,${scores})
>>>  desc
>>> log(sum(1,max(positive_reviews,0)))
>>> 0.1
>>>   
>>> 
>>> 
>>> I see this… [Solr 6.3.0]
>>> 
>>> org.apache.solr.common.SolrException: Unable to reload core 
>>> [tutors_shard1_replica11]
>>> at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:950)
>>> at 
>>> org.apache.solr.core.SolrCore.lambda$getConfListener$6(SolrCore.java:2708)
>>> at 
>>> org.apache.solr.cloud.ZkController.lambda$fireEventListeners$4(ZkController.java:2448)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: org.apache.solr.common.SolrException: Could not load conf for 
>>> core tutors_shard1_replica11: Error loading solr config from solrconfig.xml
>>> at 
>>> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:85)
>>> at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:942)
>>> ... 3 more
>>> Caused by: org.apache.solr.common.SolrException: Error loading solr config 
>>> from solrconfig.xml
>>> at 
>>> org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:187)
>>> at 
>>> org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:97)
>>> at 
>>> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:77)
>>> ... 4 more
>>> Caused by: org.apache.solr.common.SolrException: No system property or 
>>> default value specified for scores 
>>> value:sum(interaction_responsiveness_score,profile_completeness_score,school_score,us_tax_id_score,highlight_score,${scores})
>>>  desc
>>> at 
>>> org.apache.solr.util.PropertiesUtil.substituteProperty(PropertiesUtil.java:65)
>>> at org.apache.solr.util.DOMUtil.substituteProperties(DOMUtil.java:298)
>>> 
>>> wunder
>>> Walter Underwood
>>> wun...@wunderwood.org
>>> http://observer.wunderwood.org/  (my blog)
>>> 
>>> 
 On Feb 27, 2017, at 6:17 PM, Erik Hatcher  wrote:
 
 Walter -
 
 How about this, for the latter part of your request:
 
 /handler?features=a,b,c
 
  with =sum(${features}) desc
 
 That ought to do the trick.   At first I thought the #foreach nature of 
 the list of features was prohibitive, but since you’re literally plugging 
 in the exact string value and it’s used as a comma-separated list then 
 this should work.
 
 But with the just a list of subject ID’s I think you’re in custom 
 development now (or a JavaScript stage in a Fusion query pipeline ;) in 
 building a SearchComponent that takes a `features` parameter and builds 
 the sort param as needed from that.
 
Erik
 
 
 
> On Feb 27, 2017, at 7:17 PM, Walter Underwood  
> wrote:
> 
> We have documents with parameterized features. For a school subject 
> (calculus, accounting), we have three sets of features. So for subject=4 
> and subject=186, we have:
> 
> feature_a_4: 0.9
> feature_b_4: 1.6
> feature_c_4: 8.2
> feature_a_186: 3.0
> feature_b_186: 2.1
> feature_c_186: 99.2
> 
> I’d like to pass in the subject IDs and make a function query (for 
> sorting) from those, ending up with
> 
> sum(feature_x, feature_y, feature_a_4, feature_b_4, feature_c_4, 
> feature_a_186, 

Re: Using parameter values in a sort

[Erik Hatcher]

And by turning off parameter substitutions I meant disable `expandMacros` - 
https://cwiki.apache.org/confluence/display/solr/Parameter+Substitution 


Likely you haven’t and this feature should work.   I’d remove `scores` from 
your config (unless you’re going to provide a valid default, by maybe moving 
where  you put “desc” in the parameters) and provide that as a mandatory (or 
optional depending on how you arrange the params, possibly) `scores` param.


   /select?q=*:*=feature_a_1,feature_b_2
 =sum(${scores}) desc




> On Feb 27, 2017, at 9:35 PM, Erik Hatcher  wrote:
> 
> You have an empty “scores” parameter in there.  You’re not showing your full 
> search request, but did you provide that in the request?   Have you perhaps 
> turned off parameter substitutions?
> 
>   Erik
> 
> 
> 
> 
>> On Feb 27, 2017, at 9:26 PM, Walter Underwood  wrote:
>> 
>> With this in the config…
>> 
>> 
>> 
>>
>>  edismax
>>  0
>>  false
>>  id,
>>  image_thumb_large, image_thumb_medium, image_thumb_small,
>>  image_thumb_xlarge, uri, user_id, last_name, first_name,
>>  name, school, major, graduation_year, tutor_profile_id,
>>  positive_reviews, negative_reviews, gender, about_experience,
>>  about_extracurricular, time_approved
>>  *:*
>>  about_experience  about_extracurricular School  
>> Major
>>  
>>  > name="sort">sum(interaction_responsiveness_score,profile_completeness_score,school_score,us_tax_id_score,highlight_score,${scores})
>>  desc
>>  log(sum(1,max(positive_reviews,0)))
>>  0.1
>>
>> 
>> 
>> I see this… [Solr 6.3.0]
>> 
>> org.apache.solr.common.SolrException: Unable to reload core 
>> [tutors_shard1_replica11]
>>  at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:950)
>>  at 
>> org.apache.solr.core.SolrCore.lambda$getConfListener$6(SolrCore.java:2708)
>>  at 
>> org.apache.solr.cloud.ZkController.lambda$fireEventListeners$4(ZkController.java:2448)
>>  at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.solr.common.SolrException: Could not load conf for 
>> core tutors_shard1_replica11: Error loading solr config from solrconfig.xml
>>  at 
>> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:85)
>>  at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:942)
>>  ... 3 more
>> Caused by: org.apache.solr.common.SolrException: Error loading solr config 
>> from solrconfig.xml
>>  at 
>> org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:187)
>>  at 
>> org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:97)
>>  at 
>> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:77)
>>  ... 4 more
>> Caused by: org.apache.solr.common.SolrException: No system property or 
>> default value specified for scores 
>> value:sum(interaction_responsiveness_score,profile_completeness_score,school_score,us_tax_id_score,highlight_score,${scores})
>>  desc
>>  at 
>> org.apache.solr.util.PropertiesUtil.substituteProperty(PropertiesUtil.java:65)
>>  at org.apache.solr.util.DOMUtil.substituteProperties(DOMUtil.java:298)
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>> 
>>> On Feb 27, 2017, at 6:17 PM, Erik Hatcher  wrote:
>>> 
>>> Walter -
>>> 
>>> How about this, for the latter part of your request:
>>> 
>>> /handler?features=a,b,c
>>> 
>>>   with =sum(${features}) desc
>>> 
>>> That ought to do the trick.   At first I thought the #foreach nature of the 
>>> list of features was prohibitive, but since you’re literally plugging in 
>>> the exact string value and it’s used as a comma-separated list then this 
>>> should work.
>>> 
>>> But with the just a list of subject ID’s I think you’re in custom 
>>> development now (or a JavaScript stage in a Fusion query pipeline ;) in 
>>> building a SearchComponent that takes a `features` parameter and builds the 
>>> sort param as needed from that.
>>> 
>>> Erik
>>> 
>>> 
>>> 
 On Feb 27, 2017, at 7:17 PM, Walter Underwood  
 wrote:
 
 We have documents with parameterized features. For a school subject 
 (calculus, accounting), we have three sets of features. So for subject=4 
 and subject=186, we have:
 
 feature_a_4: 0.9
 feature_b_4: 1.6
 feature_c_4: 8.2
 feature_a_186: 3.0
 feature_b_186: 2.1
 feature_c_186: 99.2
 
 I’d like to pass in the subject IDs and make a function query (for 
 sorting) from those, ending up with
 
 sum(feature_x, feature_y, feature_a_4, feature_b_4, feature_c_4, 
 feature_a_186, feature_b_186, feature c_186) desc
 
 That would be used for the sort parameter.
 
 Failing that, it would be nice 

Re: Using parameter values in a sort

[Walter Underwood]

I’ve passed in a score parameter, but the solrconfig.xml is rejected before any 
requests.

Pretty ready to give up. The documentation around function queries and params 
is not working for me, though I’ve been using Solr for ten years. I have 
figured out a lot of systems. This is impenetrable.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Feb 27, 2017, at 6:35 PM, Erik Hatcher  wrote:
> 
> You have an empty “scores” parameter in there.  You’re not showing your full 
> search request, but did you provide that in the request?   Have you perhaps 
> turned off parameter substitutions?
> 
>   Erik
> 
>> On Feb 27, 2017, at 9:26 PM, Walter Underwood  wrote:
>> 
>> With this in the config…
>> 
>> 
>> 
>>
>>  edismax
>>  0
>>  false
>>  id,
>>  image_thumb_large, image_thumb_medium, image_thumb_small,
>>  image_thumb_xlarge, uri, user_id, last_name, first_name,
>>  name, school, major, graduation_year, tutor_profile_id,
>>  positive_reviews, negative_reviews, gender, about_experience,
>>  about_extracurricular, time_approved
>>  *:*
>>  about_experience  about_extracurricular School  
>> Major
>>  
>>  > name="sort">sum(interaction_responsiveness_score,profile_completeness_score,school_score,us_tax_id_score,highlight_score,${scores})
>>  desc
>>  log(sum(1,max(positive_reviews,0)))
>>  0.1
>>
>> 
>> 
>> I see this… [Solr 6.3.0]
>> 
>> org.apache.solr.common.SolrException: Unable to reload core 
>> [tutors_shard1_replica11]
>>  at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:950)
>>  at 
>> org.apache.solr.core.SolrCore.lambda$getConfListener$6(SolrCore.java:2708)
>>  at 
>> org.apache.solr.cloud.ZkController.lambda$fireEventListeners$4(ZkController.java:2448)
>>  at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.solr.common.SolrException: Could not load conf for 
>> core tutors_shard1_replica11: Error loading solr config from solrconfig.xml
>>  at 
>> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:85)
>>  at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:942)
>>  ... 3 more
>> Caused by: org.apache.solr.common.SolrException: Error loading solr config 
>> from solrconfig.xml
>>  at 
>> org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:187)
>>  at 
>> org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:97)
>>  at 
>> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:77)
>>  ... 4 more
>> Caused by: org.apache.solr.common.SolrException: No system property or 
>> default value specified for scores 
>> value:sum(interaction_responsiveness_score,profile_completeness_score,school_score,us_tax_id_score,highlight_score,${scores})
>>  desc
>>  at 
>> org.apache.solr.util.PropertiesUtil.substituteProperty(PropertiesUtil.java:65)
>>  at org.apache.solr.util.DOMUtil.substituteProperties(DOMUtil.java:298)
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>> 
>>> On Feb 27, 2017, at 6:17 PM, Erik Hatcher  wrote:
>>> 
>>> Walter -
>>> 
>>> How about this, for the latter part of your request:
>>> 
>>> /handler?features=a,b,c
>>> 
>>>   with =sum(${features}) desc
>>> 
>>> That ought to do the trick.   At first I thought the #foreach nature of the 
>>> list of features was prohibitive, but since you’re literally plugging in 
>>> the exact string value and it’s used as a comma-separated list then this 
>>> should work.
>>> 
>>> But with the just a list of subject ID’s I think you’re in custom 
>>> development now (or a JavaScript stage in a Fusion query pipeline ;) in 
>>> building a SearchComponent that takes a `features` parameter and builds the 
>>> sort param as needed from that.
>>> 
>>> Erik
>>> 
>>> 
>>> 
 On Feb 27, 2017, at 7:17 PM, Walter Underwood  
 wrote:
 
 We have documents with parameterized features. For a school subject 
 (calculus, accounting), we have three sets of features. So for subject=4 
 and subject=186, we have:
 
 feature_a_4: 0.9
 feature_b_4: 1.6
 feature_c_4: 8.2
 feature_a_186: 3.0
 feature_b_186: 2.1
 feature_c_186: 99.2
 
 I’d like to pass in the subject IDs and make a function query (for 
 sorting) from those, ending up with
 
 sum(feature_x, feature_y, feature_a_4, feature_b_4, feature_c_4, 
 feature_a_186, feature_b_186, feature c_186) desc
 
 That would be used for the sort parameter.
 
 Failing that, it would be nice so just pass in the parameterized portion, 
 like this:
 
 /handler?features=feature_a_4,feature_b_4,feature_c_4,feature_a_186,feature_b_186,feature
  c_186
 
 Right now, I can’t even make a 

Re: Using parameter values in a sort

[Erik Hatcher]

You have an empty “scores” parameter in there.  You’re not showing your full 
search request, but did you provide that in the request?   Have you perhaps 
turned off parameter substitutions?

Erik




> On Feb 27, 2017, at 9:26 PM, Walter Underwood  wrote:
> 
> With this in the config…
> 
> 
>  
> 
>   edismax
>   0
>   false
>   id,
>   image_thumb_large, image_thumb_medium, image_thumb_small,
>   image_thumb_xlarge, uri, user_id, last_name, first_name,
>   name, school, major, graduation_year, tutor_profile_id,
>   positive_reviews, negative_reviews, gender, about_experience,
>   about_extracurricular, time_approved
>   *:*
>   about_experience  about_extracurricular School  
> Major
>   
>name="sort">sum(interaction_responsiveness_score,profile_completeness_score,school_score,us_tax_id_score,highlight_score,${scores})
>  desc
>   log(sum(1,max(positive_reviews,0)))
>   0.1
> 
>  
> 
> I see this… [Solr 6.3.0]
> 
> org.apache.solr.common.SolrException: Unable to reload core 
> [tutors_shard1_replica11]
>   at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:950)
>   at 
> org.apache.solr.core.SolrCore.lambda$getConfListener$6(SolrCore.java:2708)
>   at 
> org.apache.solr.cloud.ZkController.lambda$fireEventListeners$4(ZkController.java:2448)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.solr.common.SolrException: Could not load conf for core 
> tutors_shard1_replica11: Error loading solr config from solrconfig.xml
>   at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:85)
>   at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:942)
>   ... 3 more
> Caused by: org.apache.solr.common.SolrException: Error loading solr config 
> from solrconfig.xml
>   at 
> org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:187)
>   at 
> org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:97)
>   at 
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:77)
>   ... 4 more
> Caused by: org.apache.solr.common.SolrException: No system property or 
> default value specified for scores 
> value:sum(interaction_responsiveness_score,profile_completeness_score,school_score,us_tax_id_score,highlight_score,${scores})
>  desc
>   at 
> org.apache.solr.util.PropertiesUtil.substituteProperty(PropertiesUtil.java:65)
>   at org.apache.solr.util.DOMUtil.substituteProperties(DOMUtil.java:298)
> 
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
> 
> 
>> On Feb 27, 2017, at 6:17 PM, Erik Hatcher  wrote:
>> 
>> Walter -
>> 
>> How about this, for the latter part of your request:
>> 
>>  /handler?features=a,b,c
>> 
>>with =sum(${features}) desc
>> 
>> That ought to do the trick.   At first I thought the #foreach nature of the 
>> list of features was prohibitive, but since you’re literally plugging in the 
>> exact string value and it’s used as a comma-separated list then this should 
>> work.
>> 
>> But with the just a list of subject ID’s I think you’re in custom 
>> development now (or a JavaScript stage in a Fusion query pipeline ;) in 
>> building a SearchComponent that takes a `features` parameter and builds the 
>> sort param as needed from that.
>> 
>>  Erik
>> 
>> 
>> 
>>> On Feb 27, 2017, at 7:17 PM, Walter Underwood  wrote:
>>> 
>>> We have documents with parameterized features. For a school subject 
>>> (calculus, accounting), we have three sets of features. So for subject=4 
>>> and subject=186, we have:
>>> 
>>> feature_a_4: 0.9
>>> feature_b_4: 1.6
>>> feature_c_4: 8.2
>>> feature_a_186: 3.0
>>> feature_b_186: 2.1
>>> feature_c_186: 99.2
>>> 
>>> I’d like to pass in the subject IDs and make a function query (for sorting) 
>>> from those, ending up with
>>> 
>>> sum(feature_x, feature_y, feature_a_4, feature_b_4, feature_c_4, 
>>> feature_a_186, feature_b_186, feature c_186) desc
>>> 
>>> That would be used for the sort parameter.
>>> 
>>> Failing that, it would be nice so just pass in the parameterized portion, 
>>> like this:
>>> 
>>> /handler?features=feature_a_4,feature_b_4,feature_c_4,feature_a_186,feature_b_186,feature
>>>  c_186
>>> 
>>> Right now, I can’t even make a solrconfig.xml that will load. I’ve read 
>>> everything I can find on params and function queries.
>>> 
>>> wunder
>>> Walter Underwood
>>> wun...@wunderwood.org
>>> http://observer.wunderwood.org/  (my blog)
>>> 
>>> 
>> 
> 

Re: Using parameter values in a sort

[Walter Underwood]

With this in the config…

 
  
 
   edismax
   0
   false
   id,
   image_thumb_large, image_thumb_medium, image_thumb_small,
   image_thumb_xlarge, uri, user_id, last_name, first_name,
   name, school, major, graduation_year, tutor_profile_id,
   positive_reviews, negative_reviews, gender, about_experience,
   about_extracurricular, time_approved
   *:*
   about_experience  about_extracurricular School  
Major
   
   sum(interaction_responsiveness_score,profile_completeness_score,school_score,us_tax_id_score,highlight_score,${scores})
 desc
   log(sum(1,max(positive_reviews,0)))
   0.1
 
  

I see this… [Solr 6.3.0]

org.apache.solr.common.SolrException: Unable to reload core 
[tutors_shard1_replica11]
at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:950)
at 
org.apache.solr.core.SolrCore.lambda$getConfListener$6(SolrCore.java:2708)
at 
org.apache.solr.cloud.ZkController.lambda$fireEventListeners$4(ZkController.java:2448)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Could not load conf for core 
tutors_shard1_replica11: Error loading solr config from solrconfig.xml
at 
org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:85)
at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:942)
... 3 more
Caused by: org.apache.solr.common.SolrException: Error loading solr config from 
solrconfig.xml
at 
org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:187)
at 
org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:97)
at 
org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:77)
... 4 more
Caused by: org.apache.solr.common.SolrException: No system property or default 
value specified for scores 
value:sum(interaction_responsiveness_score,profile_completeness_score,school_score,us_tax_id_score,highlight_score,${scores})
 desc
at 
org.apache.solr.util.PropertiesUtil.substituteProperty(PropertiesUtil.java:65)
at org.apache.solr.util.DOMUtil.substituteProperties(DOMUtil.java:298)

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Feb 27, 2017, at 6:17 PM, Erik Hatcher  wrote:
> 
> Walter -
> 
> How about this, for the latter part of your request:
> 
>   /handler?features=a,b,c
> 
> with =sum(${features}) desc
> 
> That ought to do the trick.   At first I thought the #foreach nature of the 
> list of features was prohibitive, but since you’re literally plugging in the 
> exact string value and it’s used as a comma-separated list then this should 
> work.
> 
> But with the just a list of subject ID’s I think you’re in custom development 
> now (or a JavaScript stage in a Fusion query pipeline ;) in building a 
> SearchComponent that takes a `features` parameter and builds the sort param 
> as needed from that.
> 
>   Erik
> 
> 
> 
>> On Feb 27, 2017, at 7:17 PM, Walter Underwood  wrote:
>> 
>> We have documents with parameterized features. For a school subject 
>> (calculus, accounting), we have three sets of features. So for subject=4 and 
>> subject=186, we have:
>> 
>> feature_a_4: 0.9
>> feature_b_4: 1.6
>> feature_c_4: 8.2
>> feature_a_186: 3.0
>> feature_b_186: 2.1
>> feature_c_186: 99.2
>> 
>> I’d like to pass in the subject IDs and make a function query (for sorting) 
>> from those, ending up with
>> 
>> sum(feature_x, feature_y, feature_a_4, feature_b_4, feature_c_4, 
>> feature_a_186, feature_b_186, feature c_186) desc
>> 
>> That would be used for the sort parameter.
>> 
>> Failing that, it would be nice so just pass in the parameterized portion, 
>> like this:
>> 
>> /handler?features=feature_a_4,feature_b_4,feature_c_4,feature_a_186,feature_b_186,feature
>>  c_186
>> 
>> Right now, I can’t even make a solrconfig.xml that will load. I’ve read 
>> everything I can find on params and function queries.
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>> 
> 

Re: Using parameter values in a sort

[Erik Hatcher]

Walter -

How about this, for the latter part of your request:

   /handler?features=a,b,c

 with =sum(${features}) desc

That ought to do the trick.   At first I thought the #foreach nature of the 
list of features was prohibitive, but since you’re literally plugging in the 
exact string value and it’s used as a comma-separated list then this should 
work.

But with the just a list of subject ID’s I think you’re in custom development 
now (or a JavaScript stage in a Fusion query pipeline ;) in building a 
SearchComponent that takes a `features` parameter and builds the sort param as 
needed from that.

Erik



> On Feb 27, 2017, at 7:17 PM, Walter Underwood  wrote:
> 
> We have documents with parameterized features. For a school subject 
> (calculus, accounting), we have three sets of features. So for subject=4 and 
> subject=186, we have:
> 
> feature_a_4: 0.9
> feature_b_4: 1.6
> feature_c_4: 8.2
> feature_a_186: 3.0
> feature_b_186: 2.1
> feature_c_186: 99.2
> 
> I’d like to pass in the subject IDs and make a function query (for sorting) 
> from those, ending up with
> 
> sum(feature_x, feature_y, feature_a_4, feature_b_4, feature_c_4, 
> feature_a_186, feature_b_186, feature c_186) desc
> 
> That would be used for the sort parameter.
> 
> Failing that, it would be nice so just pass in the parameterized portion, 
> like this:
> 
> /handler?features=feature_a_4,feature_b_4,feature_c_4,feature_a_186,feature_b_186,feature
>  c_186
> 
> Right now, I can’t even make a solrconfig.xml that will load. I’ve read 
> everything I can find on params and function queries.
> 
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
> 
> 

Re: Using parameter values in a sort

[Erick Erickson]

Hmm, the syntax can be tricky. I'd start with a vanilla solrconfig.xml and
get the syntax right just with curl or the browser, _then_ go to
trying to embed it in solrconfig.xml.

What error are you getting anyway?

Erick

On Mon, Feb 27, 2017 at 4:17 PM, Walter Underwood  wrote:
> We have documents with parameterized features. For a school subject 
> (calculus, accounting), we have three sets of features. So for subject=4 and 
> subject=186, we have:
>
> feature_a_4: 0.9
> feature_b_4: 1.6
> feature_c_4: 8.2
> feature_a_186: 3.0
> feature_b_186: 2.1
> feature_c_186: 99.2
>
> I’d like to pass in the subject IDs and make a function query (for sorting) 
> from those, ending up with
>
> sum(feature_x, feature_y, feature_a_4, feature_b_4, feature_c_4, 
> feature_a_186, feature_b_186, feature c_186) desc
>
> That would be used for the sort parameter.
>
> Failing that, it would be nice so just pass in the parameterized portion, 
> like this:
>
> /handler?features=feature_a_4,feature_b_4,feature_c_4,feature_a_186,feature_b_186,feature
>  c_186
>
> Right now, I can’t even make a solrconfig.xml that will load. I’ve read 
> everything I can find on params and function queries.
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
>

Re: OOM

[Erick Erickson]

How much memory are you giving the JVM anyway?

On Mon, Feb 27, 2017 at 5:02 PM, Alexandre Rafalovitch
 wrote:
> Marple will (probably) tell you whether you are indexing Chinese as
> English text. Unlikely it would help with OOM (though I would be happy
> to know if it did).
>
> Are you actually getting the error as you are starting? Repeatedly?
> What's the stack trace looks like?
>
> Regards,
>Alex.
> 
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 27 February 2017 at 18:57, Rick Leir  wrote:
>> Hi all,
>> We get an OOM after stopping then starting Solr (with a tiny index).  Is 
>> there something I could check quickly before I break out the Eclipse 
>> debugger? Maybe Marple could tell me about problems in the index?
>> Thanks -- Rick
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: solr warning - filling logs

[Walter Underwood]

“Brittle” means that if the node fails, your cluster cannot change until the 
node is back up. Single point of failure.

The whole point of Zookeeper is redundant protection against failure. So 
running a single Zookeeper node is just wrong. 

A Zookeeper ensemble needs to be an odd number of instances, because they will 
do majority voting. Three or five is a good number. Three works with one 
failure. Five works with one failure while you are upgrading the Zookeeper 
ensemble.

We run a three node ensemble in test and a five node ensemble in prod.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Feb 27, 2017, at 4:57 PM, Mike Thomsen  wrote:
> 
> It's a brittle ZK configuration. A typical ZK quorum is three nodes for
> most production systems. One is fine, though, for development provided the
> system it's on is not overloaded.
> 
> On Mon, Feb 27, 2017 at 6:43 PM, Rick Leir  wrote:
> 
>> Hi Mike
>> We are using a single ZK node, I think. What problems should we expect?
>> Thanks -- Rick
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: OOM

[Alexandre Rafalovitch]

Marple will (probably) tell you whether you are indexing Chinese as
English text. Unlikely it would help with OOM (though I would be happy
to know if it did).

Are you actually getting the error as you are starting? Repeatedly?
What's the stack trace looks like?

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 27 February 2017 at 18:57, Rick Leir  wrote:
> Hi all,
> We get an OOM after stopping then starting Solr (with a tiny index).  Is 
> there something I could check quickly before I break out the Eclipse 
> debugger? Maybe Marple could tell me about problems in the index?
> Thanks -- Rick
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: solr warning - filling logs

[Mike Thomsen]

It's a brittle ZK configuration. A typical ZK quorum is three nodes for
most production systems. One is fine, though, for development provided the
system it's on is not overloaded.

On Mon, Feb 27, 2017 at 6:43 PM, Rick Leir  wrote:

> Hi Mike
> We are using a single ZK node, I think. What problems should we expect?
> Thanks -- Rick
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.

Using parameter values in a sort

[Walter Underwood]

We have documents with parameterized features. For a school subject (calculus, 
accounting), we have three sets of features. So for subject=4 and subject=186, 
we have:

feature_a_4: 0.9
feature_b_4: 1.6
feature_c_4: 8.2
feature_a_186: 3.0
feature_b_186: 2.1
feature_c_186: 99.2

I’d like to pass in the subject IDs and make a function query (for sorting) 
from those, ending up with

sum(feature_x, feature_y, feature_a_4, feature_b_4, feature_c_4, feature_a_186, 
feature_b_186, feature c_186) desc

That would be used for the sort parameter.

Failing that, it would be nice so just pass in the parameterized portion, like 
this:

/handler?features=feature_a_4,feature_b_4,feature_c_4,feature_a_186,feature_b_186,feature
 c_186

Right now, I can’t even make a solrconfig.xml that will load. I’ve read 
everything I can find on params and function queries.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

OOM

[Rick Leir]

Hi all,
We get an OOM after stopping then starting Solr (with a tiny index).  Is there 
something I could check quickly before I break out the Eclipse debugger? Maybe 
Marple could tell me about problems in the index?
Thanks -- Rick
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: solr warning - filling logs

[Rick Leir]

Hi Mike
We are using a single ZK node, I think. What problems should we expect?
Thanks -- Rick
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: DIH: last_index_time not updated on if 0 docs updated

[Erick Erickson]

Seems like a legitimate request, if you can't find a JIRA feel free to open one.

And if you wanted to supply a patch, _well_ ;)

On Mon, Feb 27, 2017 at 10:37 AM, xavier jmlucjav  wrote:
> Hi,
>
> After getting our interval for calling delta index shorter and shorter, I
> have found out that last_index_time  in dataimport.properties is not
> updated every time the indexing runs, it is skipped if no docs where added.
>
> This happens at least in the following scenario:
> - running delta as full index
> ( /dataimport?command=full-import=false=true )
> - Solrcloud setup, so dataimport.properties is in zookeeper
> - Solr 5.5.0
>
> I understand skipping the commit on the index if no docs were updated is a
> nice optimization, but I believe the last_index_time info should be updated
> in all cases, so it reflects reality. We, for instance, are looking at this
> piece of information in order to do other stuff.
>
> I could not find any mention of this on Jira, so I wonder if this is
> intented or just nobody had an issue with it?
>
> xavier

DIH: last_index_time not updated on if 0 docs updated

[xavier jmlucjav]

Hi,

After getting our interval for calling delta index shorter and shorter, I
have found out that last_index_time  in dataimport.properties is not
updated every time the indexing runs, it is skipped if no docs where added.

This happens at least in the following scenario:
- running delta as full index
( /dataimport?command=full-import=false=true )
- Solrcloud setup, so dataimport.properties is in zookeeper
- Solr 5.5.0

I understand skipping the commit on the index if no docs were updated is a
nice optimization, but I believe the last_index_time info should be updated
in all cases, so it reflects reality. We, for instance, are looking at this
piece of information in order to do other stuff.

I could not find any mention of this on Jira, so I wonder if this is
intented or just nobody had an issue with it?

xavier

Re: About editing managed-schema by hand

[Erick Erickson]

see: 
https://cwiki.apache.org/confluence/display/solr/Schema+Factory+Definition+in+SolrConfig#SchemaFactoryDefinitioninSolrConfig-Switchingfromschema.xmltoManagedSchema

bq: If we can modify the schema.xml
through Schema API

You can't do this by default, but if you insist you can change
the SchemaFactory in solrcofnig.xml to use any file you
want.

Best,
Erick

On Mon, Feb 27, 2017 at 3:46 AM, Issei Nishigata  wrote:
> Thank you for your reply.
>
> If I was to say which one, I'd maybe be talking about the concept for Solr.
> I understand we should use "ClassicSchemaFactory" when we want to
> hand-edit, but why are there two files, schema.xml and managed-schema, in
> spite that we can hand-edit managed-schema? If we can modify the schema.xml
> through Schema API, I think we won't need the managed-schema, but is there
> any reason why that can't be done?
> Could you please let me know if there is any information that can clear
> things up around those details?
>
> Thanks,
> Issei
>
> 2017-02-27 1:51 GMT+09:00 Erick Erickson :
>
>> This is the sequence that gets you in trouble:
>> > start solr
>> > hand edit the schema _without_ reloading your collection or restarting
>> all your Solr instances.
>> > use the managed-schema API to make modifications.
>>
>> In this scenario your hand-edits can be lost since the in-memory version
>> of the
>> schema is written out without re-fetching it from Zookeeper.
>>
>> If you only ever hand-edit your schema, you'll be fine.
>>
>> If you conscientiously reload your collection (or restart all your Solr's)
>> after
>> you hand-edit your schema, you'll be fine even if you use the managed
>> schema
>> API calls.
>>
>> But really, if you want to hand-edit your schema why not go back to using
>> the ClassicSchemaFactory? See:
>> https://cwiki.apache.org/confluence/display/solr/
>> Schema+Factory+Definition+in+SolrConfig#SchemaFactoryDefinitioninSolrC
>> onfig-SwitchingfromManagedSchematoManuallyEditedschema.xml
>>
>> Best,
>> Erick
>>
>> On Sun, Feb 26, 2017 at 8:22 AM, Issei Nishigata 
>> wrote:
>> > Hi, All
>> >
>> > Similar questions may have been already asked, but just in case please
>> let
>> > me ask you.
>> > According to the below URL it says as "Schema modifications via the
>> Schema
>> > API will now be enabled by default.",
>> > but would there be any issues if I edited with text editor instead of
>> > Schema API?
>> >
>> > https://cwiki.apache.org/confluence/display/solr/Major+
>> Changes+from+Solr+5+to+Solr+6
>> >
>> >
>> > In the answer to the past question, it seemed okay.
>> >
>> > http://lucene.472066.n3.nabble.com/Solr-6-managed-
>> schema-amp-version-control-td4289243.html
>> >
>> >
>> > I was worried because managed-schema said "" when managed-schema was
>> > automatically generated from schema.xml.
>> > If I need to use Schema API and if I wanted to do some process that
>> cannot
>> > be done with Schema API(modifying unique key, etc), what should I do?
>> >
>> >
>> > Thanks,
>> > Issei
>>

Re: Query construction in custom Solr component plugin

[Erick Erickson]

Look for a couple of parameters, FROMLEADER and/or distrib=false.
Not quite sure which one you need though. When
a request comes in to the "aggregator" (just the first node in the cluster
that the request lands on) it sends out sub-requests to one replica in each
shard. These have to be marked do they in turn aren't sent out to other
replicas and I'm pretty sure one or both of those parameters will be set
on the sub-requests.

Best,
Erick

On Mon, Feb 27, 2017 at 12:38 AM, Baloo  wrote:
> Hi,
>
> I have developed a solr component which expands users query and adds
> additional clauses to the query. For this expansion we are making request to
> external REST api's. This query expansion logic is mainly in prepare()
> method. Everything works as expected in standalone mode. When we deploy this
> plugin in SolrCloud environment each shard is calling external REST api for
> query expansion.
>
> My question is that can we make only one call to external REST api since its
> the same request sent from each shard to external service. How can we modify
> our component to make only one call per search request ?
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Query-construction-in-custom-Solr-component-plugin-tp4322403.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Announcing Marple, a RESTful API & GUI for inspecting Lucene indexes

[Alan Woodward]

At the moment it only works with indexes accessible via the filesystem (ie via 
java.nio.Paths.get()).  But pull requests are always welcome :)

Alan Woodward
www.flax.co.uk


> On 27 Feb 2017, at 14:56, Joe Obernberger  
> wrote:
> 
> Hi Charlie - will this work with an index stored in HDFS as written by Solr 
> Cloud?
> 
> -Joe
> 
> 
> On 2/24/2017 12:24 PM, Charlie Hull wrote:
>> Hi all,
>> 
>> Very pleased to announce the first release of Marple, an open source tool 
>> for inspecting Lucene indexes. We've blogged about it here:
>> http://www.flax.co.uk/blog/2017/02/24/release-1-0-marple-lucene-index-detective/
>>  
>> which contains links to the code and released JAR.
>> 
>> This is very much a work in progress (we started it at the Lucene hackday we 
>> ran in London last autumn) so contributions, bug reports & feature requests 
>> very welcome! We'll also be talking about it at the next London Lucene/Solr 
>> Meetup on March 23rd.
>> 
>> Best
>> 
>> Charlie
> 

Re: Announcing Marple, a RESTful API & GUI for inspecting Lucene indexes

[Doug Turnbull]

Marple looks great, and if you want to work on really interesting Solr
problems, I can heartily vouch for a career in consulting and recommend
Flax as a great firm to work for!

Best
-Doug

On Fri, Feb 24, 2017 at 12:26 PM Charlie Hull  wrote:

> On 24/02/2017 17:24, Charlie Hull wrote:
> > Hi all,
> >
> > Very pleased to announce the first release of Marple, an open source
> > tool for inspecting Lucene indexes. We've blogged about it here:
> >
> http://www.flax.co.uk/blog/2017/02/24/release-1-0-marple-lucene-index-detective/
> >
> > which contains links to the code and released JAR.
> >
> > This is very much a work in progress (we started it at the Lucene
> > hackday we ran in London last autumn) so contributions, bug reports &
> > feature requests very welcome! We'll also be talking about it at the
> > next London Lucene/Solr Meetup on March 23rd.
>
> By the way, we're looking to expand the Flax team, so if you're based
> near or in Cambridge or London U.K. and want to work on interesting and
> varied search projects, get in touch!
>
> Cheers
>
> Charlie
> >
> > Best
> >
> > Charlie
>
>
> --
> Charlie Hull
> Flax - Open Source Enterprise Search
>
> tel/fax: +44 (0)8700 118334 <+44%20870%20011%208334>
> mobile:  +44 (0)7767 825828 <+44%207767%20825828>
> web: www.flax.co.uk
>

Re: Announcing Marple, a RESTful API & GUI for inspecting Lucene indexes

[Joe Obernberger]

Hi Charlie - will this work with an index stored in HDFS as written by 
Solr Cloud?


-Joe


On 2/24/2017 12:24 PM, Charlie Hull wrote:

Hi all,

Very pleased to announce the first release of Marple, an open source 
tool for inspecting Lucene indexes. We've blogged about it here:
http://www.flax.co.uk/blog/2017/02/24/release-1-0-marple-lucene-index-detective/ 


which contains links to the code and released JAR.

This is very much a work in progress (we started it at the Lucene 
hackday we ran in London last autumn) so contributions, bug reports & 
feature requests very welcome! We'll also be talking about it at the 
next London Lucene/Solr Meetup on March 23rd.


Best

Charlie

Re: Index Segments not Merging

[Mike Thomsen]

Just barely skimmed the documentation, but it looks like the tool generates
its own shards and pushes them into the collection by manipulating the
configuration of the cluster.

https://www.cloudera.com/documentation/enterprise/5-8-x/topics/search_mapreduceindexertool.html

If that reading is correct, it would stand to reason that Solr (at least as
of Solr 4.10 which is what CDH ships) would not be doing the periodic
cleanup it normally does when building shards through its APIs.

On Thu, Feb 23, 2017 at 10:01 PM, Jordan Drake 
wrote:

> We have solr with the index stored in HDFS. We are running MapReduce jobs
> to build the index using the MapReduceIndexerTool from Cloudera with the
> go-live option to merge into our live index.
>
> We are seeing an issue where the number of segments in the index never
> reduces. It continues to grow until we manually do an optimize.
>
> We are using the following solr config for merge policy
>
>
>
>
>
>
>
>
>
>
>
> * name="maxMergeAtOnce">10 name="segmentsPerTier">10 class="org.apache.lucene.index.ConcurrentMergeScheduler"> name="maxThreadCount">1 name="maxMergeCount">6*
>
> If we add documents into solr without using MapReduce the segments merge
> properly as expected.
>
> Any ideas on why we see this behavior? Does the solr index merge prevent
> the segments from merging?
>
>
> Thanks,
> Jordan
>

Re: solr warning - filling logs

[Mike Thomsen]

When you transition to an external zookeeper, you'll need at least 3 ZK
nodes. One is insufficient outside of a development environment. That's a
general requirement for any system that uses ZK.

On Sun, Feb 26, 2017 at 7:14 PM, Satya Marivada 
wrote:

> May I ask about the port scanner running? Can you please elaborate?
> Sure, will try to move out to external zookeeper
>
> On Sun, Feb 26, 2017 at 7:07 PM Dave  wrote:
>
> > You shouldn't use the embedded zookeeper with solr, it's just for
> > development not anywhere near worthy of being out in production.
> Otherwise
> > it looks like you may have a port scanner running. In any case don't use
> > the zk that comes with solr
> >
> > > On Feb 26, 2017, at 6:52 PM, Satya Marivada  >
> > wrote:
> > >
> > > Hi All,
> > >
> > > I have configured solr with SSL and enabled http authentication. It is
> > all
> > > working fine on the solr admin page, indexing and querying process. One
> > > bothering thing is that it is filling up logs every second saying no
> > > authority, I have configured host name, port and authentication
> > parameters
> > > right in all config files. Not sure, where is it coming from. Any
> > > suggestions, please. Really appreciate it. It is with sol-6.3.0 cloud
> > with
> > > embedded zookeeper. Could it be some bug with solr-6.3.0 or am I
> missing
> > > some configuration?
> > >
> > > 2017-02-26 23:32:43.660 WARN (qtp606548741-18) [c:plog s:shard1
> > > r:core_node2 x:plog_shard1_replica1] o.e.j.h.HttpParser parse
> exception:
> > > java.lang.IllegalArgumentException: No Authority for
> > > HttpChannelOverHttp@6dac689d{r=0,c=false,a=IDLE,uri=null}
> > > java.lang.IllegalArgumentException: No Authority
> > > at
> > >
> > org.eclipse.jetty.http.HostPortHttpField.(
> HostPortHttpField.java:43)
> > > at org.eclipse.jetty.http.HttpParser.parsedHeader(HttpParser.java:877)
> > > at org.eclipse.jetty.http.HttpParser.parseHeaders(
> HttpParser.java:1050)
> > > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1266)
> > > at
> > >
> > org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(
> HttpConnection.java:344)
> > > at
> > >
> > org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:227)
> > > at org.eclipse.jetty.io
> > > .AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
> > > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> > > at
> > org.eclipse.jetty.io.ssl.SslConnection.onFillable(
> SslConnection.java:186)
> > > at org.eclipse.jetty.io
> > > .AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
> > > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> > > at org.eclipse.jetty.io
> > > .SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> > > at
> > >
> > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceAndRun(ExecuteProduceConsume.java:246)
> > > at
> > >
> > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
> ExecuteProduceConsume.java:156)
> > > at
> > >
> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:654)
> > > at
> > >
> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:572)
> > > at java.lang.Thread.run(Thread.java:745)
> >
>

Query construction in custom Solr component plugin

[Baloo]

Hi, 

I have developed a solr component which expands users query and adds
additional clauses to the query. For this expansion we are making request to
external REST api's. This query expansion logic is mainly in prepare()
method. Everything works as expected in standalone mode. When we deploy this
plugin in SolrCloud environment each shard is calling external REST api for
query expansion. 

My question is that can we make only one call to external REST api since its
the same request sent from each shard to external service. How can we modify
our component to make only one call per search request ?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Query-construction-in-custom-Solr-component-plugin-tp4322403.html
Sent from the Solr - User mailing list archive at Nabble.com.

solr warning - filling logs

[Satya Marivada]

Hi All,

I have configured solr with SSL and enabled http authentication. It is all
working fine on the solr admin page, indexing and querying process. One
bothering thing is that it is filling up logs every second saying no
authority, I have configured host name, port and authentication parameters
right in all config files. Not sure, where is it coming from. Any
suggestions, please. Really appreciate it. It is with sol-6.3.0 cloud with
embedded zookeeper. Could it be some bug with solr-6.3.0 or am I missing
some configuration?

2017-02-26 23:32:43.660 WARN (qtp606548741-18) [c:plog s:shard1
r:core_node2 x:plog_shard1_replica1] o.e.j.h.HttpParser parse exception:
java.lang.IllegalArgumentException: No Authority for
HttpChannelOverHttp@6dac689d{r=0,c=false,a=IDLE,uri=null}
java.lang.IllegalArgumentException: No Authority
at
org.eclipse.jetty.http.HostPortHttpField.(HostPortHttpField.java:43)
at org.eclipse.jetty.http.HttpParser.parsedHeader(HttpParser.java:877)
at org.eclipse.jetty.http.HttpParser.parseHeaders(HttpParser.java:1050)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1266)
at
org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:344)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:227)
at org.eclipse.jetty.io
.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io
.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io
.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

Re: About editing managed-schema by hand

[Issei Nishigata]

Thank you for your reply.

If I was to say which one, I'd maybe be talking about the concept for Solr.
I understand we should use "ClassicSchemaFactory" when we want to
hand-edit, but why are there two files, schema.xml and managed-schema, in
spite that we can hand-edit managed-schema? If we can modify the schema.xml
through Schema API, I think we won't need the managed-schema, but is there
any reason why that can't be done?
Could you please let me know if there is any information that can clear
things up around those details?

Thanks,
Issei

2017-02-27 1:51 GMT+09:00 Erick Erickson :

> This is the sequence that gets you in trouble:
> > start solr
> > hand edit the schema _without_ reloading your collection or restarting
> all your Solr instances.
> > use the managed-schema API to make modifications.
>
> In this scenario your hand-edits can be lost since the in-memory version
> of the
> schema is written out without re-fetching it from Zookeeper.
>
> If you only ever hand-edit your schema, you'll be fine.
>
> If you conscientiously reload your collection (or restart all your Solr's)
> after
> you hand-edit your schema, you'll be fine even if you use the managed
> schema
> API calls.
>
> But really, if you want to hand-edit your schema why not go back to using
> the ClassicSchemaFactory? See:
> https://cwiki.apache.org/confluence/display/solr/
> Schema+Factory+Definition+in+SolrConfig#SchemaFactoryDefinitioninSolrC
> onfig-SwitchingfromManagedSchematoManuallyEditedschema.xml
>
> Best,
> Erick
>
> On Sun, Feb 26, 2017 at 8:22 AM, Issei Nishigata 
> wrote:
> > Hi, All
> >
> > Similar questions may have been already asked, but just in case please
> let
> > me ask you.
> > According to the below URL it says as "Schema modifications via the
> Schema
> > API will now be enabled by default.",
> > but would there be any issues if I edited with text editor instead of
> > Schema API?
> >
> > https://cwiki.apache.org/confluence/display/solr/Major+
> Changes+from+Solr+5+to+Solr+6
> >
> >
> > In the answer to the past question, it seemed okay.
> >
> > http://lucene.472066.n3.nabble.com/Solr-6-managed-
> schema-amp-version-control-td4289243.html
> >
> >
> > I was worried because managed-schema said "" when managed-schema was
> > automatically generated from schema.xml.
> > If I need to use Schema API and if I wanted to do some process that
> cannot
> > be done with Schema API(modifying unique key, etc), what should I do?
> >
> >
> > Thanks,
> > Issei
>

Re: Boolean expression for spatial query

[Emir Arnautovic]

Hi Michael,

I haven't been playing with spatial for a while, but if it fully 
supports WKT, you could use Intersects instead of Contains and 
MULTIPOINT instead of POINT. Something like:


fq={!field f=regionGeometry}Intersects(MULTIPOINT((x1 y1), (x2, y2)))

In any case you can use OR-ed _query_:

fq=_query_:"{!field f=regionGeometry}Contains(POINT(x1, y1))" OR 
_query_:"{!field f=regionGeometry}Contains(POINT(x2, y2))"



HTH
Emir


On 26.02.2017 07:08, Michael Dürr wrote:

Hi all,

I index documents containing a spatial field (rpt) that holds a wkt
multipolygon. In order to retrieve all documents for which a certain point
is contained within a polygon I issue the following query:

q=*:*={!field f=regionGeometry}Contains(POINT( ))

This works pretty good.

My question is: Is there any syntax to issue this query for multiple points
(i.e. return all documents for which at least one of the points is within
the document's polygon)?

E.g. something like this:

q=*:*={!field f=regionGeometry}ContainsOR(POINT( ),POINT(
),...)

If not - what other efficient options do you xuggest to do such a query?

Best regards,
Michael



--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/