date:20201210

Re: Timeout occured while waiting response from server

2020-12-10 Thread elivis

See:
https://lucene.472066.n3.nabble.com/SolrServerException-Timeout-occured-while-waiting-response-from-server-tc4464632.html

Maybe this will help somebody. I was dealing with exact same problem. We are
running on VMs, and all of our timeout problems went away after we switched
from a 5yo VmWare version to the latest Hyper-V VMs. We also made sure that
all VMs have a dedicated spindle. It appears the underlying physical disk
drive (which all VMs use) was getting overloaded with reads/writes. 




--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: SolrCloud crashing due to memory error - 'Cannot allocate memory' (errno=12)

2020-12-10 Thread Walter Underwood

How much RAM do you have on those machines? That message says you ran out.

32 GB is a HUGE heap. Unless you have a specific need for that, run with a 8 GB
heap and see how that works. 

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Dec 10, 2020, at 7:55 PM, Altamirano, Emmanuel 
>  wrote:
> 
> Hello,
>  
> We have a SolrCloud(8.6) with 3 servers with the same characteristics and 
> configuration. We assigned32GB for heap memory each, and after some short 
> period of time sending 40 concurrent requests to the SolrCloud using a load 
> balancer, we are getting the following error that shutdown each Solr Server 
> and Zookeeper:
>  
> OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory 
> req_addr: 0x bytes: 536870912 (errno = 12).
> OpenJDK 64-Bit Server VM warning: Attempt to deallocate stack guard pages 
> failed.
> OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x7edd4d9da000, 
> 12288, 0) failed; error='Cannot allocate memory' (errno=12)
>  
>  
> 20201201 10:43:29.495 [ERROR] {qtp2051853139-23369} [c:express s:shard1 
> r:core_node6 x:express_shard1_replica_n4] 
> [org.apache.solr.handler.RequestHandlerBase, 148] | 
> org.apache.solr.common.SolrException: Cannot talk to ZooKeeper - Updates are 
> disabled.
> at 
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.zkCheck(DistributedZkUpdateProcessor.java:1245)
> at 
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.setupRequest(DistributedZkUpdateProcessor.java:582)
> at 
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:239)
>  
> 
>  
> We have a one collection with one shard, almost 400 million documents 
> (~334GB).
>  
> $ sysctl vm.nr_hugepages
> vm.nr_hugepages = 32768
> $ sysctl vm.max_map_count
> vm.max_map_count = 131072
>  
> /etc/security/limits.conf
>  
> * - core unlimited
> * - data unlimited
> * - priority unlimited
> * - fsize unlimited
> * - sigpending 513928
> * - memlock unlimited
> * - nofile 131072
> * - msgqueue 819200
> * - rtprio 0
> * - stack 8192
> * - cpu unlimited
> * - rss unlimited #virtual memory unlimited
> * - locks unlimited
> * soft nproc 65536
> * hard nproc 65536
> * - nofile 131072
>  
>  
>  
> /etc/sysctl.conf
>  
> vm.nr_hugepages =  32768
> vm.max_map_count = 131072
>  
>  
> Could you please provide me some advice to fix this error?
>  
> Thanks,
>  
> Emmanuel Altamirano

SolrCloud crashing due to memory error - 'Cannot allocate memory' (errno=12)

2020-12-10 Thread Altamirano, Emmanuel

Hello,

We have a SolrCloud(8.6) with 3 servers with the same characteristics and 
configuration. We assigned 32GB for heap memory each, and after some short 
period of time sending 40 concurrent requests to the SolrCloud using a load 
balancer, we are getting the following error that shutdown each Solr Server and 
Zookeeper:

OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory 
req_addr: 0x bytes: 536870912 (errno = 12).
OpenJDK 64-Bit Server VM warning: Attempt to deallocate stack guard pages 
failed.
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x7edd4d9da000, 
12288, 0) failed; error='Cannot allocate memory' (errno=12)


20201201 10:43:29.495 [ERROR] {qtp2051853139-23369} [c:express s:shard1 
r:core_node6 x:express_shard1_replica_n4] 
[org.apache.solr.handler.RequestHandlerBase, 148] | 
org.apache.solr.common.SolrException: Cannot talk to ZooKeeper - Updates are 
disabled.
at 
org.apache.solr.update.processor.DistributedZkUpdateProcessor.zkCheck(DistributedZkUpdateProcessor.java:1245)
at 
org.apache.solr.update.processor.DistributedZkUpdateProcessor.setupRequest(DistributedZkUpdateProcessor.java:582)
at 
org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:239)

[cid:image004.jpg@01D6CF3F.27574B90]

We have a one collection with one shard, almost 400 million documents (~334GB).

$ sysctl vm.nr_hugepages
vm.nr_hugepages = 32768
$ sysctl vm.max_map_count
vm.max_map_count = 131072

/etc/security/limits.conf

* - core unlimited
* - data unlimited
* - priority unlimited
* - fsize unlimited
* - sigpending 513928
* - memlock unlimited
* - nofile 131072
* - msgqueue 819200
* - rtprio 0
* - stack 8192
* - cpu unlimited
* - rss unlimited #virtual memory unlimited
* - locks unlimited
* soft nproc 65536
* hard nproc 65536
* - nofile 131072



/etc/sysctl.conf

vm.nr_hugepages =  32768
vm.max_map_count = 131072


Could you please provide me some advice to fix this error?

Thanks,

Emmanuel Altamirano

Re: nested facets of query and terms type in JSON format

2020-12-10 Thread Arturas Mazeika

Hi Jason,

Thanks a lot for the post. Indeed the web page you are referring to has
some very nice examples. Well done.

Cheers,
Arturas

Increasing the number of threads (through the url or params section) is
unsuccessful so far. Maybe solr takes it only as a hint.

On Thu, Dec 10, 2020 at 8:01 PM Jason Gerlowski 
wrote:

> Hey Arturas,
>
> Can't help you with the secrets of Michael's inspiration (though I'm
> also curious :-p).  And I'm not sure if there's any equivalent of
> facet.threads for JSON Faceting.  You're on your own there
> unfortunately.
>
> But you (or other readers) might find this "Query Facet" example handy
> - it uses the "type": "query" syntax that MIchael mentioned. [1]
>
> [1]
> https://lucene.apache.org/solr/guide/8_5/json-facet-api.html#query-facet
>
> Best,
> Jason
>
> On Thu, Dec 3, 2020 at 5:49 PM Arturas Mazeika  wrote:
> >
> > Hi Michael,
> >
> > I wish I were able to do a percent of what you are doing. Where does your
> > inspiration come from? It is not from the manuals, cause I've checked
> > those. How do you come up with this piece of art? Did you check this from
> > the source code? Which lines revealed these secrets? I am eternally
> > grateful for your help!
> >
> > Michael, maybe you happen to know how I can plugin in facet.threads
> > parameter in that JSON body below, so the query uses more threads to
> > compute the answer? I am dying out of curiosity.
> >
> > Cheers,
> > Arturas
> >
> > On Thu, Dec 3, 2020 at 7:59 PM Michael Gibney  >
> > wrote:
> >
> > > I think the first "error" case in your set of examples above is
> closest to
> > > being correct. For "query" facet type, I think you want to explicitly
> > > specify `"type":"query"`, and specify the query itself in the `"q"`
> param,
> > > i.e.:
> > > {
> > > "query"  : "*:*",
> > > "limit"  : 0,
> > >
> > > "facet": {
> > > "aip": {
> > > "type":  "query",
> > > "q":  "cfname2:aip",
> > > "facet": {
> > > "t_buckets": {
> > > "type":  "range",
> > > "field": "t",
> > > "sort": { "t": "asc" },
> > > "start": "2018-05-02T17:00:00.000Z",
> > > "end":   "2020-11-16T21:00:00.000Z",
> > > "gap":   "+1HOUR"
> > > "limit": 1
> > > }
> > > }
> > > }
> > > }
> > > }
> > >
> > > On Thu, Dec 3, 2020 at 12:59 PM Arturas Mazeika 
> wrote:
> > >
> > > > Hi Michael,
> > > >
> > > > Thanks for helping me to figure this out.
> > > >
> > > > If I fire:
> > > >
> > > > {
> > > > "query"  : "*:*",
> > > > "limit"  : 0,
> > > >
> > > > "facet": {
> > > > "aip": { "query":  "cfname2:aip", }
> > > >
> > > > }
> > > > }
> > > >
> > > > I get
> > > >
> > > > "response": { "numFound": 20560849, "start": 0, "numFoundExact":
> true,
> > > > "docs": [] }, "facets": { "count": 20560849, "aip": { "count": 2307
> } } }
> > > >
> > > > (works). If I fire
> > > >
> > > >
> > > > {
> > > > "query"  : "*:*",
> > > > "limit"  : 0,
> > > >
> > > > "facet": {
> > > > "t_buckets": {
> > > > "type":  "range",
> > > > "field": "t",
> > > > "sort": { "t": "asc" },
> > > > "start": "2018-05-02T17:00:00.000Z",
> > > > "end":   "2020-11-16T21:00:00.000Z",
> > > > "gap":   "+1HOUR"
> > > > "limit": 1
> > > > }
> > > > }
> > > > }
> > > >
> > > > I get
> > > >
> > > > "response": { "numFound": 20560849, "start": 0, "numFoundExact":
> true,
> > > > "docs": [] }, "facets": { "count": 20560849, "t_buckets": {
> "buckets": [
> > > {
> > > > "val": "2018-05-02T17:00:00Z", "count": 150 },
> > > >
> > > > (works). If I fire:
> > > >
> > > > {
> > > > "query"  : "*:*",
> > > > "limit"  : 0,
> > > >
> > > > "facet": {
> > > > "aip": { "query":  "cfname2:aip",
> > > >
> > > > "facet": {
> > > > "t_buckets": {
> > > > "type":  "range",
> > > > "field": "t",
> > > > "sort": { "t": "asc" },
> > > > "start": "2018-05-02T17:00:00.000Z",
> > > > "end":   "2020-11-16T21:00:00.000Z",
> > > > "gap":   "+1HOUR"
> > > > "limit": 1
> > > > }
> > > > }
> > > > }
> > > > }
> > > > }
> > > >
> > > > I get
> > > >
> > > > "error": { "metadata": [ "error-class",
> > > > "org.apache.solr.common.SolrException", "root-error-class",
> > > > "org.apache.solr.common.SolrException" ], "msg": "expected facet/stat
> > > type
> > > > name, like {type:range, field:price, ...} but got null ,
> path=/facet",
> > > > "code": 400 } }
> > > >
> > > > If I fire
> > > >
> > > > {
> > > > "query"  : "*:*",
> > > > "limit"  : 0,
> > > >
> > > > "facet": {
> > > > "aip": { "quer

Re: increasing number of threads for faceting in JSON format

2020-12-10 Thread Arturas Mazeika

Hi Christine Munendra et al,

Wow, you dag into the code and checked weather threads are being blown in
range and term queries! I wish one day to be able to do the same myself.

How does one get to the level, so one can check the code herself? Is there
like a nice primer or crash course, solr 101 so to say, things you did not
learn in school about solr, but you wish you had learned web page? Well,
I'll take this opportunity to scroll through the lines in the github. Your
answer is very helpful.

Cheers,
Arturas

On Thu, Dec 10, 2020 at 7:08 PM Munendra S N 
wrote:

> Thank you Christine.
> Yeah, JSON facet does not support specifying threads.
>
>
> On Thu, Dec 10, 2020, 11:15 PM Christine Poerschke (BLOOMBERG/ LONDON) <
> cpoersc...@bloomberg.net> wrote:
>
> > Hello Arturas and Munendra!
> >
> > In the "Currently, JSON facets have support for specifying the number of
> > threads." sentence, I wonder if perhaps a "does not" got inadvertently
> > omitted i.e. "Currently, JSON facets does not have support for specifying
> > the number of threads." was intended?
> >
> > Let me share what I learnt from digging into the code:
> >
> > * "facet.threads" is for field value faceting [1] [2] but you're
> > interested in (JSON) field range faceting as well as JSON field value
> > faceting.
> >
> > * The area of the code [3] that does the JSON field range faceting shows
> > no obvious threading or parallelisation.
> >
> > Hope that helps?
> >
> > Regards,
> >
> > Christine
> >
> > [1]
> >
> https://lucene.apache.org/solr/guide/8_7/faceting.html#field-value-faceting-parameters
> > [2]
> >
> https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.7.0/solr/core/src/java/org/apache/solr/request/SimpleFacets.java
> > [3]
> >
> https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.7.0/solr/core/src/java/org/apache/solr/search/facet/FacetRangeProcessor.java#L112-L113
> >
> > From: solr-user@lucene.apache.org At: 12/03/20 22:47:35To:
> > solr-user@lucene.apache.org
> > Subject: Re: increasing number of threads for faceting in JSON format
> >
> > Hi Munedra,
> >
> > This is great that I can get things faster by reducing the gap and by
> > increasing the number of threads. How to reduce gaps I know: one can
> > replace   "gap":   "+1HOUR" with   "gap":   "+1MONTH" What should I
> change
> > in the text below to increase the number of threads from one to 20?
> >
> > Cheers,
> > Arturas
> >
> > On Thu, Dec 3, 2020 at 1:54 PM Munendra S N 
> > wrote:
> >
> > > Hi,
> > >
> > > Currently, JSON facets have support for specifying the number of
> threads.
> > > In the above request, the range facet is computed over 2 years with a
> gap
> > > of 1 hour. By reducing the number of buckets, computation should become
> > > much faster
> > >
> > > Regards,
> > > Munendra S N
> > >
> > >
> > >
> > > On Thu, Dec 3, 2020 at 1:52 PM Arturas Mazeika 
> > wrote:
> > >
> > > > Hi Solr-Users,
> > > >
> > > > I am trying to better understand the solr capabilities, how one can
> > > > formulate queries in JSON format as well as tweak parameters.
> > Currently I
> > > > have a logs collection (ca 6GB large) with a dozen of attributes
> > running
> > > in
> > > > single server mode (F:\solr_deployment\solr-8.7.0\bin\solr.cmd start
> -h
> > > > localhost -p  -m 4g)
> > > >
> > > > I am playing with faceting functionality in solr and query a couple
> of
> > > > attributes there. My typical query is:
> > > >
> > > > GET http://localhost:/solr/db/query
> > > >  HTTP/1.1
> > > > content-type: application/json
> > > >
> > > > {
> > > > "query"  : "*:*",
> > > > "limit"  : 0,
> > > > "facet": {
> > > > "t" : {
> > > > "type":  "terms",
> > > > "field": "fcomp",
> > > > "sort":  "index",
> > > >
> > > > "facet": {
> > > > "t_buckets": {
> > > > "type":  "range",
> > > > "field": "t",
> > > > "sort": { "t": "asc" },
> > > > "start": "2018-05-02T17:00:00.000Z",
> > > > "end":   "2020-11-16T21:00:00.000Z",
> > > > "gap":   "+1HOUR"
> > > > }
> > > > }
> > > > },
> > > > }
> > > > }
> > > >
> > > > not surprisingly, it takes a bit to compute the result, so I tried to
> > > > increase the number of threads. How do I do it in JSON format? I
> tried
> > > > adding
> > > >
> > > > {
> > > > "params": {
> > > > "facet.threads": 8
> > > > },
> > > > "query"  : "*:*",
> > > > ...
> > > > }
> > > >
> > > > and checked the jstack  of the solr java process, but I still
> see
> > > only
> > > > one thread working.  Can I configure params through the params
> section?
> > > >
> > > > I also tried
> > > >
> > > > {
> > > > "query"  : "*:*",
> > > > "limit"  : 0,
> > > > "facet": {
> > > > "t" : {
> > > > "type":  "terms",

Re: nested facets of query and terms type in JSON format

2020-12-10 Thread Jason Gerlowski

Hey Arturas,

Can't help you with the secrets of Michael's inspiration (though I'm
also curious :-p).  And I'm not sure if there's any equivalent of
facet.threads for JSON Faceting.  You're on your own there
unfortunately.

But you (or other readers) might find this "Query Facet" example handy
- it uses the "type": "query" syntax that MIchael mentioned. [1]

[1] https://lucene.apache.org/solr/guide/8_5/json-facet-api.html#query-facet

Best,
Jason

On Thu, Dec 3, 2020 at 5:49 PM Arturas Mazeika  wrote:
>
> Hi Michael,
>
> I wish I were able to do a percent of what you are doing. Where does your
> inspiration come from? It is not from the manuals, cause I've checked
> those. How do you come up with this piece of art? Did you check this from
> the source code? Which lines revealed these secrets? I am eternally
> grateful for your help!
>
> Michael, maybe you happen to know how I can plugin in facet.threads
> parameter in that JSON body below, so the query uses more threads to
> compute the answer? I am dying out of curiosity.
>
> Cheers,
> Arturas
>
> On Thu, Dec 3, 2020 at 7:59 PM Michael Gibney 
> wrote:
>
> > I think the first "error" case in your set of examples above is closest to
> > being correct. For "query" facet type, I think you want to explicitly
> > specify `"type":"query"`, and specify the query itself in the `"q"` param,
> > i.e.:
> > {
> > "query"  : "*:*",
> > "limit"  : 0,
> >
> > "facet": {
> > "aip": {
> > "type":  "query",
> > "q":  "cfname2:aip",
> > "facet": {
> > "t_buckets": {
> > "type":  "range",
> > "field": "t",
> > "sort": { "t": "asc" },
> > "start": "2018-05-02T17:00:00.000Z",
> > "end":   "2020-11-16T21:00:00.000Z",
> > "gap":   "+1HOUR"
> > "limit": 1
> > }
> > }
> > }
> > }
> > }
> >
> > On Thu, Dec 3, 2020 at 12:59 PM Arturas Mazeika  wrote:
> >
> > > Hi Michael,
> > >
> > > Thanks for helping me to figure this out.
> > >
> > > If I fire:
> > >
> > > {
> > > "query"  : "*:*",
> > > "limit"  : 0,
> > >
> > > "facet": {
> > > "aip": { "query":  "cfname2:aip", }
> > >
> > > }
> > > }
> > >
> > > I get
> > >
> > > "response": { "numFound": 20560849, "start": 0, "numFoundExact": true,
> > > "docs": [] }, "facets": { "count": 20560849, "aip": { "count": 2307 } } }
> > >
> > > (works). If I fire
> > >
> > >
> > > {
> > > "query"  : "*:*",
> > > "limit"  : 0,
> > >
> > > "facet": {
> > > "t_buckets": {
> > > "type":  "range",
> > > "field": "t",
> > > "sort": { "t": "asc" },
> > > "start": "2018-05-02T17:00:00.000Z",
> > > "end":   "2020-11-16T21:00:00.000Z",
> > > "gap":   "+1HOUR"
> > > "limit": 1
> > > }
> > > }
> > > }
> > >
> > > I get
> > >
> > > "response": { "numFound": 20560849, "start": 0, "numFoundExact": true,
> > > "docs": [] }, "facets": { "count": 20560849, "t_buckets": { "buckets": [
> > {
> > > "val": "2018-05-02T17:00:00Z", "count": 150 },
> > >
> > > (works). If I fire:
> > >
> > > {
> > > "query"  : "*:*",
> > > "limit"  : 0,
> > >
> > > "facet": {
> > > "aip": { "query":  "cfname2:aip",
> > >
> > > "facet": {
> > > "t_buckets": {
> > > "type":  "range",
> > > "field": "t",
> > > "sort": { "t": "asc" },
> > > "start": "2018-05-02T17:00:00.000Z",
> > > "end":   "2020-11-16T21:00:00.000Z",
> > > "gap":   "+1HOUR"
> > > "limit": 1
> > > }
> > > }
> > > }
> > > }
> > > }
> > >
> > > I get
> > >
> > > "error": { "metadata": [ "error-class",
> > > "org.apache.solr.common.SolrException", "root-error-class",
> > > "org.apache.solr.common.SolrException" ], "msg": "expected facet/stat
> > type
> > > name, like {type:range, field:price, ...} but got null , path=/facet",
> > > "code": 400 } }
> > >
> > > If I fire
> > >
> > > {
> > > "query"  : "*:*",
> > > "limit"  : 0,
> > >
> > > "facet": {
> > > "aip": { "query":  "cfname2:aip",
> > >
> > > "facet": {
> > > "type":  "range",
> > > "field": "t",
> > > "sort": { "t": "asc" },
> > > "start": "2018-05-02T17:00:00.000Z",
> > > "end":   "2020-11-16T21:00:00.000Z",
> > > "gap":   "+1HOUR"
> > > "limit": 1
> > > }
> > > }
> > > }
> > > }
> > >
> > > I get
> > >
> > > "error": { "metadata": [ "error-class",
> > > "org.apache.solr.common.SolrException", "root-error-class",
> > > "org.apache.solr.common.SolrException" ], "msg": "expected facet/stat
> > type

Re: increasing number of threads for faceting in JSON format

2020-12-10 Thread Munendra S N

Thank you Christine.
Yeah, JSON facet does not support specifying threads.


On Thu, Dec 10, 2020, 11:15 PM Christine Poerschke (BLOOMBERG/ LONDON) <
cpoersc...@bloomberg.net> wrote:

> Hello Arturas and Munendra!
>
> In the "Currently, JSON facets have support for specifying the number of
> threads." sentence, I wonder if perhaps a "does not" got inadvertently
> omitted i.e. "Currently, JSON facets does not have support for specifying
> the number of threads." was intended?
>
> Let me share what I learnt from digging into the code:
>
> * "facet.threads" is for field value faceting [1] [2] but you're
> interested in (JSON) field range faceting as well as JSON field value
> faceting.
>
> * The area of the code [3] that does the JSON field range faceting shows
> no obvious threading or parallelisation.
>
> Hope that helps?
>
> Regards,
>
> Christine
>
> [1]
> https://lucene.apache.org/solr/guide/8_7/faceting.html#field-value-faceting-parameters
> [2]
> https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.7.0/solr/core/src/java/org/apache/solr/request/SimpleFacets.java
> [3]
> https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.7.0/solr/core/src/java/org/apache/solr/search/facet/FacetRangeProcessor.java#L112-L113
>
> From: solr-user@lucene.apache.org At: 12/03/20 22:47:35To:
> solr-user@lucene.apache.org
> Subject: Re: increasing number of threads for faceting in JSON format
>
> Hi Munedra,
>
> This is great that I can get things faster by reducing the gap and by
> increasing the number of threads. How to reduce gaps I know: one can
> replace   "gap":   "+1HOUR" with   "gap":   "+1MONTH" What should I change
> in the text below to increase the number of threads from one to 20?
>
> Cheers,
> Arturas
>
> On Thu, Dec 3, 2020 at 1:54 PM Munendra S N 
> wrote:
>
> > Hi,
> >
> > Currently, JSON facets have support for specifying the number of threads.
> > In the above request, the range facet is computed over 2 years with a gap
> > of 1 hour. By reducing the number of buckets, computation should become
> > much faster
> >
> > Regards,
> > Munendra S N
> >
> >
> >
> > On Thu, Dec 3, 2020 at 1:52 PM Arturas Mazeika 
> wrote:
> >
> > > Hi Solr-Users,
> > >
> > > I am trying to better understand the solr capabilities, how one can
> > > formulate queries in JSON format as well as tweak parameters.
> Currently I
> > > have a logs collection (ca 6GB large) with a dozen of attributes
> running
> > in
> > > single server mode (F:\solr_deployment\solr-8.7.0\bin\solr.cmd start -h
> > > localhost -p  -m 4g)
> > >
> > > I am playing with faceting functionality in solr and query a couple of
> > > attributes there. My typical query is:
> > >
> > > GET http://localhost:/solr/db/query
> > >  HTTP/1.1
> > > content-type: application/json
> > >
> > > {
> > > "query"  : "*:*",
> > > "limit"  : 0,
> > > "facet": {
> > > "t" : {
> > > "type":  "terms",
> > > "field": "fcomp",
> > > "sort":  "index",
> > >
> > > "facet": {
> > > "t_buckets": {
> > > "type":  "range",
> > > "field": "t",
> > > "sort": { "t": "asc" },
> > > "start": "2018-05-02T17:00:00.000Z",
> > > "end":   "2020-11-16T21:00:00.000Z",
> > > "gap":   "+1HOUR"
> > > }
> > > }
> > > },
> > > }
> > > }
> > >
> > > not surprisingly, it takes a bit to compute the result, so I tried to
> > > increase the number of threads. How do I do it in JSON format? I tried
> > > adding
> > >
> > > {
> > > "params": {
> > > "facet.threads": 8
> > > },
> > > "query"  : "*:*",
> > > ...
> > > }
> > >
> > > and checked the jstack  of the solr java process, but I still see
> > only
> > > one thread working.  Can I configure params through the params section?
> > >
> > > I also tried
> > >
> > > {
> > > "query"  : "*:*",
> > > "limit"  : 0,
> > > "facet": {
> > > "t" : {
> > > "type":  "terms",
> > > "field": "fcomp",
> > > "sort":  "index",
> > >
> > > "facet": {
> > > "t_buckets": {
> > > "type":  "range",
> > > "field": "t",
> > > "sort": { "t": "asc" },
> > > "start": "2018-05-02T17:00:00.000Z",
> > > "end":   "2020-11-16T21:00:00.000Z",
> > > "gap":   "+1HOUR"
> > > }
> > > },
> > > "threads":8
> > > },
> > > }
> > > }
> > >
> > > but this ran in one thread as well. Can I influence the number of
> threads
> > > in the "facet" section of JSON?
> > >
> > > Cheers,
> > > Arturas
> > >
> >
>
>
>

Re: increasing number of threads for faceting in JSON format

2020-12-10 Thread Christine Poerschke (BLOOMBERG/ LONDON)

Hello Arturas and Munendra!

In the "Currently, JSON facets have support for specifying the number of 
threads." sentence, I wonder if perhaps a "does not" got inadvertently omitted 
i.e. "Currently, JSON facets does not have support for specifying the number of 
threads." was intended?

Let me share what I learnt from digging into the code:

* "facet.threads" is for field value faceting [1] [2] but you're interested in 
(JSON) field range faceting as well as JSON field value faceting.

* The area of the code [3] that does the JSON field range faceting shows no 
obvious threading or parallelisation.

Hope that helps?

Regards,

Christine

[1] 
https://lucene.apache.org/solr/guide/8_7/faceting.html#field-value-faceting-parameters
[2] 
https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.7.0/solr/core/src/java/org/apache/solr/request/SimpleFacets.java
[3] 
https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.7.0/solr/core/src/java/org/apache/solr/search/facet/FacetRangeProcessor.java#L112-L113

From: solr-user@lucene.apache.org At: 12/03/20 22:47:35To:  
solr-user@lucene.apache.org
Subject: Re: increasing number of threads for faceting in JSON format

Hi Munedra,

This is great that I can get things faster by reducing the gap and by
increasing the number of threads. How to reduce gaps I know: one can
replace   "gap":   "+1HOUR" with   "gap":   "+1MONTH" What should I change
in the text below to increase the number of threads from one to 20?

Cheers,
Arturas

On Thu, Dec 3, 2020 at 1:54 PM Munendra S N  wrote:

> Hi,
>
> Currently, JSON facets have support for specifying the number of threads.
> In the above request, the range facet is computed over 2 years with a gap
> of 1 hour. By reducing the number of buckets, computation should become
> much faster
>
> Regards,
> Munendra S N
>
>
>
> On Thu, Dec 3, 2020 at 1:52 PM Arturas Mazeika  wrote:
>
> > Hi Solr-Users,
> >
> > I am trying to better understand the solr capabilities, how one can
> > formulate queries in JSON format as well as tweak parameters. Currently I
> > have a logs collection (ca 6GB large) with a dozen of attributes running
> in
> > single server mode (F:\solr_deployment\solr-8.7.0\bin\solr.cmd start -h
> > localhost -p  -m 4g)
> >
> > I am playing with faceting functionality in solr and query a couple of
> > attributes there. My typical query is:
> >
> > GET http://localhost:/solr/db/query
> >  HTTP/1.1
> > content-type: application/json
> >
> > {
> > "query"  : "*:*",
> > "limit"  : 0,
> > "facet": {
> > "t" : {
> > "type":  "terms",
> > "field": "fcomp",
> > "sort":  "index",
> >
> > "facet": {
> > "t_buckets": {
> > "type":  "range",
> > "field": "t",
> > "sort": { "t": "asc" },
> > "start": "2018-05-02T17:00:00.000Z",
> > "end":   "2020-11-16T21:00:00.000Z",
> > "gap":   "+1HOUR"
> > }
> > }
> > },
> > }
> > }
> >
> > not surprisingly, it takes a bit to compute the result, so I tried to
> > increase the number of threads. How do I do it in JSON format? I tried
> > adding
> >
> > {
> > "params": {
> > "facet.threads": 8
> > },
> > "query"  : "*:*",
> > ...
> > }
> >
> > and checked the jstack  of the solr java process, but I still see
> only
> > one thread working.  Can I configure params through the params section?
> >
> > I also tried
> >
> > {
> > "query"  : "*:*",
> > "limit"  : 0,
> > "facet": {
> > "t" : {
> > "type":  "terms",
> > "field": "fcomp",
> > "sort":  "index",
> >
> > "facet": {
> > "t_buckets": {
> > "type":  "range",
> > "field": "t",
> > "sort": { "t": "asc" },
> > "start": "2018-05-02T17:00:00.000Z",
> > "end":   "2020-11-16T21:00:00.000Z",
> > "gap":   "+1HOUR"
> > }
> > },
> > "threads":8
> > },
> > }
> > }
> >
> > but this ran in one thread as well. Can I influence the number of threads
> > in the "facet" section of JSON?
> >
> > Cheers,
> > Arturas
> >
>

Solr Collection reload task has been in running state for a very long time

2020-12-10 Thread Moulay Hicham

Hi,

We have a solr cluster of 30 nodes with a Replication Factor =3.
Each index size is about 80GB.
Solr version is 8.1
The cluster has high TPS both in read and write.

We have recently made a schema change and uploaded it using ZKCLI
script. Then we issue a collection reload async request:
admin/collections?action=RELOAD&name=&async=1000'

When we check on the status of this request, it shows that it's still running:

admin/collections?action=REQUESTSTATUS&requestid=1000'
{
  "responseHeader":{
"status":0,
"QTime":1},
  "status":{
"state":"running",
"msg":"found [1000] in running tasks"}}

This task has been in a running state for about 5 hours so far. I am
not sure if this is expected or the status of this task failed or
completed but never reported back to zookeeper.

Also if running for that long - is it because the index is being
actively (with high TPS) updated? We have a softcommit of 10s and
hadcommit of 60s.

Please help me understand what's going on.

Thanks,
Moulay

ManagedIndexSchema takes long for larger schema changes

2020-12-10 Thread Tiziano Degaetano

Hello,

I was checking why my initial schema change is taking several minutes using the 
managed schema api.
VisualVm shows that most of the time is used in 
ManagedIndexSchema.postReadInform

[cid:image001.png@01D6CEE9.16DA5EC0]

Looking at the code shows that postReadInform is executed for every 
modification, and performs an inform on all fields.
At the end inform is called ChagesToSchema * Fields times.

I prepared a PR that changes the flow to only postReadInform once after the 
changes are done.
improve speed of large schema changes for ManagedIndexSchema · 
tizianodeg/lucene-solr@54d2161 · 
GitHub

this can dramatically decrease managed schema change from several minutes to 1 
sec [cid:image002.png@01D6CEEE.143FEB80]

I’m not sure if setLatestSchema is the right place to do the final call to 
postReadInform and also unsure if making the postReadInform public is 
acceptable.
How can I propose such an improvement? – Or should I open a Bug request for 
this?

Kind Regards,
Tiziano

Re: Solr cloud issuse: Async exception during distributed update

2020-12-10 Thread Colvin Cowie

Hi, sounds like https://issues.apache.org/jira/browse/SOLR-13963 which was
fixed in Solr 8.3.1

On Thu, 10 Dec 2020 at 06:20, Ritvik Sharma  wrote:

> Hi Houston,
> Thanks for reply
>
> We dont have this kind of field. It's a field value and it is coming
> randomly, not all the time.
> We are indexing using cloudsolrclient + spring data .  It is coming on any
> value,
>
> I am trying to do indexing of  ~30 million records. And it is coming on
> Solr cloud mode not on standalone VM.
>
> Here x.x.x.x and x.x.x.y are tlog.
>
> Remote error message: ERROR: [doc=33140886###Track] unknown field '
> https://a10.ga'
> at
>
> org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:125)
> at
>
> org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:46)
> at
>
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.directUpdate(BaseCloudSolrClient.java:549)
> at
>
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1037)
> at
>
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:897)
> at
>
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:829)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
> at
>
> org.springframework.data.solr.core.SolrTemplate.lambda$saveBeans$3(SolrTemplate.java:227)
> at
>
> org.springframework.data.solr.core.SolrTemplate.execute(SolrTemplate.java:167)
> ... 29 common frames omitted
> Caused by:
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
> from server at http://x.x.x.x:8983/solr
> /searchcollection_shard1_replica_t101: Async exception during distributed
> update: Error from server at http://x.x.x.y:8983/sol
> r/searchcollection_shard2_replica_t103/: null
>
>
>
> request: http://x.x.x.x:8983/solr/searchcollection_shard2_replica_t103/
> Remote error message: ERROR: [doc=33140886###Track] unknown field '
> https://a10.ga'
> at
>
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:665)
> at
>
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265)
> at
>
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
> at
>
> org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
> at
>
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
> at
>
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$directUpdate$0(BaseCloudSolrClient.java:525)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> ... 3 common frames omitted
>
>
> On Thu, 10 Dec 2020 at 11:38, Houston Putman 
> wrote:
>
> > Do you have a field named "314257s_seourls" in your schema?
> >
> > Is there a dynamic field you are trying to match with that name?
> >
> > - Houston
> >
> > On Thu, Dec 10, 2020 at 2:53 PM ritvik  wrote:
> >
> > > Hi ,
> > >  Please suggest, why it is happening.
> > >
> > >
> > >
> > > --
> > > Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
> > >
> >
>

Re: Timeout occured while waiting response from server

Re: SolrCloud crashing due to memory error - 'Cannot allocate memory' (errno=12)

SolrCloud crashing due to memory error - 'Cannot allocate memory' (errno=12)

Re: nested facets of query and terms type in JSON format

Re: increasing number of threads for faceting in JSON format

Re: nested facets of query and terms type in JSON format

Re: increasing number of threads for faceting in JSON format

Re: increasing number of threads for faceting in JSON format

Solr Collection reload task has been in running state for a very long time

ManagedIndexSchema takes long for larger schema changes

Re: Solr cloud issuse: Async exception during distributed update

11 matches

Site Navigation

Mail list logo

Footer information