Re: Solr 8.7.0 memory leak?

[Mike Drob]

Are you running these in docker containers?

Also, I’m assuming this is a typo but just in case the setting is Xmx :)

Can you share the OOM stack trace? It’s not always running out of memory,
sometimes Java throws OOM for file handles or threads.

Mike

On Wed, Jan 27, 2021 at 10:00 PM Luke  wrote:

> Shawn,
>
> it's killed by OOME exception. The problem is that I just created empty
> collections and the Solr JVM keeps growing and never goes down. there is no
> data at all. at the beginning, I set Xxm=6G, then 10G, now 15G, Solr 8.7
> always use all of them and it will be killed by oom.sh once jvm usage
> reachs 100%.
>
> I have another solr 8.6.2 cloud(3 nodes) in separated environment , which
> have over 100 collections, the Xxm = 6G , jvm is always 4-5G.
>
>
>
> On Thu, Jan 28, 2021 at 2:56 AM Shawn Heisey  wrote:
>
> > On 1/27/2021 5:08 PM, Luke Oak wrote:
> > > I just created a few collections and no data, memory keeps growing but
> > never go down, until I got OOM and solr is killed
> > >
> > > Any reason?
> >
> > Was Solr killed by the operating system's oom killer or did the death
> > start with a Java OutOfMemoryError exception?
> >
> > If it was the OS, then the entire system doesn't have enough memory for
> > the demands that are made on it.  The problem might be Solr, or it might
> > be something else.  You will need to either reduce the amount of memory
> > used or increase the memory in the system.
> >
> > If it was a Java OOME exception that led to Solr being killed, then some
> > resource (could be heap memory, but isn't always) will be too small and
> > will need to be increased.  To figure out what resource, you need to see
> > the exception text.  Such exceptions are not always recorded -- it may
> > occur in a section of code that has no logging.
> >
> > Thanks,
> > Shawn
> >
>

Re: Solr 8.7.0 memory leak?

[Luke]

Shawn,

it's killed by OOME exception. The problem is that I just created empty
collections and the Solr JVM keeps growing and never goes down. there is no
data at all. at the beginning, I set Xxm=6G, then 10G, now 15G, Solr 8.7
always use all of them and it will be killed by oom.sh once jvm usage
reachs 100%.

I have another solr 8.6.2 cloud(3 nodes) in separated environment , which
have over 100 collections, the Xxm = 6G , jvm is always 4-5G.



On Thu, Jan 28, 2021 at 2:56 AM Shawn Heisey  wrote:

> On 1/27/2021 5:08 PM, Luke Oak wrote:
> > I just created a few collections and no data, memory keeps growing but
> never go down, until I got OOM and solr is killed
> >
> > Any reason?
>
> Was Solr killed by the operating system's oom killer or did the death
> start with a Java OutOfMemoryError exception?
>
> If it was the OS, then the entire system doesn't have enough memory for
> the demands that are made on it.  The problem might be Solr, or it might
> be something else.  You will need to either reduce the amount of memory
> used or increase the memory in the system.
>
> If it was a Java OOME exception that led to Solr being killed, then some
> resource (could be heap memory, but isn't always) will be too small and
> will need to be increased.  To figure out what resource, you need to see
> the exception text.  Such exceptions are not always recorded -- it may
> occur in a section of code that has no logging.
>
> Thanks,
> Shawn
>

Re: Solr 8.7.0 memory leak?

[Shawn Heisey]

On 1/27/2021 5:08 PM, Luke Oak wrote:

I just created a few collections and no data, memory keeps growing but never go 
down, until I got OOM and solr is killed

Any reason?


Was Solr killed by the operating system's oom killer or did the death 
start with a Java OutOfMemoryError exception?


If it was the OS, then the entire system doesn't have enough memory for 
the demands that are made on it.  The problem might be Solr, or it might 
be something else.  You will need to either reduce the amount of memory 
used or increase the memory in the system.


If it was a Java OOME exception that led to Solr being killed, then some 
resource (could be heap memory, but isn't always) will be too small and 
will need to be increased.  To figure out what resource, you need to see 
the exception text.  Such exceptions are not always recorded -- it may 
occur in a section of code that has no logging.


Thanks,
Shawn

Solr 8.7.0 memory leak?

[Luke Oak]

Hi, I am using solr 8.7.0, centos 7, java 8.

I just created a few collections and no data, memory keeps growing but never go 
down, until I got OOM and solr is killed 

Any reason?

Thanks

Sent from my iPhone

Re: Is there way to autowarm new searcher using recently ran queries

[Joel Bernstein]

Typically what you would do is add static warming queries to warm all the
caches. These queries are hardcoded into the solrconfig.xml. You'll want to
run the facets you're using in the warming queries particularly facets on
string fields.

Once you add these it will take longer to warm the new searcher so you may
need to change the auto-commit intervals.




Joel Bernstein
http://joelsolr.blogspot.com/


On Wed, Jan 27, 2021 at 5:30 PM Pushkar Raste 
wrote:

> Hi,
>
> A rookie question. We have a Solr cluster that doesn't get too much
> traffic. We see that our queries take long time unless we run a script to
> send more traffic to Solr.
>
> We are indexing data all the time and use autoCommit.
>
> I am wondering if there is a way to warmup new searcher on commit by
> rerunning queries processed by the last searcher. May be it happens by
> default but then I can't understand why we see high query times if those
> searchers are being warmed.
>

Is there way to autowarm new searcher using recently ran queries

[Pushkar Raste]

Hi,

A rookie question. We have a Solr cluster that doesn't get too much
traffic. We see that our queries take long time unless we run a script to
send more traffic to Solr.

We are indexing data all the time and use autoCommit.

I am wondering if there is a way to warmup new searcher on commit by
rerunning queries processed by the last searcher. May be it happens by
default but then I can't understand why we see high query times if those
searchers are being warmed.

Re: Solr Slack Workspace

[Justin Sweeney]

Thanks, I joined the Relevance Slack:
https://opensourceconnections.com/slack, I definitely think a dedicated
Solr workspace would also be good allowing for channels to get involved
with development as well as user based questions.

It does seem like slack has made it increasingly difficult to create open
workspaces and not force someone to approve or only allow specific email
domains. Has anyone tried to do that recently? I tried for an hour or so
last weekend and it seemed to not be very straightforward anymore.

On Tue, Jan 26, 2021 at 12:57 PM Houston Putman 
wrote:

> There is https://solr-dev.slack.com
>
> It's not really used, but it's there and we can open it up for people to
> join and start using.
>
> On Tue, Jan 26, 2021 at 5:38 AM Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com> wrote:
>
> > Thanks ufuk. I'll take a look.
> >
> > On Tue, 26 Jan, 2021, 4:05 pm ufuk yılmaz, 
> > wrote:
> >
> > > It’s asking for a searchscale.com email address?
> > >
> > > Sent from Mail for Windows 10
> > >
> > > From: Ishan Chattopadhyaya
> > > Sent: 26 January 2021 13:33
> > > To: solr-user
> > > Subject: Re: Solr Slack Workspace
> > >
> > > There is a Slack backed by official IRC support. Please see
> > > https://lucene.472066.n3.nabble.com/Solr-Users-Slack-td4466856.html
> for
> > > details on how to join it.
> > >
> > > On Tue, 19 Jan, 2021, 2:54 pm Charlie Hull, <
> > > ch...@opensourceconnections.com>
> > > wrote:
> > >
> > > > Relevance Slack is open to anyone working on search & relevance -
> #solr
> > > is
> > > > only one of the channels, there's lots more! Hope to see you there.
> > > >
> > > > Cheers
> > > >
> > > > Charlie
> > > > https://opensourceconnections.com/slack
> > > >
> > > >
> > > > On 16/01/2021 02:18, matthew sporleder wrote:
> > > > > IRC has kind of died off,
> > > > > https://lucene.apache.org/solr/community.html has a slack
> mentioned,
> > > > > I'm on https://opensourceconnections.com/slack after taking their
> > solr
> > > > > training class and assume it's mostly open to solr community.
> > > > >
> > > > > On Fri, Jan 15, 2021 at 8:10 PM Justin Sweeney
> > > > >  wrote:
> > > > >> Hi all,
> > > > >>
> > > > >> I did some googling and didn't find anything, but is there a Slack
> > > > >> workspace for Solr? I think this could be useful to expand
> > interaction
> > > > >> within the community of Solr users and connect people solving
> > similar
> > > > >> problems.
> > > > >>
> > > > >> I'd be happy to get this setup if it does not exist already.
> > > > >>
> > > > >> Justin
> > > >
> > > >
> > > > --
> > > > Charlie Hull - Managing Consultant at OpenSource Connections Limited
> > > > 
> > > > Founding member of The Search Network  >
> > > > and co-author of Searching the Enterprise
> > > > 
> > > > tel/fax: +44 (0)8700 118334
> > > > mobile: +44 (0)7767 825828
> > > >
> > >
> > >
> >
>

Wrong HTTP status for HEAD request

[Thomas Corthals]

Hi,

In Solr 8.6.1, a GET request or a HEAD request for a non-existing term in a
managed resource (stopword or synonym) returns a HTTP status "404 Not
Found".

$ curl -i "
http://localhost:8983/solr/techproducts/schema/analysis/synonyms/english/foobar;
| head -n 1
HTTP/1.1 404 Not Found

$ curl -I "
http://localhost:8983/solr/techproducts/schema/analysis/synonyms/english/foobar;
| head -n 1
HTTP/1.1 404 Not Found

In Solr 8.7.0, the same GET request still returns "404 Not Found", but the
HEAD request now returns "200 OK" as if the term actually exists.

$ curl -i "
http://localhost:8983/solr/techproducts/schema/analysis/synonyms/english/foobar;
| head -n 1
HTTP/1.1 404 Not Found

$ curl -I "
http://localhost:8983/solr/techproducts/schema/analysis/synonyms/english/foobar;
| head -n 1
HTTP/1.1 200 OK

I presume that's a bug?

Thomas

Tweaking Shards and Replicas for high volume queries and updates

[Hollowell,Skip]

30 Dedicated physical Nodes in the Solr Cloud Cluster, all of identical 
configuration
Server01   RHEL 7.x
256GB RAM
10 2TB Spinning Disk in a RAID 10 Configuration (Leaving us 9.8TB usable per 
node)
64GB JVM Heap, Tried has high as 100GB, but it appeared that 64GB was faster.  
If we set a higher heap, do we starve the OS for caching?
Huge Pages is off on the system, and thus UseLargePages is off on Solr Startup
G1GC, Java 11  (ZGC with Java 15 and HugePages turned on was a disaster.  We 
suspect it was due to the Huge Pages configuration)
At one time we discussed putting two instances of Solr on each node, giving us 
a cloud of 60 instances instead of 30.  Load Average is high on these nodes 
during certain types of queries or updates, but CPU Load is relatively low and 
should be able to accommodate a second instance, but all the data would still 
be on the same RAID10 group of disks.
Collection 4 is the trouble collection.  It has nearly a billion documents, and 
there are between 200 and 400 million updates every day.  How do we get that 
kind of update performance, and still serve 10 million queries a day?  Schemas 
have been reviewed and re-reviewed to ensure we are only indexing and storing 
what is absolutely necessary.  What are we missing?  Do we need to revisit our 
replica policy?  Number of replicas or types of replicas (to ensure some are 
only used for reading, etc?)
[Grabbed from the Admin UI]
755.6Gb Index Size according to Solr Cloud UI
Total #docs: 371.8mn
Avg size/doc: 2.1Kb
90 Shards, 2 NRT Replicas per Shard, 1,750,612,476 documents, avg size/doc: 
1.7Kb, uses nested documents
collection-1_s69r317   31.1Gb
collection-1_s49r96 30.7Gb
collection-1_s78r154   30.2Gb
collection-1_s40r259   30.1Gb
collection-1_s9r197 29.1Gb
collection-1_s18r34 28.9Gb
120 Shards, 2 TLOG Replicas per Shard, 2,230,207,046 documents, avg size/doc: 
1.3Kb
collection-2_s78r154   22.8Gb
collection-2_s49r96 22.8Gb
collection-2_s46r331   22.8Gb
collection-2_s18r34 22.7Gb
collection-2_s109r21622.7Gb
collection-2_s104r44722.7Gb
collection-2_s15r269   22.7Gb
collection-2_s73r385   22.7Gb
120 Shards, 2 TLOG Replicas per Shard, 733,588,503 documents, avg size/doc: 
1.9Kb
collection-3_s19r277   10.6Gb
collection-3_s108r21410.6Gb
collection-3_s48r94 10.6Gb
collection-3_s109r45710.6Gb
collection-3_s47r333   10.5Gb
collection-3_s78r154   10.5Gb
collection-3_s18r34 10.5Gb
collection-3_s77r393   10.5Gb

120 Shards, 2 TLOG Replicas per Shard, 864,372,654 documents, avg size/doc: 
5.6Kb
collection-4_s109r21638.7Gb
collection-4_s100r43938.7Gb
collection-4_s49r96 38.7Gb
collection-4_s35r309   38.6Gb
collection-4_s18r34 38.6Gb
collection-4_s78r154   38.6Gb
collection-4_s7r253 38.6Gb
collection-4_s69r377   38.6Gb

Query is timing out.

[Modassar Ather]

Hi,

The boolean query with a bigger value for *rows *times out with the
following message.

The request took too long to iterate over terms. Timeout: timeoutAt

Solr version : Solr 8.6.3
Time allowed : 30
Field  : 
Query : fl:(term1 OR term2 OR . OR term1)
rows : 1
wt : json/phps

Recently we have migrated from Solr 6.5.1. The above query used to work
very fast in this Solr version.

Kindly provide your suggestions.

Best,
Modassar

Issue regarding highlighting in BlendedInfixLookupFactory

[Debarshi Das]

 Hi Lucene team,

I have been using Solr 8.4.0 and I encountered an issue where suggest
highlight feature in “BlendedInfixLookupFactory” is not working whenever I
am using “contextField”. While searching online, I came across below page
which does indicate a similar issue but for different SOLR version.

https://issues.apache.org/jira/browse/SOLR-7964

Please suggest me how to configure the SOLR or the version under 8.4.x that
might have a patch included to fix this issue, or any other version that
supports both “BlendedInfixLookupFactory” and “contextField” along with the
fix.

Regards,
Debarshi