Re: Need help on LTR

[Mohomed Rimash]

Hi Amjad, After adding the libraries into the path, Did you restart the
SOLR ?

On Tue, 19 Mar 2019 at 08:45, Amjad Khan  wrote:

> I followed the Solr LTR Documentation
>
> https://lucene.apache.org/solr/guide/7_4/learning-to-rank.html <
> https://lucene.apache.org/solr/guide/7_4/learning-to-rank.html>
>
> 1. Added library into the solr-config
> 
>regex=".*\.jar" />
>  regex="solr-ltr-\d.*\.jar" />
> 2. Successfully added feature
> 3. Get schema to see feature is available
> 4. When I try to push model I see the error below, however I added the lib
> into solr-cofig
>
> Response
> {
>   "responseHeader":{
> "status":400,
> "QTime":1},
>   "error":{
> "metadata":[
>   "error-class","org.apache.solr.common.SolrException",
>   "root-error-class","java.lang.NullPointerException"],
> "msg":"org.apache.solr.ltr.model.ModelException: Model type does not
> exist org.apache.solr.ltr.model.LinearModel",
> "code":400}}
>
> Thanks

Need help on LTR

[Amjad Khan]

I followed the Solr LTR Documentation 

https://lucene.apache.org/solr/guide/7_4/learning-to-rank.html 


1. Added library into the solr-config

  

2. Successfully added feature
3. Get schema to see feature is available
4. When I try to push model I see the error below, however I added the lib into 
solr-cofig

Response
{
  "responseHeader":{
"status":400,
"QTime":1},
  "error":{
"metadata":[
  "error-class","org.apache.solr.common.SolrException",
  "root-error-class","java.lang.NullPointerException"],
"msg":"org.apache.solr.ltr.model.ModelException: Model type does not exist 
org.apache.solr.ltr.model.LinearModel",
"code":400}}

Thanks

Fwd: Boosting Query Time Complexity

[Krishna Raj]

Hello Experts,

My goal is to understand the time complexity of the Boosting Query as part
of a search in Solr:

sort=score+desc
defType=edismax
boost=

I followed the stacktrace for the the search call and I am believe the time
complexity is as follows:

- Main Query time + multiple filters reduce the DocId set of lets say size
N.

- Boost Query iterates the N DocId set and scores using the function. Time
Complexity: O(N)

- Since Tim Sort is used worst case Time Complexity is O(NLogN).

Can I know is this is a correct understanding? *Or, does the Custom
function boost scoring happens in tandem with Document Matching ?*

*Call StackTrace:*

 
com..functions.functionvalues.PmmlComputation.pmml(PmmlComputation.java:75)
 
com..functions.functionvalues.PmmlDoubleValues.doubleVal(PmmlDoubleValues.java:23)
 
org.apache.lucene.queries.function.docvalues.DoubleDocValues.floatVal(DoubleDocValues.java:49)
 
org.apache.lucene.queries.function.valuesource.ProductFloatFunction.func(ProductFloatFunction.java:39)
 
org.apache.lucene.queries.function.valuesource.MultiFloatFunction$1.floatVal(MultiFloatFunction.java:82)
 
org.apache.lucene.queries.function.BoostedQuery$CustomScorer.score(BoostedQuery.java:129)
 org.apache.lucene.search.ConjunctionScorer.score(ConjunctionScorer.java:64)
 
org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector$1.collect(TopScoreDocCollector.java:64)
 
org.apache.lucene.search.MultiCollector$MultiLeafCollector.collect(MultiCollector.java:174)
 org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:221)
 org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
 org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
 org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
 org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:211)
 
org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1861)
 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1578)
 org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:581)
 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:528)
 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:154)
 org.apache.solr.core.SolrCore.execute(SolrCore.java:2089)
 org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)

Re: Fetching All Terms and Corresponding Documents

[Zheng Lin Edwin Yeo]

Hi,

You can take a look at MoreLikeThis from the Solr Reference Guide at
https://lucene.apache.org/solr/guide/7_7/morelikethis.html, and see if it
fits what you are looking for.

Regards,
Edwin


On Mon, 18 Mar 2019 at 22:24, Furkan KAMACI  wrote:

> Hi,
>
> I need to iterate on all terms at Solr index, and then find related
> documents for some terms that match my criteria.
>
> I know that I can send a query to *LukeRequestHandler*:
>
> */admin/luke?fl=content={distinct term count}=json*
>
> and then check my criteria. If matches, I can send a *fq* to retrieve
> related docs.
>
> However, is there any other efficient way (via rest or Solrj) for my case?
>
> Kind Regards,
> Furkan KAMACI
>

Re: Behavior of Function Query

[Zheng Lin Edwin Yeo]

Hi,

Which version of Solr are you running? How many shards/replica is this
running on, and are you running on SolrCloud?

Regards,
Edwin

On Mon, 18 Mar 2019 at 22:19, Ashish Bisht  wrote:

> Please see the below requests and response
>
> http://Sol:8983/solr/SCSpell/select?q="*internet of
>
> things*"=edismax=spellcontent=json=1=score,internet_of_things:query({!edismax
> v='"*internet of things*"'}),instant_of_things:query({!edismax v='"instant
> of things"'})
>
>
> Response contains score from function query
>
>  "fl":"score,internet_of_things:query({!edismax v='\"internet of
> things\"'}),instant_of_things:query({!edismax v='\"instant of things\"'})",
>   "rows":"1",
>   "wt":"json"}},
>   "response":{"numFound":851,"start":0,"maxScore":7.6176834,"docs":[
>   {
> "score":7.6176834,
>* "internet_of_things":7.6176834*}]
>   }}
>
>
> But if in the same request q is changed,it doesn't give score
>
> http://Sol-1:8983/solr/SCSpell/select?q="*wall
>
> street*"=edismax=spellcontent=json=1=score,internet_of_things:query({!edismax
> v='"*internet of things*"'}),instant_of_things:query({!edismax v='"instant
> of things"'})
>
>"q":"\"wall street\"",
>   "defType":"edismax",
>   "qf":"spellcontent",
>   "fl":"score,internet_of_things:query({!edismax v='\"internet of
> things\"'}),instant_of_things:query({!edismax v='\"instant of things\"'})",
>   "rows":"1",
>   "wt":"json"}},
>   "response":{"numFound":46,"start":0,"maxScore":15.670144,"docs":[
>   {
> "score":15.670144}]
>   }}
>
>
> Why score of function query is getting applied when q is a different.
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>

Re: Re: obfuscated password error

[Zheng Lin Edwin Yeo]

Hi,

Did you generate your keystore with the obfuscated password or the plain
text password?

Regards,
Edwin

On Tue, 19 Mar 2019 at 02:32, Branham, Jeremy (Experis) 
wrote:

> I’m not sure if you are sharing the trust/keystores, so I may be off-base
> here…
>
> Some thoughts –
> - Verify your VM arguments, to be sure there aren’t conflicting SSL
> properties.
> - Verify the environment is targeting the correct version of Java
> - Verify the trust/key stores exist where they are expected, and you can
> list the contents with the keytool
> - Verify the correct CA certs are trusted
>
>
> Jeremy Branham
> jb...@allstate.com
>
> On 3/18/19, 1:08 PM, "Satya Marivada"  wrote:
>
> Any suggestions please.
>
> Thanks,
> Satya
>
> On Mon, Mar 18, 2019 at 11:12 AM Satya Marivada <
> satya.chaita...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > Using solr-6.3.0, to obfuscate the password, have used jetty util to
> > generate obfuscated password
> >
> >
> > java -cp jetty-util-9.3.8.v20160314.jar
> > org.eclipse.jetty.util.security.Password mypassword
> >
> >
> > The output has been used in
> https://urldefense.proofpoint.com/v2/url?u=http-3A__solr.in.sh=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=YtmCJK2U90u6mqx-FOmBS5nqy03luM2J-Zc_LhImnG0=
> as below
> >
> >
> >
> >
> SOLR_SSL_KEY_STORE=/sanfs/mnt/vol01/solr/solr-6.3.0/server/etc/solr-ssl.keystore.jks
> >
> >
> SOLR_SSL_KEY_STORE_PASSWORD="OBF:1bcd1l161lts1ltu1uum1uvk1lq41lq61k221b9t"
> >
> >
> >
> SOLR_SSL_TRUST_STORE=/sanfs/mnt/vol01/solr/solr-6.3.0/server/etc/solr-ssl.keystore.jks
> >
> >
> >
> SOLR_SSL_TRUST_STORE_PASSWORD="OBF:1bcd1l161lts1ltu1uum1uvk1lq41lq61k221b9t"
> >
> > Solr does not start fine with below exception, any suggestions? If I
> use
> > the plain text password, it works fine. One more thing is that the
> same
> > setup with obfuscated password works in other environments except
> one which
> > got this exception. Recently system level patches are applied, just
> saying
> > though dont think that could have impact,
> >
> > Caused by: java.net.SocketException:
> > java.security.NoSuchAlgorithmException: Error constructing
> implementation
> > (algorithm: Default, provider: SunJSSE, class:
> sun.security.ssl.SSLContextIm
> > pl$DefaultSSLContext)
> > at
> > javax.net.ssl.DefaultSSLSocketFactory.throwException(
> https://urldefense.proofpoint.com/v2/url?u=http-3A__SSLSocketFactory.java=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=dud5QRNkwTMDiH04sCjNs1U9_5t8wBMxJNiyQRdjXRk=:248
> )
> > at
> > javax.net.ssl.DefaultSSLSocketFactory.createSocket(
> https://urldefense.proofpoint.com/v2/url?u=http-3A__SSLSocketFactory.java=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=dud5QRNkwTMDiH04sCjNs1U9_5t8wBMxJNiyQRdjXRk=:255
> )
> > at
> > org.apache.http.conn.ssl.SSLSocketFactory.createSocket(
> https://urldefense.proofpoint.com/v2/url?u=http-3A__SSLSocketFactory.java=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=dud5QRNkwTMDiH04sCjNs1U9_5t8wBMxJNiyQRdjXRk=:513
> )
> > at
> > org.apache.http.conn.ssl.SSLSocketFactory.createSocket(
> https://urldefense.proofpoint.com/v2/url?u=http-3A__SSLSocketFactory.java=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=dud5QRNkwTMDiH04sCjNs1U9_5t8wBMxJNiyQRdjXRk=:383
> )
> > at
> >
> org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(
> https://urldefense.proofpoint.com/v2/url?u=http-3A__DefaultClientConnectionOperator.java=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=EATR9hBi7P9kYpCcJ8maLn81bHA72GhhvwWQY0V9EQw=:165
> )
> > at
> > org.apache.http.impl.conn.ManagedClientConnectionImpl.open(
> https://urldefense.proofpoint.com/v2/url?u=http-3A__ManagedClientConnectionImpl.java=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=yuCHQjzNKMtl0uWKiDWB01ChPkiY1tCaPX8n8lhdR-s=:304
> )
> > at
> > org.apache.http.impl.client.DefaultRequestDirector.tryConnect(
> https://urldefense.proofpoint.com/v2/url?u=http-3A__DefaultRequestDirector.java=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=BuInFyYyCadGREvZzUoJMKX-9SWG7lzHzdO-A3x3rGA=:611
> )
> > at
> > org.apache.http.impl.client.DefaultRequestDirector.execute(
> 

Boosting Query Time Complexity

[Krishna Raj]

Hello Experts,

My goal is to understand the time complexity of the Boosting Query as part
of a search in Solr:

sort=score+desc
defType=edismax
boost=

I followed the stacktrace for the the search call and I am believe the time
complexity is as follows:

- Main Query time + multiple filters reduce the DocId set of lets say size
N.

- Boost Query iterates the N DocId set and scores using the function. Time
Complexity: O(N)

- Since Tim Sort is used worst case Time Complexity is O(NLogN).

Can I know is this is a correct understanding? *Or, does the Custom
function boost scoring happens in tandem with Document Matching ?*

*Call StackTrace:*

 
com..functions.functionvalues.PmmlComputation.pmml(PmmlComputation.java:75)
 
com..functions.functionvalues.PmmlDoubleValues.doubleVal(PmmlDoubleValues.java:23)
 
org.apache.lucene.queries.function.docvalues.DoubleDocValues.floatVal(DoubleDocValues.java:49)
 
org.apache.lucene.queries.function.valuesource.ProductFloatFunction.func(ProductFloatFunction.java:39)
 
org.apache.lucene.queries.function.valuesource.MultiFloatFunction$1.floatVal(MultiFloatFunction.java:82)
 
org.apache.lucene.queries.function.BoostedQuery$CustomScorer.score(BoostedQuery.java:129)
 org.apache.lucene.search.ConjunctionScorer.score(ConjunctionScorer.java:64)
 
org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector$1.collect(TopScoreDocCollector.java:64)
 
org.apache.lucene.search.MultiCollector$MultiLeafCollector.collect(MultiCollector.java:174)
 org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:221)
 org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
 org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
 org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:669)
 org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
 
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:211)
 
org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1861)
 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1578)
 org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:581)
 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:528)
 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:154)
 org.apache.solr.core.SolrCore.execute(SolrCore.java:2089)
 org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)

Re: SolrJ - CoreAdminRequest / PingRequest and HttpSolrClient baseUrl

[Jörn Franke]

Depends what you want to achieve. You can also have two different beans with 
two different clients and one accesses a core and another accesses without.

You may also want depending on your use case get all cores and ping each core 
individually to check if they are reachable.

> Am 18.03.2019 um 21:04 schrieb Markus Schuch :
> 
> Hi folks,
> 
> i have a question about how to perform a connection check with SolrJ in a 
> generic way.
> 
> I found PingRequest and CoreAdminRequest in SolrJ.
> Neither of both work in both scenarios when a SolrClient is created by 
> HttpSolrClient.Builder with AND without having the baseUrl containing the 
> path to a particular core:
> 
> 
>  *   CoreAdminRequest - is only working when no particular core is given
>  *   PingRequest - is only working when a particular core is given
> 
> Here is a sample test case demonstrating both scenarios: 
> https://gist.github.com/schuch/9d86383cb36244c1b6ec0360ebf739a5
> 
> Is this behavior intended or might it be a bug?
> Is there a better way to do a connection check?
> 
> My use case:
> I look for an improvement for the spring boot solr health indicator: 
> https://github.com/spring-projects/spring-boot/blob/master/spring-boot-project/spring-boot-actuator/src/main/java/org/springframework/boot/actuate/solr/SolrHealthIndicator.java
> 
> It currently uses the CoreAdminRequest which fails when the SolrClient points 
> to a particular core.
> 
> Many thanks in advance,
> Markus
> 
> 
> 
> Pflichtangaben anzeigen
> 
> N?here Informationen zur Datenverarbeitung im DB-Konzern finden Sie hier: 
> http://www.deutschebahn.com/de/konzern/datenschutz

Re: SolrJ - CoreAdminRequest / PingRequest and HttpSolrClient baseUrl

[Shawn Heisey]

On 3/18/2019 2:04 PM, Markus Schuch wrote:

   *   CoreAdminRequest - is only working when no particular core is given
   *   PingRequest - is only working when a particular core is given


That sounds like what I would expect to happen.

Using HTTP (not SolrJ), the CoreAdmin API is accessed through a URL that 
does NOT contain a core name in the URL path.  If you initialize your 
HttpSolrClient using a URL that contains the core name, you won't be 
able to use that client for CoreAdmin requests -- because CoreAdmin 
requests do not use that kind of URL.


Similarly, the Ping handler is accessed through a URL that DOES contain 
the core name in the URL path.  You can use a client initialized without 
a core name by providing the core name when you execute the request. 
See the following code:


@Test
public void testPingWithoutCore() throws IOException, 
SolrServerException {

String coreToTest = "techproducts";
HttpSolrClient client = new 
HttpSolrClient.Builder().withBaseSolrUrl("http://localhost:8983/solr;).build();

SolrPing ping = new SolrPing();
SolrPingResponse response = ping.process(client, coreToTest);
assertEquals(0, response.getStatus());
}

SolrJ - CoreAdminRequest / PingRequest and HttpSolrClient baseUrl

[Markus Schuch]

Hi folks,

i have a question about how to perform a connection check with SolrJ in a 
generic way.

I found PingRequest and CoreAdminRequest in SolrJ.
Neither of both work in both scenarios when a SolrClient is created by 
HttpSolrClient.Builder with AND without having the baseUrl containing the path 
to a particular core:


  *   CoreAdminRequest - is only working when no particular core is given
  *   PingRequest - is only working when a particular core is given

Here is a sample test case demonstrating both scenarios: 
https://gist.github.com/schuch/9d86383cb36244c1b6ec0360ebf739a5

Is this behavior intended or might it be a bug?
Is there a better way to do a connection check?

My use case:
I look for an improvement for the spring boot solr health indicator: 
https://github.com/spring-projects/spring-boot/blob/master/spring-boot-project/spring-boot-actuator/src/main/java/org/springframework/boot/actuate/solr/SolrHealthIndicator.java

It currently uses the CoreAdminRequest which fails when the SolrClient points 
to a particular core.

Many thanks in advance,
Markus



Pflichtangaben anzeigen

N?here Informationen zur Datenverarbeitung im DB-Konzern finden Sie hier: 
http://www.deutschebahn.com/de/konzern/datenschutz

Re: Java Heap Error

[wclarke]

I was mistaken xms and xmx are equal.  I was looking in the wrong spot.

I will look at the rest.

Thanks!



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Garbage Collection Metrics

[Jeff Courtade]

yes i needed to see behavior ovr time from 14 servers and correlate it.

Putting it into zabbix gave me behavior type graphs... like this unusual
gc activity...

[image: image.png]

On Mon, Mar 18, 2019 at 1:32 PM Erick Erickson 
wrote:

> GCViewer will work on the GC logging file created by Solr. It has some
> nice summaries, particularly of stop-the-world GC events.
>
> From there, you can pinpoint the exact times from manual inspection of the
> GC log. It usually looks like this:
>
> Your replica went into recovery when again? sometime between 14:00 and
> 15:00.
>
> GCViewer: Oh, look. Sometime between 14:30 and 14:35 there was a 20 second
> stop-the-world GC pause.
>
> Ok, back to the Solr log and I see “leader initiated recovery” sent by the
> leader to this replica at 14:32. Oh look, the GC log shows the
> stop-the-world GC pause starting at 14:31:45. Let's tune GC.
>
> That’s if you want to pin it exactly, but usually it looks more like this:
>
> Oh, look. GCViewer shows long stop-the-world GC pauses about the time your
> replica went into recovery, let’s start tuning GC.
>
> There are other products that give you a good way to navigate the GC
> events, GCViewer is free though.
>
> Best,
> Erick
>
> > On Mar 18, 2019, at 10:17 AM, Jeff Courtade 
> wrote:
> >
> > So,
> >
> > I had a problem when at a customer site. They use zabbix for data
> > collection and alerting.
> >
> > The solr server had been setup to use only jmx metrics.
> >
> > the jvm was unstable and would lock up for a period of time and the
> metrics
> > and counters would be all screwed up. Because it was using jmx to alert
> it
> > was screwing up as  the jvm needed to be working to be used.
> >
> > So I turned on gclogging and wrote a script to collect data points about
> > for instance how long the jvm was stopped in the last minute.
> >
> > I eventually got the gc tuned and behaving well but it was difficult.
> >
> >
> > turn on gcloging
> >
> > i use these options
> >
> > -Xloggc:../var/logs/gclog.log \
> > -XX:+PrintHeapAtGC \
> > -XX:+PrintGCDetails \
> > -XX:+PrintGCDateStamps \
> > -XX:+PrintGCTimeStamps \
> > -XX:+PrintTenuringDistribution \
> > -XX:+PrintGCApplicationStoppedTime \
> > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 \
> > -XX:GCLogFileSize=100M
> >
> > in the solr runtme users crontab on each system 
> >
> > * * * * * nohup /opt/scripts/getstats-gclog &
> >
> > this is the script
> >
> > /opt/scripts/getstats-gclog
> > #/bin/bash -x
> > #
> > # get some statistics
> > #
> > # GC time stamp
> > # 2018-06-27T12:52:57.200+
> > #
> > FIVEMIN=`date --date '5 minutes ago' +'%Y-%m-%dT%H:%M'`
> > FOURMIN=`date --date '4 minutes ago' +'%Y-%m-%dT%H:%M'`
> > THREEMIN=`date --date '3 minutes ago' +'%Y-%m-%dT%H:%M'`
> > TWOMIN=`date --date '2 minutes ago' +'%Y-%m-%dT%H:%M'`
> > ONEMIN=`date --date '1 minute ago' +'%Y-%m-%dT%H:%M'`
> > YEAR=`date --date '1 minute ago' +'%Y'`
> > MONTH=`date --date '1 minute ago' +'%m'`
> > DAY=`date --date '1 minute ago' +'%d'`
> > HOUR=`date --date '1 minute ago' +'%H'`
> > MINUTE=`date --date '1 minute ago' +'%M'`
> > SECOND=`date --date '1 minute ago' +'%S'`
> > #
> > #
> > STATSDIR=/opt/stats/
> > WORKDIR=/$STATSDIR/working_gc
> > #
> > #
> > LOGDIR=/u01/app/solr/var/logs/
> > LOGNAME=gclog.log
> > Prep() {
> > mkdir -p $STATSDIR
> > chmod 755 $STATSDIR
> > mkdir -p $WORKDIR
> > chmod 755 $WORKDIR
> > }
> > GetStats() {
> > cd $WORKDIR
> > grep $ONEMIN $LOGDIR/$LOGNAME.*|grep stopped|awk '{print $11}'>ALLGC
> > COUNT=`cat ALLGC |wc -l`
> > # number under .00X XDecimalplaces U3D for example
> > U3D=`grep "^0\.00[1-9]" ALLGC|wc -l`
> > if [ -z $U3D ]
> > then
> >  U3D=0
> > fi
> > U2D=`grep "^0\.0[1-9]" ALLGC|wc -l`
> > if [ -z $U2D ]
> > then
> >  U2D=0
> > fi
> > U1D=`grep "^0\.[1-9]" ALLGC|wc -l`
> > if [ -z $U1D ]
> > then
> >  U1D=0
> > fi
> > O1S=`cat ALLGC | grep -v "^0\."|wc -l`
> > if [ -z $O1S ]
> > then
> >  O1S=0
> > fi
> > O10S=`grep "[0-9]\+[0-9]\.[0-9]*" ALLGC|wc -l`
> > if [ -z $O10S ]
> > then
> >  O10S=0
> > fi
> > cat ALLGC | grep -v "^0\.">OVER1SECDATA
> >
> > TOTAL=0
> > COUNT=0
> > while read DAT
> > do
> > TOTAL=`echo "$TOTAL + $DAT"|bc`
> > COUNT=`expr $COUNT + 1`
> > done <$WORKDIR/ALLGC
> > #AO1S=$(printf "%.2f\n"  `echo "scale=10;$COUNT/60" | bc`)
> > #AVGQT=$(printf "%.0f\n"  `echo "scale=10;$TOTAL/$COUNT"|bc`)
> > TOTSTOP=$TOTAL
> > AVGSTOPT=`echo "scale=7;$TOTAL/$COUNT"|bc`
> > if [ -z $AVGSTOPT ]
> > then
> >  AVGSTOPT=0
> > fi
> > # get top gc times
> > #
> >
> > #echo 0.000>ALLGCU1S
> > #echo 0.000>ALLGCO1S
> >
> > grep '^0.' $WORKDIR/ALLGC >ALLGCU1S
> > grep -v '^0.' $WORKDIR/ALLGC >ALLGCO1S
> > TOPGCTIMEU1S=`cat $WORKDIR/ALLGCU1S |sort |tail -1`
> > if [ -z $TOPGCTIMEU1S ]
> > then
> >  TOPGCTIMEU1S=0
> > fi
> > TOPGCTIMEO1S=`cat $WORKDIR/ALLGCO1S |sort |tail -1`
> > if [ -z $TOPGCTIMEO1S ]
> > then
> >  TOPGCTIMEO1S=0
> > fi
> >
> > }
> > PrintStats() {
> > #
> > ## stats
> > #COUNT= total number of garbage collection this minute

Re: Re: obfuscated password error

[Branham, Jeremy (Experis)]

I’m not sure if you are sharing the trust/keystores, so I may be off-base here…

Some thoughts –
- Verify your VM arguments, to be sure there aren’t conflicting SSL properties.
- Verify the environment is targeting the correct version of Java
- Verify the trust/key stores exist where they are expected, and you can list 
the contents with the keytool
- Verify the correct CA certs are trusted

 
Jeremy Branham
jb...@allstate.com

On 3/18/19, 1:08 PM, "Satya Marivada"  wrote:

Any suggestions please.

Thanks,
Satya

On Mon, Mar 18, 2019 at 11:12 AM Satya Marivada 
wrote:

> Hi All,
>
> Using solr-6.3.0, to obfuscate the password, have used jetty util to
> generate obfuscated password
>
>
> java -cp jetty-util-9.3.8.v20160314.jar
> org.eclipse.jetty.util.security.Password mypassword
>
>
> The output has been used in 
https://urldefense.proofpoint.com/v2/url?u=http-3A__solr.in.sh=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=YtmCJK2U90u6mqx-FOmBS5nqy03luM2J-Zc_LhImnG0=
 as below
>
>
>
> 
SOLR_SSL_KEY_STORE=/sanfs/mnt/vol01/solr/solr-6.3.0/server/etc/solr-ssl.keystore.jks
>
> SOLR_SSL_KEY_STORE_PASSWORD="OBF:1bcd1l161lts1ltu1uum1uvk1lq41lq61k221b9t"
>
>
> 
SOLR_SSL_TRUST_STORE=/sanfs/mnt/vol01/solr/solr-6.3.0/server/etc/solr-ssl.keystore.jks
>
>
> 
SOLR_SSL_TRUST_STORE_PASSWORD="OBF:1bcd1l161lts1ltu1uum1uvk1lq41lq61k221b9t"
>
> Solr does not start fine with below exception, any suggestions? If I use
> the plain text password, it works fine. One more thing is that the same
> setup with obfuscated password works in other environments except one 
which
> got this exception. Recently system level patches are applied, just saying
> though dont think that could have impact,
>
> Caused by: java.net.SocketException:
> java.security.NoSuchAlgorithmException: Error constructing implementation
> (algorithm: Default, provider: SunJSSE, class: 
sun.security.ssl.SSLContextIm
> pl$DefaultSSLContext)
> at
> 
javax.net.ssl.DefaultSSLSocketFactory.throwException(https://urldefense.proofpoint.com/v2/url?u=http-3A__SSLSocketFactory.java=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=dud5QRNkwTMDiH04sCjNs1U9_5t8wBMxJNiyQRdjXRk=:248)
> at
> 
javax.net.ssl.DefaultSSLSocketFactory.createSocket(https://urldefense.proofpoint.com/v2/url?u=http-3A__SSLSocketFactory.java=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=dud5QRNkwTMDiH04sCjNs1U9_5t8wBMxJNiyQRdjXRk=:255)
> at
> 
org.apache.http.conn.ssl.SSLSocketFactory.createSocket(https://urldefense.proofpoint.com/v2/url?u=http-3A__SSLSocketFactory.java=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=dud5QRNkwTMDiH04sCjNs1U9_5t8wBMxJNiyQRdjXRk=:513)
> at
> 
org.apache.http.conn.ssl.SSLSocketFactory.createSocket(https://urldefense.proofpoint.com/v2/url?u=http-3A__SSLSocketFactory.java=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=dud5QRNkwTMDiH04sCjNs1U9_5t8wBMxJNiyQRdjXRk=:383)
> at
> 
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(https://urldefense.proofpoint.com/v2/url?u=http-3A__DefaultClientConnectionOperator.java=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=EATR9hBi7P9kYpCcJ8maLn81bHA72GhhvwWQY0V9EQw=:165)
> at
> 
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(https://urldefense.proofpoint.com/v2/url?u=http-3A__ManagedClientConnectionImpl.java=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=yuCHQjzNKMtl0uWKiDWB01ChPkiY1tCaPX8n8lhdR-s=:304)
> at
> 
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(https://urldefense.proofpoint.com/v2/url?u=http-3A__DefaultRequestDirector.java=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=BuInFyYyCadGREvZzUoJMKX-9SWG7lzHzdO-A3x3rGA=:611)
> at
> 
org.apache.http.impl.client.DefaultRequestDirector.execute(https://urldefense.proofpoint.com/v2/url?u=http-3A__DefaultRequestDirector.java=DwIBaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=Ix7ZcyM45ms93i2fWx4SNPgiLA7TGHVDOjCklcxbvLs=BuInFyYyCadGREvZzUoJMKX-9SWG7lzHzdO-A3x3rGA=:446)
> at
> 

Re: Re: Garbage Collection Metrics

[Branham, Jeremy (Experis)]

I get these metrics by pushing the JMX data into Graphite, then use the 
non-negative derivative function on the GC ‘time’ metric.
It essentially shows the amount of change on a counter, at the specific time it 
occurred. 
 
Jeremy Branham
jb...@allstate.com

On 3/18/19, 12:06 PM, "Jeff Courtade"  wrote:

The only way I found to track GC times was by truning on GC logging and the
writing cronjob data collection script and graphing it in zabbix

On Mon, Mar 18, 2019 at 12:34 PM Erick Erickson 
wrote:

> Attachments are pretty aggressively stripped by the apache mail server, so
> it didn’t come through.
>
> That said, I’m not sure how much use just the last GC time is. What do you
> want it for? This
> sounds a bit like an XY problem.
>
> Best,
> Erick
>
> > On Mar 17, 2019, at 2:43 PM, Karthik K G  wrote:
> >
> > Hi Team,
> >
> > I was looking for Old GC duration time metrics, but all I could find was
> the API for this "/solr/admin/metrics?wt=json=jvm=gc.G1-
> Old-Generation", but I am not sure if this is for
> 'gc_g1_gen_o_lastgc_duration'. I tried to hookup the IP to the jconsole 
and
> was looking for the metrics, but all I could see was the collection time
> but not last GC duration as attached in the screenshot. Can you please 
help
> here with finding the correct metrics. I strongly believe we are not
> capturing this information. Please correct me if I am wrong.
> >
> > Thanks & Regards,
> > Karthik
>
>

Re: obfuscated password error

[Satya Marivada]

Any suggestions please.

Thanks,
Satya

On Mon, Mar 18, 2019 at 11:12 AM Satya Marivada 
wrote:

> Hi All,
>
> Using solr-6.3.0, to obfuscate the password, have used jetty util to
> generate obfuscated password
>
>
> java -cp jetty-util-9.3.8.v20160314.jar
> org.eclipse.jetty.util.security.Password mypassword
>
>
> The output has been used in solr.in.sh as below
>
>
>
> SOLR_SSL_KEY_STORE=/sanfs/mnt/vol01/solr/solr-6.3.0/server/etc/solr-ssl.keystore.jks
>
> SOLR_SSL_KEY_STORE_PASSWORD="OBF:1bcd1l161lts1ltu1uum1uvk1lq41lq61k221b9t"
>
>
> SOLR_SSL_TRUST_STORE=/sanfs/mnt/vol01/solr/solr-6.3.0/server/etc/solr-ssl.keystore.jks
>
>
> SOLR_SSL_TRUST_STORE_PASSWORD="OBF:1bcd1l161lts1ltu1uum1uvk1lq41lq61k221b9t"
>
> Solr does not start fine with below exception, any suggestions? If I use
> the plain text password, it works fine. One more thing is that the same
> setup with obfuscated password works in other environments except one which
> got this exception. Recently system level patches are applied, just saying
> though dont think that could have impact,
>
> Caused by: java.net.SocketException:
> java.security.NoSuchAlgorithmException: Error constructing implementation
> (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextIm
> pl$DefaultSSLContext)
> at
> javax.net.ssl.DefaultSSLSocketFactory.throwException(SSLSocketFactory.java:248)
> at
> javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:255)
> at
> org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactory.java:513)
> at
> org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactory.java:383)
> at
> org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:165)
> at
> org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
> at
> org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
> at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
> at
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:498)
> ... 11 more
> Caused by: java.security.NoSuchAlgorithmException: Error constructing
> implementation (algorithm: Default, provider: SunJSSE, class:
> sun.security.ssl.SSLContextImpl$DefaultSSLContext)
> at java.security.Provider$Service.newInstance(Provider.java:1617)
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
> at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
> at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96)
>
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
> at org.eclipse.jetty.start.Main.start(Main.java:457)
> at org.eclipse.jetty.start.Main.main(Main.java:75)
> Caused by: java.io.IOException: Keystore was tampered with, or password
> was incorrect
> at
> sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:785)
> at
> sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
> at
> sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
> at
> sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
> at java.security.KeyStore.load(KeyStore.java:1445)
> at
> sun.security.ssl.TrustManagerFactoryImpl.getCacertsKeyStore(TrustManagerFactoryImpl.java:226)
> at
> sun.security.ssl.SSLContextImpl$DefaultManagersHolder.getTrustManagers(SSLContextImpl.java:877)
> at
> sun.security.ssl.SSLContextImpl$DefaultManagersHolder.(SSLContextImpl.java:854)
> at
> sun.security.ssl.SSLContextImpl$DefaultSSLContext.(SSLContextImpl.java:1019)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 

Re: Unsubscribing

[Erick Erickson]

Please follow the instructions here: 
http://lucene.apache.org/solr/community.html#mailing-lists-irc. You must use 
the _exact_ same e-mail as you used to subscribe.

If the initial try doesn't work and following the suggestions at the "problems" 
link doesn't work for you, let us know. But note you need to show us the 
_entire_ return header to allow anyone to diagnose the problem.

Best,
Erick



> On Mar 18, 2019, at 10:09 AM, Monique Monteiro  
> wrote:
> 
> How can I unsubscribe from this list?
> 
> Thanks in advance,
> 
> -- 
> Monique Monteiro
> Twitter: http://twitter.com/monilouise

Re: Garbage Collection Metrics

[Erick Erickson]

GCViewer will work on the GC logging file created by Solr. It has some nice 
summaries, particularly of stop-the-world GC events.

From there, you can pinpoint the exact times from manual inspection of the GC 
log. It usually looks like this:

Your replica went into recovery when again? sometime between 14:00 and 15:00.

GCViewer: Oh, look. Sometime between 14:30 and 14:35 there was a 20 second 
stop-the-world GC pause. 

Ok, back to the Solr log and I see “leader initiated recovery” sent by the 
leader to this replica at 14:32. Oh look, the GC log shows the stop-the-world 
GC pause starting at 14:31:45. Let's tune GC.

That’s if you want to pin it exactly, but usually it looks more like this:

Oh, look. GCViewer shows long stop-the-world GC pauses about the time your 
replica went into recovery, let’s start tuning GC.

There are other products that give you a good way to navigate the GC events, 
GCViewer is free though.

Best,
Erick

> On Mar 18, 2019, at 10:17 AM, Jeff Courtade  wrote:
> 
> So,
> 
> I had a problem when at a customer site. They use zabbix for data
> collection and alerting.
> 
> The solr server had been setup to use only jmx metrics.
> 
> the jvm was unstable and would lock up for a period of time and the metrics
> and counters would be all screwed up. Because it was using jmx to alert it
> was screwing up as  the jvm needed to be working to be used.
> 
> So I turned on gclogging and wrote a script to collect data points about
> for instance how long the jvm was stopped in the last minute.
> 
> I eventually got the gc tuned and behaving well but it was difficult.
> 
> 
> turn on gcloging
> 
> i use these options
> 
> -Xloggc:../var/logs/gclog.log \
> -XX:+PrintHeapAtGC \
> -XX:+PrintGCDetails \
> -XX:+PrintGCDateStamps \
> -XX:+PrintGCTimeStamps \
> -XX:+PrintTenuringDistribution \
> -XX:+PrintGCApplicationStoppedTime \
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 \
> -XX:GCLogFileSize=100M
> 
> in the solr runtme users crontab on each system 
> 
> * * * * * nohup /opt/scripts/getstats-gclog &
> 
> this is the script
> 
> /opt/scripts/getstats-gclog
> #/bin/bash -x
> #
> # get some statistics
> #
> # GC time stamp
> # 2018-06-27T12:52:57.200+
> #
> FIVEMIN=`date --date '5 minutes ago' +'%Y-%m-%dT%H:%M'`
> FOURMIN=`date --date '4 minutes ago' +'%Y-%m-%dT%H:%M'`
> THREEMIN=`date --date '3 minutes ago' +'%Y-%m-%dT%H:%M'`
> TWOMIN=`date --date '2 minutes ago' +'%Y-%m-%dT%H:%M'`
> ONEMIN=`date --date '1 minute ago' +'%Y-%m-%dT%H:%M'`
> YEAR=`date --date '1 minute ago' +'%Y'`
> MONTH=`date --date '1 minute ago' +'%m'`
> DAY=`date --date '1 minute ago' +'%d'`
> HOUR=`date --date '1 minute ago' +'%H'`
> MINUTE=`date --date '1 minute ago' +'%M'`
> SECOND=`date --date '1 minute ago' +'%S'`
> #
> #
> STATSDIR=/opt/stats/
> WORKDIR=/$STATSDIR/working_gc
> #
> #
> LOGDIR=/u01/app/solr/var/logs/
> LOGNAME=gclog.log
> Prep() {
> mkdir -p $STATSDIR
> chmod 755 $STATSDIR
> mkdir -p $WORKDIR
> chmod 755 $WORKDIR
> }
> GetStats() {
> cd $WORKDIR
> grep $ONEMIN $LOGDIR/$LOGNAME.*|grep stopped|awk '{print $11}'>ALLGC
> COUNT=`cat ALLGC |wc -l`
> # number under .00X XDecimalplaces U3D for example
> U3D=`grep "^0\.00[1-9]" ALLGC|wc -l`
> if [ -z $U3D ]
> then
>  U3D=0
> fi
> U2D=`grep "^0\.0[1-9]" ALLGC|wc -l`
> if [ -z $U2D ]
> then
>  U2D=0
> fi
> U1D=`grep "^0\.[1-9]" ALLGC|wc -l`
> if [ -z $U1D ]
> then
>  U1D=0
> fi
> O1S=`cat ALLGC | grep -v "^0\."|wc -l`
> if [ -z $O1S ]
> then
>  O1S=0
> fi
> O10S=`grep "[0-9]\+[0-9]\.[0-9]*" ALLGC|wc -l`
> if [ -z $O10S ]
> then
>  O10S=0
> fi
> cat ALLGC | grep -v "^0\.">OVER1SECDATA
> 
> TOTAL=0
> COUNT=0
> while read DAT
> do
> TOTAL=`echo "$TOTAL + $DAT"|bc`
> COUNT=`expr $COUNT + 1`
> done <$WORKDIR/ALLGC
> #AO1S=$(printf "%.2f\n"  `echo "scale=10;$COUNT/60" | bc`)
> #AVGQT=$(printf "%.0f\n"  `echo "scale=10;$TOTAL/$COUNT"|bc`)
> TOTSTOP=$TOTAL
> AVGSTOPT=`echo "scale=7;$TOTAL/$COUNT"|bc`
> if [ -z $AVGSTOPT ]
> then
>  AVGSTOPT=0
> fi
> # get top gc times
> #
> 
> #echo 0.000>ALLGCU1S
> #echo 0.000>ALLGCO1S
> 
> grep '^0.' $WORKDIR/ALLGC >ALLGCU1S
> grep -v '^0.' $WORKDIR/ALLGC >ALLGCO1S
> TOPGCTIMEU1S=`cat $WORKDIR/ALLGCU1S |sort |tail -1`
> if [ -z $TOPGCTIMEU1S ]
> then
>  TOPGCTIMEU1S=0
> fi
> TOPGCTIMEO1S=`cat $WORKDIR/ALLGCO1S |sort |tail -1`
> if [ -z $TOPGCTIMEO1S ]
> then
>  TOPGCTIMEO1S=0
> fi
> 
> }
> PrintStats() {
> #
> ## stats
> #COUNT= total number of garbage collection this minute
> ## U3d = Total number of GC that are under 0.00Xseconds
> # U2D total number of GC that are under 0.0X seconds
> # U1D total number of GC that are under 0.X seconds
> # O1S total number of GC that are over 1 second but under 10
> # O10S total number of GC that are over 10 seconds
> # TOTSTOPT the total time stopped for all GCs
> # AVGSTOPT the average time of all the GCs
> # TOPGCTIMEU1S the highest GC time Under 1 sec this minute
> # TOPGCTIMEO1S the highest GC time Over 1 sec this minute
> echo $COUNT $U3D $U2D $U1D $O1S $O10S $TOTSTOP 

Unsubscribing

[Monique Monteiro]

How can I unsubscribe from this list?

Thanks in advance,

-- 
Monique Monteiro
Twitter: http://twitter.com/monilouise

Re: Garbage Collection Metrics

[Jeff Courtade]

So,

I had a problem when at a customer site. They use zabbix for data
collection and alerting.

The solr server had been setup to use only jmx metrics.

the jvm was unstable and would lock up for a period of time and the metrics
and counters would be all screwed up. Because it was using jmx to alert it
was screwing up as  the jvm needed to be working to be used.

So I turned on gclogging and wrote a script to collect data points about
for instance how long the jvm was stopped in the last minute.

I eventually got the gc tuned and behaving well but it was difficult.


turn on gcloging

i use these options

-Xloggc:../var/logs/gclog.log \
-XX:+PrintHeapAtGC \
-XX:+PrintGCDetails \
-XX:+PrintGCDateStamps \
-XX:+PrintGCTimeStamps \
-XX:+PrintTenuringDistribution \
-XX:+PrintGCApplicationStoppedTime \
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 \
-XX:GCLogFileSize=100M

in the solr runtme users crontab on each system 

* * * * * nohup /opt/scripts/getstats-gclog &

this is the script

/opt/scripts/getstats-gclog
#/bin/bash -x
#
# get some statistics
#
# GC time stamp
# 2018-06-27T12:52:57.200+
#
FIVEMIN=`date --date '5 minutes ago' +'%Y-%m-%dT%H:%M'`
FOURMIN=`date --date '4 minutes ago' +'%Y-%m-%dT%H:%M'`
THREEMIN=`date --date '3 minutes ago' +'%Y-%m-%dT%H:%M'`
TWOMIN=`date --date '2 minutes ago' +'%Y-%m-%dT%H:%M'`
ONEMIN=`date --date '1 minute ago' +'%Y-%m-%dT%H:%M'`
YEAR=`date --date '1 minute ago' +'%Y'`
MONTH=`date --date '1 minute ago' +'%m'`
DAY=`date --date '1 minute ago' +'%d'`
HOUR=`date --date '1 minute ago' +'%H'`
MINUTE=`date --date '1 minute ago' +'%M'`
SECOND=`date --date '1 minute ago' +'%S'`
#
#
STATSDIR=/opt/stats/
WORKDIR=/$STATSDIR/working_gc
#
#
LOGDIR=/u01/app/solr/var/logs/
LOGNAME=gclog.log
Prep() {
mkdir -p $STATSDIR
chmod 755 $STATSDIR
mkdir -p $WORKDIR
chmod 755 $WORKDIR
}
GetStats() {
cd $WORKDIR
grep $ONEMIN $LOGDIR/$LOGNAME.*|grep stopped|awk '{print $11}'>ALLGC
COUNT=`cat ALLGC |wc -l`
# number under .00X XDecimalplaces U3D for example
U3D=`grep "^0\.00[1-9]" ALLGC|wc -l`
if [ -z $U3D ]
then
  U3D=0
fi
U2D=`grep "^0\.0[1-9]" ALLGC|wc -l`
if [ -z $U2D ]
then
  U2D=0
fi
U1D=`grep "^0\.[1-9]" ALLGC|wc -l`
if [ -z $U1D ]
then
  U1D=0
fi
O1S=`cat ALLGC | grep -v "^0\."|wc -l`
if [ -z $O1S ]
then
  O1S=0
fi
O10S=`grep "[0-9]\+[0-9]\.[0-9]*" ALLGC|wc -l`
if [ -z $O10S ]
then
  O10S=0
fi
cat ALLGC | grep -v "^0\.">OVER1SECDATA

TOTAL=0
COUNT=0
while read DAT
do
TOTAL=`echo "$TOTAL + $DAT"|bc`
COUNT=`expr $COUNT + 1`
done <$WORKDIR/ALLGC
#AO1S=$(printf "%.2f\n"  `echo "scale=10;$COUNT/60" | bc`)
#AVGQT=$(printf "%.0f\n"  `echo "scale=10;$TOTAL/$COUNT"|bc`)
TOTSTOP=$TOTAL
AVGSTOPT=`echo "scale=7;$TOTAL/$COUNT"|bc`
if [ -z $AVGSTOPT ]
then
  AVGSTOPT=0
fi
# get top gc times
#

#echo 0.000>ALLGCU1S
#echo 0.000>ALLGCO1S

grep '^0.' $WORKDIR/ALLGC >ALLGCU1S
grep -v '^0.' $WORKDIR/ALLGC >ALLGCO1S
TOPGCTIMEU1S=`cat $WORKDIR/ALLGCU1S |sort |tail -1`
if [ -z $TOPGCTIMEU1S ]
then
  TOPGCTIMEU1S=0
fi
TOPGCTIMEO1S=`cat $WORKDIR/ALLGCO1S |sort |tail -1`
if [ -z $TOPGCTIMEO1S ]
then
  TOPGCTIMEO1S=0
fi

}
PrintStats() {
#
## stats
#COUNT= total number of garbage collection this minute
## U3d = Total number of GC that are under 0.00Xseconds
# U2D total number of GC that are under 0.0X seconds
# U1D total number of GC that are under 0.X seconds
# O1S total number of GC that are over 1 second but under 10
# O10S total number of GC that are over 10 seconds
# TOTSTOPT the total time stopped for all GCs
# AVGSTOPT the average time of all the GCs
# TOPGCTIMEU1S the highest GC time Under 1 sec this minute
# TOPGCTIMEO1S the highest GC time Over 1 sec this minute
echo $COUNT $U3D $U2D $U1D $O1S $O10S $TOTSTOP $AVGSTOPT $TOPGCTIMEU1S
$TOPGCTIMEO1S >$STATSDIR/GCSTATS
#echo $COUNT $U3D $U2D $U1D $O1S $O10S $TOTSTOP $AVGSTOPT $TOPGCTIMEU1S
$TOPGCTIMEO1S
}

Prep
GetStats
PrintStats

then in the zabbix-agentd.conf
 add these parameters

# total number of GC in the last minute
UserParameter=gc-num,cat /opt/stats/GCSTATS |awk '{print $1}'
# total number of GC 0.00[1-9] - 0.000 second in the last minute
UserParameter=gc-n3d,cat /opt/stats/GCSTATS |awk '{print $2}'
# total number of GC 0.0[1-9] - 0.009 second in the last minute
UserParameter=gc-n2d,cat /opt/stats/GCSTATS |awk '{print $3}'
# total number of GC  0.[1-9] - 0.09 second in the last minute
UserParameter=gc-n1d,cat /opt/stats/GCSTATS |awk '{print $4}'
# total number of GC [1-9].X seconds in the last minute
UserParameter=gc-no1s,cat /opt/stats/GCSTATS |awk '{print $5}'
# total number of GC OVER 10  seconds in the last minute
UserParameter=gc-no10s,cat /opt/stats/GCSTATS |awk '{print $6}'
# these are all 0.000 time
# Total time the JVM was stopped for GC in the last minute
UserParameter=gc-tst,cat /opt/stats/GCSTATS |awk '{print $7}'
# Average time the JVM was stopped for ALL GC in the last minute
UserParameter=gc-ast,cat /opt/stats/GCSTATS |awk '{print $8}'
# Highest Time the JVM was stopped for GC under 1 second

Re: Garbage Collection Metrics

[Jeff Courtade]

The only way I found to track GC times was by truning on GC logging and the
writing cronjob data collection script and graphing it in zabbix

On Mon, Mar 18, 2019 at 12:34 PM Erick Erickson 
wrote:

> Attachments are pretty aggressively stripped by the apache mail server, so
> it didn’t come through.
>
> That said, I’m not sure how much use just the last GC time is. What do you
> want it for? This
> sounds a bit like an XY problem.
>
> Best,
> Erick
>
> > On Mar 17, 2019, at 2:43 PM, Karthik K G  wrote:
> >
> > Hi Team,
> >
> > I was looking for Old GC duration time metrics, but all I could find was
> the API for this "/solr/admin/metrics?wt=json=jvm=gc.G1-
> Old-Generation", but I am not sure if this is for
> 'gc_g1_gen_o_lastgc_duration'. I tried to hookup the IP to the jconsole and
> was looking for the metrics, but all I could see was the collection time
> but not last GC duration as attached in the screenshot. Can you please help
> here with finding the correct metrics. I strongly believe we are not
> capturing this information. Please correct me if I am wrong.
> >
> > Thanks & Regards,
> > Karthik
>
>

Re: Garbage Collection Metrics

[Erick Erickson]

Attachments are pretty aggressively stripped by the apache mail server, so it 
didn’t come through.

That said, I’m not sure how much use just the last GC time is. What do you want 
it for? This
sounds a bit like an XY problem.

Best,
Erick

> On Mar 17, 2019, at 2:43 PM, Karthik K G  wrote:
> 
> Hi Team,
> 
> I was looking for Old GC duration time metrics, but all I could find was the 
> API for this "/solr/admin/metrics?wt=json=jvm=gc.G1- 
> Old-Generation", but I am not sure if this is for 
> 'gc_g1_gen_o_lastgc_duration'. I tried to hookup the IP to the jconsole and 
> was looking for the metrics, but all I could see was the collection time but 
> not last GC duration as attached in the screenshot. Can you please help here 
> with finding the correct metrics. I strongly believe we are not capturing 
> this information. Please correct me if I am wrong.
> 
> Thanks & Regards,
> Karthik

Re: Spellchecker -File based vs Index based

[Erick Erickson]

Two guesses:
1> you have something different in your spellcheck config .vs. index config.
2> you don’t have the word in your file for the file-based spellcheck, thus
 Solr has no way of knowing the word is correctly spelled.

> On Mar 17, 2019, at 11:56 PM, Ashish Bisht  wrote:
> 
> Hi,
> 
> I am seeing difference in file based spellcheck and index based spellcheck
> implementations.
> 
> Using index based
> http://:8983/solr/SCSpell/spell?q=*intnet of
> things*=edismax=spellcontent=json=0=true=*default*=AND
> 
> 
>  "suggestions":[
>  "intnet",{
>"numFound":10,
>"startOffset":0,
>"endOffset":6,
>"origFreq
> 
> 
> Suggestion get build up only for wrong word.
> 
> 
> But while suing file based,they get build up for right words too which
> messes collations
> 
> http://:8983/solr/SCSpell/spell?q=intnet%20of%20things=edismax=spellcontent=json=0&=true=*file*=AND
> 
> "suggestion":["*internet*",
>  "contnet",
>  "intel",
>  "intent",
>  "intert",
>  "intelect",
>  "intended",
>  "intented",
>  "interest",
>  "botnets"]},
>  "*of*",{
>"numFound":8,
>"startOffset":7,
>"endOffset":9,
>"suggestion":["ofc",
>  "off",
>  "ohf",
> .
>  "soft"]},
> "*things*",{
>"numFound":10,
>"startOffset":10,
>"endOffset":16,
>"suggestion":["thing",
>  "brings",
>  "think",
>  "thinkers",
>  .
> 
> 
> 
> Is there any property in file based which I use to fix this
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Java Heap Error

[Erick Erickson]

1> set Xmx and Xms equal. That’s not the root of our problem, but it doesn’t 
help either.

2> Crack the documents outside of Solr. It’s quite easy to write a SolJ program 
to do this, 
here’s an example with the reasons why it’s advisable to do this outside of 
Solr.:

https://lucidworks.com/2012/02/14/indexing-with-solrj/

or use a service, I think the Tika project has one.

Best,
Erick



> On Mar 18, 2019, at 8:46 AM, wclarke  wrote:
> 
> Hi,
> 
> I am doing a recursive search on a local hard drive through DIH with Tika. 
> It starts slow and then it dies at Thread-14 every time with a Java Heap
> error.  My Xmx is at 10g, my Xms is at 512m (which might be the problem). 
> It takes hours to fail after it has fetched over 300,000 items and processed
> over 162,000.  Any ideas?
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr index slow response

[Walter Underwood]

Solr is not designed to have consistent response times for updates. You are 
expecting Solr to do something that it does not do.

About Xms and Xmx, the JVM will continue to allocate memory until it hits the 
max. After it hits the max, it will start to collect garbage. A smaller Xms 
just wastes time doing allocations after the JVM is running. Avoid that by 
making Xms and Xms the same.

We run all of our JVMs with 8 GB of heap and the G1 collector. You probably do 
not need more than 8 GB unless you are doing high-cardinality facets or some 
other memory-hungry querying.

The first step would be to use a good configuration. We start our Java 8 JVMs 
with these parameters:

SOLR_HEAP=8g
# Use G1 GC  -- wunder 2017-01-23
# Settings from https://wiki.apache.org/solr/ShawnHeisey
GC_TUNE=" \
-XX:+UseG1GC \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=8m \
-XX:MaxGCPauseMillis=200 \
-XX:+UseLargePages \
-XX:+AggressiveOpts \
“

Use SSD for disks, with total space about 3X as big as the expected index size.

Have RAM not used by Solr or the OS that is equal to the expected index size.

After that, let’s figure out what the real requirement is. If you must have 
consistent response times for update requests, you’ll need to do that outside 
of Solr. But if you need high data import rates, we can probably help.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Mar 18, 2019, at 8:31 AM, Aaron Yingcai Sun  wrote:
> 
> Hello, Chris
> 
> 
> Thanks for the tips.  So I tried to set it as you suggested, not see too much 
> improvement.  Since I don't need it visible immediately, softCommit is 
> disabled totally.
> 
> The slow response is happening every few seconds,  if it happens hourly I 
> would suspect the hourly auto-commit.  But it happen much more frequently.  I 
> don't see any CPU/RAM/NETWORK IO/DISK IO bottleneck on OS level.  It just 
> looks like solr server is blocking internally itself.
> 
> 
> <   ${solr.autoCommit.maxTime:360}
> ---
>>  ${solr.autoCommit.maxTime:15000}
> 16c16
> <   true
> ---
>>  false
> 
> 
> 
> 190318-162811.610-189982 DBG1:doc_count: 10 , doc_size: 539  KB, Res code: 
> 200, QTime: 1405 ms, Request time: 1407 ms.
> 190318-162811.636-189968 DBG1:doc_count: 10 , doc_size: 465  KB, Res code: 
> 200, QTime: 1357 ms, Request time: 1360 ms.
> 190318-162811.732-189968 DBG1:doc_count: 10 , doc_size: 473  KB, Res code: 
> 200, QTime: 90 ms, Request time: 92 ms.
> 190318-162811.995-189981 DBG1:doc_count: 10 , doc_size: 610  KB, Res code: 
> 200, QTime: 5306 ms, Request time: 5308 ms.
> 190318-162814.873-190003 DBG1:doc_count: 10 , doc_size: 508  KB, Res code: 
> 200, QTime: 4775 ms, Request time: 4777 ms.
> 190318-162814.889-189972 DBG1:doc_count: 10 , doc_size: 563  KB, Res code: 
> 200, QTime: 20222 ms, Request time: 20224 ms.
> 190318-162814.975-191817 DBG1:doc_count: 10 , doc_size: 539  KB, Res code: 
> 200, QTime: 27732 ms, Request time: 27735 ms.
> 190318-162814.975-189958 DBG1:doc_count: 10 , doc_size: 616  KB, Res code: 
> 200, QTime: 28106 ms, Request time: 28109 ms.
> 190318-162814.975-190004 DBG1:doc_count: 10 , doc_size: 473  KB, Res code: 
> 200, QTime: 16703 ms, Request time: 16706 ms.
> 190318-162814.982-189963 DBG1:doc_count: 10 , doc_size: 540  KB, Res code: 
> 200, QTime: 28216 ms, Request time: 28218 ms.
> 190318-162814.988-190007 DBG1:doc_count: 10 , doc_size: 673  KB, Res code: 
> 200, QTime: 28133 ms, Request time: 28136 ms.
> 190318-162814.993-189962 DBG1:doc_count: 10 , doc_size: 631  KB, Res code: 
> 200, QTime: 27909 ms, Request time: 27912 ms.
> 190318-162814.996-191818 DBG1:doc_count: 10 , doc_size: 529  KB, Res code: 
> 200, QTime: 28172 ms, Request time: 28174 ms.
> 190318-162815.056-189986 DBG1:doc_count: 10 , doc_size: 602  KB, Res code: 
> 200, QTime: 11375 ms, Request time: 11378 ms.
> 190318-162815.100-189988 DBG1:doc_count: 10 , doc_size: 530  KB, Res code: 
> 200, QTime: 8663 ms, Request time: 8666 ms.
> 190318-162815.275-189980 DBG1:doc_count: 10 , doc_size: 553  KB, Res code: 
> 200, QTime: 27526 ms, Request time: 27528 ms.
> 190318-162815.283-189997 DBG1:doc_count: 10 , doc_size: 600  KB, Res code: 
> 200, QTime: 27529 ms, Request time: 27535 ms.
> 190318-162815.318-189961 DBG1:doc_count: 10 , doc_size: 661  KB, Res code: 
> 200, QTime: 16617 ms, Request time: 16621 ms.
> 190318-162815.484-189952 DBG1:doc_count: 10 , doc_size: 549  KB, Res code: 
> 200, QTime: 11707 ms, Request time: 11711 ms.
> 190318-162815.485-189993 DBG1:doc_count: 10 , doc_size: 618  KB, Res code: 
> 200, QTime: 28315 ms, Request time: 28321 ms.
> 190318-162816.216-189972 DBG1:doc_count: 10 , doc_size: 493  KB, Res code: 
> 200, QTime: 1320 ms, Request time: 1322 ms.
> 190318-162816.354-189972 DBG1:doc_count: 10 , doc_size: 631  KB, Res code: 
> 200, QTime: 130 ms, Request time: 132 ms.
> 190318-162816.471-189972 DBG1:doc_count: 10 , doc_size: 563  KB, Res code: 
> 200, QTime: 111 ms, Request time: 113 ms.
> 

Java Heap Error

[wclarke]

Hi,

I am doing a recursive search on a local hard drive through DIH with Tika. 
It starts slow and then it dies at Thread-14 every time with a Java Heap
error.  My Xmx is at 10g, my Xms is at 512m (which might be the problem). 
It takes hours to fail after it has fetched over 300,000 items and processed
over 162,000.  Any ideas?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr index slow response

[Aaron Yingcai Sun]

Hello, Chris


Thanks for the tips.  So I tried to set it as you suggested, not see too much 
improvement.  Since I don't need it visible immediately, softCommit is disabled 
totally.

The slow response is happening every few seconds,  if it happens hourly I would 
suspect the hourly auto-commit.  But it happen much more frequently.  I don't 
see any CPU/RAM/NETWORK IO/DISK IO bottleneck on OS level.  It just looks like 
solr server is blocking internally itself.


<   ${solr.autoCommit.maxTime:360}
---
>   ${solr.autoCommit.maxTime:15000}
16c16
<   true
---
>   false



190318-162811.610-189982 DBG1:doc_count: 10 , doc_size: 539  KB, Res code: 200, 
QTime: 1405 ms, Request time: 1407 ms.
190318-162811.636-189968 DBG1:doc_count: 10 , doc_size: 465  KB, Res code: 200, 
QTime: 1357 ms, Request time: 1360 ms.
190318-162811.732-189968 DBG1:doc_count: 10 , doc_size: 473  KB, Res code: 200, 
QTime: 90 ms, Request time: 92 ms.
190318-162811.995-189981 DBG1:doc_count: 10 , doc_size: 610  KB, Res code: 200, 
QTime: 5306 ms, Request time: 5308 ms.
190318-162814.873-190003 DBG1:doc_count: 10 , doc_size: 508  KB, Res code: 200, 
QTime: 4775 ms, Request time: 4777 ms.
190318-162814.889-189972 DBG1:doc_count: 10 , doc_size: 563  KB, Res code: 200, 
QTime: 20222 ms, Request time: 20224 ms.
190318-162814.975-191817 DBG1:doc_count: 10 , doc_size: 539  KB, Res code: 200, 
QTime: 27732 ms, Request time: 27735 ms.
190318-162814.975-189958 DBG1:doc_count: 10 , doc_size: 616  KB, Res code: 200, 
QTime: 28106 ms, Request time: 28109 ms.
190318-162814.975-190004 DBG1:doc_count: 10 , doc_size: 473  KB, Res code: 200, 
QTime: 16703 ms, Request time: 16706 ms.
190318-162814.982-189963 DBG1:doc_count: 10 , doc_size: 540  KB, Res code: 200, 
QTime: 28216 ms, Request time: 28218 ms.
190318-162814.988-190007 DBG1:doc_count: 10 , doc_size: 673  KB, Res code: 200, 
QTime: 28133 ms, Request time: 28136 ms.
190318-162814.993-189962 DBG1:doc_count: 10 , doc_size: 631  KB, Res code: 200, 
QTime: 27909 ms, Request time: 27912 ms.
190318-162814.996-191818 DBG1:doc_count: 10 , doc_size: 529  KB, Res code: 200, 
QTime: 28172 ms, Request time: 28174 ms.
190318-162815.056-189986 DBG1:doc_count: 10 , doc_size: 602  KB, Res code: 200, 
QTime: 11375 ms, Request time: 11378 ms.
190318-162815.100-189988 DBG1:doc_count: 10 , doc_size: 530  KB, Res code: 200, 
QTime: 8663 ms, Request time: 8666 ms.
190318-162815.275-189980 DBG1:doc_count: 10 , doc_size: 553  KB, Res code: 200, 
QTime: 27526 ms, Request time: 27528 ms.
190318-162815.283-189997 DBG1:doc_count: 10 , doc_size: 600  KB, Res code: 200, 
QTime: 27529 ms, Request time: 27535 ms.
190318-162815.318-189961 DBG1:doc_count: 10 , doc_size: 661  KB, Res code: 200, 
QTime: 16617 ms, Request time: 16621 ms.
190318-162815.484-189952 DBG1:doc_count: 10 , doc_size: 549  KB, Res code: 200, 
QTime: 11707 ms, Request time: 11711 ms.
190318-162815.485-189993 DBG1:doc_count: 10 , doc_size: 618  KB, Res code: 200, 
QTime: 28315 ms, Request time: 28321 ms.
190318-162816.216-189972 DBG1:doc_count: 10 , doc_size: 493  KB, Res code: 200, 
QTime: 1320 ms, Request time: 1322 ms.
190318-162816.354-189972 DBG1:doc_count: 10 , doc_size: 631  KB, Res code: 200, 
QTime: 130 ms, Request time: 132 ms.
190318-162816.471-189972 DBG1:doc_count: 10 , doc_size: 563  KB, Res code: 200, 
QTime: 111 ms, Request time: 113 ms.
190318-162816.586-189972 DBG1:doc_count: 10 , doc_size: 554  KB, Res code: 200, 
QTime: 110 ms, Request time: 111 ms.
190318-162816.716-189972 DBG1:doc_count: 10 , doc_size: 590  KB, Res code: 200, 
QTime: 124 ms, Request time: 125 ms.
190318-162820.494-189972 DBG1:doc_count: 10 , doc_size: 583  KB, Res code: 200, 
QTime: 3772 ms, Request time: 3774 ms.
190318-162820.574-189953 DBG1:doc_count: 10 , doc_size: 550  KB, Res code: 200, 
QTime: 32802 ms, Request time: 32804 ms.
190318-162820.609-189991 DBG1:doc_count: 10 , doc_size: 586  KB, Res code: 200, 
QTime: 33787 ms, Request time: 33790 ms.
190318-162820.621-189976 DBG1:doc_count: 10 , doc_size: 572  KB, Res code: 200, 
QTime: 33397 ms, Request time: 33400 ms.
190318-162820.622-189985 DBG1:doc_count: 10 , doc_size: 498  KB, Res code: 200, 
QTime: 32917 ms, Request time: 32919 ms.
190318-162820.987-190005 DBG1:doc_count: 10 , doc_size: 629  KB, Res code: 200, 
QTime: 22207 ms, Request time: 22209 ms.
190318-162821.028-189979 DBG1:doc_count: 10 , doc_size: 584  KB, Res code: 200, 
QTime: 22800 ms, Request time: 22802 ms.
190318-162821.056-189948 DBG1:doc_count: 10 , doc_size: 670  KB, Res code: 200, 
QTime: 34193 ms, Request time: 34195 ms.
190318-162821.062-189983 DBG1:doc_count: 10 , doc_size: 675  KB, Res code: 200, 
QTime: 22250 ms, Request time: 22252 ms.




BRs

//Aaron


From: Chris Ulicny 
Sent: Monday, March 18, 2019 2:54:25 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr index slow response

One other thing to look at besides the heap is your commit settings. We've
experienced something similar, 

obfuscated password error

[Satya Marivada]

Hi All,

Using solr-6.3.0, to obfuscate the password, have used jetty util to
generate obfuscated password


java -cp jetty-util-9.3.8.v20160314.jar
org.eclipse.jetty.util.security.Password mypassword


The output has been used in solr.in.sh as below


SOLR_SSL_KEY_STORE=/sanfs/mnt/vol01/solr/solr-6.3.0/server/etc/solr-ssl.keystore.jks

SOLR_SSL_KEY_STORE_PASSWORD="OBF:1bcd1l161lts1ltu1uum1uvk1lq41lq61k221b9t"

SOLR_SSL_TRUST_STORE=/sanfs/mnt/vol01/solr/solr-6.3.0/server/etc/solr-ssl.keystore.jks

SOLR_SSL_TRUST_STORE_PASSWORD="OBF:1bcd1l161lts1ltu1uum1uvk1lq41lq61k221b9t"

Solr does not start fine with below exception, any suggestions? If I use
the plain text password, it works fine. One more thing is that the same
setup with obfuscated password works in other environments except one which
got this exception. Recently system level patches are applied, just saying
though dont think that could have impact,

Caused by: java.net.SocketException:
java.security.NoSuchAlgorithmException: Error constructing implementation
(algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextIm
pl$DefaultSSLContext)
at
javax.net.ssl.DefaultSSLSocketFactory.throwException(SSLSocketFactory.java:248)
at
javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:255)
at
org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactory.java:513)
at
org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactory.java:383)
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:165)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:498)
... 11 more
Caused by: java.security.NoSuchAlgorithmException: Error constructing
implementation (algorithm: Default, provider: SunJSSE, class:
sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at java.security.Provider$Service.newInstance(Provider.java:1617)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96)


at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
at org.eclipse.jetty.start.Main.start(Main.java:457)
at org.eclipse.jetty.start.Main.main(Main.java:75)
Caused by: java.io.IOException: Keystore was tampered with, or password was
incorrect
at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:785)
at
sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at
sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
at
sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at
sun.security.ssl.TrustManagerFactoryImpl.getCacertsKeyStore(TrustManagerFactoryImpl.java:226)
at
sun.security.ssl.SSLContextImpl$DefaultManagersHolder.getTrustManagers(SSLContextImpl.java:877)
at
sun.security.ssl.SSLContextImpl$DefaultManagersHolder.(SSLContextImpl.java:854)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.(SSLContextImpl.java:1019)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.security.Provider$Service.newInstance(Provider.java:1595)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)

Thanks,
Satya

Fetching All Terms and Corresponding Documents

[Furkan KAMACI]

Hi,

I need to iterate on all terms at Solr index, and then find related
documents for some terms that match my criteria.

I know that I can send a query to *LukeRequestHandler*:

*/admin/luke?fl=content={distinct term count}=json*

and then check my criteria. If matches, I can send a *fq* to retrieve
related docs.

However, is there any other efficient way (via rest or Solrj) for my case?

Kind Regards,
Furkan KAMACI

Behavior of Function Query

[Ashish Bisht]

Please see the below requests and response

http://Sol:8983/solr/SCSpell/select?q="*internet of
things*"=edismax=spellcontent=json=1=score,internet_of_things:query({!edismax
v='"*internet of things*"'}),instant_of_things:query({!edismax v='"instant
of things"'})


Response contains score from function query

 "fl":"score,internet_of_things:query({!edismax v='\"internet of
things\"'}),instant_of_things:query({!edismax v='\"instant of things\"'})",
  "rows":"1",
  "wt":"json"}},
  "response":{"numFound":851,"start":0,"maxScore":7.6176834,"docs":[
  {
"score":7.6176834,
   * "internet_of_things":7.6176834*}]
  }}


But if in the same request q is changed,it doesn't give score

http://Sol-1:8983/solr/SCSpell/select?q="*wall
street*"=edismax=spellcontent=json=1=score,internet_of_things:query({!edismax
v='"*internet of things*"'}),instant_of_things:query({!edismax v='"instant
of things"'})

   "q":"\"wall street\"",
  "defType":"edismax",
  "qf":"spellcontent",
  "fl":"score,internet_of_things:query({!edismax v='\"internet of
things\"'}),instant_of_things:query({!edismax v='\"instant of things\"'})",
  "rows":"1",
  "wt":"json"}},
  "response":{"numFound":46,"start":0,"maxScore":15.670144,"docs":[
  {
"score":15.670144}]
  }}


Why score of function query is getting applied when q is a different.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr index slow response

[Chris Ulicny]

One other thing to look at besides the heap is your commit settings. We've
experienced something similar, and changing commit settings alleviated the
issue.

Are you opening a search on every hardcommit? If so, you might want to
reconsider and use the softcommit for the hourly creation of a new searcher.

The hardcommit interval should be much lower. Probably something on the
order of seconds (15000) instead of hours (currently 360). When the
hard commit fires, numerous merges might be firing off in the background
due to the volume of documents you are indexing, which might explain the
periodic bad response times shown in your logs.

It would depend on your specific scenario, but here's our setup. During
long periods of constant indexing of documents to a staging collection (~2
billion documents), we have following commit settings

softcommit: 360ms (for periodic validation of data, since it's not in
production)
hardcommit: openSearcher -> false, 15000ms (no document limit)

This makes the documents available for searching every hour, but doesn't
result in the large bursts of IO due to the infrequent hard commits.

For more info, Erick Erickson has a great write up:
https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

Best,
Chris

On Mon, Mar 18, 2019 at 9:36 AM Aaron Yingcai Sun  wrote:

> Hi, Emir,
>
> My system used to run with max 32GB, the response time is bad as well.
> swap is set to 4GB, there 3.2 free, I doubt swap would affect it since
> there is such huge free memory.
>
> I could try to with set Xms and Xmx to the same value, but I doubt how
> much would that change the response time.
>
>
> BRs
>
> //Aaron
>
> 
> From: Emir Arnautović 
> Sent: Monday, March 18, 2019 2:19:19 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr index slow response
>
> Hi Aaron,
> Without looking too much into numbers, my bet would be that it is large
> heap that is causing issues. I would decrease is significantly (<30GB) and
> see if it is enough for your max load. Also, disable swap or reduce
> swappiness to min.
>
> In any case, you should install some monitoring tool that would help you
> do better analysis when you run into problems. One such tool is our
> monitoring solution: https://sematext.com/spm
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
> > On 18 Mar 2019, at 13:14, Aaron Yingcai Sun  wrote:
> >
> > Hello, Emir,
> >
> > Thanks for the reply, this is the solr version and heap info, standalone
> single solr server. I don't have monitor tool connected. only look at
> 'top', has not seen cpu spike so far, when the slow response happens, cpu
> usage is not high at all, around 30%.
> >
> >
> > # curl 'http://.../solr/admin/info/system?wt=json=true'
> > {
> >  "responseHeader":{
> >"status":0,
> >"QTime":27},
> >  "mode":"std",
> >  "solr_home":"/ardome/solr",
> >  "lucene":{
> >"solr-spec-version":"6.5.1",
> >"solr-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 -
> jimczi - 2017-04-21 12:23:42",
> >"lucene-spec-version":"6.5.1",
> >"lucene-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75
> - jimczi - 2017-04-21 12:17:15"},
> >  "jvm":{
> >"version":"1.8.0_144 25.144-b01",
> >"name":"Oracle Corporation Java HotSpot(TM) 64-Bit Server VM",
> >"spec":{
> >  "vendor":"Oracle Corporation",
> >  "name":"Java Platform API Specification",
> >  "version":"1.8"},
> >"jre":{
> >  "vendor":"Oracle Corporation",
> >  "version":"1.8.0_144"},
> >"vm":{
> >  "vendor":"Oracle Corporation",
> >  "name":"Java HotSpot(TM) 64-Bit Server VM",
> >  "version":"25.144-b01"},
> >"processors":32,
> >"memory":{
> >  "free":"69.1 GB",
> >  "total":"180.2 GB",
> >  "max":"266.7 GB",
> >  "used":"111 GB (%41.6)",
> >  "raw":{
> >"free":74238728336,
> >"total":193470136320,
> >"max":286331502592,
> >"used":119231407984,
> >"used%":41.64103736566334}},
> >"jmx":{
> >
> "bootclasspath":"/usr/java/jdk1.8.0_144/jre/lib/resources.jar:/usr/java/jdk1.8.0_144/jre/lib/rt.jar:/usr/java/jdk1.8.0_144/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_144/jre/lib/jsse.jar:/usr/java/jdk1.8.0_144/jre/lib/jce.jar:/usr/java/jdk1.8.0_144/jre/lib/charsets.jar:/usr/java/jdk1.8.0_144/jre/lib/jfr.jar:/usr/java/jdk1.8.0_144/jre/classes",
> >  "classpath":"...",
> >  "commandLineArgs":["-Xms100G",
> >"-Xmx300G",
> >"-DSTOP.PORT=8079",
> >"-DSTOP.KEY=..",
> >"-Dsolr.solr.home=..",
> >"-Djetty.port=8983"],
> >  "startTime":"2019-03-18T09:35:27.892Z",
> >  "upTimeMS":9258422}},
> >  "system":{
> >"name":"Linux",
> >"arch":"amd64",
> >"availableProcessors":32,
> >"systemLoadAverage":14.72,
> >

Re: Solr index slow response

[Emir Arnautović]

4GB swap on 400GB machine does not make much sense, so disable it. Even 4GB, 
some pages might be swapped, and if those are some Solr pages, it’ll affect 
Solr.

Setting Xms and Xmx to the same value will not solve your issue but you will 
avoid heap resize when your heap reaches Xms.

Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 18 Mar 2019, at 14:36, Aaron Yingcai Sun  wrote:
> 
> Hi, Emir,
> 
> My system used to run with max 32GB, the response time is bad as well.  swap 
> is set to 4GB, there 3.2 free, I doubt swap would affect it since there is 
> such huge free memory.
> 
> I could try to with set Xms and Xmx to the same value, but I doubt how much 
> would that change the response time.
> 
> 
> BRs
> 
> //Aaron
> 
> 
> From: Emir Arnautović 
> Sent: Monday, March 18, 2019 2:19:19 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr index slow response
> 
> Hi Aaron,
> Without looking too much into numbers, my bet would be that it is large heap 
> that is causing issues. I would decrease is significantly (<30GB) and see if 
> it is enough for your max load. Also, disable swap or reduce swappiness to 
> min.
> 
> In any case, you should install some monitoring tool that would help you do 
> better analysis when you run into problems. One such tool is our monitoring 
> solution: https://sematext.com/spm
> 
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> 
> 
> 
>> On 18 Mar 2019, at 13:14, Aaron Yingcai Sun  wrote:
>> 
>> Hello, Emir,
>> 
>> Thanks for the reply, this is the solr version and heap info, standalone 
>> single solr server. I don't have monitor tool connected. only look at 'top', 
>> has not seen cpu spike so far, when the slow response happens, cpu usage is 
>> not high at all, around 30%.
>> 
>> 
>> # curl 'http://.../solr/admin/info/system?wt=json=true'
>> {
>> "responseHeader":{
>>   "status":0,
>>   "QTime":27},
>> "mode":"std",
>> "solr_home":"/ardome/solr",
>> "lucene":{
>>   "solr-spec-version":"6.5.1",
>>   "solr-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - 
>> jimczi - 2017-04-21 12:23:42",
>>   "lucene-spec-version":"6.5.1",
>>   "lucene-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - 
>> jimczi - 2017-04-21 12:17:15"},
>> "jvm":{
>>   "version":"1.8.0_144 25.144-b01",
>>   "name":"Oracle Corporation Java HotSpot(TM) 64-Bit Server VM",
>>   "spec":{
>> "vendor":"Oracle Corporation",
>> "name":"Java Platform API Specification",
>> "version":"1.8"},
>>   "jre":{
>> "vendor":"Oracle Corporation",
>> "version":"1.8.0_144"},
>>   "vm":{
>> "vendor":"Oracle Corporation",
>> "name":"Java HotSpot(TM) 64-Bit Server VM",
>> "version":"25.144-b01"},
>>   "processors":32,
>>   "memory":{
>> "free":"69.1 GB",
>> "total":"180.2 GB",
>> "max":"266.7 GB",
>> "used":"111 GB (%41.6)",
>> "raw":{
>>   "free":74238728336,
>>   "total":193470136320,
>>   "max":286331502592,
>>   "used":119231407984,
>>   "used%":41.64103736566334}},
>>   "jmx":{
>> 
>> "bootclasspath":"/usr/java/jdk1.8.0_144/jre/lib/resources.jar:/usr/java/jdk1.8.0_144/jre/lib/rt.jar:/usr/java/jdk1.8.0_144/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_144/jre/lib/jsse.jar:/usr/java/jdk1.8.0_144/jre/lib/jce.jar:/usr/java/jdk1.8.0_144/jre/lib/charsets.jar:/usr/java/jdk1.8.0_144/jre/lib/jfr.jar:/usr/java/jdk1.8.0_144/jre/classes",
>> "classpath":"...",
>> "commandLineArgs":["-Xms100G",
>>   "-Xmx300G",
>>   "-DSTOP.PORT=8079",
>>   "-DSTOP.KEY=..",
>>   "-Dsolr.solr.home=..",
>>   "-Djetty.port=8983"],
>> "startTime":"2019-03-18T09:35:27.892Z",
>> "upTimeMS":9258422}},
>> "system":{
>>   "name":"Linux",
>>   "arch":"amd64",
>>   "availableProcessors":32,
>>   "systemLoadAverage":14.72,
>>   "version":"3.0.101-311.g08a8a9d-default",
>>   "committedVirtualMemorySize":2547960700928,
>>   "freePhysicalMemorySize":4530696192,
>>   "freeSwapSpaceSize":3486846976,
>>   "processCpuLoad":0.3257436126790475,
>>   "processCpuTime":9386945000,
>>   "systemCpuLoad":0.3279781055816521,
>>   "totalPhysicalMemorySize":406480175104,
>>   "totalSwapSpaceSize":4302303232,
>>   "maxFileDescriptorCount":32768,
>>   "openFileDescriptorCount":385,
>>   "uname":"Linux ... 3.0.101-311.g08a8a9d-default #1 SMP Wed Dec 14 10:15:37 
>> UTC 2016 (08a8a9d) x86_64 x86_64 x86_64 GNU/Linux\n",
>>   "uptime":" 13:09pm  up 5 days 21:23,  7 users,  load average: 14.72, 
>> 12.28, 11.48\n"}}
>> 
>> 
>> 
>> 
>> 
>> From: Emir Arnautović 
>> Sent: Monday, March 18, 2019 12:10:30 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr index slow response
>> 
>> Hi Aaron,
>> Which version of Solr? How did you configure your heap? Is it standalone 
>> Solr 

Re: Solr index slow response

[Emir Arnautović]

Hi Aaron,
You are right - large heap means that there will be no major GC all the time, 
but eventually it will happen and then the larger the heap the longer it will 
take. So with 300GB heap it takes observed 300s. If you used to run on 32GB 
heap and it was slow, it probably means that heap is too small for your load, 
but if you did not run into OOM, then it means it is starving but still can 
handle the load.
In any case, I would not go with super large heap. One option is to split your 
server to more Solr instances. That would let you run more instances with 
smaller heap. I am not sure about your business case and if it is possible to 
do it without switching to SolrCloud, but if you have 100s of clients that are 
completely separate, you can simply split clients to several Solr instances 
that are running on the same server and have some logic that routes client to 
the right instance.
If your average doc size is 5MB, then I would reduce the number of documents 
and that will reduce some load to heap. Unfortunately, the only way to answer 
similar questions is to run some tests.

Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 18 Mar 2019, at 14:30, Aaron Yingcai Sun  wrote:
> 
> I'm a bit confused, why large heap size would make it slower?  Isn't that 
> give it enough room to make it not busy doing GC all the time?
> 
> My http/json request contains 100 documents, the total size of the 100 
> documents is around 5M, there are ~100 client sending those requests 
> continuously.
> 
> Previously the JVM is set to max 32 GB ,  the speed was even worse,  now it's 
> running with min 100GB, max 300GB, it use around 100GB.
> 
> 
> this page suggest to use smaller number of documents per request,   
> https://wiki.apache.org/solr/SolrPerformanceProblems
> 
> SolrPerformanceProblems - Solr 
> Wiki
> wiki.apache.org
> General information. There is a performance bug that makes *everything* slow 
> in versions 6.4.0 and 6.4.1. The problem is fixed in 6.4.2. It is described 
> by SOLR-10130.This is highly version specific, so if you are not running one 
> of the affected versions, don't worry about it.
> 
> So I try to reduce the number, still I could get lots of large response QTime:
> 
> 190318-142652.695-160214 DBG1:doc_count: 10 , doc_size: 609  KB, Res code: 
> 200, QTime: 47918 ms, Request time: 47921 ms.
> 190318-142652.704-160179 DBG1:doc_count: 10 , doc_size: 568  KB, Res code: 
> 200, QTime: 36919 ms, Request time: 36922 ms.
> 190318-142652.780-160197 DBG1:doc_count: 10 , doc_size: 609  KB, Res code: 
> 200, QTime: 36082 ms, Request time: 36084 ms.
> 190318-142652.859-160200 DBG1:doc_count: 10 , doc_size: 569  KB, Res code: 
> 200, QTime: 36880 ms, Request time: 36882 ms.
> 190318-142653.131-160148 DBG1:doc_count: 10 , doc_size: 608  KB, Res code: 
> 200, QTime: 37222 ms, Request time: 37224 ms.
> 190318-142653.154-160211 DBG1:doc_count: 10 , doc_size: 541  KB, Res code: 
> 200, QTime: 37241 ms, Request time: 37243 ms.
> 190318-142653.223-163490 DBG1:doc_count: 10 , doc_size: 589  KB, Res code: 
> 200, QTime: 37174 ms, Request time: 37176 ms.
> 190318-142653.359-160154 DBG1:doc_count: 10 , doc_size: 592  KB, Res code: 
> 200, QTime: 37008 ms, Request time: 37011 ms.
> 190318-142653.497-163491 DBG1:doc_count: 10 , doc_size: 583  KB, Res code: 
> 200, QTime: 24828 ms, Request time: 24830 ms.
> 190318-142653.987-160208 DBG1:doc_count: 10 , doc_size: 669  KB, Res code: 
> 200, QTime: 23900 ms, Request time: 23902 ms.
> 190318-142654.114-160208 DBG1:doc_count: 10 , doc_size: 544  KB, Res code: 
> 200, QTime: 121 ms, Request time: 122 ms.
> 190318-142654.233-160208 DBG1:doc_count: 10 , doc_size: 536  KB, Res code: 
> 200, QTime: 113 ms, Request time: 115 ms.
> 190318-142654.354-160208 DBG1:doc_count: 10 , doc_size: 598  KB, Res code: 
> 200, QTime: 116 ms, Request time: 117 ms.
> 190318-142654.466-160208 DBG1:doc_count: 10 , doc_size: 546  KB, Res code: 
> 200, QTime: 107 ms, Request time: 108 ms.
> 190318-142654.586-160208 DBG1:doc_count: 10 , doc_size: 566  KB, Res code: 
> 200, QTime: 114 ms, Request time: 115 ms.
> 190318-142654.687-160208 DBG1:doc_count: 10 , doc_size: 541  KB, Res code: 
> 200, QTime: 96 ms, Request time: 98 ms.
> 190318-142654.768-160208 DBG1:doc_count: 10 , doc_size: 455  KB, Res code: 
> 200, QTime: 75 ms, Request time: 77 ms.
> 190318-142654.870-160208 DBG1:doc_count: 10 , doc_size: 538  KB, Res code: 
> 200, QTime: 97 ms, Request time: 98 ms.
> 190318-142654.967-160208 DBG1:doc_count: 10 , doc_size: 539  KB, Res code: 
> 200, QTime: 92 ms, Request time: 93 ms.
> 190318-142655.096-160208 DBG1:doc_count: 10 , doc_size: 672  KB, Res code: 
> 200, QTime: 124 ms, Request time: 125 ms.
> 190318-142655.210-160208 DBG1:doc_count: 10 , doc_size: 605  KB, Res code: 
> 200, QTime: 108 ms, Request time: 110 ms.
> 190318-142655.304-160208 

Re: Solr index slow response

[Aaron Yingcai Sun]

Hi, Emir,

My system used to run with max 32GB, the response time is bad as well.  swap is 
set to 4GB, there 3.2 free, I doubt swap would affect it since there is such 
huge free memory.

I could try to with set Xms and Xmx to the same value, but I doubt how much 
would that change the response time.


BRs

//Aaron


From: Emir Arnautović 
Sent: Monday, March 18, 2019 2:19:19 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr index slow response

Hi Aaron,
Without looking too much into numbers, my bet would be that it is large heap 
that is causing issues. I would decrease is significantly (<30GB) and see if it 
is enough for your max load. Also, disable swap or reduce swappiness to min.

In any case, you should install some monitoring tool that would help you do 
better analysis when you run into problems. One such tool is our monitoring 
solution: https://sematext.com/spm

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 18 Mar 2019, at 13:14, Aaron Yingcai Sun  wrote:
>
> Hello, Emir,
>
> Thanks for the reply, this is the solr version and heap info, standalone 
> single solr server. I don't have monitor tool connected. only look at 'top', 
> has not seen cpu spike so far, when the slow response happens, cpu usage is 
> not high at all, around 30%.
>
>
> # curl 'http://.../solr/admin/info/system?wt=json=true'
> {
>  "responseHeader":{
>"status":0,
>"QTime":27},
>  "mode":"std",
>  "solr_home":"/ardome/solr",
>  "lucene":{
>"solr-spec-version":"6.5.1",
>"solr-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - 
> jimczi - 2017-04-21 12:23:42",
>"lucene-spec-version":"6.5.1",
>"lucene-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - 
> jimczi - 2017-04-21 12:17:15"},
>  "jvm":{
>"version":"1.8.0_144 25.144-b01",
>"name":"Oracle Corporation Java HotSpot(TM) 64-Bit Server VM",
>"spec":{
>  "vendor":"Oracle Corporation",
>  "name":"Java Platform API Specification",
>  "version":"1.8"},
>"jre":{
>  "vendor":"Oracle Corporation",
>  "version":"1.8.0_144"},
>"vm":{
>  "vendor":"Oracle Corporation",
>  "name":"Java HotSpot(TM) 64-Bit Server VM",
>  "version":"25.144-b01"},
>"processors":32,
>"memory":{
>  "free":"69.1 GB",
>  "total":"180.2 GB",
>  "max":"266.7 GB",
>  "used":"111 GB (%41.6)",
>  "raw":{
>"free":74238728336,
>"total":193470136320,
>"max":286331502592,
>"used":119231407984,
>"used%":41.64103736566334}},
>"jmx":{
>  
> "bootclasspath":"/usr/java/jdk1.8.0_144/jre/lib/resources.jar:/usr/java/jdk1.8.0_144/jre/lib/rt.jar:/usr/java/jdk1.8.0_144/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_144/jre/lib/jsse.jar:/usr/java/jdk1.8.0_144/jre/lib/jce.jar:/usr/java/jdk1.8.0_144/jre/lib/charsets.jar:/usr/java/jdk1.8.0_144/jre/lib/jfr.jar:/usr/java/jdk1.8.0_144/jre/classes",
>  "classpath":"...",
>  "commandLineArgs":["-Xms100G",
>"-Xmx300G",
>"-DSTOP.PORT=8079",
>"-DSTOP.KEY=..",
>"-Dsolr.solr.home=..",
>"-Djetty.port=8983"],
>  "startTime":"2019-03-18T09:35:27.892Z",
>  "upTimeMS":9258422}},
>  "system":{
>"name":"Linux",
>"arch":"amd64",
>"availableProcessors":32,
>"systemLoadAverage":14.72,
>"version":"3.0.101-311.g08a8a9d-default",
>"committedVirtualMemorySize":2547960700928,
>"freePhysicalMemorySize":4530696192,
>"freeSwapSpaceSize":3486846976,
>"processCpuLoad":0.3257436126790475,
>"processCpuTime":9386945000,
>"systemCpuLoad":0.3279781055816521,
>"totalPhysicalMemorySize":406480175104,
>"totalSwapSpaceSize":4302303232,
>"maxFileDescriptorCount":32768,
>"openFileDescriptorCount":385,
>"uname":"Linux ... 3.0.101-311.g08a8a9d-default #1 SMP Wed Dec 14 10:15:37 
> UTC 2016 (08a8a9d) x86_64 x86_64 x86_64 GNU/Linux\n",
>"uptime":" 13:09pm  up 5 days 21:23,  7 users,  load average: 14.72, 
> 12.28, 11.48\n"}}
>
>
>
>
> 
> From: Emir Arnautović 
> Sent: Monday, March 18, 2019 12:10:30 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr index slow response
>
> Hi Aaron,
> Which version of Solr? How did you configure your heap? Is it standalone Solr 
> or SolrCloud? A single server? Do you use some monitoring tool? Do you see 
> some spikes, pauses or CPU usage is constant?
>
> Thanks,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
>> On 18 Mar 2019, at 11:47, Aaron Yingcai Sun  wrote:
>>
>> Hello, Solr!
>>
>>
>> We are having some performance issue when try to send documents for solr to 
>> index. The repose time is very slow and unpredictable some time.
>>
>>
>> Solr server is running on a quit powerful server, 32 cpus, 400GB RAM, while 

Re: Solr index slow response

[Aaron Yingcai Sun]

I'm a bit confused, why large heap size would make it slower?  Isn't that give 
it enough room to make it not busy doing GC all the time?

My http/json request contains 100 documents, the total size of the 100 
documents is around 5M, there are ~100 client sending those requests 
continuously.

Previously the JVM is set to max 32 GB ,  the speed was even worse,  now it's 
running with min 100GB, max 300GB, it use around 100GB.


this page suggest to use smaller number of documents per request,   
https://wiki.apache.org/solr/SolrPerformanceProblems

SolrPerformanceProblems - Solr 
Wiki
wiki.apache.org
General information. There is a performance bug that makes *everything* slow in 
versions 6.4.0 and 6.4.1. The problem is fixed in 6.4.2. It is described by 
SOLR-10130.This is highly version specific, so if you are not running one of 
the affected versions, don't worry about it.

So I try to reduce the number, still I could get lots of large response QTime:

190318-142652.695-160214 DBG1:doc_count: 10 , doc_size: 609  KB, Res code: 200, 
QTime: 47918 ms, Request time: 47921 ms.
190318-142652.704-160179 DBG1:doc_count: 10 , doc_size: 568  KB, Res code: 200, 
QTime: 36919 ms, Request time: 36922 ms.
190318-142652.780-160197 DBG1:doc_count: 10 , doc_size: 609  KB, Res code: 200, 
QTime: 36082 ms, Request time: 36084 ms.
190318-142652.859-160200 DBG1:doc_count: 10 , doc_size: 569  KB, Res code: 200, 
QTime: 36880 ms, Request time: 36882 ms.
190318-142653.131-160148 DBG1:doc_count: 10 , doc_size: 608  KB, Res code: 200, 
QTime: 37222 ms, Request time: 37224 ms.
190318-142653.154-160211 DBG1:doc_count: 10 , doc_size: 541  KB, Res code: 200, 
QTime: 37241 ms, Request time: 37243 ms.
190318-142653.223-163490 DBG1:doc_count: 10 , doc_size: 589  KB, Res code: 200, 
QTime: 37174 ms, Request time: 37176 ms.
190318-142653.359-160154 DBG1:doc_count: 10 , doc_size: 592  KB, Res code: 200, 
QTime: 37008 ms, Request time: 37011 ms.
190318-142653.497-163491 DBG1:doc_count: 10 , doc_size: 583  KB, Res code: 200, 
QTime: 24828 ms, Request time: 24830 ms.
190318-142653.987-160208 DBG1:doc_count: 10 , doc_size: 669  KB, Res code: 200, 
QTime: 23900 ms, Request time: 23902 ms.
190318-142654.114-160208 DBG1:doc_count: 10 , doc_size: 544  KB, Res code: 200, 
QTime: 121 ms, Request time: 122 ms.
190318-142654.233-160208 DBG1:doc_count: 10 , doc_size: 536  KB, Res code: 200, 
QTime: 113 ms, Request time: 115 ms.
190318-142654.354-160208 DBG1:doc_count: 10 , doc_size: 598  KB, Res code: 200, 
QTime: 116 ms, Request time: 117 ms.
190318-142654.466-160208 DBG1:doc_count: 10 , doc_size: 546  KB, Res code: 200, 
QTime: 107 ms, Request time: 108 ms.
190318-142654.586-160208 DBG1:doc_count: 10 , doc_size: 566  KB, Res code: 200, 
QTime: 114 ms, Request time: 115 ms.
190318-142654.687-160208 DBG1:doc_count: 10 , doc_size: 541  KB, Res code: 200, 
QTime: 96 ms, Request time: 98 ms.
190318-142654.768-160208 DBG1:doc_count: 10 , doc_size: 455  KB, Res code: 200, 
QTime: 75 ms, Request time: 77 ms.
190318-142654.870-160208 DBG1:doc_count: 10 , doc_size: 538  KB, Res code: 200, 
QTime: 97 ms, Request time: 98 ms.
190318-142654.967-160208 DBG1:doc_count: 10 , doc_size: 539  KB, Res code: 200, 
QTime: 92 ms, Request time: 93 ms.
190318-142655.096-160208 DBG1:doc_count: 10 , doc_size: 672  KB, Res code: 200, 
QTime: 124 ms, Request time: 125 ms.
190318-142655.210-160208 DBG1:doc_count: 10 , doc_size: 605  KB, Res code: 200, 
QTime: 108 ms, Request time: 110 ms.
190318-142655.304-160208 DBG1:doc_count: 10 , doc_size: 481  KB, Res code: 200, 
QTime: 89 ms, Request time: 90 ms.
190318-142655.410-160208 DBG1:doc_count: 10 , doc_size: 468  KB, Res code: 200, 
QTime: 101 ms, Request time: 102 ms.



From: Toke Eskildsen 
Sent: Monday, March 18, 2019 2:13:12 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr index slow response

On Mon, 2019-03-18 at 10:47 +, Aaron Yingcai Sun wrote:
> Solr server is running on a quit powerful server, 32 cpus, 400GB RAM,
> while 300 GB is reserved for solr, [...]

300GB for Solr sounds excessive.

> Our application send 100 documents to solr per request, json encoded.
> the size is around 5M each time. some times the response time is
> under 1 seconds, some times could be 300 seconds, the slow response
> happens very often.
> ...
> There are around 100 clients sending those documents at the same
> time, but each for the client is blocking call which wait the http
> response then send the next one.

100 clients * 5MB/batch = at most 500MB. Or maybe you meant 100 clients
* 100 documents * 5MB/document = at most 50GB? Either way it is a long
way from 300GB and the stats you provide further down the thread
indicates that you are overprovisioning quite a lot:

"memory":{
  "free":"69.1 GB",
  "total":"180.2 GB",
  "max":"266.7 GB",
  "used":"111 GB (%41.6)",

Intermittent slow response times are a known effect of having 

Re: Solr index slow response

[Emir Arnautović]

One more thing - it is considered a good practice to use the same value for Xmx 
and Xms.

Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 18 Mar 2019, at 14:19, Emir Arnautović  
> wrote:
> 
> Hi Aaron,
> Without looking too much into numbers, my bet would be that it is large heap 
> that is causing issues. I would decrease is significantly (<30GB) and see if 
> it is enough for your max load. Also, disable swap or reduce swappiness to 
> min.
> 
> In any case, you should install some monitoring tool that would help you do 
> better analysis when you run into problems. One such tool is our monitoring 
> solution: https://sematext.com/spm
> 
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> 
> 
> 
>> On 18 Mar 2019, at 13:14, Aaron Yingcai Sun  wrote:
>> 
>> Hello, Emir,
>> 
>> Thanks for the reply, this is the solr version and heap info, standalone 
>> single solr server. I don't have monitor tool connected. only look at 'top', 
>> has not seen cpu spike so far, when the slow response happens, cpu usage is 
>> not high at all, around 30%.
>> 
>> 
>> # curl 'http://.../solr/admin/info/system?wt=json=true'
>> {
>> "responseHeader":{
>>   "status":0,
>>   "QTime":27},
>> "mode":"std",
>> "solr_home":"/ardome/solr",
>> "lucene":{
>>   "solr-spec-version":"6.5.1",
>>   "solr-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - 
>> jimczi - 2017-04-21 12:23:42",
>>   "lucene-spec-version":"6.5.1",
>>   "lucene-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - 
>> jimczi - 2017-04-21 12:17:15"},
>> "jvm":{
>>   "version":"1.8.0_144 25.144-b01",
>>   "name":"Oracle Corporation Java HotSpot(TM) 64-Bit Server VM",
>>   "spec":{
>> "vendor":"Oracle Corporation",
>> "name":"Java Platform API Specification",
>> "version":"1.8"},
>>   "jre":{
>> "vendor":"Oracle Corporation",
>> "version":"1.8.0_144"},
>>   "vm":{
>> "vendor":"Oracle Corporation",
>> "name":"Java HotSpot(TM) 64-Bit Server VM",
>> "version":"25.144-b01"},
>>   "processors":32,
>>   "memory":{
>> "free":"69.1 GB",
>> "total":"180.2 GB",
>> "max":"266.7 GB",
>> "used":"111 GB (%41.6)",
>> "raw":{
>>   "free":74238728336,
>>   "total":193470136320,
>>   "max":286331502592,
>>   "used":119231407984,
>>   "used%":41.64103736566334}},
>>   "jmx":{
>> 
>> "bootclasspath":"/usr/java/jdk1.8.0_144/jre/lib/resources.jar:/usr/java/jdk1.8.0_144/jre/lib/rt.jar:/usr/java/jdk1.8.0_144/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_144/jre/lib/jsse.jar:/usr/java/jdk1.8.0_144/jre/lib/jce.jar:/usr/java/jdk1.8.0_144/jre/lib/charsets.jar:/usr/java/jdk1.8.0_144/jre/lib/jfr.jar:/usr/java/jdk1.8.0_144/jre/classes",
>> "classpath":"...",
>> "commandLineArgs":["-Xms100G",
>>   "-Xmx300G",
>>   "-DSTOP.PORT=8079",
>>   "-DSTOP.KEY=..",
>>   "-Dsolr.solr.home=..",
>>   "-Djetty.port=8983"],
>> "startTime":"2019-03-18T09:35:27.892Z",
>> "upTimeMS":9258422}},
>> "system":{
>>   "name":"Linux",
>>   "arch":"amd64",
>>   "availableProcessors":32,
>>   "systemLoadAverage":14.72,
>>   "version":"3.0.101-311.g08a8a9d-default",
>>   "committedVirtualMemorySize":2547960700928,
>>   "freePhysicalMemorySize":4530696192,
>>   "freeSwapSpaceSize":3486846976,
>>   "processCpuLoad":0.3257436126790475,
>>   "processCpuTime":9386945000,
>>   "systemCpuLoad":0.3279781055816521,
>>   "totalPhysicalMemorySize":406480175104,
>>   "totalSwapSpaceSize":4302303232,
>>   "maxFileDescriptorCount":32768,
>>   "openFileDescriptorCount":385,
>>   "uname":"Linux ... 3.0.101-311.g08a8a9d-default #1 SMP Wed Dec 14 10:15:37 
>> UTC 2016 (08a8a9d) x86_64 x86_64 x86_64 GNU/Linux\n",
>>   "uptime":" 13:09pm  up 5 days 21:23,  7 users,  load average: 14.72, 
>> 12.28, 11.48\n"}}
>> 
>> 
>> 
>> 
>> 
>> From: Emir Arnautović 
>> Sent: Monday, March 18, 2019 12:10:30 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr index slow response
>> 
>> Hi Aaron,
>> Which version of Solr? How did you configure your heap? Is it standalone 
>> Solr or SolrCloud? A single server? Do you use some monitoring tool? Do you 
>> see some spikes, pauses or CPU usage is constant?
>> 
>> Thanks,
>> Emir
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>> 
>> 
>> 
>>> On 18 Mar 2019, at 11:47, Aaron Yingcai Sun  wrote:
>>> 
>>> Hello, Solr!
>>> 
>>> 
>>> We are having some performance issue when try to send documents for solr to 
>>> index. The repose time is very slow and unpredictable some time.
>>> 
>>> 
>>> Solr server is running on a quit powerful server, 32 cpus, 400GB RAM, while 
>>> 300 GB is reserved for solr, while this happening, cpu usage is around 30%, 
>>> mem 

Re: Different behavior when using function queries

[Erik Hatcher]

If you have no documents in the results, there’s nothing to attach the function 
result to.`fl` is field list of fields to show in matched documents.   You 
have no matches documents. 

Erik

> On Mar 18, 2019, at 07:55, Ashish Bisht  wrote:
> 
> Can someone please explain the below behavior.For different q parameter
> function query response differs although function queries are same
> 
> http://:8983/solr/SCSpell/select?q="*market
> place*"=edismax=spellcontent=json=1=internet_of_things:if(exists(query({!edismax
> v='"internet of
> things"'})),true,false),instant_of_things:if(exists(query({!edismax
> v='"instant of things"'})),true,false)
> 
> Response contains function query results
> 
> "response":{"numFound":80,"start":0,"docs":[
>  {
>"internet_of_things":false,
>"instant_of_things":false}]
>  }}
> 
> wheras for different q
> 
> http://:8983/solr/SCSpell/select?q="*intent of
> things*"=edismax=spellcontent=json=1=internet_of_things:if(exists(query({!edismax
> v='"internet of
> things"'})),true,false),instant_of_things:if(exists(query({!edismax
> v='"instant of things"'})),true,false)
> 
> Response doesnot contain function query results
> 
> "response":{"numFound":0,"start":0,"docs":[]
>  }}
> 
> 
> From the results it looks like if the results of q doesn't yield result
> function queries don't work.
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr index slow response

[Emir Arnautović]

Hi Aaron,
Without looking too much into numbers, my bet would be that it is large heap 
that is causing issues. I would decrease is significantly (<30GB) and see if it 
is enough for your max load. Also, disable swap or reduce swappiness to min.

In any case, you should install some monitoring tool that would help you do 
better analysis when you run into problems. One such tool is our monitoring 
solution: https://sematext.com/spm

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 18 Mar 2019, at 13:14, Aaron Yingcai Sun  wrote:
> 
> Hello, Emir,
> 
> Thanks for the reply, this is the solr version and heap info, standalone 
> single solr server. I don't have monitor tool connected. only look at 'top', 
> has not seen cpu spike so far, when the slow response happens, cpu usage is 
> not high at all, around 30%.
> 
> 
> # curl 'http://.../solr/admin/info/system?wt=json=true'
> {
>  "responseHeader":{
>"status":0,
>"QTime":27},
>  "mode":"std",
>  "solr_home":"/ardome/solr",
>  "lucene":{
>"solr-spec-version":"6.5.1",
>"solr-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - 
> jimczi - 2017-04-21 12:23:42",
>"lucene-spec-version":"6.5.1",
>"lucene-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - 
> jimczi - 2017-04-21 12:17:15"},
>  "jvm":{
>"version":"1.8.0_144 25.144-b01",
>"name":"Oracle Corporation Java HotSpot(TM) 64-Bit Server VM",
>"spec":{
>  "vendor":"Oracle Corporation",
>  "name":"Java Platform API Specification",
>  "version":"1.8"},
>"jre":{
>  "vendor":"Oracle Corporation",
>  "version":"1.8.0_144"},
>"vm":{
>  "vendor":"Oracle Corporation",
>  "name":"Java HotSpot(TM) 64-Bit Server VM",
>  "version":"25.144-b01"},
>"processors":32,
>"memory":{
>  "free":"69.1 GB",
>  "total":"180.2 GB",
>  "max":"266.7 GB",
>  "used":"111 GB (%41.6)",
>  "raw":{
>"free":74238728336,
>"total":193470136320,
>"max":286331502592,
>"used":119231407984,
>"used%":41.64103736566334}},
>"jmx":{
>  
> "bootclasspath":"/usr/java/jdk1.8.0_144/jre/lib/resources.jar:/usr/java/jdk1.8.0_144/jre/lib/rt.jar:/usr/java/jdk1.8.0_144/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_144/jre/lib/jsse.jar:/usr/java/jdk1.8.0_144/jre/lib/jce.jar:/usr/java/jdk1.8.0_144/jre/lib/charsets.jar:/usr/java/jdk1.8.0_144/jre/lib/jfr.jar:/usr/java/jdk1.8.0_144/jre/classes",
>  "classpath":"...",
>  "commandLineArgs":["-Xms100G",
>"-Xmx300G",
>"-DSTOP.PORT=8079",
>"-DSTOP.KEY=..",
>"-Dsolr.solr.home=..",
>"-Djetty.port=8983"],
>  "startTime":"2019-03-18T09:35:27.892Z",
>  "upTimeMS":9258422}},
>  "system":{
>"name":"Linux",
>"arch":"amd64",
>"availableProcessors":32,
>"systemLoadAverage":14.72,
>"version":"3.0.101-311.g08a8a9d-default",
>"committedVirtualMemorySize":2547960700928,
>"freePhysicalMemorySize":4530696192,
>"freeSwapSpaceSize":3486846976,
>"processCpuLoad":0.3257436126790475,
>"processCpuTime":9386945000,
>"systemCpuLoad":0.3279781055816521,
>"totalPhysicalMemorySize":406480175104,
>"totalSwapSpaceSize":4302303232,
>"maxFileDescriptorCount":32768,
>"openFileDescriptorCount":385,
>"uname":"Linux ... 3.0.101-311.g08a8a9d-default #1 SMP Wed Dec 14 10:15:37 
> UTC 2016 (08a8a9d) x86_64 x86_64 x86_64 GNU/Linux\n",
>"uptime":" 13:09pm  up 5 days 21:23,  7 users,  load average: 14.72, 
> 12.28, 11.48\n"}}
> 
> 
> 
> 
> 
> From: Emir Arnautović 
> Sent: Monday, March 18, 2019 12:10:30 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr index slow response
> 
> Hi Aaron,
> Which version of Solr? How did you configure your heap? Is it standalone Solr 
> or SolrCloud? A single server? Do you use some monitoring tool? Do you see 
> some spikes, pauses or CPU usage is constant?
> 
> Thanks,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> 
> 
> 
>> On 18 Mar 2019, at 11:47, Aaron Yingcai Sun  wrote:
>> 
>> Hello, Solr!
>> 
>> 
>> We are having some performance issue when try to send documents for solr to 
>> index. The repose time is very slow and unpredictable some time.
>> 
>> 
>> Solr server is running on a quit powerful server, 32 cpus, 400GB RAM, while 
>> 300 GB is reserved for solr, while this happening, cpu usage is around 30%, 
>> mem usage is 34%.  io also look ok according to iotop. SSD disk.
>> 
>> 
>> Our application send 100 documents to solr per request, json encoded. the 
>> size is around 5M each time. some times the response time is under 1 
>> seconds, some times could be 300 seconds, the slow response happens very 
>> often.
>> 
>> 
>> "Soft AutoCommit: disabled", "Hard AutoCommit: if uncommited 

Re: Solr index slow response

[Toke Eskildsen]

On Mon, 2019-03-18 at 10:47 +, Aaron Yingcai Sun wrote:
> Solr server is running on a quit powerful server, 32 cpus, 400GB RAM,
> while 300 GB is reserved for solr, [...]

300GB for Solr sounds excessive.

> Our application send 100 documents to solr per request, json encoded.
> the size is around 5M each time. some times the response time is
> under 1 seconds, some times could be 300 seconds, the slow response
> happens very often.
> ...
> There are around 100 clients sending those documents at the same
> time, but each for the client is blocking call which wait the http
> response then send the next one.

100 clients * 5MB/batch = at most 500MB. Or maybe you meant 100 clients
* 100 documents * 5MB/document = at most 50GB? Either way it is a long
way from 300GB and the stats you provide further down the thread
indicates that you are overprovisioning quite a lot:

"memory":{
  "free":"69.1 GB",
  "total":"180.2 GB",
  "max":"266.7 GB",
  "used":"111 GB (%41.6)",

Intermittent slow response times are a known effect of having large
Java heaps, due to stop-the-world garbage collections. 

Try dialing Xmx _way_ down: If your batches are only 5MB each, try
Xmx=20g or less. I know that the stats above says that Solr uses 111GB,
but the JVM has a tendency to expand the heap quite a lot when it is
getting hammered. If you want to check beforehand, you can see how much
memeory is freed from full GCs in the GC-log.

- Toke Eskildsen, Royal Danish Library

Re: Solr index slow response

[Aaron Yingcai Sun]

Hello, Emir,

Thanks for the reply, this is the solr version and heap info, standalone single 
solr server. I don't have monitor tool connected. only look at 'top', has not 
seen cpu spike so far, when the slow response happens, cpu usage is not high at 
all, around 30%.


# curl 'http://.../solr/admin/info/system?wt=json=true'
{
  "responseHeader":{
"status":0,
"QTime":27},
  "mode":"std",
  "solr_home":"/ardome/solr",
  "lucene":{
"solr-spec-version":"6.5.1",
"solr-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - 
jimczi - 2017-04-21 12:23:42",
"lucene-spec-version":"6.5.1",
"lucene-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - 
jimczi - 2017-04-21 12:17:15"},
  "jvm":{
"version":"1.8.0_144 25.144-b01",
"name":"Oracle Corporation Java HotSpot(TM) 64-Bit Server VM",
"spec":{
  "vendor":"Oracle Corporation",
  "name":"Java Platform API Specification",
  "version":"1.8"},
"jre":{
  "vendor":"Oracle Corporation",
  "version":"1.8.0_144"},
"vm":{
  "vendor":"Oracle Corporation",
  "name":"Java HotSpot(TM) 64-Bit Server VM",
  "version":"25.144-b01"},
"processors":32,
"memory":{
  "free":"69.1 GB",
  "total":"180.2 GB",
  "max":"266.7 GB",
  "used":"111 GB (%41.6)",
  "raw":{
"free":74238728336,
"total":193470136320,
"max":286331502592,
"used":119231407984,
"used%":41.64103736566334}},
"jmx":{
  
"bootclasspath":"/usr/java/jdk1.8.0_144/jre/lib/resources.jar:/usr/java/jdk1.8.0_144/jre/lib/rt.jar:/usr/java/jdk1.8.0_144/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_144/jre/lib/jsse.jar:/usr/java/jdk1.8.0_144/jre/lib/jce.jar:/usr/java/jdk1.8.0_144/jre/lib/charsets.jar:/usr/java/jdk1.8.0_144/jre/lib/jfr.jar:/usr/java/jdk1.8.0_144/jre/classes",
  "classpath":"...",
  "commandLineArgs":["-Xms100G",
"-Xmx300G",
"-DSTOP.PORT=8079",
"-DSTOP.KEY=..",
"-Dsolr.solr.home=..",
"-Djetty.port=8983"],
  "startTime":"2019-03-18T09:35:27.892Z",
  "upTimeMS":9258422}},
  "system":{
"name":"Linux",
"arch":"amd64",
"availableProcessors":32,
"systemLoadAverage":14.72,
"version":"3.0.101-311.g08a8a9d-default",
"committedVirtualMemorySize":2547960700928,
"freePhysicalMemorySize":4530696192,
"freeSwapSpaceSize":3486846976,
"processCpuLoad":0.3257436126790475,
"processCpuTime":9386945000,
"systemCpuLoad":0.3279781055816521,
"totalPhysicalMemorySize":406480175104,
"totalSwapSpaceSize":4302303232,
"maxFileDescriptorCount":32768,
"openFileDescriptorCount":385,
"uname":"Linux ... 3.0.101-311.g08a8a9d-default #1 SMP Wed Dec 14 10:15:37 
UTC 2016 (08a8a9d) x86_64 x86_64 x86_64 GNU/Linux\n",
"uptime":" 13:09pm  up 5 days 21:23,  7 users,  load average: 14.72, 12.28, 
11.48\n"}}





From: Emir Arnautović 
Sent: Monday, March 18, 2019 12:10:30 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr index slow response

Hi Aaron,
Which version of Solr? How did you configure your heap? Is it standalone Solr 
or SolrCloud? A single server? Do you use some monitoring tool? Do you see some 
spikes, pauses or CPU usage is constant?

Thanks,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 18 Mar 2019, at 11:47, Aaron Yingcai Sun  wrote:
>
> Hello, Solr!
>
>
> We are having some performance issue when try to send documents for solr to 
> index. The repose time is very slow and unpredictable some time.
>
>
> Solr server is running on a quit powerful server, 32 cpus, 400GB RAM, while 
> 300 GB is reserved for solr, while this happening, cpu usage is around 30%, 
> mem usage is 34%.  io also look ok according to iotop. SSD disk.
>
>
> Our application send 100 documents to solr per request, json encoded. the 
> size is around 5M each time. some times the response time is under 1 seconds, 
> some times could be 300 seconds, the slow response happens very often.
>
>
> "Soft AutoCommit: disabled", "Hard AutoCommit: if uncommited for 360ms; 
> if 100 uncommited docs"
>
>
> There are around 100 clients sending those documents at the same time, but 
> each for the client is blocking call which wait the http response then send 
> the next one.
>
>
> I tried to make the number of documents smaller in one request, such as 20, 
> but  still I see slow response time to time, like 80 seconds.
>
>
> Would you help to give some hint how improve the response time?  solr does 
> not seems very loaded, there must be a way to make the response faster.
>
>
> BRs
>
> //Aaron
>
>
>

Different behavior when using function queries

[Ashish Bisht]

Can someone please explain the below behavior.For different q parameter
function query response differs although function queries are same

http://:8983/solr/SCSpell/select?q="*market
place*"=edismax=spellcontent=json=1=internet_of_things:if(exists(query({!edismax
v='"internet of
things"'})),true,false),instant_of_things:if(exists(query({!edismax
v='"instant of things"'})),true,false)

Response contains function query results

 "response":{"numFound":80,"start":0,"docs":[
  {
"internet_of_things":false,
"instant_of_things":false}]
  }}

wheras for different q

http://:8983/solr/SCSpell/select?q="*intent of
things*"=edismax=spellcontent=json=1=internet_of_things:if(exists(query({!edismax
v='"internet of
things"'})),true,false),instant_of_things:if(exists(query({!edismax
v='"instant of things"'})),true,false)

Response doesnot contain function query results
  
"response":{"numFound":0,"start":0,"docs":[]
  }}


>From the results it looks like if the results of q doesn't yield result
function queries don't work.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr index slow response

[Emir Arnautović]

Hi Aaron,
Which version of Solr? How did you configure your heap? Is it standalone Solr 
or SolrCloud? A single server? Do you use some monitoring tool? Do you see some 
spikes, pauses or CPU usage is constant?

Thanks,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 18 Mar 2019, at 11:47, Aaron Yingcai Sun  wrote:
> 
> Hello, Solr!
> 
> 
> We are having some performance issue when try to send documents for solr to 
> index. The repose time is very slow and unpredictable some time.
> 
> 
> Solr server is running on a quit powerful server, 32 cpus, 400GB RAM, while 
> 300 GB is reserved for solr, while this happening, cpu usage is around 30%, 
> mem usage is 34%.  io also look ok according to iotop. SSD disk.
> 
> 
> Our application send 100 documents to solr per request, json encoded. the 
> size is around 5M each time. some times the response time is under 1 seconds, 
> some times could be 300 seconds, the slow response happens very often.
> 
> 
> "Soft AutoCommit: disabled", "Hard AutoCommit: if uncommited for 360ms; 
> if 100 uncommited docs"
> 
> 
> There are around 100 clients sending those documents at the same time, but 
> each for the client is blocking call which wait the http response then send 
> the next one.
> 
> 
> I tried to make the number of documents smaller in one request, such as 20, 
> but  still I see slow response time to time, like 80 seconds.
> 
> 
> Would you help to give some hint how improve the response time?  solr does 
> not seems very loaded, there must be a way to make the response faster.
> 
> 
> BRs
> 
> //Aaron
> 
> 
> 

Solr index slow response

[Aaron Yingcai Sun]

Hello, Solr!


We are having some performance issue when try to send documents for solr to 
index. The repose time is very slow and unpredictable some time.


Solr server is running on a quit powerful server, 32 cpus, 400GB RAM, while 300 
GB is reserved for solr, while this happening, cpu usage is around 30%, mem 
usage is 34%.  io also look ok according to iotop. SSD disk.


Our application send 100 documents to solr per request, json encoded. the size 
is around 5M each time. some times the response time is under 1 seconds, some 
times could be 300 seconds, the slow response happens very often.


"Soft AutoCommit: disabled", "Hard AutoCommit: if uncommited for 360ms; if 
100 uncommited docs"


There are around 100 clients sending those documents at the same time, but each 
for the client is blocking call which wait the http response then send the next 
one.


I tried to make the number of documents smaller in one request, such as 20, but 
 still I see slow response time to time, like 80 seconds.


Would you help to give some hint how improve the response time?  solr does not 
seems very loaded, there must be a way to make the response faster.


BRs

//Aaron

Re: Solr or SolrJ Atomic Update

[THIERRY BOUCHENY]

Hi all,

Ok, I found my problem, it was a silly one as I expected !!! In my schema the  
“uniqueKey” was spelled with a small cap k !! 

Thierry

> On 15 Mar 2019, at 19:03, THIERRY BOUCHENY  wrote:
> 
> Hello,
> 
> I have spent a few hours trying to understand why I get this error.
> 
> RunUpdateProcessor has received an AddUpdateCommand containing a document 
> that appears to still contain Atomic document update operations, most likely 
> because DistributedUpdateProcessorFactory was explicitly disabled from this 
> updateRequestProcessorChain
> 
> I am trying to do a part update of a record Using either Solo ( through a 
> POST update ) or SolrJ, in both cases I have the same error. It must be 
> configuration problem but I can’t find out what is is, even after having sent 
> a few hours on the web.
> 
> I run Solr on 2 different servers one with solr 5.4.1 and an other with solr 
> 7.5.0 with the same problem.
> 
> Adding a new document works fine but trying to update the text field always 
> come back with this error.
> 
> My SolrJ code is quite simple.
> 
>   HttpSolrClient client = new 
> HttpSolrClient.Builder(baseurl).build();
>   client.setParser(new XMLResponseParser());
> 
>   SolrQuery query = new SolrQuery();
>   query.set("q", "userid:18278456");
>   QueryResponse response = client.query(query);
>
>   SolrDocumentList docList = response.getResults();
>   System.out.println("docList: " + docList.size());   
>   if( docList.size() == 0  )
>   {
>   SolrInputDocument doc = new SolrInputDocument();
>   doc.addField("userid", "18278456");
>   doc.addField("text", _data);
>   client.add(doc);
>   client.commit();
>   }
>   else
>   {
>   for (SolrDocument doc : docList)
>   {
>   System.out.println("existing doc id: " + 
> doc.get("id"));
>   SolrInputDocument _updatedoc = new 
> SolrInputDocument();
>   Map partialUpdate = new 
> HashMap();
>   partialUpdate.put("set", _data);
>   _updatedoc.addField("id", doc.get("id"));
>   _updatedoc.addField("text", partialUpdate);
>   client.add(_updatedoc);
>   client.commit();
>   break;
>   }
>   }
> 
> And my POST solr requests
> 
> for a Add ( this works ) Content-Type: application/json
> 
> {{solrdomainurl}}/testtika/update?commit=true
> 
> With the raw body [{"userid":"18278456","text":"test”}]
> 
> For part update ( responds with the above error ) Content-Type: 
> application/json
> 
> With the raw body 
> [{"id":"e841a2b5-a48d-47ef-b019-8f8d41e92655","text":{"set":"other"}}]
> 
> 
> id is my uniqueId
> 
> Defined as follow in my schema.xml
> 
>default="NEW" />
> 
>   id
> 
>
> 
> I won’t put the solrconfig.xml in the email but after using my custom one 
> with unsuccessful tries I took the one in exemple/example-DIH/solr folder 
> with the same result.
> 
> Any idea what I could do wrong is welcome !! Thanks in advance.
> 
> Thierry
> 

Re: Solr 7.6 Shard name - possible issue?

[Jan Høydahl]

It's just a name, so as long as Solr agrees with itself and ZK that this is the 
core name you're fine. I have no idea why the numbering seems to jump in that 
way though.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 18. mar. 2019 kl. 02:15 skrev Dave Durbin :
> 
> I have a collection with 16 shards. Originally it had 2 shards and then each 
> of those was split into 8 shards.
> 
> When I look at data stored on the servers, I have a couple of shard replicas 
> that have slightly odd names. For example while most of the shard replicas 
> look like this:
> 
> collection-name-2018-10-30_shard1_2_replica_n1
> 
> I have a couple who’s replica number exceeds a couple of hundred.
> 
> collection-name-2018-10-30_shard2_1_replica_n213
> 
> Does this seem reasonable? Does it suggest a problem with these shard 
> replicas or this shard in general ? 
> 
> Thanks
> Dave
> 
> 
> 
> -- 
> *P.S. We've launched a new blog to share the latest ideas and case studies 
> from our team. Check it out here: product.canva.com 
> . ***
> ** Empowering the 
> world to design
> Also, we're hiring. Apply here! 
> 
>  
>   
>     
>   
> 
> 
> 
> 
> 
> 

Spellchecker -File based vs Index based

[Ashish Bisht]

Hi,

I am seeing difference in file based spellcheck and index based spellcheck
implementations.

Using index based
http://:8983/solr/SCSpell/spell?q=*intnet of
things*=edismax=spellcontent=json=0=true=*default*=AND


  "suggestions":[
  "intnet",{
"numFound":10,
"startOffset":0,
"endOffset":6,
"origFreq


Suggestion get build up only for wrong word.


But while suing file based,they get build up for right words too which
messes collations

http://:8983/solr/SCSpell/spell?q=intnet%20of%20things=edismax=spellcontent=json=0&=true=*file*=AND

 "suggestion":["*internet*",
  "contnet",
  "intel",
  "intent",
  "intert",
  "intelect",
  "intended",
  "intented",
  "interest",
  "botnets"]},
  "*of*",{
"numFound":8,
"startOffset":7,
"endOffset":9,
"suggestion":["ofc",
  "off",
  "ohf",
 .
  "soft"]},
 "*things*",{
"numFound":10,
"startOffset":10,
"endOffset":16,
"suggestion":["thing",
  "brings",
  "think",
  "thinkers",
  .



Is there any property in file based which I use to fix this



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html