Solr Cloud version is 8.5. I have also attached the solr log with gc
enabled and our app log which shows that there was SocketTimeoutException.
On Fri, Jul 23, 2021 at 2:31 PM Pratik Patel <pra...@semandex.net> wrote:
> Hi All,
>
> *tl;dr* : running into long GC pauses and solr client socket timeouts
> when indexing bulk of documents into solr. Commit strategy in essence is to
> do hard commits at the interval of 50k documents (maxDocs=50k) and disable
> soft commit altogether during bulk indexing. Simple solr cloud set up with
> one node and one shard.
>
> *Details*:
> We have about 6 million documents which we are trying to index into solr.
> From these, about 500k documents have a text field which holds Abstracts of
> scientific papers/Articles. We extract keywords from these Abstracts and we
> index these keywords as well into solr.
>
> We have a many to many kind of relationship between Articles and keywords.
> To store this, we have following structure.
>
> Article documents
> Keyword documents
> Article-Keyword Join documents
>
> We use block join to index Articles with "Article-Keyword" join documents
> and Keyword documents are indexed independently.
>
> In other words, we have blocks of "Article + Article-Keyword Joins" and we
> have Keyword documents(they hold some additional metadata about keyword ).
>
> We have a bulk processing operation which creates these documents and
> indexes them into solr. During this bulk indexing, we don't need documents
> to be searchable. We need to search against them only after ALL the
> documents are indexed.
>
> *Based on this, this is our current strategy. *
> Soft commits are disabled and Hard commits are done at an interval of 50k
> documents with openSearcher=false. Our code triggers explicit commits 4
> times after various stages of bulk indexing. Transaction logs are enabled
> and have default settings.
>
> <autoCommit>
> <maxTime>${solr.autoCommit.maxTime:-1}</maxTime>
> <maxDocs>${solr.autoCommit.maxDocs:50000}</maxDocs>
> <openSearcher>false</openSearcher>
> </autoCommit>
>
> <autoSoftCommit>
> <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
> </autoSoftCommit>
>
> Other Environmental Details:
> Xms=8g and Xmx=14g, solr client socketTimeout=7 minutes and
> zkClienttimeout=2 mins
> Our indexing operation triggers many "add" operations in parallel using
> RxJava (15 to 30 threads) each "add" operation is passed about 1000
> documents.
>
> Currently, when we run this indexing operation, we notice that after a
> while solr goes into long GC pauses (longer than our sockeTimeout of 7
> minutes) and we get SocketTimeoutExceptions.
>
> *What could be causing such long GC pauses?*
>
> *Does this commit strategy make sense ? If not, what is the recommended
> strategy that we can look into? *
>
> *Any help on this is much appreciated. Thanks.*
>
>
Processed 199300 items
2021-07-23T17:38:24,889 [RxCachedThreadScheduler-108] ERROR
org.apache.solr.client.solrj.impl.BaseCloudSolrClient - Request to collection
[collection_store_60f97f9e6f56251cd6eda174] failed due to (0)
java.net.SocketTimeoutException: Read timed out, retry=0 commError=false
errorCode=0
2021-07-23T17:38:24,890 [RxCachedThreadScheduler-108] INFO
org.apache.solr.client.solrj.impl.BaseCloudSolrClient - request was not
communication error it seems
2021-07-23T17:38:24,890 [RxCachedThreadScheduler-108] ERROR
net.semandex.utils.solrlib.SolrEnv - Error indexing solr docs Timeout occurred
while waiting response from server at:
http://solr:8081/solr/collection_store_60f97f9e6f56251cd6eda174_shard1_replica_n1
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Timeout
occurred while waiting response from server at:
http://solr:8081/solr/collection_store_60f97f9e6f56251cd6eda174_shard1_replica_n1
at
org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:125)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:46)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.directUpdate(BaseCloudSolrClient.java:559)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1046)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:906)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:838)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
net.semandex.salsa.core.solr.concepts.BaseConceptSolrCollection._indexConcepts(BaseConceptSolrCollection.java:246)
~[core-2.11.0.jar:?]
at
net.semandex.salsa.core.solr.concepts.BaseConceptSolrCollection.indexConcepts(BaseConceptSolrCollection.java:257)
~[core-2.11.0.jar:?]
at
net.semandex.salsa.api.libs.processing.SolrKeywordProcessor.processDocs(SolrKeywordProcessor.kt:56)
~[classes/:?]
at
net.semandex.salsa.api.libs.processing.SolrKeywordProcessor.access$processDocs(SolrKeywordProcessor.kt:20)
~[classes/:?]
at
net.semandex.salsa.api.libs.processing.SolrKeywordProcessor$start$1.invoke(SolrKeywordProcessor.kt:45)
~[classes/:?]
at
net.semandex.salsa.api.libs.processing.SolrKeywordProcessor$start$1.invoke(SolrKeywordProcessor.kt:20)
~[classes/:?]
at
net.semandex.salsa.api.libs.processing.SolrKeywordProcessor$sam$io_reactivex_functions_Consumer$0.accept(SolrKeywordProcessor.kt)
~[classes/:?]
at net.semandex.salsa.libs.RxWorker.doOnNext(RxWorker.java:47)
~[salsa-lib-0.12.1.jar:?]
at
net.semandex.salsa.libs.RxProgressiveWorker.lambda$start$3(RxProgressiveWorker.java:83)
~[salsa-lib-0.12.1.jar:?]
at
io.reactivex.internal.operators.parallel.ParallelPeek$ParallelPeekSubscriber.onNext(ParallelPeek.java:148)
[rxjava-2.2.17.jar:?]
at
io.reactivex.internal.operators.parallel.ParallelRunOn$RunOnSubscriber.run(ParallelRunOn.java:273)
[rxjava-2.2.17.jar:?]
at
io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
[rxjava-2.2.17.jar:?]
at
io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
[rxjava-2.2.17.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_275]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_275]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occurred
while waiting response from server at:
http://solr:8081/solr/collection_store_60f97f9e6f56251cd6eda174_shard1_replica_n1
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:676)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$directUpdate$0(BaseCloudSolrClient.java:533)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_275]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
... 3 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_275]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
~[?:1.8.0_275]
at java.net.SocketInputStream.read(SocketInputStream.java:171)
~[?:1.8.0_275]
at java.net.SocketInputStream.read(SocketInputStream.java:141)
~[?:1.8.0_275]
at
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
~[httpclient-4.5.10.jar:4.5.10]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:564)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$directUpdate$0(BaseCloudSolrClient.java:533)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_275]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
... 3 more
2021-07-23T17:38:24,895 [RxCachedThreadScheduler-108] ERROR
InsightApi.SolrKeywordProcessor - Error in processing Timeout occurred while
waiting response from server at:
http://solr:8081/solr/collection_store_60f97f9e6f56251cd6eda174_shard1_replica_n1
net.semandex.salsa.core.exceptions.SalsaCoreException: Timeout occurred while
waiting response from server at:
http://solr:8081/solr/collection_store_60f97f9e6f56251cd6eda174_shard1_replica_n1
at
net.semandex.salsa.core.solr.concepts.BaseConceptSolrCollection._indexConcepts(BaseConceptSolrCollection.java:249)
~[core-2.11.0.jar:?]
at
net.semandex.salsa.core.solr.concepts.BaseConceptSolrCollection.indexConcepts(BaseConceptSolrCollection.java:257)
~[core-2.11.0.jar:?]
at
net.semandex.salsa.api.libs.processing.SolrKeywordProcessor.processDocs(SolrKeywordProcessor.kt:56)
~[classes/:?]
at
net.semandex.salsa.api.libs.processing.SolrKeywordProcessor.access$processDocs(SolrKeywordProcessor.kt:20)
~[classes/:?]
at
net.semandex.salsa.api.libs.processing.SolrKeywordProcessor$start$1.invoke(SolrKeywordProcessor.kt:45)
~[classes/:?]
at
net.semandex.salsa.api.libs.processing.SolrKeywordProcessor$start$1.invoke(SolrKeywordProcessor.kt:20)
~[classes/:?]
at
net.semandex.salsa.api.libs.processing.SolrKeywordProcessor$sam$io_reactivex_functions_Consumer$0.accept(SolrKeywordProcessor.kt)
~[classes/:?]
at net.semandex.salsa.libs.RxWorker.doOnNext(RxWorker.java:47)
~[salsa-lib-0.12.1.jar:?]
at
net.semandex.salsa.libs.RxProgressiveWorker.lambda$start$3(RxProgressiveWorker.java:83)
~[salsa-lib-0.12.1.jar:?]
at
io.reactivex.internal.operators.parallel.ParallelPeek$ParallelPeekSubscriber.onNext(ParallelPeek.java:148)
[rxjava-2.2.17.jar:?]
at
io.reactivex.internal.operators.parallel.ParallelRunOn$RunOnSubscriber.run(ParallelRunOn.java:273)
[rxjava-2.2.17.jar:?]
at
io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
[rxjava-2.2.17.jar:?]
at
io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
[rxjava-2.2.17.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_275]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_275]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException:
Timeout occurred while waiting response from server at:
http://solr:8081/solr/collection_store_60f97f9e6f56251cd6eda174_shard1_replica_n1
at
org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:125)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:46)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.directUpdate(BaseCloudSolrClient.java:559)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1046)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:906)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:838)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
net.semandex.salsa.core.solr.concepts.BaseConceptSolrCollection._indexConcepts(BaseConceptSolrCollection.java:246)
~[core-2.11.0.jar:?]
... 18 more
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occurred
while waiting response from server at:
http://solr:8081/solr/collection_store_60f97f9e6f56251cd6eda174_shard1_replica_n1
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:676)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$directUpdate$0(BaseCloudSolrClient.java:533)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_275]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
... 3 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_275]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
~[?:1.8.0_275]
at java.net.SocketInputStream.read(SocketInputStream.java:171)
~[?:1.8.0_275]
at java.net.SocketInputStream.read(SocketInputStream.java:141)
~[?:1.8.0_275]
at
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
~[httpclient-4.5.10.jar:4.5.10]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:564)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$directUpdate$0(BaseCloudSolrClient.java:533)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_275]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
... 3 more
2021-07-23T17:38:25,447 [RxCachedThreadScheduler-108] INFO
InsightApi.SolrKeywordProcessor - Processed 199400 items
2021-07-23T17:38:25,883 [RxCachedThreadScheduler-108] INFO
InsightApi.SolrKeywordProcessor - Processed 199500 items
2021-07-23T17:38:26,330 [RxCachedThreadScheduler-91] ERROR
org.apache.solr.client.solrj.impl.BaseCloudSolrClient - Request to collection
[collection_store_60f97f9e6f56251cd6eda174] failed due to (0)
java.net.SocketTimeoutException: Read timed out, retry=0 commError=false
errorCode=0
2021-07-23T17:38:26,330 [RxCachedThreadScheduler-91] INFO
org.apache.solr.client.solrj.impl.BaseCloudSolrClient - request was not
communication error it seems
2021-07-23T17:38:26,330 [RxCachedThreadScheduler-91] ERROR
net.semandex.utils.solrlib.SolrEnv - Error indexing solr docs Timeout occurred
while waiting response from server at:
http://solr:8081/solr/collection_store_60f97f9e6f56251cd6eda174_shard1_replica_n1
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Timeout
occurred while waiting response from server at:
http://solr:8081/solr/collection_store_60f97f9e6f56251cd6eda174_shard1_replica_n1
at
org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:125)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:46)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.directUpdate(BaseCloudSolrClient.java:559)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1046)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:906)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:838)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
net.semandex.salsa.core.solr.concepts.BaseConceptSolrCollection._indexConcepts(BaseConceptSolrCollection.java:246)
~[core-2.11.0.jar:?]
at
net.semandex.salsa.core.solr.concepts.BaseConceptSolrCollection.updateConcepts(BaseConceptSolrCollection.java:297)
~[core-2.11.0.jar:?]
at
net.semandex.salsa.api.libs.processing.SolrKeywordProcessor.processDocs(SolrKeywordProcessor.kt:54)
~[classes/:?]
at
net.semandex.salsa.api.libs.processing.SolrKeywordProcessor.access$processDocs(SolrKeywordProcessor.kt:20)
~[classes/:?]
at
net.semandex.salsa.api.libs.processing.SolrKeywordProcessor$start$1.invoke(SolrKeywordProcessor.kt:45)
~[classes/:?]
at
net.semandex.salsa.api.libs.processing.SolrKeywordProcessor$start$1.invoke(SolrKeywordProcessor.kt:20)
~[classes/:?]
at
net.semandex.salsa.api.libs.processing.SolrKeywordProcessor$sam$io_reactivex_functions_Consumer$0.accept(SolrKeywordProcessor.kt)
~[classes/:?]
at net.semandex.salsa.libs.RxWorker.doOnNext(RxWorker.java:47)
~[salsa-lib-0.12.1.jar:?]
at
net.semandex.salsa.libs.RxProgressiveWorker.lambda$start$3(RxProgressiveWorker.java:83)
~[salsa-lib-0.12.1.jar:?]
at
io.reactivex.internal.operators.parallel.ParallelPeek$ParallelPeekSubscriber.onNext(ParallelPeek.java:148)
[rxjava-2.2.17.jar:?]
at
io.reactivex.internal.operators.parallel.ParallelRunOn$RunOnSubscriber.run(ParallelRunOn.java:273)
[rxjava-2.2.17.jar:?]
at
io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
[rxjava-2.2.17.jar:?]
at
io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
[rxjava-2.2.17.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_275]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_275]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occurred
while waiting response from server at:
http://solr:8081/solr/collection_store_60f97f9e6f56251cd6eda174_shard1_replica_n1
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:676)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$directUpdate$0(BaseCloudSolrClient.java:533)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_275]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
... 3 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_275]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
~[?:1.8.0_275]
at java.net.SocketInputStream.read(SocketInputStream.java:171)
~[?:1.8.0_275]
at java.net.SocketInputStream.read(SocketInputStream.java:141)
~[?:1.8.0_275]
at
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
~[httpcore-4.4.12.jar:4.4.12]
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
~[httpclient-4.5.10.jar:4.5.10]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
~[httpclient-4.5.10.jar:4.5.10]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:564)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$directUpdate$0(BaseCloudSolrClient.java:533)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_275]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[solr-solrj-8.5.0.jar:8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 -
romseygeek - 2020-03-13 09:38:26]
... 3 more2021-07-23 17:32:02.495 INFO (qtp473053293-138)
[c:collection_store_60f97f9e6f56251cd6eda174 s:shard1 r:core_node2
x:collection_store_60f97f9e6f56251cd6eda174_shard1_replica_n1]
o.a.s.u.p.LogUpdateProcessorFactory
[collection_store_60f97f9e6f56251cd6eda174_shard1_replica_n1] webapp=/solr
path=/update
params={wt=javabin&version=2}{add=[60f99fbb07d582e9ae21e22760f97f9e6f56251cd6eda174
(1706097662795710464), 60f99fc107d582e9ae22727d60f97f9e6f56251cd6eda174
(1706097662796759040), 60f99fc107d582e9ae2272aa60f97f9e6f56251cd6eda174
(1706097662797807616), 60f9a00c07d582e9ae29285060f97f9e6f56251cd6eda174
(1706097662799904768), 60f9a01b07d582e9ae2acef660f97f9e6f56251cd6eda174
(1706097662799904769), 60f9a01d07d582e9ae2af4b660f97f9e6f56251cd6eda174
(1706097662802001920), 60f9a01e07d582e9ae2b030360f97f9e6f56251cd6eda174
(1706097662803050496), 60f9a02307d582e9ae2b917160f97f9e6f56251cd6eda174
(1706097662804099072), 60f9a02307d582e9ae2b977260f97f9e6f56251cd6eda174
(1706097662804099073), 60f9a02407d582e9ae2ba5e060f97f9e6f56251cd6eda174
(1706097662806196224), ... (100 adds)]} 0 206
[2889.119s][info ][gc,marking ] GC(3060) Concurrent Mark From Roots
3063.575ms
[2889.119s][info ][gc,marking ] GC(3060) Concurrent Preclean
[2889.121s][info ][gc,marking ] GC(3060) Concurrent Preclean 2.505ms
[2889.121s][info ][gc,marking ] GC(3060) Concurrent Mark (2886.055s,
2889.121s) 3066.210ms
[2889.122s][info ][gc,start ] GC(3060) Pause Remark
[2889.180s][info ][gc,stringtable] GC(3060) Cleaned string and symbol table,
strings: 34954 processed, 0 removed, symbols: 171224 processed, 0 removed
[2889.183s][info ][gc ] GC(3060) Pause Remark 9838M->7486M(14056M)
61.308ms
[2889.183s][info ][gc,cpu ] GC(3060) User=0.29s Sys=1.32s Real=0.06s
[2889.183s][info ][gc,marking ] GC(3060) Concurrent Rebuild Remembered Sets
[2890.777s][info ][gc,marking ] GC(3060) Concurrent Rebuild Remembered
Sets 1594.073ms
[2890.778s][info ][gc,start ] GC(3060) Pause Cleanup
[2890.780s][info ][gc ] GC(3060) Pause Cleanup
8056M->8056M(14056M) 2.682ms
[2890.780s][info ][gc,cpu ] GC(3060) User=0.03s Sys=0.00s Real=0.00s
[2890.780s][info ][gc,marking ] GC(3060) Concurrent Cleanup for Next Mark
[2890.801s][info ][gc,marking ] GC(3060) Concurrent Cleanup for Next Mark
20.947ms
[2890.802s][info ][gc ] GC(3060) Concurrent Cycle 4763.562ms
2021-07-23 17:32:04.547 INFO (commitScheduler-27-thread-1) [ ]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@5b444a61[collection_store_60f97f9e6f56251cd6eda174_shard1_replica_n1]
realtime]
2021-07-23 17:32:05.632 INFO (commitScheduler-27-thread-1) [ ]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
2021-07-23 17:32:05.632 INFO (commitScheduler-27-thread-1) [ ]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1706097666301100032,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2021-07-23 17:32:05.632 INFO (commitScheduler-27-thread-1) [ ]
o.a.s.u.SolrIndexWriter Calling setCommitData with
IW:org.apache.solr.update.SolrIndexWriter@7e3e7df9
commitCommandVersion:1706097666301100032
[2910.596s][info ][gc,start ] GC(3064) Pause Young (Prepare Mixed) (G1
Evacuation Pause)
[2910.596s][info ][gc,task ] GC(3064) Using 28 workers of 28 for
evacuation
[2910.775s][info ][gc,phases ] GC(3064) Pre Evacuate Collection Set:
6.0ms
[2910.775s][info ][gc,phases ] GC(3064) Evacuate Collection Set: 162.7ms
[2910.775s][info ][gc,phases ] GC(3064) Post Evacuate Collection Set:
4.6ms
[2910.775s][info ][gc,phases ] GC(3064) Other: 5.7ms
[2910.775s][info ][gc,heap ] GC(3064) Eden regions: 647->0(95)
[2910.775s][info ][gc,heap ] GC(3064) Survivor regions: 56->80(88)
[2910.775s][info ][gc,heap ] GC(3064) Old regions: 1083->1098
[2910.775s][info ][gc,heap ] GC(3064) Humongous regions: 720->720
[2910.775s][info ][gc,metaspace ] GC(3064) Metaspace:
68174K->68174K(1112064K)
[2910.775s][info ][gc ] GC(3064) Pause Young (Prepare Mixed) (G1
Evacuation Pause) 10022M->7588M(14056M) 179.136ms
[2910.775s][info ][gc,cpu ] GC(3064) User=3.14s Sys=1.32s Real=0.18s
[2917.676s][info ][gc,start ] GC(3065) Pause Young (Mixed) (G1
Evacuation Pause)
[2917.676s][info ][gc,task ] GC(3065) Using 28 workers of 28 for
evacuation
[2917.858s][info ][gc,phases ] GC(3065) Pre Evacuate Collection Set:
5.8ms
[2917.858s][info ][gc,phases ] GC(3065) Evacuate Collection Set: 168.4ms
[2917.858s][info ][gc,phases ] GC(3065) Post Evacuate Collection Set:
3.6ms
[2917.858s][info ][gc,phases ] GC(3065) Other: 4.4ms
[2917.858s][info ][gc,heap ] GC(3065) Eden regions: 95->0(1067)
[2917.858s][info ][gc,heap ] GC(3065) Survivor regions: 80->10(22)
[2917.858s][info ][gc,heap ] GC(3065) Old regions: 1098->1164
[2917.858s][info ][gc,heap ] GC(3065) Humongous regions: 720->720
[2917.858s][info ][gc,metaspace ] GC(3065) Metaspace:
68174K->68174K(1112064K)
[2917.858s][info ][gc ] GC(3065) Pause Young (Mixed) (G1
Evacuation Pause) 7968M->7571M(14056M) 182.500ms
[2917.858s][info ][gc,cpu ] GC(3065) User=3.61s Sys=1.08s Real=0.18s
2021-07-23 17:33:37.049 INFO (qtp473053293-135) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
[3007.619s][info ][gc,start ] GC(3066) Pause Young (Normal) (G1
Evacuation Pause)
[3007.619s][info ][gc,task ] GC(3066) Using 28 workers of 28 for
evacuation
[3007.686s][info ][gc,phases ] GC(3066) Pre Evacuate Collection Set:
6.1ms
[3007.686s][info ][gc,phases ] GC(3066) Evacuate Collection Set: 55.1ms
[3007.686s][info ][gc,phases ] GC(3066) Post Evacuate Collection Set:
4.7ms
[3007.686s][info ][gc,phases ] GC(3066) Other: 1.0ms
[3007.686s][info ][gc,heap ] GC(3066) Eden regions: 1067->0(171)
[3007.686s][info ][gc,heap ] GC(3066) Survivor regions: 10->4(135)
[3007.686s][info ][gc,heap ] GC(3066) Old regions: 1164->1164
[3007.686s][info ][gc,heap ] GC(3066) Humongous regions: 720->720
[3007.686s][info ][gc,metaspace ] GC(3066) Metaspace:
68194K->68194K(1112064K)
[3007.686s][info ][gc ] GC(3066) Pause Young (Normal) (G1
Evacuation Pause) 11839M->7549M(14056M) 67.044ms
[3007.686s][info ][gc,cpu ] GC(3066) User=1.01s Sys=0.46s Real=0.07s
2021-07-23 17:34:07.512 INFO (qtp473053293-27) [ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with
params action=CLUSTERSTATUS&wt=json&_=1627058716478 and sendToOCPQueue=true
2021-07-23 17:34:07.519 INFO (qtp473053293-27) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={action=CLUSTERSTATUS&wt=json&_=1627058716478} status=0 QTime=7
2021-07-23 17:34:07.543 INFO (qtp473053293-641) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system
params={wt=javabin&version=2&_=1627058716478} status=0 QTime=9
2021-07-23 17:34:07.545 INFO (qtp473053293-122) [ ]
o.a.s.h.a.AdminHandlersProxy Fetched response from 1 nodes: [solr:8081_solr]
2021-07-23 17:34:07.545 INFO (qtp473053293-122) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system
params={nodes=solr:8081_solr&wt=json&_=1627058716478} status=0 QTime=20
2021-07-23 17:34:07.571 INFO (qtp473053293-108) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={prefix=CONTAINER.fs,org.eclipse.jetty.server.handler.DefaultHandler.get-requests,INDEX.sizeInBytes,SEARCHER.searcher.numDocs,SEARCHER.searcher.deletedDocs,SEARCHER.searcher.warmupTime&wt=javabin&version=2&_=1627059754491}
status=0 QTime=39
2021-07-23 17:34:07.572 INFO (qtp473053293-636) [ ]
o.a.s.h.a.AdminHandlersProxy Fetched response from 1 nodes: [solr:8081_solr]
2021-07-23 17:34:07.572 INFO (qtp473053293-636) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={nodes=solr:8081_solr&prefix=CONTAINER.fs,org.eclipse.jetty.server.handler.DefaultHandler.get-requests,INDEX.sizeInBytes,SEARCHER.searcher.numDocs,SEARCHER.searcher.deletedDocs,SEARCHER.searcher.warmupTime&wt=json&_=1627059754491}
status=0 QTime=46
[3024.262s][info ][gc,start ] GC(3067) Pause Young (Normal) (G1
Evacuation Pause)
[3024.262s][info ][gc,task ] GC(3067) Using 28 workers of 28 for
evacuation
[3024.278s][info ][gc,phases ] GC(3067) Pre Evacuate Collection Set:
4.7ms
[3024.278s][info ][gc,phases ] GC(3067) Evacuate Collection Set: 7.5ms
[3024.278s][info ][gc,phases ] GC(3067) Post Evacuate Collection Set:
2.8ms
[3024.278s][info ][gc,phases ] GC(3067) Other: 0.8ms
[3024.278s][info ][gc,heap ] GC(3067) Eden regions: 171->0(173)
[3024.278s][info ][gc,heap ] GC(3067) Survivor regions: 4->2(22)
[3024.278s][info ][gc,heap ] GC(3067) Old regions: 1164->1164
[3024.278s][info ][gc,heap ] GC(3067) Humongous regions: 720->720
[3024.278s][info ][gc,metaspace ] GC(3067) Metaspace:
68198K->68198K(1112064K)
[3024.278s][info ][gc ] GC(3067) Pause Young (Normal) (G1
Evacuation Pause) 8233M->7541M(14056M) 15.933ms
[3024.278s][info ][gc,cpu ] GC(3067) User=0.19s Sys=0.01s Real=0.02s
[3041.309s][info ][gc,start ] GC(3068) Pause Young (Concurrent Start)
(G1 Evacuation Pause)
[3041.309s][info ][gc,task ] GC(3068) Using 28 workers of 28 for
evacuation
[3041.322s][info ][gc,phases ] GC(3068) Pre Evacuate Collection Set:
1.4ms
[3041.322s][info ][gc,phases ] GC(3068) Evacuate Collection Set: 9.8ms
[3041.322s][info ][gc,phases ] GC(3068) Post Evacuate Collection Set:
1.0ms
[3041.322s][info ][gc,phases ] GC(3068) Other: 1.1ms
[3041.322s][info ][gc,heap ] GC(3068) Eden regions: 173->0(173)
[3041.322s][info ][gc,heap ] GC(3068) Survivor regions: 2->2(22)
[3041.323s][info ][gc,heap ] GC(3068) Old regions: 1164->1164
[3041.323s][info ][gc,heap ] GC(3068) Humongous regions: 720->720
[3041.323s][info ][gc,metaspace ] GC(3068) Metaspace:
68198K->68198K(1112064K)
[3041.323s][info ][gc ] GC(3068) Pause Young (Concurrent Start)
(G1 Evacuation Pause) 8233M->7541M(14056M) 13.478ms
[3041.323s][info ][gc,cpu ] GC(3068) User=0.17s Sys=0.01s Real=0.01s
[3041.323s][info ][gc ] GC(3069) Concurrent Cycle
[3041.323s][info ][gc,marking ] GC(3069) Concurrent Clear Claimed Marks
[3041.323s][info ][gc,marking ] GC(3069) Concurrent Clear Claimed Marks
0.057ms
[3041.323s][info ][gc,marking ] GC(3069) Concurrent Scan Root Regions
[3041.354s][info ][gc,marking ] GC(3069) Concurrent Scan Root Regions
30.806ms
[3041.354s][info ][gc,marking ] GC(3069) Concurrent Mark (3041.354s)
[3041.354s][info ][gc,marking ] GC(3069) Concurrent Mark From Roots
[3041.354s][info ][gc,task ] GC(3069) Using 7 workers of 7 for marking
[3043.326s][info ][gc,marking ] GC(3069) Concurrent Mark From Roots
1971.886ms
[3043.326s][info ][gc,marking ] GC(3069) Concurrent Preclean
[3043.328s][info ][gc,marking ] GC(3069) Concurrent Preclean 1.959ms
[3043.328s][info ][gc,marking ] GC(3069) Concurrent Mark (3041.354s,
3043.328s) 1973.982ms
[3043.330s][info ][gc,start ] GC(3069) Pause Remark
[3043.379s][info ][gc,stringtable] GC(3069) Cleaned string and symbol table,
strings: 34960 processed, 0 removed, symbols: 171229 processed, 5 removed
[3043.382s][info ][gc ] GC(3069) Pause Remark 7615M->5875M(14056M)
52.268ms
[3043.382s][info ][gc,cpu ] GC(3069) User=0.26s Sys=1.08s Real=0.05s
[3043.382s][info ][gc,marking ] GC(3069) Concurrent Rebuild Remembered Sets
2021-07-23 17:34:37.082 INFO (qtp473053293-109) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.collection1.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.collection1.shard1.replica_n1:QUERY./select.requests&key=solr.core.collection_store_60f9ce1bfbcaec40b56a190e.shard1.replica_n1:QUERY./select.requests&key=solr.core.collection_store_60f97f8f6f56251cd6eda173.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.collection1.shard1.replica_n1:UPDATE./update.requests&key=solr.core.collection_store_60f97f8f6f56251cd6eda173.shard1.replica_n1:UPDATE./update.requests&key=solr.core.collection_store_60f97f9e6f56251cd6eda174.shard1.replica_n1:UPDATE./update.requests&key=solr.core.collection_store_60f97f9e6f56251cd6eda174.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.collection_store_60f9ce1bfbcaec40b56a190e.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.collection_store_60f9ce1bfbcaec40b56a190e.shard1.replica_n1:UPDATE./update.requests&key=solr.core.collection_store_60f97f9e6f56251cd6eda174.shard1.replica_n1:QUERY./select.requests&key=solr.core.collection_store_60f97f8f6f56251cd6eda173.shard1.replica_n1:QUERY./select.requests}
status=0 QTime=28
2021-07-23 17:34:37.084 INFO (qtp473053293-636) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
[3044.783s][info ][gc,marking ] GC(3069) Concurrent Rebuild Remembered
Sets 1400.791ms
[3044.783s][info ][gc,start ] GC(3069) Pause Cleanup
[3044.786s][info ][gc ] GC(3069) Pause Cleanup
5931M->5931M(14056M) 2.669ms
[3044.786s][info ][gc,cpu ] GC(3069) User=0.04s Sys=0.00s Real=0.01s
[3044.786s][info ][gc,marking ] GC(3069) Concurrent Cleanup for Next Mark
[3044.808s][info ][gc,marking ] GC(3069) Concurrent Cleanup for Next Mark
21.401ms
[3044.808s][info ][gc ] GC(3069) Concurrent Cycle 3485.122ms
[3059.375s][info ][gc,start ] GC(3070) Pause Young (Prepare Mixed) (G1
Evacuation Pause)
[3059.375s][info ][gc,task ] GC(3070) Using 28 workers of 28 for
evacuation
[3059.397s][info ][gc,phases ] GC(3070) Pre Evacuate Collection Set:
1.9ms
[3059.397s][info ][gc,phases ] GC(3070) Evacuate Collection Set: 14.0ms
[3059.397s][info ][gc,phases ] GC(3070) Post Evacuate Collection Set:
2.2ms
[3059.397s][info ][gc,phases ] GC(3070) Other: 3.6ms
[3059.397s][info ][gc,heap ] GC(3070) Eden regions: 173->0(173)
[3059.397s][info ][gc,heap ] GC(3070) Survivor regions: 2->2(22)
[3059.397s][info ][gc,heap ] GC(3070) Old regions: 1137->1137
[3059.397s][info ][gc,heap ] GC(3070) Humongous regions: 312->312
[3059.397s][info ][gc,metaspace ] GC(3070) Metaspace:
68201K->68201K(1112064K)
[3059.397s][info ][gc ] GC(3070) Pause Young (Prepare Mixed) (G1
Evacuation Pause) 6493M->5801M(14056M) 21.774ms
[3059.397s][info ][gc,cpu ] GC(3070) User=0.25s Sys=0.05s Real=0.02s
[3078.451s][info ][gc,start ] GC(3071) Pause Young (Mixed) (G1
Evacuation Pause)
[3078.451s][info ][gc,task ] GC(3071) Using 28 workers of 28 for
evacuation
[3078.618s][info ][gc,phases ] GC(3071) Pre Evacuate Collection Set:
21.2ms
[3078.618s][info ][gc,phases ] GC(3071) Evacuate Collection Set: 141.5ms
[3078.619s][info ][gc,phases ] GC(3071) Post Evacuate Collection Set:
4.0ms
[3078.619s][info ][gc,phases ] GC(3071) Other: 1.0ms
[3078.619s][info ][gc,heap ] GC(3071) Eden regions: 173->0(172)
[3078.619s][info ][gc,heap ] GC(3071) Survivor regions: 2->3(22)
[3078.619s][info ][gc,heap ] GC(3071) Old regions: 1137->1068
[3078.619s][info ][gc,heap ] GC(3071) Humongous regions: 312->312
[3078.619s][info ][gc,metaspace ] GC(3071) Metaspace:
68201K->68201K(1112064K)
[3078.619s][info ][gc ] GC(3071) Pause Young (Mixed) (G1
Evacuation Pause) 6493M->5527M(14056M) 167.807ms
[3078.619s][info ][gc,cpu ] GC(3071) User=2.61s Sys=1.27s Real=0.17s
[3098.203s][info ][gc,start ] GC(3072) Pause Young (Mixed) (G1
Evacuation Pause)
[3098.203s][info ][gc,task ] GC(3072) Using 28 workers of 28 for
evacuation
[3098.427s][info ][gc,phases ] GC(3072) Pre Evacuate Collection Set:
22.9ms
[3098.427s][info ][gc,phases ] GC(3072) Evacuate Collection Set: 194.0ms
[3098.427s][info ][gc,phases ] GC(3072) Post Evacuate Collection Set:
3.8ms
[3098.427s][info ][gc,phases ] GC(3072) Other: 3.9ms
[3098.427s][info ][gc,heap ] GC(3072) Eden regions: 172->0(195)
[3098.427s][info ][gc,heap ] GC(3072) Survivor regions: 3->3(22)
[3098.427s][info ][gc,heap ] GC(3072) Old regions: 1068->1047
[3098.427s][info ][gc,heap ] GC(3072) Humongous regions: 312->312
[3098.427s][info ][gc,metaspace ] GC(3072) Metaspace:
68201K->68201K(1112064K)
[3098.428s][info ][gc ] GC(3072) Pause Young (Mixed) (G1
Evacuation Pause) 6215M->5443M(14056M) 224.752ms
[3098.428s][info ][gc,cpu ] GC(3072) User=3.83s Sys=1.54s Real=0.23s
2021-07-23 17:35:37.121 INFO (qtp473053293-109) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.collection1.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.collection1.shard1.replica_n1:QUERY./select.requests&key=solr.core.collection_store_60f9ce1bfbcaec40b56a190e.shard1.replica_n1:QUERY./select.requests&key=solr.core.collection_store_60f97f8f6f56251cd6eda173.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.collection1.shard1.replica_n1:UPDATE./update.requests&key=solr.core.collection_store_60f97f8f6f56251cd6eda173.shard1.replica_n1:UPDATE./update.requests&key=solr.core.collection_store_60f97f9e6f56251cd6eda174.shard1.replica_n1:UPDATE./update.requests&key=solr.core.collection_store_60f97f9e6f56251cd6eda174.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.collection_store_60f9ce1bfbcaec40b56a190e.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.collection_store_60f9ce1bfbcaec40b56a190e.shard1.replica_n1:UPDATE./update.requests&key=solr.core.collection_store_60f97f9e6f56251cd6eda174.shard1.replica_n1:QUERY./select.requests&key=solr.core.collection_store_60f97f8f6f56251cd6eda173.shard1.replica_n1:QUERY./select.requests}
status=0 QTime=30
2021-07-23 17:35:37.124 INFO (qtp473053293-636) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
[3121.795s][info ][gc,start ] GC(3073) Pause Young (Normal) (G1
Evacuation Pause)
[3121.795s][info ][gc,task ] GC(3073) Using 28 workers of 28 for
evacuation
[3121.840s][info ][gc,phases ] GC(3073) Pre Evacuate Collection Set:
6.3ms
[3121.840s][info ][gc,phases ] GC(3073) Evacuate Collection Set: 36.2ms
[3121.840s][info ][gc,phases ] GC(3073) Post Evacuate Collection Set:
2.0ms
[3121.840s][info ][gc,phases ] GC(3073) Other: 1.1ms
[3121.840s][info ][gc,heap ] GC(3073) Eden regions: 195->0(267)
[3121.840s][info ][gc,heap ] GC(3073) Survivor regions: 3->3(25)
[3121.840s][info ][gc,heap ] GC(3073) Old regions: 1047->1047
[3121.840s][info ][gc,heap ] GC(3073) Humongous regions: 312->312
[3121.840s][info ][gc,metaspace ] GC(3073) Metaspace:
68202K->68202K(1112064K)
[3121.840s][info ][gc ] GC(3073) Pause Young (Normal) (G1
Evacuation Pause) 6223M->5443M(14056M) 45.714ms
[3121.840s][info ][gc,cpu ] GC(3073) User=0.77s Sys=0.18s Real=0.04s
[3155.479s][info ][gc,start ] GC(3074) Pause Young (Normal) (G1
Evacuation Pause)
[3155.479s][info ][gc,task ] GC(3074) Using 28 workers of 28 for
evacuation
[3155.499s][info ][gc,phases ] GC(3074) Pre Evacuate Collection Set:
6.1ms
[3155.499s][info ][gc,phases ] GC(3074) Evacuate Collection Set: 9.9ms
[3155.499s][info ][gc,phases ] GC(3074) Post Evacuate Collection Set:
2.4ms
[3155.499s][info ][gc,phases ] GC(3074) Other: 1.0ms
[3155.499s][info ][gc,heap ] GC(3074) Eden regions: 267->0(303)
[3155.499s][info ][gc,heap ] GC(3074) Survivor regions: 3->3(34)
[3155.499s][info ][gc,heap ] GC(3074) Old regions: 1047->1047
[3155.499s][info ][gc,heap ] GC(3074) Humongous regions: 312->312
[3155.499s][info ][gc,metaspace ] GC(3074) Metaspace:
68204K->68204K(1112064K)
[3155.499s][info ][gc ] GC(3074) Pause Young (Normal) (G1
Evacuation Pause) 6511M->5444M(14056M) 19.438ms
[3155.499s][info ][gc,cpu ] GC(3074) User=0.20s Sys=0.06s Real=0.02s
2021-07-23 17:36:37.165 INFO (qtp473053293-138) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.collection1.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.collection1.shard1.replica_n1:QUERY./select.requests&key=solr.core.collection_store_60f9ce1bfbcaec40b56a190e.shard1.replica_n1:QUERY./select.requests&key=solr.core.collection_store_60f97f8f6f56251cd6eda173.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.collection1.shard1.replica_n1:UPDATE./update.requests&key=solr.core.collection_store_60f97f8f6f56251cd6eda173.shard1.replica_n1:UPDATE./update.requests&key=solr.core.collection_store_60f97f9e6f56251cd6eda174.shard1.replica_n1:UPDATE./update.requests&key=solr.core.collection_store_60f97f9e6f56251cd6eda174.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.collection_store_60f9ce1bfbcaec40b56a190e.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.collection_store_60f9ce1bfbcaec40b56a190e.shard1.replica_n1:UPDATE./update.requests&key=solr.core.collection_store_60f97f9e6f56251cd6eda174.shard1.replica_n1:QUERY./select.requests&key=solr.core.collection_store_60f97f8f6f56251cd6eda173.shard1.replica_n1:QUERY./select.requests}
status=0 QTime=31
2021-07-23 17:36:37.167 INFO (qtp473053293-109) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
[3196.104s][info ][gc,start ] GC(3075) Pause Young (Normal) (G1
Evacuation Pause)
[3196.104s][info ][gc,task ] GC(3075) Using 28 workers of 28 for
evacuation
[3196.125s][info ][gc,phases ] GC(3075) Pre Evacuate Collection Set:
6.0ms
[3196.125s][info ][gc,phases ] GC(3075) Evacuate Collection Set: 11.1ms
[3196.125s][info ][gc,phases ] GC(3075) Post Evacuate Collection Set:
2.4ms
[3196.125s][info ][gc,phases ] GC(3075) Other: 1.0ms
[3196.125s][info ][gc,heap ] GC(3075) Eden regions: 303->0(383)
[3196.125s][info ][gc,heap ] GC(3075) Survivor regions: 3->3(39)
[3196.125s][info ][gc,heap ] GC(3075) Old regions: 1047->1047
[3196.125s][info ][gc,heap ] GC(3075) Humongous regions: 312->312
[3196.125s][info ][gc,metaspace ] GC(3075) Metaspace:
68204K->68204K(1112064K)
[3196.125s][info ][gc ] GC(3075) Pause Young (Normal) (G1
Evacuation Pause) 6656M->5446M(14056M) 20.747ms
[3196.125s][info ][gc,cpu ] GC(3075) User=0.19s Sys=0.05s Real=0.03s
2021-07-23 17:37:37.214 INFO (qtp473053293-138) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.collection1.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.collection1.shard1.replica_n1:QUERY./select.requests&key=solr.core.collection_store_60f9ce1bfbcaec40b56a190e.shard1.replica_n1:QUERY./select.requests&key=solr.core.collection_store_60f97f8f6f56251cd6eda173.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.collection1.shard1.replica_n1:UPDATE./update.requests&key=solr.core.collection_store_60f97f8f6f56251cd6eda173.shard1.replica_n1:UPDATE./update.requests&key=solr.core.collection_store_60f97f9e6f56251cd6eda174.shard1.replica_n1:UPDATE./update.requests&key=solr.core.collection_store_60f97f9e6f56251cd6eda174.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.collection_store_60f9ce1bfbcaec40b56a190e.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.collection_store_60f9ce1bfbcaec40b56a190e.shard1.replica_n1:UPDATE./update.requests&key=solr.core.collection_store_60f97f9e6f56251cd6eda174.shard1.replica_n1:QUERY./select.requests&key=solr.core.collection_store_60f97f8f6f56251cd6eda173.shard1.replica_n1:QUERY./select.requests}
status=0 QTime=41
2021-07-23 17:37:37.217 INFO (qtp473053293-109) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
[3250.713s][info ][gc,start ] GC(3076) Pause Young (Normal) (G1
Evacuation Pause)
[3250.713s][info ][gc,task ] GC(3076) Using 28 workers of 28 for
evacuation
[3250.734s][info ][gc,phases ] GC(3076) Pre Evacuate Collection Set:
6.0ms
[3250.734s][info ][gc,phases ] GC(3076) Evacuate Collection Set: 12.0ms
[3250.734s][info ][gc,phases ] GC(3076) Post Evacuate Collection Set:
2.6ms
[3250.734s][info ][gc,phases ] GC(3076) Other: 1.0ms
[3250.734s][info ][gc,heap ] GC(3076) Eden regions: 383->0(412)
[3250.734s][info ][gc,heap ] GC(3076) Survivor regions: 3->4(49)
[3250.734s][info ][gc,heap ] GC(3076) Old regions: 1047->1047
[3250.734s][info ][gc,heap ] GC(3076) Humongous regions: 312->312
[3250.734s][info ][gc,metaspace ] GC(3076) Metaspace:
68211K->68211K(1112064K)
[3250.734s][info ][gc ] GC(3076) Pause Young (Normal) (G1
Evacuation Pause) 6978M->5447M(14056M) 21.703ms
[3250.734s][info ][gc,cpu ] GC(3076) User=0.22s Sys=0.08s Real=0.03s
2021-07-23 17:38:25.353 INFO (qtp473053293-151)
[c:collection_store_60f97f9e6f56251cd6eda174 s:shard1 r:core_node2
x:collection_store_60f97f9e6f56251cd6eda174_shard1_replica_n1]
o.a.s.u.p.LogUpdateProcessorFactory
[collection_store_60f97f9e6f56251cd6eda174_shard1_replica_n1] webapp=/solr
path=/update
params={wt=javabin&version=2}{add=[60f9a06807d582e9ae3260e560f97f9e6f56251cd6eda174
(1706098064277635072), 60f9a06d07d582e9ae32d7e760f97f9e6f56251cd6eda174
(1706098064282877952), 60f9a07307d582e9ae3377cb60f97f9e6f56251cd6eda174
(1706098064283926528), 60f9a07407d582e9ae3383b460f97f9e6f56251cd6eda174
(1706098064284975104), 60f9a07407d582e9ae3383fc60f97f9e6f56251cd6eda174
(1706098064286023680), 60f9a07407d582e9ae33874460f97f9e6f56251cd6eda174
(1706098064287072256), 60f9a07507d582e9ae33a27660f97f9e6f56251cd6eda174
(1706098064288120832), 60f9a07507d582e9ae33a2c560f97f9e6f56251cd6eda174
(1706098064289169408), 60f9a07507d582e9ae33a2f960f97f9e6f56251cd6eda174
(1706098064290217984), 60f9a07507d582e9ae33a74c60f97f9e6f56251cd6eda174
(1706098064291266560), ... (100 adds)]} 0 183
