[jira] [Commented] (SOLR-9521) Warm first searcher with queries from log file
[
https://issues.apache.org/jira/browse/SOLR-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496817#comment-15496817
]
Russell Black commented on SOLR-9521:
-
Our production system running a 3.6 solr cluster does something similar. The
first searcher warms itself by using the cache from another running server in
the cluster. It works by providing a cache implementation that keeps track of
the query params for each cache entry:
{code:xml}
{code}
Then there is a request handler that serves up the top search queries from the
cache:
{code:xml}
{code}
And a firstSearcher listener that fetches the cache queries from the cluster's
{{/remoteautowarm}} handler.
{code:xml}
http://10.100.25.25/solr/remoteautowarm
static firstSearcher warming in solrconfig.xml
{code}
I have been using this code in production on a 3.6 cluster for several years.
If you are interested, I can share the code.
> Warm first searcher with queries from log file
> --
>
> Key: SOLR-9521
> URL: https://issues.apache.org/jira/browse/SOLR-9521
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 6.2
>Reporter: Russell Black
>Priority: Minor
> Labels: performance
> Attachments: LogFileWarmer.java
>
>
> The only way I know of to warm the first searcher before it is opened, is to
> hard code warming queries in
> {code:xml} class="solr.QuerySenderListener">{code}
> For our application, hard-coding is not a good solution since some of our
> more expensive filter queries change periodically.
> I have created a plugin that warms the first searcher from the most recent
> entries in the log files. In my opinion this capability ought to be part of
> solr, so I'm contributing my code in the hopes that it can be incorporated.
> It works by wrapping an instance of QuerySenderListener, setting its
> "queries" argument from recent queries from the log files. It is configured
> like this:
> {code:xml}
>
>
>
> logs/solr.log
> logs/solr.log.1
> logs/solr.log.2
> logs/solr.log.3
> logs/solr.log.4
> logs/solr.log.5
> logs/solr.log.6
> logs/solr.log.7
> logs/solr.log.8
> logs/solr.log.9
>
>
> 100
>
> 100
>
> path=/select params=\{([^ ]+)\}
>
>
> solrprice asc
> rocksweight asc
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9521) Warm first searcher with queries from log file
[
https://issues.apache.org/jira/browse/SOLR-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496647#comment-15496647
]
Russell Black commented on SOLR-9521:
-
bq. I wonder if we can get rid of specifying the log file names in
solrconfig.xml or at least make them optional.
All the parameters do have default values except for "fallback". See the first
xml comment in the example config in the description. All arguments shown
there are set to their default values.
bq. Please be sure to use the "async" parameter for all collection APIs because
the same shardHandlerFactory is used for all collection admin requests as well
as the internal core admin requests performed as part of the collection APIs.
The short timeouts are only used in the /select SearchHandler:
{code:xml}
...
1500
1500
{code}
I have confirmed with the debugger that the
{{OverseerCollectionMessageHandler}} which appears to service the collections
APIs does not use the same ShardHandlerFactory.
> Warm first searcher with queries from log file
> --
>
> Key: SOLR-9521
> URL: https://issues.apache.org/jira/browse/SOLR-9521
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 6.2
>Reporter: Russell Black
>Priority: Minor
> Labels: performance
> Attachments: LogFileWarmer.java
>
>
> The only way I know of to warm the first searcher before it is opened, is to
> hard code warming queries in
> {code:xml} class="solr.QuerySenderListener">{code}
> For our application, hard-coding is not a good solution since some of our
> more expensive filter queries change periodically.
> I have created a plugin that warms the first searcher from the most recent
> entries in the log files. In my opinion this capability ought to be part of
> solr, so I'm contributing my code in the hopes that it can be incorporated.
> It works by wrapping an instance of QuerySenderListener, setting its
> "queries" argument from recent queries from the log files. It is configured
> like this:
> {code:xml}
>
>
>
> logs/solr.log
> logs/solr.log.1
> logs/solr.log.2
> logs/solr.log.3
> logs/solr.log.4
> logs/solr.log.5
> logs/solr.log.6
> logs/solr.log.7
> logs/solr.log.8
> logs/solr.log.9
>
>
> 100
>
> 100
>
> path=/select params=\{([^ ]+)\}
>
>
> solrprice asc
> rocksweight asc
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9521) Warm first searcher with queries from log file
[
https://issues.apache.org/jira/browse/SOLR-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496221#comment-15496221
]
David Smiley commented on SOLR-9521:
It's exciting to finally see a solution to this problem :-) I had worked on
Endeca years ago and it had this feature.
> Warm first searcher with queries from log file
> --
>
> Key: SOLR-9521
> URL: https://issues.apache.org/jira/browse/SOLR-9521
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 6.2
>Reporter: Russell Black
>Priority: Minor
> Labels: performance
> Attachments: LogFileWarmer.java
>
>
> The only way I know of to warm the first searcher before it is opened, is to
> hard code warming queries in
> {code:xml} class="solr.QuerySenderListener">{code}
> For our application, hard-coding is not a good solution since some of our
> more expensive filter queries change periodically.
> I have created a plugin that warms the first searcher from the most recent
> entries in the log files. In my opinion this capability ought to be part of
> solr, so I'm contributing my code in the hopes that it can be incorporated.
> It works by wrapping an instance of QuerySenderListener, setting its
> "queries" argument from recent queries from the log files. It is configured
> like this:
> {code:xml}
>
>
>
> logs/solr.log
> logs/solr.log.1
> logs/solr.log.2
> logs/solr.log.3
> logs/solr.log.4
> logs/solr.log.5
> logs/solr.log.6
> logs/solr.log.7
> logs/solr.log.8
> logs/solr.log.9
>
>
> 100
>
> 100
>
> path=/select params=\{([^ ]+)\}
>
>
> solrprice asc
> rocksweight asc
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9521) Warm first searcher with queries from log file
[
https://issues.apache.org/jira/browse/SOLR-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15495307#comment-15495307
]
Erick Erickson commented on SOLR-9521:
--
I was wondering if, rather than mine the log files an alternate approach would
be to serialize out the top N filterCache and/or queryResultCache entries and
use those for firstSearcher? That would require the system to be up and running
at least once, but so does mining log files. Plus it "should" be more robust
than creating a regex for mining the log file.
This is just off the top of my head, I haven't thought about the details, e.g.
when to serialize that out. At core close? With a collections API call? As part
of the autowarming when a new searcher is opened?
Throwing this out for discussion, no real opinion either way.
> Warm first searcher with queries from log file
> --
>
> Key: SOLR-9521
> URL: https://issues.apache.org/jira/browse/SOLR-9521
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 6.2
>Reporter: Russell Black
>Priority: Minor
> Labels: performance
> Attachments: LogFileWarmer.java
>
>
> The only way I know of to warm the first searcher before it is opened, is to
> hard code warming queries in
> {code:xml} class="solr.QuerySenderListener">{code}
> For our application, hard-coding is not a good solution since some of our
> more expensive filter queries change periodically.
> I have created a plugin that warms the first searcher from the most recent
> entries in the log files. In my opinion this capability ought to be part of
> solr, so I'm contributing my code in the hopes that it can be incorporated.
> It works by wrapping an instance of QuerySenderListener, setting its
> "queries" argument from recent queries from the log files. It is configured
> like this:
> {code:xml}
>
>
>
> logs/solr.log
> logs/solr.log.1
> logs/solr.log.2
> logs/solr.log.3
> logs/solr.log.4
> logs/solr.log.5
> logs/solr.log.6
> logs/solr.log.7
> logs/solr.log.8
> logs/solr.log.9
>
>
> 100
>
> 100
>
> path=/select params=\{([^ ]+)\}
>
>
> solrprice asc
> rocksweight asc
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9521) Warm first searcher with queries from log file
[
https://issues.apache.org/jira/browse/SOLR-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15495269#comment-15495269
]
Shalin Shekhar Mangar commented on SOLR-9521:
-
Thanks Russel this is very useful!
I wonder if we can get rid of specifying the log file names in solrconfig.xml
or at least make them optional. Now that Solr has got rid of the war and we
control Jetty and logging, it should be possible to automatically detect the
logging directory and use the recent log files.
bq. We have a socketTimeout of 1500 ms on the SolrClient, a socketTimeout of
1000 ms on the shardHandlerFactory
Please be sure to use the "async" parameter for all collection APIs because the
same shardHandlerFactory is used for all collection admin requests as well as
the internal core admin requests performed as part of the collection APIs.
To Shawn's point of optimize -- those use the update shard handler and won't be
affected by HttpShardHandler.
> Warm first searcher with queries from log file
> --
>
> Key: SOLR-9521
> URL: https://issues.apache.org/jira/browse/SOLR-9521
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 6.2
>Reporter: Russell Black
>Priority: Minor
> Labels: performance
> Attachments: LogFileWarmer.java
>
>
> The only way I know of to warm the first searcher before it is opened, is to
> hard code warming queries in
> {code:xml} class="solr.QuerySenderListener">{code}
> For our application, hard-coding is not a good solution since some of our
> more expensive filter queries change periodically.
> I have created a plugin that warms the first searcher from the most recent
> entries in the log files. In my opinion this capability ought to be part of
> solr, so I'm contributing my code in the hopes that it can be incorporated.
> It works by wrapping an instance of QuerySenderListener, setting its
> "queries" argument from recent queries from the log files. It is configured
> like this:
> {code:xml}
>
>
>
> logs/solr.log
> logs/solr.log.1
> logs/solr.log.2
> logs/solr.log.3
> logs/solr.log.4
> logs/solr.log.5
> logs/solr.log.6
> logs/solr.log.7
> logs/solr.log.8
> logs/solr.log.9
>
>
> 100
>
> 100
>
> path=/select params=\{([^ ]+)\}
>
>
> solrprice asc
> rocksweight asc
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9521) Warm first searcher with queries from log file
[
https://issues.apache.org/jira/browse/SOLR-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15494472#comment-15494472
]
Shawn Heisey commented on SOLR-9521:
Those are exceptionally short timeouts. Configuring timeouts that low is
likely to cause issues.
I understand a desire to ensure that no single component can tie things up for
very long ... but I'd go with 30 seconds minimum for the socket timeouts, and
probably a 5-10 second minimum for timeAllowed.
What happens if you make a request that takes *minutes* ... like an optimize?
> Warm first searcher with queries from log file
> --
>
> Key: SOLR-9521
> URL: https://issues.apache.org/jira/browse/SOLR-9521
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 6.2
>Reporter: Russell Black
>Priority: Minor
> Labels: performance
> Attachments: LogFileWarmer.java
>
>
> The only way I know of to warm the first searcher before it is opened, is to
> hard code warming queries in
> {code:xml} class="solr.QuerySenderListener">{code}
> For our application, hard-coding is not a good solution since some of our
> more expensive filter queries change periodically.
> I have created a plugin that warms the first searcher from the most recent
> entries in the log files. In my opinion this capability ought to be part of
> solr, so I'm contributing my code in the hopes that it can be incorporated.
> It works by wrapping an instance of QuerySenderListener, setting its
> "queries" argument from recent queries from the log files. It is configured
> like this:
> {code:xml}
>
>
>
> logs/solr.log
> logs/solr.log.1
> logs/solr.log.2
> logs/solr.log.3
> logs/solr.log.4
> logs/solr.log.5
> logs/solr.log.6
> logs/solr.log.7
> logs/solr.log.8
> logs/solr.log.9
>
>
> 100
>
> 100
>
> path=/select params=\{([^ ]+)\}
>
>
> solrprice asc
> rocksweight asc
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9521) Warm first searcher with queries from log file
[
https://issues.apache.org/jira/browse/SOLR-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1549#comment-1549
]
Russell Black commented on SOLR-9521:
-
We are not restarting often. We have been running a large solr 6 cluster (80
shards) for a few days. We have a socketTimeout of 1500 ms on the SolrClient,
a socketTimeout of 1000 ms on the shardHandlerFactory, and we are sending
timeAllowed=1000 with every search. Average response time is 30 ms. We
restarted solr on all the shards for the first time under a live load
yesterday. We restarted them 3 at a time.
After the restart, the cold firstSearcher immediately started taking traffic.
The load on the servers went way up, compared to its pre-restart load, and all
queries were completing in 1000 ms, presumably with partial results. This high
load continued for 20 minutes until finally I switched our site to point to our
old search system. We were later able to get them back online by gradually
transitioning the traffic back to the new system. (I can control what
percentage of searches are directed to the new solr cluster and change it in
real time).
My belief (and I could be wrong) is that the caches could never get populated
after the restart because queries were being aborted early by the timeAllowed
parameter and caches can't populate with partial results. Hence my desire to
properly warm the first searcher.
> Warm first searcher with queries from log file
> --
>
> Key: SOLR-9521
> URL: https://issues.apache.org/jira/browse/SOLR-9521
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 6.2
>Reporter: Russell Black
>Priority: Minor
> Labels: performance
> Attachments: LogFileWarmer.java
>
>
> The only way I know of to warm the first searcher before it is opened, is to
> hard code warming queries in
> {code:xml} class="solr.QuerySenderListener">{code}
> For our application, hard-coding is not a good solution since some of our
> more expensive filter queries change periodically.
> I have created a plugin that warms the first searcher from the most recent
> entries in the log files. In my opinion this capability ought to be part of
> solr, so I'm contributing my code in the hopes that it can be incorporated.
> It works by wrapping an instance of QuerySenderListener, setting its
> "queries" argument from recent queries from the log files. It is configured
> like this:
> {code:xml}
>
>
>
> logs/solr.log
> logs/solr.log.1
> logs/solr.log.2
> logs/solr.log.3
> logs/solr.log.4
> logs/solr.log.5
> logs/solr.log.6
> logs/solr.log.7
> logs/solr.log.8
> logs/solr.log.9
>
>
> 100
>
> 100
>
> path=/select params=\{([^ ]+)\}
>
>
> solrprice asc
> rocksweight asc
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9521) Warm first searcher with queries from log file
[
https://issues.apache.org/jira/browse/SOLR-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15494133#comment-15494133
]
Erick Erickson commented on SOLR-9521:
--
How often are you restarting Solr and why? Historically firstSearcher hasn't
gotten this kind of attention b/c the assumption is that restarting Solr is
fairly rare. So I'm curious what your processes are that this is noticeable.
Not disagreeing mind you, this looks useful, just wondering at the environment.
> Warm first searcher with queries from log file
> --
>
> Key: SOLR-9521
> URL: https://issues.apache.org/jira/browse/SOLR-9521
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 6.2
>Reporter: Russell Black
>Priority: Minor
> Labels: performance
> Attachments: LogFileWarmer.java
>
>
> The only way I know of to warm the first searcher before it is opened, is to
> hard code warming queries in
> {code:xml} class="solr.QuerySenderListener">{code}
> For our application, hard-coding is not a good solution since some of our
> more expensive filter queries change periodically.
> I have created a plugin that warms the first searcher from the most recent
> entries in the log files. In my opinion this capability ought to be part of
> solr, so I'm contributing my code in the hopes that it can be incorporated.
> It works by wrapping an instance of QuerySenderListener, setting its
> "queries" argument from recent queries from the log files. It is configured
> like this:
> {code:xml}
>
>
>
> logs/solr.log
> logs/solr.log.1
> logs/solr.log.2
> logs/solr.log.3
> logs/solr.log.4
> logs/solr.log.5
> logs/solr.log.6
> logs/solr.log.7
> logs/solr.log.8
> logs/solr.log.9
>
>
> 100
>
> 100
>
> path=/select params=\{([^ ]+)\}
>
>
> solrprice asc
> rocksweight asc
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
