Re: SolrJ optimize method -- not returning immediately when the "wait" options are false
On 11/8/2016 4:27 PM, Yonik Seeley wrote: > So no, you should not expect to see an immediate return with > waitSearcher=false since it only represents the > open-and-register-searcher part. That's too bad. I already have the SolrJ call happening in a separate thread, so it doesn't really affect my *current* program much. For a future version of the program, I have a question: If I have a SolrJ optimize running in a background thread, can I call close() on SolrClient and HttpClient objects (and remove all references to them) while that's happening and have all the objects properly turn into garbage, possibly at a later time when the optimize finishes? The 40+ seconds for the optimizes I have mentioned is the hot shard. I've seen optimizes on a cold shard take well over an hour. Basically, I want to be able to have my program dismantle all the objects (threads, SolrClient, HttpClient, JDBC, etc) related to building an individual index, reload its configuration, and set everything back up again, without stopping the program. Currently I do that by restarting the program entirely, but I want something much more dynamic, with a web interface to control it. I'd like it to be able to do this even when a LONG background optimize is happening. I do have a possible alternate plan: Use a client with a very short socket timeout for optimizes and ignore the exception. I'm worried that I won't be able to tell the difference between a problem with the optimize call and the socket timeout being reached. Thanks, Shawn
Re: SolrJ optimize method -- not returning immediately when the "wait" options are false
https://issues.apache.org/jira/browse/SOLR-2018 There used to be a waitFlush parameter (wait until the IndexWriter has written all the changes) as well as a waitSearcher parameter (wait until a new searcher has been registered... i.e. whatever changes you made will be guaranteed to be visible). The waitFlush parameter was removed because it was never implemented (we always wait until IW has flushed). So no, you should not expect to see an immediate return with waitSearcher=false since it only represents the open-and-register-searcher part. -Yonik On Tue, Nov 8, 2016 at 5:55 PM, Shawn Heiseywrote: > I have this code in my SolrJ program: > > LOG.info("{}: background optimizing", logPrefix); > myOptimizeSolrClient.optimize(myName, false, false); > elapsedMillis = (System.nanoTime() - startNanos) / 100; > LOG.info("{}: Background optimize completed, elapsed={}", logPrefix, > elapsedMillis); > > This is what I get when this code runs. I expected it to return > immediately, but it took 49 seconds: > > INFO - 2016-11-08 15:10:56.316; 409; shard.c.inc.inclive.optimize; > shard.c.inc.inclive: Background optimize completed, elapsed=49339 > > I'm using SolrJ 5.5.3, and the SolrClient object is HttpSolrClient. I > have not tried 6.x versions. The server that this is talking to is > 5.3.2-SNAPSHOT. > > I found this in solr.log: > > 2016-11-08 15:10:56.315 INFO (qtp1164175787-708968) [ x:inclive] > org.apache.solr.update.processor.LogUpdateProcessor [inclive] > webapp=/solr path=/update > params={optimize=true=1=true=javabin=2} > {optimize=} 0 49338 > > It looks like waitSearcher is not being set properly by the SolrJ code. > I could not see any obvious problem in the master branch, which I > realize is not the same as the 5.5 code I'm running. > > I did try the request manually, both with waitSearcher set to true and > to false, and in both cases, the request DID wait until the optimize was > finished before it returned a response. So even if the SolrJ problem is > fixed, Solr itself will not work the way I'm expecting. Is it correct > to expect an immediate return for optimize when waitSearcher is false? > > I am not in a position to try this in 6.x versions. Is there anyone out > there who does have a 6.x index they can try it on, see if it's still a > problem? > > Thanks, > Shawn >
Re: SolrJ optimize method -- not returning immediately when the "wait" options are false
On 11/8/2016 3:55 PM, Shawn Heisey wrote: > I am not in a position to try this in 6.x versions. Is there anyone > out there who does have a 6.x index they can try it on, see if it's > still a problem? I upgraded a dev version of the program to SolrJ 6.2.1 (newest currently available via ivy), the server still received waitSearcher=true, even though my code explicitly said false. Optimizing the index took 46 seconds. It's the server side that I can't try on 6.x without a lot of work. Thanks, Shawn
SolrJ optimize method -- not returning immediately when the "wait" options are false
I have this code in my SolrJ program: LOG.info("{}: background optimizing", logPrefix); myOptimizeSolrClient.optimize(myName, false, false); elapsedMillis = (System.nanoTime() - startNanos) / 100; LOG.info("{}: Background optimize completed, elapsed={}", logPrefix, elapsedMillis); This is what I get when this code runs. I expected it to return immediately, but it took 49 seconds: INFO - 2016-11-08 15:10:56.316; 409; shard.c.inc.inclive.optimize; shard.c.inc.inclive: Background optimize completed, elapsed=49339 I'm using SolrJ 5.5.3, and the SolrClient object is HttpSolrClient. I have not tried 6.x versions. The server that this is talking to is 5.3.2-SNAPSHOT. I found this in solr.log: 2016-11-08 15:10:56.315 INFO (qtp1164175787-708968) [ x:inclive] org.apache.solr.update.processor.LogUpdateProcessor [inclive] webapp=/solr path=/update params={optimize=true=1=true=javabin=2} {optimize=} 0 49338 It looks like waitSearcher is not being set properly by the SolrJ code. I could not see any obvious problem in the master branch, which I realize is not the same as the 5.5 code I'm running. I did try the request manually, both with waitSearcher set to true and to false, and in both cases, the request DID wait until the optimize was finished before it returned a response. So even if the SolrJ problem is fixed, Solr itself will not work the way I'm expecting. Is it correct to expect an immediate return for optimize when waitSearcher is false? I am not in a position to try this in 6.x versions. Is there anyone out there who does have a 6.x index they can try it on, see if it's still a problem? Thanks, Shawn