Hi,
The only thing that I spot is that you use both auto-commit with 900 sec
frequency AND commitWithin. Solr is smart enough to skip empty commits. But
if auto-commit kicks in during the doc add / delete, there will be at least
two commits ongoing. Could you change you Full recovery case to commit
eventually from the client code? Then you won't need the autoCommit section.
On Mon, Mar 7, 2016 at 1:11 PM, Gerald Reinhart <gerald.reinh...@kelkoo.com>
wrote:
>
> Hi,
>
> To give you some context, we are migrating from Solr4 and solr5,
> the client code and the configuration haven't changed but now we are
> facing this problem. We have already checked the commit behaviour
> configuration and it seems good.
>
> Here it is :
>
> Server side, we have 2 collections (main and temp with blue and green
> aliases) :
>
> solrconfig.xml:
>
> <!-- The default high-performance update handler -->
> <updateHandler class="solr.DirectUpdateHandler2">
> (...)
> <autoCommit>
> <maxTime>900000</maxTime>
> <openSearcher>false</openSearcher>
> </autoCommit>
>
> <!-- Soft commit: commented => not used -->
>
> </updateHandler>
>
> Client side, we have 2 different modes:
>
> 1 - Full recovery :
>
> - Delete all documents from the temp collection
> solrClient.deleteByQuery("*:*")
>
> - Add all new documents in temp collection (can be more
> than 5Millions),
> solrClient.add(doc, -1) // commitWithinMs == -1
>
> - Commit when all documents are added
> solrClient.commit(false,false) // waitFlush == false ,
> waitSearcher == false
>
> - Swap blue and green using "create alias" command
>
> - Reload the temp collection to clean the cache. This is
> at this point we have the issue.
>
> 2 - Incremental :
>
> - Add or delete documents from the main collection
> solrClient.add(doc, 1800000) // commitWithin
> == 30 mn
> solrClient.deleteById(doc, 1800000) // commitWithin == 30 mn
>
> Maybe you will spot something obviously wrong ?
>
> Thanks
>
> Gérald and Elodie
>
>
>
>
> On 03/04/2016 12:41 PM, Dmitry Kan wrote:
>
>> Hi,
>>
>> Check the the autoCommit and autoSoftCommit nodes in the solrconfig.xml.
>> Set them to reasonable values. The idea is that if you commit too often,
>> searchers will be warmed up and thrown away. If at any point in time you
>> get overlapping commits, there will be several searchers sitting on the
>> deck.
>>
>> Dmitry
>>
>> On Mon, Feb 29, 2016 at 4:20 PM, Gerald Reinhart <
>> gerald.reinh...@kelkoo.com
>>
>>> wrote:
>>> Hi,
>>>
>>> We are facing an issue during a migration from Solr4 to Solr5.
>>>
>>> Given
>>> - migration from solr 4.10.4 to 5.4.1
>>> - 2 collections
>>> - cloud with one leader and several replicas
>>> - in solrconfig.xml: maxWarmingSearchers=1
>>> - no code change
>>>
>>> When collection reload using /admin/collections using solrj
>>>
>>> Then
>>>
>>> 2016-02-29 13:42:49,011 [http-8080-3] INFO
>>> org.apache.solr.core.CoreContainer:reload:848 - Reloading SolrCore
>>> 'fr_blue' using configuration from collection fr_blue
>>> 2016-02-29 13:42:45,428 [http-8080-6] INFO
>>> org.apache.solr.search.SolrIndexSearcher:<init>:237 - Opening
>>> Searcher@58b65fc[fr_blue] main
>>> (...)
>>> 2016-02-29 13:42:49,077 [http-8080-3] WARN
>>> org.apache.solr.core.SolrCore:getSearcher:1762 - [fr_blue] Error
>>> opening new searcher. exceeded limit of maxWarmingSearchers=1, try again
>>> later.
>>> 2016-02-29 13:42:49,091 [http-8080-3] ERROR
>>> org.apache.solr.handler.RequestHandlerBase:log:139 -
>>> org.apache.solr.common.SolrException: Error handling 'reload' action
>>> at
>>>
>>>
>>> org.apache.solr.handler.admin.CoreAdminHandler.handleReloadAction(CoreAdminHandler.java:770)
>>> at
>>>
>>>
>>> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:230)
>>> at
>>>
>>>
>>> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:184)
>>> at
>>>
>>>
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
>>> at
>>>
>>>
>>> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:664)
>>> at
>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:438)
>>> at
>>>
>>>
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:223)
>>> at
>>>
>>>
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
>>> at
>>>
>>>
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>> at
>>>
>>>
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>> at
>>>
>>>
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>> at
>>>
>>>
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>> at
>>>
>>>
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>> at
>>>
>>>
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>> at
>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
>>> at
>>>
>>>
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>> at
>>>
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>>> at
>>>
>>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
>>> at
>>>
>>>
>>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
>>> at
>>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: org.apache.solr.common.SolrException: Unable to reload core
>>> [fr_blue]
>>> at
>>> org.apache.solr.core.CoreContainer.reload(CoreContainer.java:854)
>>> at
>>>
>>>
>>> org.apache.solr.handler.admin.CoreAdminHandler.handleReloadAction(CoreAdminHandler.java:768)
>>> ... 20 more
>>> Caused by: org.apache.solr.common.SolrException: Error opening new
>>> searcher. exceeded limit of maxWarmingSearchers=1, try again later.
>>> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1764)
>>> at org.apache.solr.core.SolrCore.reload(SolrCore.java:474)
>>> at
>>> org.apache.solr.core.CoreContainer.reload(CoreContainer.java:849)
>>> ... 21 more
>>>
>>>
>>> Thanks
>>>
>>>
>>> Gérald and Elodie
>>>
>>>
>>> Kelkoo SAS
>>> Société par Actions Simplifiée
>>> Au capital de € 4.168.964,30
>>> Siège social : 158 Ter Rue du Temple 75003 Paris
>>> 425 093 069 RCS Paris
>>>
>>> Ce message et les pièces jointes sont confidentiels et établis à
>>> l'attention exclusive de leurs destinataires. Si vous n'êtes pas le
>>> destinataire de ce message, merci de le détruire et d'en avertir
>>> l'expéditeur.
>>>
>>>
>>
>>
>
> --
>
> Kelkoo
>
>
>
> *Gérald Reinhart *Software engineer
>
> *E*gerald.reinh...@kelkoo.com <mailto:steve.con...@kelkoo.com>
> *Y!Messenger*gerald.reinhart
>
> *A*Rue des Méridiens 38130 Echirolles
>
>
>
>
>
>
>
> Kelkoo SAS
> Société par Actions Simplifiée
> Au capital de € 4.168.964,30
> Siège social : 158 Ter Rue du Temple 75003 Paris
> 425 093 069 RCS Paris
>
> Ce message et les pièces jointes sont confidentiels et établis à
> l'attention exclusive de leurs destinataires. Si vous n'êtes pas le
> destinataire de ce message, merci de le détruire et d'en avertir
> l'expéditeur.
>
--
Dmitry Kan
Luke Toolbox: http://github.com/DmitryKey/luke
Blog: http://dmitrykan.blogspot.com
Twitter: http://twitter.com/dmitrykan
SemanticAnalyzer: www.semanticanalyzer.info
