Just trying to think through why the connection pool isn't resetting when it hits this error. Don't suppose you have a stack trace on hand that I could have a look at?
On 17/09/2013, at 12:21 AM, Ngan Pham <[email protected]> wrote: > When it happens, the only way to fix is to restart the app ASAP. > > On Sep 16, 2013, at 3:04 AM, "Pat Allan" <[email protected]> wrote: > >> Hi Ngan >> >> I'm not sure why it's cropped up again - but I'm thinking it's not worth >> worrying about unless it reoccurs a bit more regularly. The error isn't >> lasting for more than half a minute or so, right? Or did it persist for a >> while? >> >> -- >> Pat >> >> On 13/09/2013, at 12:18 AM, Ngan Pham wrote: >> >>> Once, out of 10 deploys. >>> >>> On Sep 12, 2013, at 5:36 AM, "Pat Allan" <[email protected]> wrote: >>> >>>> Hi Ngan >>>> >>>> How often is it recurring now that it's returned? >>>> >>>> -- >>>> Pat >>>> >>>> On 12/09/2013, at 4:44 PM, Ngan wrote: >>>> >>>>> Hey Pat, >>>>> >>>>> Just had the "no enabled local indexes to search" come back again (after >>>>> a while of not seeing it after deploy). Any thoughts on why this is >>>>> still happening? >>>>> >>>>> Thanks, >>>>> Ngan >>>>> >>>>> On Monday, August 12, 2013 9:24:01 AM UTC-7, Ngan wrote: >>>>> You are awesome! Thanks! >>>>> >>>>> I've deployed to production and everything works great. I'll be keeping >>>>> an eye out for my original problem of errors on deployment. >>>>> >>>>> Can't wait for 2.1.1 :-) >>>>> >>>>> On Monday, August 12, 2013 5:08:54 AM UTC-7, Pat Allan wrote: >>>>> The change was pretty simple: >>>>> https://github.com/pat/thinking-sphinx/commit/d5249ea0215128d5d1f916ee42882932ddb86ba7 >>>>> >>>>> >>>>> To use it (for the 2.x releases of Thinking Sphinx): >>>>> >>>>> gem 'thinking-sphinx', '~> 2.1.0', >>>>> :git => 'git://github.com/pat/thinking-sphinx.git', >>>>> :branch => 'v2', >>>>> :ref => '64da4dc7ff' >>>>> >>>>> And then in an initialiser: >>>>> >>>>> ThinkingSphinx.persistence_enabled = false >>>>> >>>>> Will be keen to hear if everything returns to normal for you! >>>>> >>>>> On 12/08/2013, at 12:19 AM, Ngan Pham wrote: >>>>> >>>>> > I see... >>>>> > >>>>> > So I've tested it out and it's a bit of a disaster for my situation. >>>>> > Some search requests are getting dropped and the server that runs >>>>> > sphinx is hitting memory limits. I could, in theory, solve the problem >>>>> > by increasing ram, how ever we have fallback servers...and it would >>>>> > suck to have to increase those as well. I'll do some research on how >>>>> > people handle large sphinx requests...like airbnb. >>>>> > >>>>> > As for my 2 cents: I never felt that there was that much of an issue >>>>> > with the cost of opening a connection to sphinx for every request. It >>>>> > is very fast for us currently, and shaving 50-80% off something that >>>>> > already only takes milliseconds while trading for more memory usage and >>>>> > less scalability isn't worth for me specifically. It would awesome if >>>>> > you added a configuration option to disable persistent connections! I >>>>> > will buy you a beer! :) >>>>> > >>>>> >> On Aug 10, 2013, at 10:26 PM, Pat Allan <[email protected]> >>>>> >> wrote: >>>>> >> >>>>> >> And the benefit is faster query times - the socket isn't being >>>>> >> setup/cleaned up on every single search request. From the quick >>>>> >> testing I did back when I made this change, this made a noticeable >>>>> >> difference with Sphinx query times (no numbers at hand, but I think it >>>>> >> was at least a 50% improvement, if not closer to 80%). >>>>> >> >>>>> >> -- >>>>> >> Pat >>>>> >> >>>>> >>> On 11/08/2013, at 3:15 PM, Ngan Pham wrote: >>>>> >>> >>>>> >>> Hm…I was afraid of that… >>>>> >>> So, we have 9 application servers with 10-20 processes each. At high >>>>> >>> traffic times, we'd be seeing 181 processes for searchd? Won't that >>>>> >>> blow memory up like crazy? Is this normally something that everyone >>>>> >>> deals with? >>>>> >>> >>>>> >>> Just curious…what are the benefits of persisted connection pools? >>>>> >>> >>>>> >>> As always, thanks for the quick response Pat! >>>>> >>> >>>>> >>> - Ngan >>>>> >>>> On Saturday, August 10, 2013 at 10:11 PM, Pat Allan wrote: >>>>> >>>> >>>>> >>>> Hi Ngan >>>>> >>>> >>>>> >>>> There's not really any documentation around the changes, I'm afraid… >>>>> >>>> but what you're seeing is a process per thread of your app, plus a >>>>> >>>> master daemon process, due to the persisted connection pool. >>>>> >>>> >>>>> >>>> All the logic for this connection pool can be found in >>>>> >>>> ThinkingSphinx::Connection: >>>>> >>>> https://github.com/pat/thinking-sphinx/blob/v2/lib/thinking_sphinx/connection.rb >>>>> >>>> >>>>> >>>> >>>>> >>>> The HISTORY file has a list of all relevant changes though - here's >>>>> >>>> the list of what's changed between 2.0.14 and 2.1.0 - but this and >>>>> >>>> some delta refactoring are the biggest items: >>>>> >>>> >>>>> >>>> * Removed plugin support - Thinking Sphinx is now gem-only across >>>>> >>>> all branches. >>>>> >>>> * ThinkingSphinx::Version and the thinking_sphinx:version task have >>>>> >>>> been removed - it's a gem, it has a version number. >>>>> >>>> * Updating Riddle to 1.5.6 or newer. >>>>> >>>> * Requires ActiveRecord ~> 2.1 for TS 1.x releases (earlier versions >>>>> >>>> were considered unsupported a few releases ago). >>>>> >>>> * Allow custom Riddle controllers - useful for Flying Sphinx to take >>>>> >>>> over management of Sphinx daemon/indexing actions. >>>>> >>>> * Rejigged delta support to be generic, with local job classes that >>>>> >>>> provide a clean, simple interface for third-party libraries. >>>>> >>>> * Add hooks for anything that needs to happen before indexing (such >>>>> >>>> as clearing out existing delta jobs). >>>>> >>>> * Connection pool for all Sphinx client communication, with new >>>>> >>>> connections built if there's any connection-related (as opposed to >>>>> >>>> syntax) issues. >>>>> >>>> * Multiple-field search conditions can be done with arrays of field >>>>> >>>> names as keys in the :conditions hash (Alex Dowad). >>>>> >>>> * Removed named capture in regular expressions to maintain MRI 1.8 >>>>> >>>> support (Michael Wintrant). >>>>> >>>> * Support new JDBC configuration style (Kyle Stevens). >>>>> >>>> >>>>> >>>> -- >>>>> >>>> Pat >>>>> >>>> >>>>> >>>>> On 11/08/2013, at 2:39 PM, Ngan wrote: >>>>> >>>>> >>>>> >>>>> Hi Pat, >>>>> >>>>> >>>>> >>>>> Thanks for the help. I tried upgrading to 2.1.0...and I'm noticing >>>>> >>>>> multiple instance of searched running now. Is that normal? Would >>>>> >>>>> you be able to point me to documentation of major changes with >>>>> >>>>> 2.1.0? >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Ngan >>>>> >>>>> >>>>> >>>>> On Friday, August 9, 2013 7:57:51 AM UTC-7, Pat Allan wrote: >>>>> >>>>> Hi Ngan >>>>> >>>>> >>>>> >>>>> In 2.1.0 there's been some patches that deal with these kinds of >>>>> >>>>> errors - TS will now retry searches if an error crops up on the >>>>> >>>>> client connection (which is also persisted per thread in a >>>>> >>>>> connection pool, saving socket setup/teardown time). If an error >>>>> >>>>> crops up, a new connection is made, but if the error persists, >>>>> >>>>> it'll still get raised… >>>>> >>>>> >>>>> >>>>> Give 2.1.0 a spin, see if that helps matters. >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Pat >>>>> >>>>> >>>>> >>>>>> On 09/08/2013, at 10:46 PM, Ngan wrote: >>>>> >>>>>> >>>>> >>>>>> Forgot to add the trace: >>>>> >>>>>> >>>>> >>>>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:438:in >>>>> >>>>>> `block in populate' >>>>> >>>>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:606:in >>>>> >>>>>> `call' >>>>> >>>>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:606:in >>>>> >>>>>> `retry_on_stale_index' >>>>> >>>>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:426:in >>>>> >>>>>> `populate' >>>>> >>>>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:104:in >>>>> >>>>>> `to_a' >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> On Friday, August 9, 2013 5:45:55 AM UTC-7, Ngan wrote: >>>>> >>>>>> Hi, >>>>> >>>>>> >>>>> >>>>>> We reindex our entire index pretty often (once every 3 minutes) >>>>> >>>>>> because we have a pretty small data collection and we don't want >>>>> >>>>>> to use delayed delta. I notice however, that every once in a >>>>> >>>>>> while, when we deploy our application and it happens to be the >>>>> >>>>>> same time the reindexing is about to rotate, we'll get this error >>>>> >>>>>> "no enabled local indexes to search" every time we hit sphinx >>>>> >>>>>> there afterwards. When this happens, we have to restart our app so >>>>> >>>>>> that it picks up the new indexes. We are reindexing with rotate so >>>>> >>>>>> the existing index should still be there and the rotation should >>>>> >>>>>> be seamless. Any ideas on why this happens? And if there's >>>>> >>>>>> anything to do about it? >>>>> >>>>>> >>>>> >>>>>> Thanks, >>>>> >>>>>> Ngan >>>>> >>>>>> >>>>> >>>>>> -- >>>>> >>>>>> You received this message because you are subscribed to the Google >>>>> >>>>>> Groups "Thinking Sphinx" group. >>>>> >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>> >>>>>> send an email to [email protected]. >>>>> >>>>>> To post to this group, send email to [email protected]. >>>>> >>>>>> Visit this group at >>>>> >>>>>> http://groups.google.com/group/thinking-sphinx. >>>>> >>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> You received this message because you are subscribed to the Google >>>>> >>>>> Groups "Thinking Sphinx" group. >>>>> >>>>> To unsubscribe from this group and stop receiving emails from it, >>>>> >>>>> send an email to [email protected]. >>>>> >>>>> To post to this group, send email to [email protected]. >>>>> >>>>> Visit this group at http://groups.google.com/group/thinking-sphinx. >>>>> >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>> >>>>> >>>> -- >>>>> >>>> You received this message because you are subscribed to a topic in >>>>> >>>> the Google Groups "Thinking Sphinx" group. >>>>> >>>> To unsubscribe from this topic, visit >>>>> >>>> https://groups.google.com/d/topic/thinking-sphinx/rPsXVX8wlHU/unsubscribe. >>>>> >>>> >>>>> >>>> To unsubscribe from this group and all its topics, send an email to >>>>> >>>> [email protected]. >>>>> >>>> To post to this group, send email to [email protected]. >>>>> >>>> Visit this group at http://groups.google.com/group/thinking-sphinx. >>>>> >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>> >>>>> >>> >>>>> >>> -- >>>>> >>> You received this message because you are subscribed to the Google >>>>> >>> Groups "Thinking Sphinx" group. >>>>> >>> To unsubscribe from this group and stop receiving emails from it, >>>>> >>> send an email to [email protected]. >>>>> >>> To post to this group, send email to [email protected]. >>>>> >>> Visit this group at http://groups.google.com/group/thinking-sphinx. >>>>> >>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >> >>>>> >> -- >>>>> >> You received this message because you are subscribed to a topic in the >>>>> >> Google Groups "Thinking Sphinx" group. >>>>> >> To unsubscribe from this topic, visit >>>>> >> https://groups.google.com/d/topic/thinking-sphinx/rPsXVX8wlHU/unsubscribe. >>>>> >> >>>>> >> To unsubscribe from this group and all its topics, send an email to >>>>> >> [email protected]. >>>>> >> To post to this group, send email to [email protected]. >>>>> >> Visit this group at http://groups.google.com/group/thinking-sphinx. >>>>> >> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >> >>>>> >> >>>>> > >>>>> > -- >>>>> > You received this message because you are subscribed to the Google >>>>> > Groups "Thinking Sphinx" group. >>>>> > To unsubscribe from this group and stop receiving emails from it, send >>>>> > an email to [email protected]. >>>>> > To post to this group, send email to [email protected]. >>>>> > Visit this group at http://groups.google.com/group/thinking-sphinx. >>>>> > For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Thinking Sphinx" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at http://groups.google.com/group/thinking-sphinx. >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "Thinking Sphinx" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/thinking-sphinx/rPsXVX8wlHU/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/thinking-sphinx. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Thinking Sphinx" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/thinking-sphinx. >>> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Thinking Sphinx" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/thinking-sphinx/rPsXVX8wlHU/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/thinking-sphinx. >> For more options, visit https://groups.google.com/groups/opt_out. > > > -- > You received this message because you are subscribed to the Google Groups > "Thinking Sphinx" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/thinking-sphinx. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/thinking-sphinx. For more options, visit https://groups.google.com/groups/opt_out.
