I see... So I've tested it out and it's a bit of a disaster for my situation. Some search requests are getting dropped and the server that runs sphinx is hitting memory limits. I could, in theory, solve the problem by increasing ram, how ever we have fallback servers...and it would suck to have to increase those as well. I'll do some research on how people handle large sphinx requests...like airbnb.
As for my 2 cents: I never felt that there was that much of an issue with the cost of opening a connection to sphinx for every request. It is very fast for us currently, and shaving 50-80% off something that already only takes milliseconds while trading for more memory usage and less scalability isn't worth for me specifically. It would awesome if you added a configuration option to disable persistent connections! I will buy you a beer! :) > On Aug 10, 2013, at 10:26 PM, Pat Allan <[email protected]> wrote: > > And the benefit is faster query times - the socket isn't being setup/cleaned > up on every single search request. From the quick testing I did back when I > made this change, this made a noticeable difference with Sphinx query times > (no numbers at hand, but I think it was at least a 50% improvement, if not > closer to 80%). > > -- > Pat > >> On 11/08/2013, at 3:15 PM, Ngan Pham wrote: >> >> Hm…I was afraid of that… >> So, we have 9 application servers with 10-20 processes each. At high >> traffic times, we'd be seeing 181 processes for searchd? Won't that blow >> memory up like crazy? Is this normally something that everyone deals with? >> >> Just curious…what are the benefits of persisted connection pools? >> >> As always, thanks for the quick response Pat! >> >> - Ngan >>> On Saturday, August 10, 2013 at 10:11 PM, Pat Allan wrote: >>> >>> Hi Ngan >>> >>> There's not really any documentation around the changes, I'm afraid… but >>> what you're seeing is a process per thread of your app, plus a master >>> daemon process, due to the persisted connection pool. >>> >>> All the logic for this connection pool can be found in >>> ThinkingSphinx::Connection: >>> https://github.com/pat/thinking-sphinx/blob/v2/lib/thinking_sphinx/connection.rb >>> >>> The HISTORY file has a list of all relevant changes though - here's the >>> list of what's changed between 2.0.14 and 2.1.0 - but this and some delta >>> refactoring are the biggest items: >>> >>> * Removed plugin support - Thinking Sphinx is now gem-only across all >>> branches. >>> * ThinkingSphinx::Version and the thinking_sphinx:version task have been >>> removed - it's a gem, it has a version number. >>> * Updating Riddle to 1.5.6 or newer. >>> * Requires ActiveRecord ~> 2.1 for TS 1.x releases (earlier versions were >>> considered unsupported a few releases ago). >>> * Allow custom Riddle controllers - useful for Flying Sphinx to take over >>> management of Sphinx daemon/indexing actions. >>> * Rejigged delta support to be generic, with local job classes that provide >>> a clean, simple interface for third-party libraries. >>> * Add hooks for anything that needs to happen before indexing (such as >>> clearing out existing delta jobs). >>> * Connection pool for all Sphinx client communication, with new connections >>> built if there's any connection-related (as opposed to syntax) issues. >>> * Multiple-field search conditions can be done with arrays of field names >>> as keys in the :conditions hash (Alex Dowad). >>> * Removed named capture in regular expressions to maintain MRI 1.8 support >>> (Michael Wintrant). >>> * Support new JDBC configuration style (Kyle Stevens). >>> >>> -- >>> Pat >>> >>>> On 11/08/2013, at 2:39 PM, Ngan wrote: >>>> >>>> Hi Pat, >>>> >>>> Thanks for the help. I tried upgrading to 2.1.0...and I'm noticing >>>> multiple instance of searched running now. Is that normal? Would you be >>>> able to point me to documentation of major changes with 2.1.0? >>>> >>>> Thanks, >>>> Ngan >>>> >>>> On Friday, August 9, 2013 7:57:51 AM UTC-7, Pat Allan wrote: >>>> Hi Ngan >>>> >>>> In 2.1.0 there's been some patches that deal with these kinds of errors - >>>> TS will now retry searches if an error crops up on the client connection >>>> (which is also persisted per thread in a connection pool, saving socket >>>> setup/teardown time). If an error crops up, a new connection is made, but >>>> if the error persists, it'll still get raised… >>>> >>>> Give 2.1.0 a spin, see if that helps matters. >>>> >>>> -- >>>> Pat >>>> >>>>> On 09/08/2013, at 10:46 PM, Ngan wrote: >>>>> >>>>> Forgot to add the trace: >>>>> >>>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:438:in `block >>>>> in populate' >>>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:606:in `call' >>>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:606:in >>>>> `retry_on_stale_index' >>>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:426:in >>>>> `populate' >>>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:104:in `to_a' >>>>> >>>>> >>>>> >>>>> On Friday, August 9, 2013 5:45:55 AM UTC-7, Ngan wrote: >>>>> Hi, >>>>> >>>>> We reindex our entire index pretty often (once every 3 minutes) because >>>>> we have a pretty small data collection and we don't want to use delayed >>>>> delta. I notice however, that every once in a while, when we deploy our >>>>> application and it happens to be the same time the reindexing is about to >>>>> rotate, we'll get this error "no enabled local indexes to search" every >>>>> time we hit sphinx there afterwards. When this happens, we have to >>>>> restart our app so that it picks up the new indexes. We are reindexing >>>>> with rotate so the existing index should still be there and the rotation >>>>> should be seamless. Any ideas on why this happens? And if there's >>>>> anything to do about it? >>>>> >>>>> Thanks, >>>>> Ngan >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Thinking Sphinx" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at http://groups.google.com/group/thinking-sphinx. >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Thinking Sphinx" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/thinking-sphinx. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Thinking Sphinx" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/thinking-sphinx/rPsXVX8wlHU/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/thinking-sphinx. >>> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Thinking Sphinx" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/thinking-sphinx. >> For more options, visit https://groups.google.com/groups/opt_out. > > -- > You received this message because you are subscribed to a topic in the Google > Groups "Thinking Sphinx" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/thinking-sphinx/rPsXVX8wlHU/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/thinking-sphinx. > For more options, visit https://groups.google.com/groups/opt_out. > > -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/thinking-sphinx. For more options, visit https://groups.google.com/groups/opt_out.
