Hi Scott Great to know TS is appreciated :)
I don’t think I’ve come across this issue before… and a search of the Sphinx forum doesn’t turn up any likely settings either. The default is to accept as many connections as possible <http://sphinxsearch.com/docs/current.html#conf-max-children>, and with TS’s connection pooling, I think it should create a connection per thread - and it’ll likely hold onto each of those connections while those threads are present. Each connection, in turn, creates a searchd child process - so perhaps you’re running out of resources on your machine to handle a large number of children processes? A potential way around this is to turn off TS’s persistent connections: thus, each searchd child process will be created as needed, but then released again: ThinkingSphinx::Connection.persistent = false Now, this means that every single search request is going to have the TCP overhead of setting up the connection to Sphinx. If this situation where you’re updating all of your items is done via a rake task, perhaps it’s worth only trying that setting within the task? Also: TS should generally be threadsafe - though that probably doesn’t apply to the persistent setting, as that’s expected to be set once and not touched after that. Hope this helps! — Pat > On 16 Jul 2016, at 6:07 PM, [email protected] wrote: > > Hi, > > To start I wanted to thank Pat for Thinking Sphinx - wonderful bit of open > source. > > I have a fairly large pool of items - 2387868 - that are each processed by 1 > or more queries. Given that each query takes about .1 seconds to run I'm > using a threaded approach to run them (thread pool style architecture with > each item being popped off a redis queue). > > What I'm seeing is a large number of ThinkingSphinx::SphinxError: Too many > connections. Here's a full stack trace: > > SELECT * FROM `unique_link_core` WHERE MATCH('@site_id 37 @url_no_www > *\"debit ATM\"*') AND `sphinx_deleted` = 0 LIMIT 0, 20 > ThinkingSphinx::SphinxError: Too many connections - SELECT * FROM > `unique_link_core` WHERE MATCH('@site_id 37 @url_no_www *\"debit ATM\"*') AND > `sphinx_deleted` = 0 LIMIT 0, 20; SHOW META > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/connection.rb:93:in > `rescue in query' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/connection.rb:96:in > `query' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/connection.rb:77:in > `query_all' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/search/batch_inquirer.rb:17:in > `block in results' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/connection.rb:39:in > `block in take' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/innertube-1.1.0/lib/innertube.rb:138:in > `take' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/connection.rb:37:in > `take' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/search/batch_inquirer.rb:16:in > `results' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/middlewares/inquirer.rb:9:in > `block in call' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/activesupport-4.1.4/lib/active_support/notifications.rb:159:in > `block in instrument' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/activesupport-4.1.4/lib/active_support/notifications/instrumenter.rb:20:in > `instrument' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/activesupport-4.1.4/lib/active_support/notifications.rb:159:in > `instrument' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/logger.rb:3:in > `log' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/middlewares/inquirer.rb:8:in > `call' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/middlewares/geographer.rb:11:in > `call' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/middlewares/sphinxql.rb:14:in > `call' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/middleware-0.1.0/lib/middleware/runner.rb:31:in > `call' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/middleware-0.1.0/lib/middleware/builder.rb:102:in > `call' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/search.rb:77:in > `populate' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/search.rb:56:in > `meta' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/masks/pagination_mask.rb:43:in > `total_entries' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/search.rb:124:in > `block in method_missing' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/search.rb:123:in > `each' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/search.rb:123:in > `method_missing' > from > /home/banks/srv/shared/bundle/ruby/1.9.1/gems/thinking-sphinx-3.2.0/lib/thinking_sphinx/active_record/base.rb:24:in > `search_count' > > My db connection pool is set to 1000 in database.yml but I haven't found any > option for setting connections for sphinx itself. Is there some option I > missed? > > I'm only running with 10 threads at present (down from 20 when I first saw > this). > > Any thoughts? Is Thinking Sphinx threadsafe or do I have a configuration > setting wrong perhaps ? > > Appreciated. I'm glad to see your recent trip the states was enjoyable. > > Scott > > -- > You received this message because you are subscribed to the Google Groups > "Thinking Sphinx" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/thinking-sphinx > <https://groups.google.com/group/thinking-sphinx>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/thinking-sphinx. For more options, visit https://groups.google.com/d/optout.
