Re: Distributed search and haproxy and connection build up
Have you played around with the option httpclose or the option forceclose configuration options in HAProxy (both documented here: http://haproxy.1wt.eu/download/1.3/doc/configuration.txt)? -Tim On Wed, Feb 10, 2010 at 10:05 AM, Ian Connor ian.con...@gmail.com wrote: Thanks, I bypassed haproxy as a test and it did reduce the number of connections - but it did not seem as those these connections were hurting anything. Ian. On Tue, Feb 9, 2010 at 11:01 PM, Lance Norskog goks...@gmail.com wrote: This goes through the Apache Commons HTTP client library: http://hc.apache.org/httpclient-3.x/ We used 'balance' at another project and did not have any problems. On Tue, Feb 9, 2010 at 5:54 AM, Ian Connor ian.con...@gmail.com wrote: I have been using distributed search with haproxy but noticed that I am suffering a little from tcp connections building up waiting for the OS level closing/time out: netstat -a ... tcp6 1 0 10.0.16.170%34654:53789 10.0.16.181%363574:8893 CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:43932 10.0.16.181%363574:8890 CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:43190 10.0.16.181%363574:8895 CLOSE_WAIT tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:53770 TIME_WAIT tcp6 1 0 10.0.16.170%34654:41782 10.0.16.181%363574: CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:52169 10.0.16.181%363574:8890 CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:55947 10.0.16.181%363574:8887 CLOSE_WAIT tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:54040 TIME_WAIT tcp6 1 0 10.0.16.170%34654:40030 10.0.16.160%363574:8984 CLOSE_WAIT ... Digging a little into the haproxy documentation, it seems that they do not support persistent connections. Does solr normally persist the connections between shards (would this problem happen even without haproxy)? Ian. -- Lance Norskog goks...@gmail.com -- Regards, Ian Connor
Re: Distributed search and haproxy and connection build up
Not yet - but thanks for the link. I think that the OS also has a timeout that keeps it around even after this event and with heavy traffic I have seen this build up. Having said all this, the performance impact after testing was negligible for us but I thought I would post that haproxy can cause large numbers of connections on a busy site. Going directly to shards does cut the number of connections down a lot if someone else finds this to be a problem. I am looking forward to distribution under 1.5 where the | option allows redundancy in the request. This will solve the persistence problem while still allowing failover for the shard requests. Even after 1.5, I would then still advocate haproxy between ruby (or your http stack) and solr. It is just when Solr is sharding the request it can keep its connections open and save some resources here. Ian. On Thu, Feb 11, 2010 at 11:49 AM, Tim Underwood timunderw...@gmail.comwrote: Have you played around with the option httpclose or the option forceclose configuration options in HAProxy (both documented here: http://haproxy.1wt.eu/download/1.3/doc/configuration.txt)? -Tim On Wed, Feb 10, 2010 at 10:05 AM, Ian Connor ian.con...@gmail.com wrote: Thanks, I bypassed haproxy as a test and it did reduce the number of connections - but it did not seem as those these connections were hurting anything. Ian. On Tue, Feb 9, 2010 at 11:01 PM, Lance Norskog goks...@gmail.com wrote: This goes through the Apache Commons HTTP client library: http://hc.apache.org/httpclient-3.x/ We used 'balance' at another project and did not have any problems. On Tue, Feb 9, 2010 at 5:54 AM, Ian Connor ian.con...@gmail.com wrote: I have been using distributed search with haproxy but noticed that I am suffering a little from tcp connections building up waiting for the OS level closing/time out: netstat -a ... tcp6 1 0 10.0.16.170%34654:53789 10.0.16.181%363574:8893 CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:43932 10.0.16.181%363574:8890 CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:43190 10.0.16.181%363574:8895 CLOSE_WAIT tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:53770 TIME_WAIT tcp6 1 0 10.0.16.170%34654:41782 10.0.16.181%363574: CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:52169 10.0.16.181%363574:8890 CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:55947 10.0.16.181%363574:8887 CLOSE_WAIT tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:54040 TIME_WAIT tcp6 1 0 10.0.16.170%34654:40030 10.0.16.160%363574:8984 CLOSE_WAIT ... Digging a little into the haproxy documentation, it seems that they do not support persistent connections. Does solr normally persist the connections between shards (would this problem happen even without haproxy)? Ian. -- Lance Norskog goks...@gmail.com -- Regards, Ian Connor
Re: Distributed search and haproxy and connection build up
Thanks, I bypassed haproxy as a test and it did reduce the number of connections - but it did not seem as those these connections were hurting anything. Ian. On Tue, Feb 9, 2010 at 11:01 PM, Lance Norskog goks...@gmail.com wrote: This goes through the Apache Commons HTTP client library: http://hc.apache.org/httpclient-3.x/ We used 'balance' at another project and did not have any problems. On Tue, Feb 9, 2010 at 5:54 AM, Ian Connor ian.con...@gmail.com wrote: I have been using distributed search with haproxy but noticed that I am suffering a little from tcp connections building up waiting for the OS level closing/time out: netstat -a ... tcp6 1 0 10.0.16.170%34654:53789 10.0.16.181%363574:8893 CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:43932 10.0.16.181%363574:8890 CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:43190 10.0.16.181%363574:8895 CLOSE_WAIT tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:53770 TIME_WAIT tcp6 1 0 10.0.16.170%34654:41782 10.0.16.181%363574: CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:52169 10.0.16.181%363574:8890 CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:55947 10.0.16.181%363574:8887 CLOSE_WAIT tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:54040 TIME_WAIT tcp6 1 0 10.0.16.170%34654:40030 10.0.16.160%363574:8984 CLOSE_WAIT ... Digging a little into the haproxy documentation, it seems that they do not support persistent connections. Does solr normally persist the connections between shards (would this problem happen even without haproxy)? Ian. -- Lance Norskog goks...@gmail.com -- Regards, Ian Connor
Distributed search and haproxy and connection build up
I have been using distributed search with haproxy but noticed that I am suffering a little from tcp connections building up waiting for the OS level closing/time out: netstat -a ... tcp6 1 0 10.0.16.170%34654:53789 10.0.16.181%363574:8893 CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:43932 10.0.16.181%363574:8890 CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:43190 10.0.16.181%363574:8895 CLOSE_WAIT tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:53770 TIME_WAIT tcp6 1 0 10.0.16.170%34654:41782 10.0.16.181%363574: CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:52169 10.0.16.181%363574:8890 CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:55947 10.0.16.181%363574:8887 CLOSE_WAIT tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:54040 TIME_WAIT tcp6 1 0 10.0.16.170%34654:40030 10.0.16.160%363574:8984 CLOSE_WAIT ... Digging a little into the haproxy documentation, it seems that they do not support persistent connections. Does solr normally persist the connections between shards (would this problem happen even without haproxy)? Ian.
Re: Distributed search and haproxy and connection build up
This goes through the Apache Commons HTTP client library: http://hc.apache.org/httpclient-3.x/ We used 'balance' at another project and did not have any problems. On Tue, Feb 9, 2010 at 5:54 AM, Ian Connor ian.con...@gmail.com wrote: I have been using distributed search with haproxy but noticed that I am suffering a little from tcp connections building up waiting for the OS level closing/time out: netstat -a ... tcp6 1 0 10.0.16.170%34654:53789 10.0.16.181%363574:8893 CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:43932 10.0.16.181%363574:8890 CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:43190 10.0.16.181%363574:8895 CLOSE_WAIT tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:53770 TIME_WAIT tcp6 1 0 10.0.16.170%34654:41782 10.0.16.181%363574: CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:52169 10.0.16.181%363574:8890 CLOSE_WAIT tcp6 1 0 10.0.16.170%34654:55947 10.0.16.181%363574:8887 CLOSE_WAIT tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:54040 TIME_WAIT tcp6 1 0 10.0.16.170%34654:40030 10.0.16.160%363574:8984 CLOSE_WAIT ... Digging a little into the haproxy documentation, it seems that they do not support persistent connections. Does solr normally persist the connections between shards (would this problem happen even without haproxy)? Ian. -- Lance Norskog goks...@gmail.com
