Re: Telling HC to make new connections all the time
On Wed, Jun 22, 2016 at 11:26 AM, Oleg Kalnichevskiwrote: > On Wed, 2016-06-22 at 07:07 -0400, Benson Margulies wrote: >> On Wed, Jun 22, 2016 at 3:05 AM, Stefan Magnus Landrø >> wrote: >> > Hi Benson, >> > >> > Have you considered implementing a custom >> > https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/DnsResolver.html >> > that handles DNS changes in a better way than the default implementation? >> > You could probably use http://www.xbill.org/dnsjava/ and lookup names >> > directly with the amazon NS? >> >> Thanks for reminding me to ask about that. Does HC call the resolver >> as part of deciding whether to keep a connection or not? > > No, it does not, but one could make it do so by using a custom > ConnectionReuseStrategy. The javadoc for ConnectionReuseStrategy implies that it is called at the end of the request. Am I misreading it? > > Oleg > >> The problem >> as we see it is this: the DNS mapping for an ELB changes, and a minute >> later the connections to it become useless but don't necessarily get >> properly closed with a FIN by the ELB. I've written some goofy code >> that notices when the address resolution changes for any of the ELB's >> we care about, and when it does it calls 'closeIdleConnections' with a >> time of 0. On the other hand, I don't think we're currently having any >> problems getting InetAddress.getByName to keep up with Amazon, but >> your suggestion would be safer. >> >> >> > >> > Stefan >> > >> > 2016-06-20 19:57 GMT+02:00 Benson Margulies : >> > >> >> We use the async http components library to connect various components >> >> in a system that is deployed, in some cases, on AWS. When it's >> >> deployed on AWS, there are ELBs involved. >> >> >> >> Like some other people, we've having various trauma dealing with the >> >> very dynamic environment of AWS ELBs, which includes constant DNS >> >> changes and some other issues. So far, we've been making our HC >> >> configuration more and more complex. I'm wondering if we should back >> >> off and make it simpler -- make a new connection for each request. >> >> Obviously, this will be somewhat slower and more expensive, but at the >> >> moment we might prefer a slower and more expensive approach that works >> >> to one that occasionally gets lost. >> >> >> >> I appreciate that even making new connections all the time won't >> >> compensate for all possible DNS problems -- if Java itself caches a >> >> stale name->address translation, making a new connection will fail >> >> just as effectively as anything else. >> >> >> >> Does anyone out there have advice to share (other than running away from >> >> ELB)? >> >> >> >> - >> >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org >> >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org >> >> >> >> >> > >> > >> > -- >> > BEKK Open >> > http://open.bekk.no >> > >> > TesTcl - a unit test framework for iRules >> > http://testcl.com >> >> - >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org >> > > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Telling HC to make new connections all the time
On Wed, 2016-06-22 at 07:07 -0400, Benson Margulies wrote: > On Wed, Jun 22, 2016 at 3:05 AM, Stefan Magnus Landrø >wrote: > > Hi Benson, > > > > Have you considered implementing a custom > > https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/DnsResolver.html > > that handles DNS changes in a better way than the default implementation? > > You could probably use http://www.xbill.org/dnsjava/ and lookup names > > directly with the amazon NS? > > Thanks for reminding me to ask about that. Does HC call the resolver > as part of deciding whether to keep a connection or not? No, it does not, but one could make it do so by using a custom ConnectionReuseStrategy. Oleg > The problem > as we see it is this: the DNS mapping for an ELB changes, and a minute > later the connections to it become useless but don't necessarily get > properly closed with a FIN by the ELB. I've written some goofy code > that notices when the address resolution changes for any of the ELB's > we care about, and when it does it calls 'closeIdleConnections' with a > time of 0. On the other hand, I don't think we're currently having any > problems getting InetAddress.getByName to keep up with Amazon, but > your suggestion would be safer. > > > > > > Stefan > > > > 2016-06-20 19:57 GMT+02:00 Benson Margulies : > > > >> We use the async http components library to connect various components > >> in a system that is deployed, in some cases, on AWS. When it's > >> deployed on AWS, there are ELBs involved. > >> > >> Like some other people, we've having various trauma dealing with the > >> very dynamic environment of AWS ELBs, which includes constant DNS > >> changes and some other issues. So far, we've been making our HC > >> configuration more and more complex. I'm wondering if we should back > >> off and make it simpler -- make a new connection for each request. > >> Obviously, this will be somewhat slower and more expensive, but at the > >> moment we might prefer a slower and more expensive approach that works > >> to one that occasionally gets lost. > >> > >> I appreciate that even making new connections all the time won't > >> compensate for all possible DNS problems -- if Java itself caches a > >> stale name->address translation, making a new connection will fail > >> just as effectively as anything else. > >> > >> Does anyone out there have advice to share (other than running away from > >> ELB)? > >> > >> - > >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org > >> > >> > > > > > > -- > > BEKK Open > > http://open.bekk.no > > > > TesTcl - a unit test framework for iRules > > http://testcl.com > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Telling HC to make new connections all the time
On Wed, Jun 22, 2016 at 3:05 AM, Stefan Magnus Landrøwrote: > Hi Benson, > > Have you considered implementing a custom > https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/DnsResolver.html > that handles DNS changes in a better way than the default implementation? > You could probably use http://www.xbill.org/dnsjava/ and lookup names > directly with the amazon NS? Thanks for reminding me to ask about that. Does HC call the resolver as part of deciding whether to keep a connection or not? The problem as we see it is this: the DNS mapping for an ELB changes, and a minute later the connections to it become useless but don't necessarily get properly closed with a FIN by the ELB. I've written some goofy code that notices when the address resolution changes for any of the ELB's we care about, and when it does it calls 'closeIdleConnections' with a time of 0. On the other hand, I don't think we're currently having any problems getting InetAddress.getByName to keep up with Amazon, but your suggestion would be safer. > > Stefan > > 2016-06-20 19:57 GMT+02:00 Benson Margulies : > >> We use the async http components library to connect various components >> in a system that is deployed, in some cases, on AWS. When it's >> deployed on AWS, there are ELBs involved. >> >> Like some other people, we've having various trauma dealing with the >> very dynamic environment of AWS ELBs, which includes constant DNS >> changes and some other issues. So far, we've been making our HC >> configuration more and more complex. I'm wondering if we should back >> off and make it simpler -- make a new connection for each request. >> Obviously, this will be somewhat slower and more expensive, but at the >> moment we might prefer a slower and more expensive approach that works >> to one that occasionally gets lost. >> >> I appreciate that even making new connections all the time won't >> compensate for all possible DNS problems -- if Java itself caches a >> stale name->address translation, making a new connection will fail >> just as effectively as anything else. >> >> Does anyone out there have advice to share (other than running away from >> ELB)? >> >> - >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org >> >> > > > -- > BEKK Open > http://open.bekk.no > > TesTcl - a unit test framework for iRules > http://testcl.com - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Telling HC to make new connections all the time
Hi Benson, Have you considered implementing a custom https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/DnsResolver.html that handles DNS changes in a better way than the default implementation? You could probably use http://www.xbill.org/dnsjava/ and lookup names directly with the amazon NS? Stefan 2016-06-20 19:57 GMT+02:00 Benson Margulies: > We use the async http components library to connect various components > in a system that is deployed, in some cases, on AWS. When it's > deployed on AWS, there are ELBs involved. > > Like some other people, we've having various trauma dealing with the > very dynamic environment of AWS ELBs, which includes constant DNS > changes and some other issues. So far, we've been making our HC > configuration more and more complex. I'm wondering if we should back > off and make it simpler -- make a new connection for each request. > Obviously, this will be somewhat slower and more expensive, but at the > moment we might prefer a slower and more expensive approach that works > to one that occasionally gets lost. > > I appreciate that even making new connections all the time won't > compensate for all possible DNS problems -- if Java itself caches a > stale name->address translation, making a new connection will fail > just as effectively as anything else. > > Does anyone out there have advice to share (other than running away from > ELB)? > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > -- BEKK Open http://open.bekk.no TesTcl - a unit test framework for iRules http://testcl.com
Re: Telling HC to make new connections all the time
On Tue, 2016-06-21 at 08:28 -0400, Benson Margulies wrote: > Oleg, If I set a TTL, do I also need an IdleConnectionMonitorThread ( > http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html#d5e652 > )? thanks, benson. > TTL setting only ensures connections cannot get re-used beyond their TTL but they can still be kept alive in the pool until the pool manager discards them either while trying to find a valid connection or while evicting expired ones. One might still need IdleConnectionMonitorThread to ensure eviction of expired connections when the connection manager becomes idle after a period of heavy activity. Oleg > > On Tue, Jun 21, 2016 at 4:41 AM, Oleg Kalnichevskiwrote: > > On Mon, 2016-06-20 at 13:57 -0400, Benson Margulies wrote: > >> We use the async http components library to connect various components > >> in a system that is deployed, in some cases, on AWS. When it's > >> deployed on AWS, there are ELBs involved. > >> > >> Like some other people, we've having various trauma dealing with the > >> very dynamic environment of AWS ELBs, which includes constant DNS > >> changes and some other issues. So far, we've been making our HC > >> configuration more and more complex. I'm wondering if we should back > >> off and make it simpler -- make a new connection for each request. > >> Obviously, this will be somewhat slower and more expensive, but at the > >> moment we might prefer a slower and more expensive approach that works > >> to one that occasionally gets lost. > >> > >> I appreciate that even making new connections all the time won't > >> compensate for all possible DNS problems -- if Java itself caches a > >> stale name->address translation, making a new connection will fail > >> just as effectively as anything else. > >> > >> Does anyone out there have advice to share (other than running away from > >> ELB)? > >> > > > > Hi Benson > > > > I personally would consider disabling connection persistence extreme and > > unnecessary. Consider setting the TTL (total time to live) of persistent > > connections to some low value (say, 10 seconds or even less) instead. > > This should improve distribution of connections across the cluster of > > servers behind a load balancer. > > > > Hope it helps > > > > Oleg > > > > > > - > > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Telling HC to make new connections all the time
Oleg, If I set a TTL, do I also need an IdleConnectionMonitorThread ( http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html#d5e652 )? thanks, benson. On Tue, Jun 21, 2016 at 4:41 AM, Oleg Kalnichevskiwrote: > On Mon, 2016-06-20 at 13:57 -0400, Benson Margulies wrote: >> We use the async http components library to connect various components >> in a system that is deployed, in some cases, on AWS. When it's >> deployed on AWS, there are ELBs involved. >> >> Like some other people, we've having various trauma dealing with the >> very dynamic environment of AWS ELBs, which includes constant DNS >> changes and some other issues. So far, we've been making our HC >> configuration more and more complex. I'm wondering if we should back >> off and make it simpler -- make a new connection for each request. >> Obviously, this will be somewhat slower and more expensive, but at the >> moment we might prefer a slower and more expensive approach that works >> to one that occasionally gets lost. >> >> I appreciate that even making new connections all the time won't >> compensate for all possible DNS problems -- if Java itself caches a >> stale name->address translation, making a new connection will fail >> just as effectively as anything else. >> >> Does anyone out there have advice to share (other than running away from >> ELB)? >> > > Hi Benson > > I personally would consider disabling connection persistence extreme and > unnecessary. Consider setting the TTL (total time to live) of persistent > connections to some low value (say, 10 seconds or even less) instead. > This should improve distribution of connections across the cluster of > servers behind a load balancer. > > Hope it helps > > Oleg > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Telling HC to make new connections all the time
On Mon, 2016-06-20 at 13:57 -0400, Benson Margulies wrote: > We use the async http components library to connect various components > in a system that is deployed, in some cases, on AWS. When it's > deployed on AWS, there are ELBs involved. > > Like some other people, we've having various trauma dealing with the > very dynamic environment of AWS ELBs, which includes constant DNS > changes and some other issues. So far, we've been making our HC > configuration more and more complex. I'm wondering if we should back > off and make it simpler -- make a new connection for each request. > Obviously, this will be somewhat slower and more expensive, but at the > moment we might prefer a slower and more expensive approach that works > to one that occasionally gets lost. > > I appreciate that even making new connections all the time won't > compensate for all possible DNS problems -- if Java itself caches a > stale name->address translation, making a new connection will fail > just as effectively as anything else. > > Does anyone out there have advice to share (other than running away from ELB)? > Hi Benson I personally would consider disabling connection persistence extreme and unnecessary. Consider setting the TTL (total time to live) of persistent connections to some low value (say, 10 seconds or even less) instead. This should improve distribution of connections across the cluster of servers behind a load balancer. Hope it helps Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Telling HC to make new connections all the time
We use the async http components library to connect various components in a system that is deployed, in some cases, on AWS. When it's deployed on AWS, there are ELBs involved. Like some other people, we've having various trauma dealing with the very dynamic environment of AWS ELBs, which includes constant DNS changes and some other issues. So far, we've been making our HC configuration more and more complex. I'm wondering if we should back off and make it simpler -- make a new connection for each request. Obviously, this will be somewhat slower and more expensive, but at the moment we might prefer a slower and more expensive approach that works to one that occasionally gets lost. I appreciate that even making new connections all the time won't compensate for all possible DNS problems -- if Java itself caches a stale name->address translation, making a new connection will fail just as effectively as anything else. Does anyone out there have advice to share (other than running away from ELB)? - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org