Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Filip Hanik - Dev wrote: ok, there are two very simple memory friendly ways to do sticky load balancing. And as a matter of fact, this is how some hardware loadbalancers do it. 1. Set a cookie on the clients machine - no server memory to hold a map 2. If the client doesn't accept cookies, do a simple sticky load balancing based on the IP of the client request. Again, no memory map needed. The current jvmRoute addition to JSESSIONID is not really needed, since it doesn't add that much of a benefit over the two options above. So right then and there, there is one less thing to configure. It's cool to have one less thing to configure, but it seems to me jvmRoute is the most reliable and efficient way of doing stickiness (the cookie way is intrusive, and the IP way is highly inaccurate). Rmy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Tim Funk wrote: *Changes to tomcat* Add a proxy mode flag to allow for the X- headers to pass authentication and other variables. Add to the manager(?) app and method to expose all the URL spaces availble. Minor changes to fix getRemoteAddr() to show the client, not the apache server. Pros - Simple(?) and not tomcat specific. Cons - Duplicate (re)parsing of headers, probably higher latency than jk. The changes would have to be simple, and non intrusive performance wise. Otherwise, I'm going to prefer AJP ;) Speaking about performance, did anyone do a comparison of mod_proxy against mod_jk to see how good/bad it is ? This is really important information IMO, and I don't see how a decision can be made without it. Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Filip Hanik wrote: really, so then there is no fail over. Yes there are. In Jk default is to fail over always. The JK2 has a routeRedirect to handle such cases (but not strictly). If the routeRedirect is down it will still fail over, which is probably incorrect. cause that is what fail over does, redirects you to another server. and with session replication in place, you should be good to go Sure if you've set up many-to-many session replication. We should have something like: If we have a session route but the worker is down If there is a routeRedirect then If the routeRedirect worker is down return 500 Else return routeRedirect Else Fail over to another worker Else return sessionWorker Basically it means that if the redirection worker is down don't fail over, but rather break the transaction. It enables single session replication node. I don't care if the fail over will be turned on by default or not, but would like to have a control in cases where either there are: A) session replication is in place on all nodes B) session replication is done on a single node C) there is no session replication at all Like said the JK presumes there is A, JK2 handles both A and B cases (but it should C too). MT. smime.p7s Description: S/MIME cryptographic signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Mladen Turk wrote: Filip Hanik wrote: really, so then there is no fail over. Yes there are. In Jk default is to fail over always. The JK2 has a routeRedirect to handle such cases (but not strictly). If the routeRedirect is down it will still fail over, which is probably incorrect. cause that is what fail over does, redirects you to another server. and with session replication in place, you should be good to go Sure if you've set up many-to-many session replication. We should have something like: If we have a session route but the worker is down If there is a routeRedirect then If the routeRedirect worker is down return 500 Else return routeRedirect Else Fail over to another worker Else return sessionWorker Basically it means that if the redirection worker is down don't fail over, but rather break the transaction. It enables single session replication node. I don't care if the fail over will be turned on by default or not, but would like to have a control in cases where either there are: A) session replication is in place on all nodes B) session replication is done on a single node C) there is no session replication at all Like said the JK presumes there is A, JK2 handles both A and B cases (but it should C too). MT. I am looking to get the sessionid: - By reading request_rec-unparsed_uri or request_rec-uri. - By reading the cooky from request_rec-headers_in. How do I read the sessionid in the response? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Remy Maucherat wrote: Speaking about performance, did anyone do a comparison of mod_proxy against mod_jk to see how good/bad it is ? This is really important information IMO, and I don't see how a decision can be made without it. Results a quite impressive, good question Remy :) Tomcat 5.0.27 + Apache 2.0.50 All tests done on a clean startup with empty logs. ProxyPass /servlets-examples/ http://localhost:8080/servlets-examples/ ProxyPassReverse /servlets-examples/ http://localhost:8080/servlets-examples/ Ab -n 1000 Time taken for tests: 4.226077 seconds Mod_jk-1.26 JkWorkersFile conf/workers.properties JkMount /servlets-examples/* ajp13 Time taken for tests: 2.824061 seconds A lot, lot faster. What is more interesting is when you increase the number of loops forom 1000 to 1 Ab -n 1 Time taken for tests: 239.614549 seconds Complete requests: 1 Failed requests:7011 Using mod_jk: Ab -n 1 Time taken for tests: 21.811363 seconds Complete requests: 1 Failed requests:0 So, mod_proxy is a lot slower and doesn't handle load. (Perhaps increasing http listeners on TC would help). MT. smime.p7s Description: S/MIME cryptographic signature
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
In previous mail I forgot one crucial result: Ab -n 1000 directly to TC Time taken for tests: 1.882708 seconds Ab -n 1 directly to TC Time taken for tests: 17.244797 seconds I'll leave the calculation to others. MT. smime.p7s Description: S/MIME cryptographic signature
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
jean-frederic clere wrote: I am looking to get the sessionid: - By reading request_rec-unparsed_uri or request_rec-uri. - By reading the cooky from request_rec-headers_in. How do I read the sessionid in the response? Good point. We'll need something like mod_proxy_html or something like that to parse the headers returned from Remote. Cannot speak for sure, cause I still have some problems figuring out the mod_proxy algorithm. It's getting more and more complicated :). MT. smime.p7s Description: S/MIME cryptographic signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Remy Maucherat wrote, On 7/21/2004 11:33 PM: The changes would have to be simple, and non intrusive performance wise. Otherwise, I'm going to prefer AJP ;) Speaking about performance, did anyone do a comparison of mod_proxy against mod_jk to see how good/bad it is ? This is really important information IMO, and I don't see how a decision can be made without it. I did a couple quick tests using a few different sized static documents using ab, Tomcat 5.0.27, Apache 2.0.50 both on the same single CPU machine. Depending on the document size, mod_proxy was anywhere from 50% (very small document of a few bytes) to 10% (30k document) slower than mod_jk. By comparison, going directly to Tomcat about twice as fast as using mod_jk, and going directly to Apache is another 30% faster than that. Watching CPU utilization when switching between mod_jk and mod_proxy showed that Tomcat uses a lot more CPU when using mod_proxy than when using mod_jk. So while the performance hit isn't negligible, it's not too bad. It does show that it will be worthwhile to a proxy_ajp_module, but at the same time I think that using plain http provides more than enough performance for the majority of users out there, and you can't argue with the fact that the work will benefit a lot more than just Tomcat and lessen development time. I could see people using the proposed load-balancing functionality along with mod_proxy to balance requests to clusters of many other application servers (PHP, perl, CGI, etc) as well. -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Ab -n 1 Time taken for tests: 239.614549 seconds Complete requests: 1 Failed requests:7011 So, mod_proxy is a lot slower and doesn't handle load. (Perhaps increasing http listeners on TC would help). Increasing the maxThreads to 350 and acceptCount to 300 doesn't solve the issue. On large load I'm getting the following using mod_proxy: [error] (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted. : proxy: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed Anyone has a clue where and why those error messages comes from? MT. smime.p7s Description: S/MIME cryptographic signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Mladen Turk wrote: Ab -n 1 Time taken for tests: 239.614549 seconds Complete requests: 1 Failed requests:7011 So, mod_proxy is a lot slower and doesn't handle load. (Perhaps increasing http listeners on TC would help). Increasing the maxThreads to 350 and acceptCount to 300 doesn't solve the issue. On large load I'm getting the following using mod_proxy: [error] (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted. : proxy: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed Anyone has a clue where and why those error messages comes from? Could you try ab -k? - mod_proxy will reuse the socket - MT. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
jean-frederic clere wrote: [error] (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted. : proxy: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed Anyone has a clue where and why those error messages comes from? Could you try ab -k? - mod_proxy will reuse the socket - Still the same messages. When running ab on the same host where apache and TC are located I'm getting between 3-16 errors on '-n 1000', But when using remote machine and use the ab from remote I'm getting around 50% error requests; cca. 450 - 520 requests gets failed. The number of failed requests gets lower and lower when I run same ab in a series, and it get more and more random, varying from 0 - 50 %. Interesting is that rising concurrency level to 20 or more, lowers the error number, but It's still around 10% (1120 from 1 request are 502). Very strange and totally unusable at least on WIN32. MT. smime.p7s Description: S/MIME cryptographic signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Mladen Turk wrote: jean-frederic clere wrote: [error] (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted. : proxy: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed Anyone has a clue where and why those error messages comes from? Could you try ab -k? - mod_proxy will reuse the socket - Still the same messages. When running ab on the same host where apache and TC are located I'm getting between 3-16 errors on '-n 1000', But when using remote machine and use the ab from remote I'm getting around 50% error requests; cca. 450 - 520 requests gets failed. The number of failed requests gets lower and lower when I run same ab in a series, and it get more and more random, varying from 0 - 50 %. Interesting is that rising concurrency level to 20 or more, lowers the error number, but It's still around 10% (1120 from 1 request are 502). Very strange and totally unusable at least on WIN32. Well ab running on Win32 didn't very stable ;( - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Mladen Turk wrote: jean-frederic clere wrote: [error] (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted. : proxy: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed Anyone has a clue where and why those error messages comes from? Could you try ab -k? - mod_proxy will reuse the socket - Still the same messages. When running ab on the same host where apache and TC are located I'm getting between 3-16 errors on '-n 1000', Don't run ab on the same host, otherwise you are benching ab. But when using remote machine and use the ab from remote I'm getting around 50% error requests; cca. 450 - 520 requests gets failed. The number of failed requests gets lower and lower when I run same ab in a series, and it get more and more random, varying from 0 - 50 %. Interesting is that rising concurrency level to 20 or more, lowers the error number, but It's still around 10% (1120 from 1 request are 502). I also I have some (40) errors with concurrency 300 but Tomcat and Apache are in 2 different machines: +++ [Thu Jul 22 11:39:39 2004] [error] [client 172.25.182.35] proxy: DNS lookup failure for: pgtr0327.mch.fsc.net returned by /examples/servlet/HelloWorldExample +++ Very strange and totally unusable at least on WIN32. Use a real operating system. MT. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Henri Gomez wrote: Very strange and totally unusable at least on WIN32. Well ab running on Win32 didn't very stable ;( Yeah, sure :) WTF then mod_jk doesn't produce such errors ? BTW, the errors reported comes from mod_proxy. MT. smime.p7s Description: S/MIME cryptographic signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Filip Hanik - Dev wrote: I suppose in this case the load balancer would run HOOK_MIDDLE, and sticky would run HOOK_LAST. cool, and then have the server just try them in that order? ie, if the sticky server went down, it just takes the next one from the list (and that list should be ordered well since it comes from the LB algorithm) correct assumption? Yes. In other words, you start with a list of IPs (supplied by the config, resolved by DNS), and each load balancer module reorders the IP addresses in turn, as needed. So a load balancer module will reorder the IPs least loaded to most loaded. Then the sticky module will either do nothing to list, or it will notice a sticky session, and move the sticky IP address to the top of the list. Thing is for the sticky module to be effective, it must run last, otherwise the load balancer will undo sticky's work. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Tim Funk wrote: I'm not sure of the status so far, but I'd like to summarize a strawman. I have no idea how to code this at this time or if it can be done. No problem, we drill down into the details as we go along :) *Config* [Feel free to change the names] ProxyClient http://server1/config.xml ProxyClient http://server2/config.xml ProxyClient http://server3/another/config.xml ProxyUpdate /proxyconfig The config should follow the established config for proxy, which would be this: ProxyPass /myWebapp ajp://server1/myWebapp At the moment, server1 would resolve to multiple names, however a userful addition would be ability to do this: ProxyPass /myWebapp ajp://server1 server2 server3/myWebapp which is the way LDAP URLs handle multiple possible servers. *Startup* Apache polls each proxy-client for existence and config at the URL defined by te ProxyClient directive. Each proxy-client states the paths/extension/mimetype/??? it can serve as well as its weight. It is the job of proxy_ajp to understand what ajp://server1 server2 server3/myWebapp means. What proxy_ajp can do, if it has not already done so, is poll the servers in the server list and say hey guys, which of you lot serves myWebapp? All of you? Ok cool bananas - the top of the list of IPs I have been given is you server1, so go ahead, knock yourself out. *On going* Apache periodically polls the config URL for changes. --or-- A proxy-client can join, remove, alter URL availability by using making a request to apache defined at ProxyUpdate. proxy_ajp need not poll - it can do so as the request arrives. If proxy_ajp connects to the backend, and that backend returns 5xx, then proxy_ajp says Oops, obviously not serving any more, let me try the next IP in my list and see if that one is willing to serve my request. *Edge cases* Authentication, environment variables, and other system parameters would be passed to the proxy via X-Headers. Apache would need to deny/filter these header names from outside clients. I don't understand - are these headers created by tomcat, and passed towards the client, or created by the client, and passed towards tomcat? Either way, funky header handling would be done by proxy_ajp. *stickyness* unknown how to fix this in a generic manner Using a completely separate and independant module called proxy_sticky, which uses a hook I will be putting into proxy sometime this weekend. :) *Failover* 2 cases 1) server unavailable - easy to recover since a new connection is made 2) server available but doesn't handle request anymore - Proxy-client returns a custom header to let apache move to next node? No need for a custom header - just interpret the response codes correctly. 5xx means try the next server in the list, if you run out of list, pass the error back up to the client. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Remy Maucherat wrote: It's cool to have one less thing to configure, but it seems to me jvmRoute is the most reliable and efficient way of doing stickiness Can you describe the jvmRoute method to me? (the cookie way is intrusive, and the IP way is highly inaccurate). I agree on the IP way being inaccurate (and have argued long and hard against people who couldn't understand that the same browser could easily come in from one of many IP addresses, changing all the time as they go). Currently tomcat handles sessions as either a) a cookie or b) a parameter. We could easily teach proxy_sticky that stickiness is based on either of these two. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Mladen Turk wrote: BTW, the errors reported comes from mod_proxy. What are the errors though, and do they come from mod_proxy or mod_proxy_http? It would be a huge help to the people using proxy (ie for non tomcat related stuff) if we could find and fix these error conditions under load. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Graham Leggett wrote: Remy Maucherat wrote: It's cool to have one less thing to configure, but it seems to me jvmRoute is the most reliable and efficient way of doing stickiness Can you describe the jvmRoute method to me? It's really dumb: we append the node name to the session id when it's generated (it ends up as the JSESSIONID cookie value). Of course, the downside is that you have to configure stuff on each node :/ Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
jean-frederic clere wrote: I also I have some (40) errors with concurrency 300 but Tomcat and Apache are in 2 different machines: +++ [Thu Jul 22 11:39:39 2004] [error] [client 172.25.182.35] proxy: DNS lookup failure for: pgtr0327.mch.fsc.net returned by ^^ /examples/servlet/HelloWorldExample +++ Very strange and totally unusable at least on WIN32. Use a real operating system. Or a decent DNS server :) Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Graham Leggett wrote: jean-frederic clere wrote: I also I have some (40) errors with concurrency 300 but Tomcat and Apache are in 2 different machines: +++ [Thu Jul 22 11:39:39 2004] [error] [client 172.25.182.35] proxy: DNS lookup failure for: pgtr0327.mch.fsc.net returned by ^^ /examples/servlet/HelloWorldExample +++ Very strange and totally unusable at least on WIN32. Use a real operating system. Or a decent DNS server :) It's not normal there's a DNS lookup on each request. Why does it happen ? +1 on using a real OS ;) Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Graham Leggett wrote:
Mladen Turk wrote:
BTW, the errors reported comes from mod_proxy.
What are the errors though, and do they come from mod_proxy
or mod_proxy_http?
All are exactly the same:
[error] (OS 10048)Only one usage of each socket address (protocol/network
address/port) is normally permitted. : proxy: HTTP: attempt to connect to
127.0.0.1:8080 (localhost) failed
It would be a huge help to the people using proxy (ie for non
tomcat related stuff) if we could find and fix these error
conditions under load.
We had a same problen in jk for over two years now. The problem is that you
will need at least:
Line 1037 in proxy_util.c:
/* make the connection out of the socket */
do {
rv = apr_socket_connect(*newsock, backend_addr);
} while (APR_STATUS_IS_EINTR(rv));
But again it might be just an attemp to connect to an nonclosed connection
(as BTW the error suggests), or perhaps connection beeing in the process of
disconnection, but cause of high load you have a race condition.
MT.
smime.p7s
Description: S/MIME cryptographic signature
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Remy Maucherat wrote: +1 on using a real OS ;) Well, you could also use a real programming language for start ;-). MT. smime.p7s Description: S/MIME cryptographic signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Remy Maucherat wrote: DNS lookup failure for: pgtr0327.mch.fsc.net returned by ^^ It's not normal there's a DNS lookup on each request. Why does it happen ? In the config it was set to connect to a DNS name, which has to be resolved - but httpd doesn't do any caching of this (which in theory would be the job of a caching DNS server), thus a lookup on each request. Perhaps an enhancement to proxy_http would be to support DNS caching for a period of time. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Mladen Turk wrote:
All are exactly the same:
[error] (OS 10048)Only one usage of each socket address (protocol/network
address/port) is normally permitted. : proxy: HTTP: attempt to connect to
127.0.0.1:8080 (localhost) failed
Ok.
We had a same problen in jk for over two years now. The problem is that you
will need at least:
Line 1037 in proxy_util.c:
/* make the connection out of the socket */
do {
rv = apr_socket_connect(*newsock, backend_addr);
} while (APR_STATUS_IS_EINTR(rv));
But again it might be just an attemp to connect to an nonclosed connection
(as BTW the error suggests), or perhaps connection beeing in the process of
disconnection, but cause of high load you have a race condition.
So if I committed the above patch to httpd v2.1.0-dev would you be in a
position to test it?
Regards,
Graham
--
smime.p7s
Description: S/MIME Cryptographic Signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Remy Maucherat wrote: Graham Leggett wrote: jean-frederic clere wrote: I also I have some (40) errors with concurrency 300 but Tomcat and Apache are in 2 different machines: +++ [Thu Jul 22 11:39:39 2004] [error] [client 172.25.182.35] proxy: DNS lookup failure for: pgtr0327.mch.fsc.net returned by ^^ /examples/servlet/HelloWorldExample +++ Very strange and totally unusable at least on WIN32. Use a real operating system. Or a decent DNS server :) It's not normal there's a DNS lookup on each request. Why does it happen ? Not for each request but each time http makes a new connection to Tomcat. We have to cache the result of apr_sockaddr_info_get(). +1 on using a real OS ;) Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Mladen Turk wrote:
/* make the connection out of the socket */
do {
rv = apr_socket_connect(*newsock, backend_addr);
} while (APR_STATUS_IS_EINTR(rv));
One further question (I am not 100% clued up on the workings of apr's
socket handling) - would a situation ever arise where
APR-STATUS_IS_EINTR(rv) would always return non zero, causing an
infinite loop?
Regards,
Graham
--
smime.p7s
Description: S/MIME Cryptographic Signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
jean-frederic clere wrote: Not for each request but each time http makes a new connection to Tomcat. We have to cache the result of apr_sockaddr_info_get(). Added to bugzilla as a request for enhancement (so this doesn't fall through the cracks). Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Graham Leggett wrote: So if I committed the above patch to httpd v2.1.0-dev would you be in a position to test it? No, I've tested it. Still has the same error messages. OS error 10048 means: Typically, only one usage of each socket address (protocol/IP address/port) is permitted. This error occurs if an application attempts to bind a socket to an IP address/port that has already been used for an existing socket, or a socket that was not closed properly, or one that is still in the process of closing. For server applications that need to bind multiple sockets to the same port number, consider using setsockopt (SO_REUSEADDR). MT. smime.p7s Description: S/MIME cryptographic signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Mladen Turk wrote:
We had a same problen in jk for over two years now. The problem is that you
will need at least:
Line 1037 in proxy_util.c:
/* make the connection out of the socket */
do {
rv = apr_socket_connect(*newsock, backend_addr);
} while (APR_STATUS_IS_EINTR(rv));
Added to bugzilla as 30260.
Regards,
Graham
--
smime.p7s
Description: S/MIME Cryptographic Signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Mladen Turk wrote: OS error 10048 means: Typically, only one usage of each socket address (protocol/IP address/port) is permitted. This error occurs if an application attempts to bind a socket to an IP address/port that has already been used for an existing socket, or a socket that was not closed properly, or one that is still in the process of closing. For server applications that need to bind multiple sockets to the same port number, consider using setsockopt (SO_REUSEADDR). Could this be the problem? Maybe proxy is not closing sockets properly. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Graham Leggett wrote: jean-frederic clere wrote: Not for each request but each time http makes a new connection to Tomcat. We have to cache the result of apr_sockaddr_info_get(). Added to bugzilla as a request for enhancement (so this doesn't fall through the cracks). That is PR 30259. Regards, Graham -- - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Please help me to unsubscribe me from the tomcat and relatives forum. I already send a lot of mails to the mayordomo with the words unsubscribe but im still receiving mail. Please help me. thanks fernando R Torrijos _ Do You Yahoo!? Información de Estados Unidos y América Latina, en Yahoo! Noticias. Visítanos en http://noticias.espanol.yahoo.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Fernando R. Torrijos wrote: Please help me to unsubscribe me from the tomcat and relatives forum. I already send a lot of mails to the mayordomo with the words unsubscribe but im still receiving mail. Please help me. Please follow the instructions at the bottom of the emails you have received, which are: To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Graham Leggett wrote: used for an existing socket, or a socket that was not closed properly, or one that is still in the process of closing. For server applications that need to bind multiple sockets to the same port number, consider using setsockopt (SO_REUSEADDR). Could this be the problem? Maybe proxy is not closing sockets properly. If you turn the loglevel to debug then there is no error messages (although everything is by the order of magnitude slower), so the closing algorithm is correct. The problem is IMHO that you are using a socket (presuming it is free) still served by the bucket brigade, but I may be wrong. MT. smime.p7s Description: S/MIME cryptographic signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Mladen Turk wrote: If you turn the loglevel to debug then there is no error messages (although everything is by the order of magnitude slower), so the closing algorithm is correct. The problem is IMHO that you are using a socket (presuming it is free) still served by the bucket brigade, but I may be wrong. MT. Can you look at the comments at http://issues.apache.org/bugzilla/show_bug.cgi?id=30260 - apparently this may be a Windows specific problem. I'm a unix type, so the windows issues are not something I am familiar with. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Remy Maucherat wrote: Graham Leggett wrote: Remy Maucherat wrote: It's cool to have one less thing to configure, but it seems to me jvmRoute is the most reliable and efficient way of doing stickiness Can you describe the jvmRoute method to me? It's really dumb: we append the node name to the session id when it's generated (it ends up as the JSESSIONID cookie value). Of course, the downside is that you have to configure stuff on each node :/ Or we could have a reasonable default - like the ajp ip/port of the host ( or md5 of it ). Costin - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Graham Leggett wrote:
Can you look at the comments at
http://issues.apache.org/bugzilla/show_bug.cgi?id=30260 -
apparently this may be a Windows specific problem.
Could be, or not.
If for example after connect I write:
if (rv == 730048) {
apr_socket_close(*newsock);
*newsock = NULL;
continue;
}
It goes in an infinite loop, but it should not.
I'm a unix type, so the windows issues are not something I am
familiar with.
Think that has nothing to do with windows (I even lowered the thread count
to 50), but not sure.
If someone wishes to check that on some worker mpm with high thread count.
It serves 1995 conections, either ab -n 2000 or 2 x ab -n 1000 (the first
one has no errors, the second has 5 errors).
After that any attempt to connect to mod_proxy returns 503.
I've spend couple of hours trying to find the bug, or at least the reason,
but I'm slightly loosing the interest.
Regards,
MT.
smime.p7s
Description: S/MIME cryptographic signature
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Filip Hanik wrote: The current jvmRoute addition to JSESSIONID is not really needed, since it doesn't add that much of a benefit over the two options above. So right then and there, there is one less thing to configure. Ok, If we'll make a lb for a mod_proxy, then at least it will need a balance load factor, not just sticky sessions and simple round robin. Also the JSESSIONID then does not to be harcoded, but rather configurable. MT. smime.p7s Description: S/MIME cryptographic signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
I didn't say anything about just simple round robin the algorithm for distribution can be entirely separate from the stickiness as they are two separate things. The distribution algorithm, (round robin, load, random, etc) is separate and should not be confused. Stickyness means that if I have been to one server, I should go to that server again and again until that server fails or meets another criteria to not receive requests. Filip - Original Message - From: Mladen Turk [EMAIL PROTECTED] To: 'Tomcat Developers List' [EMAIL PROTECTED] Sent: Wednesday, July 21, 2004 11:30 AM Subject: RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev Filip Hanik wrote: The current jvmRoute addition to JSESSIONID is not really needed, since it doesn't add that much of a benefit over the two options above. So right then and there, there is one less thing to configure. Ok, If we'll make a lb for a mod_proxy, then at least it will need a balance load factor, not just sticky sessions and simple round robin. Also the JSESSIONID then does not to be harcoded, but rather configurable. MT. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Filip Hanik wrote: I didn't say anything about just simple round robin the algorithm for distribution can be entirely separate from the stickiness as they are two separate things. The distribution algorithm, (round robin, load, random, etc) is separate and should not be confused. Stickyness means that if I have been to one server, I should go to that server again and again until that server fails or meets another criteria to not receive requests. Ok, just wanted to clear if we are going to make another compromise :), since sticky sessions are tightly coupled with the load balancer itself and the way it decides the client route. MT. smime.p7s Description: S/MIME cryptographic signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Mladen Turk wrote: Ok, just wanted to clear if we are going to make another compromise :), since sticky sessions are tightly coupled with the load balancer itself and the way it decides the client route. In theory sticky sessions shouldn't be tightly coupled like this - it should be a case of plan a) stick to the same server, else revert to plan b). In this case plan B could be simple DNS round robin, or it could be a load balancer, in other words another module entirely. This raises a question - what order do the modules run in? In theory the sticky module should run last, in other words a load balancer (or DNS round robin, or whatever) orders the IPs based on some criteria (load, or round robin, or whatever), and then the sticky module takes the sticky server identified by the cookie/parameter and moves that sticky IP on top of the list. I suppose in this case the load balancer would run HOOK_MIDDLE, and sticky would run HOOK_LAST. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Dobry den, nizsie uvedeny mail NEBOL DORUCENY. Dopiste, prosim, k predmetu mailu NIE SPAM a znova ho poslite. V buducich mailoch uz NIE SPAM pisat nemusite. Antispamovy filter Oddych.sk Nedoruceny mail: GL Mladen Turk wrote: Ok, just wanted to clear if we are going to make another compromise :), since sticky sessions are tightly coupled with the load balancer itself and the way it decides the client route. GL In theory sticky sessions shouldn't be tightly coupled like this - it GL should be a case of plan a) stick to the same server, else revert to GL plan b). GL In this case plan B could be simple DNS round robin, or it could be a GL load balancer, in other words another module entirely. GL This raises a question - what order do the modules run in? In theory the GL sticky module should run last, in other words a load balancer (or DNS GL round robin, or whatever) orders the IPs based on some criteria (load, GL or round robin, or whatever), and then the sticky module takes the GL sticky server identified by the cookie/parameter and moves that sticky GL IP on top of the list. GL I suppose in this case the load balancer would run HOOK_MIDDLE, and GL sticky would run HOOK_LAST. GL Regards, GL Graham GL -- -=x=- Skontrolované antivírovým programom NOD32 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Graham Leggett wrote: sticky sessions are tightly coupled with the load balancer itself and the way it decides the client route. In theory sticky sessions shouldn't be tightly coupled like this - it should be a case of plan a) stick to the same server, else revert to plan b). Yes, but why would you wish to separate those? The stickiness is the part of load balancer, and merely flags the connection to a particular remote so that the load balancer can skip the Remote selection, cause it already selected the Remote in some previous transaction. Something different is the way the load balancer selects the Remote, and that can be separated if the separation is needed at all, but I doubt that. I suppose in this case the load balancer would run HOOK_MIDDLE, and sticky would run HOOK_LAST. I'm not that familiar with mod_proxy code, so please no hooks, not jet :) MT. smime.p7s Description: S/MIME cryptographic signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Dobry den, nizsie uvedeny mail NEBOL DORUCENY. Dopiste, prosim, k predmetu mailu NIE SPAM a znova ho poslite. V buducich mailoch uz NIE SPAM pisat nemusite. Antispamovy filter Oddych.sk Nedoruceny mail: MT Graham Leggett wrote: sticky sessions are tightly coupled with the load balancer itself and the way it decides the client route. In theory sticky sessions shouldn't be tightly coupled like this - it should be a case of plan a) stick to the same server, else revert to plan b). MT Yes, but why would you wish to separate those? MT The stickiness is the part of load balancer, and merely flags the connection MT to a particular remote so that the load balancer can skip the Remote MT selection, cause it already selected the Remote in some previous MT transaction. MT Something different is the way the load balancer selects the Remote, and MT that can be separated if the separation is needed at all, but I doubt that. I suppose in this case the load balancer would run HOOK_MIDDLE, and sticky would run HOOK_LAST. MT I'm not that familiar with mod_proxy code, so please no hooks, not jet :) MT MT. -=x=- Skontrolované antivírovým programom NOD32 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Mladen Turk wrote: Yes, but why would you wish to separate those? Because they are two separate behaviours that could quite easily be used independantly of each other. I would probably use the stickiness long before I started messing around with load balancing. I'm not that familiar with mod_proxy code, so please no hooks, not jet :) I am familiar with the code, so don't be afraid of the hooks, as I'm quite willing to tell you where things go :) Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Graham Leggett wrote: Yes, but why would you wish to separate those? Because they are two separate behaviours that could quite easily be used independantly of each other. OK, it makes sense. I'm not that familiar with mod_proxy code, so please no hooks, not jet :) I am familiar with the code, so don't be afraid of the hooks, as I'm quite willing to tell you where things go :) So, where do you see a lb code in the mod_proxy tree. Do you have some ideas where will it fit? It would be also good if you could make some p-code of the mod_proxy or some data flow diagram. Also do you need our support on coding? It would be perhaps better that you write the function prototypes that need some lb behavior returnig somethig like not_implemented or just mark some /* TODO: ...*/ in the code. MT. smime.p7s Description: S/MIME cryptographic signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
I suppose in this case the load balancer would run HOOK_MIDDLE, and sticky would run HOOK_LAST. cool, and then have the server just try them in that order? ie, if the sticky server went down, it just takes the next one from the list (and that list should be ordered well since it comes from the LB algorithm) correct assumption? Filip - Original Message - From: Mladen Turk [EMAIL PROTECTED] To: 'Tomcat Developers List' [EMAIL PROTECTED] Sent: Wednesday, July 21, 2004 1:12 PM Subject: RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev Graham Leggett wrote: Yes, but why would you wish to separate those? Because they are two separate behaviours that could quite easily be used independantly of each other. OK, it makes sense. I'm not that familiar with mod_proxy code, so please no hooks, not jet :) I am familiar with the code, so don't be afraid of the hooks, as I'm quite willing to tell you where things go :) So, where do you see a lb code in the mod_proxy tree. Do you have some ideas where will it fit? It would be also good if you could make some p-code of the mod_proxy or some data flow diagram. Also do you need our support on coding? It would be perhaps better that you write the function prototypes that need some lb behavior returnig somethig like not_implemented or just mark some /* TODO: ...*/ in the code. MT. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Dobry den, nizsie uvedeny mail NEBOL DORUCENY. Dopiste, prosim, k predmetu mailu NIE SPAM a znova ho poslite. V buducich mailoch uz NIE SPAM pisat nemusite. Antispamovy filter Oddych.sk Nedoruceny mail: I suppose in this case the load balancer would run HOOK_MIDDLE, and sticky would run HOOK_LAST. FHD cool, and then have the server just try them in that order? FHD ie, if the sticky server went down, it just takes the next one FHD from the FHD list (and that list should be ordered well since it comes from the LB algorithm) FHD correct assumption? FHD Filip FHD - Original Message - FHD From: Mladen Turk [EMAIL PROTECTED] FHD To: 'Tomcat Developers List' [EMAIL PROTECTED] FHD Sent: Wednesday, July 21, 2004 1:12 PM FHD Subject: RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev FHD Graham Leggett wrote: Yes, but why would you wish to separate those? Because they are two separate behaviours that could quite easily be used independantly of each other. FHD OK, it makes sense. I'm not that familiar with mod_proxy code, so please no hooks, not jet :) I am familiar with the code, so don't be afraid of the hooks, as I'm quite willing to tell you where things go :) FHD So, where do you see a lb code in the mod_proxy tree. Do you have some ideas FHD where will it fit? FHD It would be also good if you could make some p-code of the mod_proxy or some FHD data flow diagram. FHD Also do you need our support on coding? FHD It would be perhaps better that you write the function prototypes that need FHD some lb behavior returnig somethig like not_implemented or just mark some /* FHD TODO: ...*/ in the code. FHD MT. FHD - FHD To unsubscribe, e-mail: [EMAIL PROTECTED] FHD For additional commands, e-mail: FHD [EMAIL PROTECTED] -=x=- Skontrolované antivírovým programom NOD32 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Filip Hanik wrote: I suppose in this case the load balancer would run HOOK_MIDDLE, and sticky would run HOOK_LAST. cool, and then have the server just try them in that order? ie, if the sticky server went down, it just takes the next one from the list (and that list should be ordered well since it comes from the LB algorithm) correct assumption? Think that it should return 'Server Busy' in case the sticky is not available, cause you may be in the middle of transaction while the other server might have no clue about that. MT. smime.p7s Description: S/MIME cryptographic signature
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
really, so then there is no fail over. cause that is what fail over does, redirects you to another server. and with session replication in place, you should be good to go FIlip - Original Message - From: Mladen Turk [EMAIL PROTECTED] To: 'Tomcat Developers List' [EMAIL PROTECTED] Sent: Wednesday, July 21, 2004 2:00 PM Subject: RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev Filip Hanik wrote: I suppose in this case the load balancer would run HOOK_MIDDLE, and sticky would run HOOK_LAST. cool, and then have the server just try them in that order? ie, if the sticky server went down, it just takes the next one from the list (and that list should be ordered well since it comes from the LB algorithm) correct assumption? Think that it should return 'Server Busy' in case the sticky is not available, cause you may be in the middle of transaction while the other server might have no clue about that. MT. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
Filip Hanik - Dev wrote: I didn't say anything about just simple round robin the algorithm for distribution can be entirely separate from the stickiness as they are two separate things. The distribution algorithm, (round ron, load, random, etc) is separate and should not be confused. Stickyness means that if I have been to one server, I should go to that server again and again until that server fails or meets another criteria to not receive requests. And this another criteria is a _very_ important use case ( and one of the reasons I am insisting on having some form of dynamic configuration/jmx/whatever ). Many large servers ( banks, etc ) would not want to lose client sessions when they upgrade a worker in the pool - so they need to be able to tell that a particular instance should not be included in the round robin for new sessions, but only get the requests from previous sessions. They also need to be able to add more workers, or remove some workers from the pool. I know, the jk implementation is very ugly - but IMO this use case is quite important and it shouldn't be lost. Costin Filip - Original Message - From: Mladen Turk [EMAIL PROTECTED] To: 'Tomcat Developers List' [EMAIL PROTECTED] Sent: Wednesday, July 21, 2004 11:30 AM Subject: RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev Filip Hanik wrote: The current jvmRoute addition to JSESSIONID is not really needed, since it doesn't add that much of a benefit over the two options above. So right then and there, there is one less thing to configure. Ok, If we'll make a lb for a mod_proxy, then at least it will need a balance load factor, not just sticky sessions and simple round robin. Also the JSESSIONID then does not to be harcoded, but rather configurable. MT. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
I'm not sure of the status so far, but I'd like to summarize a strawman. I have no idea how to code this at this time or if it can be done. -- *Config* [Feel free to change the names] ProxyClient http://server1/config.xml ProxyClient http://server2/config.xml ProxyClient http://server3/another/config.xml ProxyUpdate /proxyconfig *Startup* Apache polls each proxy-client for existence and config at the URL defined by te ProxyClient directive. Each proxy-client states the paths/extension/mimetype/??? it can serve as well as its weight. *On going* Apache periodically polls the config URL for changes. --or-- A proxy-client can join, remove, alter URL availability by using making a request to apache defined at ProxyUpdate. *Edge cases* Authentication, environment variables, and other system parameters would be passed to the proxy via X-Headers. Apache would need to deny/filter these header names from outside clients. *stickyness* unknown how to fix this in a generic manner *Failover* 2 cases 1) server unavailable - easy to recover since a new connection is made 2) server available but doesn't handle request anymore - Proxy-client returns a custom header to let apache move to next node? *Changes to tomcat* Add a proxy mode flag to allow for the X- headers to pass authentication and other variables. Add to the manager(?) app and method to expose all the URL spaces availble. Minor changes to fix getRemoteAddr() to show the client, not the apache server. Pros - Simple(?) and not tomcat specific. Cons - Duplicate (re)parsing of headers, probably higher latency than jk. -Tim - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev
added to summarization, missing stickiness 1. Set a cookie on the clients machine - no server memory to hold a map 2. If the client doesn't accept cookies, do a simple sticky load balancing based on the IP of the client request. Again, no memory map needed. -Original Message- From: Tim Funk [mailto:[EMAIL PROTECTED] Sent: Wednesday, July 21, 2004 8:17 PM To: Tomcat Developers List Subject: Re: Simple Sticky LB WAS: Invitation to HTTPD commiters in tomcat-dev I'm not sure of the status so far, but I'd like to summarize a strawman. I have no idea how to code this at this time or if it can be done. -- *Config* [Feel free to change the names] ProxyClient http://server1/config.xml ProxyClient http://server2/config.xml ProxyClient http://server3/another/config.xml ProxyUpdate /proxyconfig *Startup* Apache polls each proxy-client for existence and config at the URL defined by te ProxyClient directive. Each proxy-client states the paths/extension/mimetype/??? it can serve as well as its weight. *On going* Apache periodically polls the config URL for changes. --or-- A proxy-client can join, remove, alter URL availability by using making a request to apache defined at ProxyUpdate. *Edge cases* Authentication, environment variables, and other system parameters would be passed to the proxy via X-Headers. Apache would need to deny/filter these header names from outside clients. *stickyness* unknown how to fix this in a generic manner *Failover* 2 cases 1) server unavailable - easy to recover since a new connection is made 2) server available but doesn't handle request anymore - Proxy-client returns a custom header to let apache move to next node? *Changes to tomcat* Add a proxy mode flag to allow for the X- headers to pass authentication and other variables. Add to the manager(?) app and method to expose all the URL spaces availble. Minor changes to fix getRemoteAddr() to show the client, not the apache server. Pros - Simple(?) and not tomcat specific. Cons - Duplicate (re)parsing of headers, probably higher latency than jk. -Tim - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --- Incoming mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.721 / Virus Database: 477 - Release Date: 7/16/2004 --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.721 / Virus Database: 477 - Release Date: 7/16/2004 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
