Re: reversed proxy stopped working with tomcat cluster
Hi Rainer, Well, I did make some bloody operation by eliminating the VmWare machine as the reverse proxy on my Mac. /Following Chris advise. :)/ Because apache2 on my Mac was not compiled with --enable-proxy, etc, I downloaded the version 2.2.11 and compiled it with the three proxy enablers /--enable-proxy, --enable-proxy-balancer, and --enable-proxy- http/. I copied the reverse proxy config part and the JkMount directives from the VmWare virtual machine into the httpd.conf file on the Mac. Now I have two machines: - A PC with Windows 2000 and a Mac with OSX 10.5.6. The PC has apache2 + mod_jk + tomcat (1 instance). - A Mac that has apache2 + mod_jk + tomcat (3 instances). The Mac also acts as a reverse proxy. Fired up the newly compiled apache2 on the Mac and now everything works like charm. It is failing over nicely when I stop the selected tomcat node. I still have to test the fail over when I stop apache on the PC in the middle of a session. I changed two things: 1. moved the reverse proxy configuration off from the Windows Xp inside the VmWare Fussion machine on the Mac, so the Mac itself became the reverse proxy. Nothing was done to the tomcats or to the apache2 on the PC, not even stopping them. 2. On the Mac I installed a newer version of apache2. /2.2.11 compared to 2.2.9/ My guess is the VmWare update and the WinXp update on the virtual machine from December changed somehow the security settings that did not allow the information to flow back to the reverse proxy. Of course, I am not sure about it. :) Theoretically it should have worked without any problems. If I have time I will install a 2.2.11 apache2 on the XP and move back to it the reverse proxy configuration from the Mac, to see if that helps or not. Thanks ahead, János On Mar 25, 2009, at 6:49 PM, Rainer Jung wrote: On 20.03.2009 18:48, János Löbb wrote: Hi, I have two real machines. One of them is a Windows XP running Apache 2.2.10 + mod_jk /release date of 10/30/2008/ + Tomcat 6.0.16. The other one is a Mac with OSX 10.5.6 with Apache 2.2.9 + mod_jk 1.2.26 and Tomcat 6.0.16. The XP machine runs one instance of Tomcat /node4/. The Mac runs 3, - node1,2,3. Altogether 4 tomcats and they form a cluster. ... In front of these two machines is an Apache 2.2.10 in a VMWare Fusion virtual Machine on Windows XP in the Mac and configured as a Reverse proxy Here is the reverse proxy config from the httpd.conf snip ProxyRequests Off Proxy balancer://pathCluster BalancerMember http://bml0039.yalepath.org loadfactor=10 BalancerMember http://bml0073.yalepath.org loadfactor=10 ProxySet lbmethod=bytraffic /Proxy ProxyPass /tc/ balancer://pathCluster/ ProxyPassReverse /tc/ balancer://pathCluster/ Location /balancer-manager SetHandler balancer-manager Order Deny,Allow Allow from .yalepath.org /Location /snip Here is the access_log from the reverse proxy: snip 10.84.2.195 - - [20/Mar/2009:12:09:48 -0400] GET /tc/CassetteLabeler HTTP/1.1 302 - Returns with a trailing slash redirect 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] GET /tc/ CassetteLabeler/ HTTP/1.1 200 1506 Returns Content with 1506 Bytes 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] GET /CassetteLabeler/pathology.labeler.labeler.Labeler/Labeler.html HTTP/1.1 404 260 Wrong request, Prefix /tc/ missing. You need to find out, where this request comes from. Likely it is either generated by someone using a link in the page returned by the previous request, or that page contains a meta tag in the head, which triggers a redirect to this wrong URL. So retrieve /tc/CassetteLabeler/ and have a look at the content to identify, whether the wrong URL /CassetteLabeler/ pathology.labeler.labeler.Labeler/Labeler.html is in the page. If no, we have to think further. If yes, you have to ask, why is this wrong URL in there. The question is, how your application generates this URL. If it generates absolute URLs instead of relative ones, then you can easily run into trouble, when the path of the real application is different from the path used from the outside. It is not unlikely though, that in such a case the app or the framework used has a configurable path prefix. But first check, if you can verify the wrong URL is somewhere in the content of the page. The request from the reverse proxy did go to the XP machine: Here are the log entries from the access log: snip 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] GET /CassetteLabeler HTTP/1.1 302 - That's only the first request, answered with the trailing slash redirect. The next request is the interesting one (the one answered with status 200). It also has to appear somewhere in your 2nd laqyer httpd logs. Here are the entries from mod_jk.log of this XP machine: snip [Fri Mar 20 12:09:49.254 2009] [852:1808] [debug] jk_uri_worker_map.c (682): Attempting to map URI '/CassetteLabeler' from 5 maps Again that's the
Re: reversed proxy stopped working with tomcat cluster
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 János, On 3/20/2009 1:48 PM, János Löbb wrote: Connector port=8280 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8643 / Connector port=8209 protocol=AJP/1.3 redirectPort=8643 / [snip] ProxyRequests Off Proxy balancer://pathCluster BalancerMember http://bml0039.yalepath.org loadfactor=10 BalancerMember http://bml0073.yalepath.org loadfactor=10 Note that you have configured your Tomcat to listen for HTTP on port 8280, but your Proxy configuration points to port 80. This isn't going to work. Here is the workers.property file: snip bml0073:local janos$ cat apache2/conf/workers.properties worker.list = lb,jkstatus This looks like workers.properties, which is a mod_jk thing. You are using mod_proxy_http. This isn't going to work for two reasons: 1. mod_proxy_http uses HTTP as the protocol and mod_jk uses AJP 2. mod_proxy_http doesn't read workers.properties, so the file is not relevant Similar workers.properies on the XP machine but the hosts are logically different If your Windows XP machine runs Tomcat only, then the presence of a workers.properties file is not relevant. Well, this setup worked in December. Very unlikely. The request from the reverse proxy did go to the XP machine: Here are the log entries from the access log: snip 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] GET /CassetteLabeler HTTP/1.1 302 - /snip Wait... are you running 3 copies of Apache httpd? No wonder there's tons and tons configuration. You only need one single httpd instance: the one doing the load-balancing. Configure /all/ of the TC instances that are part of the balancing setup in that one httpd. What you've done is set up load balancing across two httpd instances that then load-balance to a set of webapp servers. I can only think of a single reason to ever do that (you want lopsided load balancing) but it still doesn't make any sense. Lose the additional httpd instances and simplify your life, man. Then there won't be so much confusion over port numbers, forwarding, proxying, etc. Here are the entries from mod_jk.log of this XP machine: snip [Fri Mar 20 12:09:49.254 2009] [852:1808] [debug] jk_uri_worker_map.c (682): Attempting to map URI '/CassetteLabeler' from 5 maps Ugh. Okay, so you are using mod_proxy_http for the main httpd and mod_jk from the other ones. That's not confusing at all. I would highly recommend that you simplify your configuration unless you have a real need to complicate it. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAknKMrUACgkQ9CaO5/Lv0PBjrACgvAP1zKwFa2wYlx7O4cvPx7DC JhAAoLd4B7HnrjSX+8f7IcoivwJcpiDs =/8Rs -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: reversed proxy stopped working with tomcat cluster
Hi Chris, In our production environment we have 3 Xserves, all running apache2 , mod+jk and Tomcat. There are about 5-6 web applications running on these 3 servers standalone, -non-clustered - and there will be more. The there Xserves are doing other things too, not just web serving. That is the reason I cannot reduce the number of apache2 instances. They grandfathered in themselves :) This CassetteLabeler app is our first application that needs to be up all the time and that is why we turned to clustering, using the existing setups on these servers and hoping that just by small config changes here and there we can get our clustered app runing without worrying about the non-clustered apps. You know, the goat has to have a full stomach and the cabbage had to remain too :) . Not to destroy accidently anything in the production environment, I created a small test environment, simulating our production environment. Because I do not have 3 machines, I used the VmWare Fussion on my Mac with Windows XP to create the third machine, and to simplify things I do not run mod_jk and Tomcat on it just Apache2. The four Tomcats are forming a Tomcat Cluster. Because the individual Apaches have to communicate to their own Tomcats via their own mod_jk for the non-clustered webapps /non-clustered because they cannot be serialized/ to reach those apps, the users connect via: http://balancemember1.yalepath.org/Standalone1 kind of URL and it goes through port 80 to mod_jk and mod_jk passes it via ajp to to the appropriate tomcat. We need to preserve this behavior for the clustered app, that is why in the reversed proxy configuration the default port 80 is used. I do not think that I can changed that one. The reason for the reverse proxy is, that two of the machines in production are behind a Pix firewall and the users of the clustered app can get to it by hitting the reverse proxy. So in theory a user would hit a url like: http://reverse_proxy.yalepath.org/tc/CassetteLabeler The reverse proxy using mod_proxy, mod_proxy_balancer, mod_proxy_http would transfer it to one of the balancemembers via port 80 by the selected lbmethod. It would arrive there as: http://balancememberX.yalepath.org/CassetteLabeler The selected balance member seeing that there is a JkMount for that request would involve mod_jk and mod_jk would forward it via the ajp connector to one of the selected tomcats from the cluster. /Or that is how I imagine it :)/ Looks like the initial request is received by the reverse proxy. It selects a balance member and sends the request to it correctly, that is stripping off the /tc. Tha balancemember involves mod_jk and mod_jk forwards it to one of the clustered tomcats. Looking the debug info tomcat is doing its job and even asks for the body and receives it. Then tomcat is closing the connection to the balance member. That is where things get out of control. Then the selected balance member instead of continue with the earlier selected tomcat instance - although stickiness is specified - selects another tomcat node and sends the SAME info to it. This second tomcat does exactly as the first one did and it closes the connection. At that point the balancemember /or the reverse proxy/ sends the non-formated URL from the reverse proxy to the Apache of the second tomcat node and of course that cannot be resolved so I get a 404 error. So, I am looking why stickiness is not working, why the selected balancemember is not satisfied with the firstly selected tomcat node and just finishing the whole session with it, so the right info can flow back on the chain to the clients browser and paint the first form of the web app. Now I configured stickyness even in ProxyPass on the reverse proxy, but no cigar. ProxyRequests Off Proxy balancer://pathCluster BalancerMember http://bml0039.yalepath.org loadfactor=10 BalancerMember http://bml0073.yalepath.org loadfactor=10 ProxySet lbmethod=bytraffic /Proxy ProxyPass /tc/balancer://pathCluster/ stickysession=JSESSIONID ProxyPassReverse/tc/balancer://pathCluster/ #ProxyHTMLLogVerbose On Location /balancer-manager SetHandler balancer-manager Order Deny,Allow Allow from .yalepath.org /Location I would like to emphasize that when I request the web app from the individual balance members using of course port 80, like: http://balancemember1.yalepath.org/CassetteLabeler or http://balancemember2.yalepath.org/Cassettelabeler , everything is working right. The problem is somewhere as the info is flowing back in the chain to the reverse proxy. Thanks ahead, János On Mar 25, 2009, at 9:33 AM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 János, On 3/20/2009 1:48 PM, János Löbb wrote: Connector port=8280 protocol=HTTP/1.1
Re: reversed proxy stopped working with tomcat cluster
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 János, On 3/25/2009 11:53 AM, János Löbb wrote: In our production environment we have 3 Xserves, all running apache2 , mod+jk and Tomcat. There are about 5-6 web applications running on these 3 servers standalone, -non-clustered - and there will be more. The there Xserves are doing other things too, not just web serving. That is the reason I cannot reduce the number of apache2 instances. They grandfathered in themselves :) Just because they're there doesn't mean you have to use them. As long as the Tomcat ports are directly accessible from the load balancer, you can simply skip the intermediate httpd instances. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAknKbBgACgkQ9CaO5/Lv0PBk7gCdG8EQqggQ1ngeRcHMUjTH+M55 c/MAoLTqMlyXNOCmj7Uu27Z1AYjd+hp4 =9lj/ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: reversed proxy stopped working with tomcat cluster
On 20.03.2009 18:48, János Löbb wrote: Hi, I have two real machines. One of them is a Windows XP running Apache 2.2.10 + mod_jk /release date of 10/30/2008/ + Tomcat 6.0.16. The other one is a Mac with OSX 10.5.6 with Apache 2.2.9 + mod_jk 1.2.26 and Tomcat 6.0.16. The XP machine runs one instance of Tomcat /node4/. The Mac runs 3, - node1,2,3. Altogether 4 tomcats and they form a cluster. ... In front of these two machines is an Apache 2.2.10 in a VMWare Fusion virtual Machine on Windows XP in the Mac and configured as a Reverse proxy Here is the reverse proxy config from the httpd.conf snip ProxyRequests Off Proxy balancer://pathCluster BalancerMember http://bml0039.yalepath.org loadfactor=10 BalancerMember http://bml0073.yalepath.org loadfactor=10 ProxySet lbmethod=bytraffic /Proxy ProxyPass /tc/ balancer://pathCluster/ ProxyPassReverse /tc/ balancer://pathCluster/ Location /balancer-manager SetHandler balancer-manager Order Deny,Allow Allow from .yalepath.org /Location /snip Here is the access_log from the reverse proxy: snip 10.84.2.195 - - [20/Mar/2009:12:09:48 -0400] GET /tc/CassetteLabeler HTTP/1.1 302 - Returns with a trailing slash redirect 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] GET /tc/CassetteLabeler/ HTTP/1.1 200 1506 Returns Content with 1506 Bytes 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] GET /CassetteLabeler/pathology.labeler.labeler.Labeler/Labeler.html HTTP/1.1 404 260 Wrong request, Prefix /tc/ missing. You need to find out, where this request comes from. Likely it is either generated by someone using a link in the page returned by the previous request, or that page contains a meta tag in the head, which triggers a redirect to this wrong URL. So retrieve /tc/CassetteLabeler/ and have a look at the content to identify, whether the wrong URL /CassetteLabeler/pathology.labeler.labeler.Labeler/Labeler.html is in the page. If no, we have to think further. If yes, you have to ask, why is this wrong URL in there. The question is, how your application generates this URL. If it generates absolute URLs instead of relative ones, then you can easily run into trouble, when the path of the real application is different from the path used from the outside. It is not unlikely though, that in such a case the app or the framework used has a configurable path prefix. But first check, if you can verify the wrong URL is somewhere in the content of the page. The request from the reverse proxy did go to the XP machine: Here are the log entries from the access log: snip 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] GET /CassetteLabeler HTTP/1.1 302 - That's only the first request, answered with the trailing slash redirect. The next request is the interesting one (the one answered with status 200). It also has to appear somewhere in your 2nd laqyer httpd logs. Here are the entries from mod_jk.log of this XP machine: snip [Fri Mar 20 12:09:49.254 2009] [852:1808] [debug] jk_uri_worker_map.c (682): Attempting to map URI '/CassetteLabeler' from 5 maps Again that's the first request [Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c (608): status = 302 [Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c (615): Number of headers is = 1 [Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c (671): Header[0] [Location] = [http://bml0039.yalepath.org/CassetteLabeler/] Aha, we get back a redirect (302) to localtion http://bml0039.yalepath.org/CassetteLabeler/ (which will be further rewritten by ProxyPassReverse). But that we already know. The next request is the interesting one. I have a production environment - just with 3 Macs and just one tomcat per machine forming a cluster - where a very similar configuration runs fine. The only difference is that the sticky parameters are NOT with the load balancer but with the real workers in the workers.properties file, This is very likely not the cause. and the reverse proxy configuration somewhat different. Here it is: snip IfModule mod_proxy_balancer.c ProxyPass /apps/ balancer://balancer-group/ stickysession=JSESSIONID ProxyPassReverse /apps/ balancer://balancer-group/ Proxy balancer://balancer-group BalancerMember http://pathsrv1.yalepath.org loadfactor=10 BalancerMember http://raid2b.yalepath.org loadfactor=10 BalancerMember http://reptile.yalepath.org loadfactor=10 /Proxy /IfModule Looks pretty analogous at first sight. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
reversed proxy stopped working with tomcat cluster
Hi, I have two real machines. One of them is a Windows XP running Apache 2.2.10 + mod_jk /release date of 10/30/2008/ + Tomcat 6.0.16. The other one is a Mac with OSX 10.5.6 with Apache 2.2.9 + mod_jk 1.2.26 and Tomcat 6.0.16. The XP machine runs one instance of Tomcat /node4/. The Mac runs 3, - node1,2,3. Altogether 4 tomcats and they form a cluster. Here is one server .xml from node3. snip ?xml version='1.0' encoding='utf-8'? Server port=8205 shutdown=SHUTDOWN Listener className=org.apache.catalina.core.AprLifecycleListener SSLEngine=on / Listener className=org.apache.catalina.core.JasperListener / Listener className=org.apache.catalina.mbeans.ServerLifecycleListener / Listener className =org.apache.catalina.mbeans.GlobalResourcesLifecycleListener / GlobalNamingResources Resource name=UserDatabase auth=Container type=org.apache.catalina.UserDatabase description=User database that can be updated and saved factory=org.apache.catalina.users.MemoryUserDatabaseFactory pathname=conf/tomcat-users.xml / /GlobalNamingResources Service name=Catalina Connector port=8280 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8643 / Connector port=8209 protocol=AJP/1.3 redirectPort=8643 / Engine name=Catalina defaultHost=localhost jvmRoute=node3 Cluster className=org.apache.catalina.ha.tcp.SimpleTcpCluster/ Realm className=org.apache.catalina.realm.UserDatabaseRealm resourceName=UserDatabase/ Host name=localhost appBase=webapps unpackWARs=true autoDeploy=true xmlValidation=false xmlNamespaceAware=false /Host /Engine /Service /Server snip In front of these two machines is an Apache 2.2.10 in a VMWare Fusion virtual Machine on Windows XP in the Mac and configured as a Reverse proxy Here is the reverse proxy config from the httpd.conf snip ProxyRequests Off Proxy balancer://pathCluster BalancerMember http://bml0039.yalepath.org loadfactor=10 BalancerMember http://bml0073.yalepath.org loadfactor=10 ProxySet lbmethod=bytraffic /Proxy ProxyPass /tc/balancer://pathCluster/ ProxyPassReverse/tc/balancer://pathCluster/ Location /balancer-manager SetHandler balancer-manager Order Deny,Allow Allow from .yalepath.org /Location /snip Here is the workers.property file: snip bml0073:local janos$ cat apache2/conf/workers.properties worker.list = lb,jkstatus worker.lb.type=lb worker.lb.balance_workers=node1,node2,node3,node4 worker.lb.sticky_session = True worker.lb.sticky_session_force = False worker.jkstatus.type=status worker.node1.type = ajp13 worker.node1.host = localhost worker.node1.port = 8009 worker.node1.lbfactor = 1 worker.node1.redirect=node2 worker.node2.type = ajp13 worker.node2.host = localhost worker.node2.port = 8109 worker.node2.lbfactor = 1 worker.node2.redirect=node3 worker.node3.type = ajp13 worker.node3.host = localhost worker.node3.port = 8209 worker.node3.lbfactor = 1 worker.node3.redirect=node4 worker.node4.type = ajp13 worker.node4.host = bml0039.yalepath.org worker.node4.port = 8309 worker.node4.lbfactor = 1 worker.node4.redirect=node1 /snip Similar workers.properies on the XP machine but the hosts are logically different Well, this setup worked in December. The only change I made is that the sticky parameters now assigned to the loadbalancing worker 'lb' and not to the individual real workers. /Thanks to pointing that out :) / There were some updates to the Mac OS and also to the VmWare Fussion. I fired up this cluster yesterday and I realized it does not work anymore. Here is the access_log from the reverse proxy: snip 10.84.2.195 - - [20/Mar/2009:12:09:48 -0400] GET /tc/CassetteLabeler HTTP/1.1 302 - 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] GET /tc/CassetteLabeler/ HTTP/1.1 200 1506 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] GET /CassetteLabeler/ pathology.labeler.labeler.Labeler/Labeler.html HTTP/1.1 404 260 /snip here are the entries from the error log: snip [Fri Mar 20 12:08:30 2009] [notice] Apache/2.2.10 (Win32) configured -- resuming normal operations [Fri Mar 20 12:08:30 2009] [notice] Server built: Oct 10 2008 12:39:04 [Fri Mar 20 12:08:31 2009] [notice] Parent: Created child process 2564 [Fri Mar 20 12:08:31 2009] [notice] Child 2564: Child process is running [Fri Mar 20 12:08:31 2009] [notice] Child 2564: Acquired the start mutex. [Fri Mar 20 12:08:31 2009] [notice] Child 2564: Starting 64 worker threads. [Fri Mar 20 12:08:31 2009] [notice] Child 2564: Starting thread to listen on port 80. [Fri Mar 20 12:09:32 2009] [error] [client 10.84.2.195] File does not exist: C:/Program Files/Apache Software Foundation/Apache2.2/htdocs/ favicon.ico [Fri Mar 20 12:09:35 2009] [error] [client 10.84.2.195] File does not exist: C:/Program
