RE: SEVERE message from DeltaManager
Thankyou very much for your diagnosis here, Mark. I will investigate the proposed solution and let you know how it goes. Cheers, Matt. -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Tuesday, 20 July 2010 3:07 AM To: Tomcat Users List Subject: Re: SEVERE message from DeltaManager On 16/07/2010 10:19, Mark Thomas wrote: On 16/07/2010 06:49, Matt Peterson wrote: While load testing our clustered Tomcats, we are seeing the following stack trace in our catalina.out occasionally, but not regularly: Jul 16, 2010 3:34:49 PM org.apache.catalina.ha.session.DeltaManager messageReceived SEVERE: Manager [localhost#/urs]: Unable to receive message through TCP channel java.lang.IllegalStateException: removeAttribute: Session already invalidated snip/ Under what conditions would this occur? Could it be that a session diff is being transmitted, but the session it relates to has been invalidated by the time the diff is processed (via a user logout for example)? Or could it be that a timeout has been reached??? Someone at $work has been doing a load test with tc Server (which has identical code to Tomcat in this area) and seen the same issue. I know it isn't due to timeout since the sessions are only a few seconds old when it happens. My current guess is that the messages are not being processed in the same order as they are sent. I need to dig into this more to figure out if this is a configuration issue or a bug. I did wonder if switching to channel send options 6 would fix it. I'll get them to try that and see. Matt, Testing shows that it is caused by using async session replication. If you use synchronous replication that ensures messages are processed on the receiving nodes in the order they are sent. Asynchronous replication in conjunction with the fact the the receiving node uses a thread pool to process messages means that it is possible for messages to be processed out of sequence. If a session invalidate is processed before and update then you'll see this error. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: HTTP connector to be aware of proxied SSL requests
This is *open* source... Thx Capt. Obvious - very helpful ;-) OK, so I now understand why it was chosen to perform the redirection in the Connector rather than in a Valve; to remove unnecessary processing keeping the redirect response as efficient as possible. I might lodge an enhancement for the connector to have the redirect configurable so that it can be disabled via an element attribute. The redirecting can then be done as a valve instead. We are using an F5 LB which does not support AJP. So that option will not work for us. The other option of using multiple HTTP Connectors is doable, but adds a lot of config management overhead (and points of possible failure/error) which is not very popular with those responsible for that management. But that is an internal issue which I need to deal with if this prob is deemed to be worth the worry. Out of interest, what are some of the security risks around non-trusted proxies injecting the x-forwarded-* headers? Thanks for your help, Matt. -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Thursday, 17 June 2010 10:28 PM To: Tomcat Users List Subject: Re: HTTP connector to be aware of proxied SSL requests On 17/06/2010 01:41, Matt Peterson wrote: I can't find any documentation on the order of events for the Connector, so I'm not sure what other decisions get made based on the request attributes, but assume there are others. This is *open* source... Is there another solution to handling proxied SSL requests so that Catalina as well as our apps are aware that the requests are secure??? One possibility is to have two Connectors (1 using the secure, scheme and serverPort attributes for secure and 1 for non-secure) and have the LB connect to the appropriate Connector depending on the request. But this effectively doubles the amount of config needed to be managed (2nd set of config for LB + 2nd connector), which is considerable when dealing with 6 TC clusters each with their own set of LB config. The other option would be to proxy using AJP rather than HTTP (if the load-balancer supports it) since AJP passes SSL info as part of the protocol. If you want to use mixed HTTP/HTTPS in the LB and just HTTP on Tomcat than multiple connectors is usually what I'd recommend. Should I lodge an enhancement request for the Connector to become aware of proxied SSL requests (perhaps via an injected x-forwarded-proto header, ala WebLogic)? You can, not sure how much traction it would get. Both the logic and configuration is non-trivial to ensure only trusted proxies set the header. We try to keep the connector code fairly slick. This feels like more than we would want to add (bearing in mind this is just instinct - I haven't looked at any code at ths point). You might have better luck with an option to defer the redirection with the / to later in the processing chain. That would be simpler to implement but would add some extra processing that currently is bypassed by doing the rediection as early as possible. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: HTTP connector to be aware of proxied SSL requests
Hi Cyrille, We have the RemoteIpValve implemented already, thanks. The behaviour we are seeing is occurring in the Connector, before the request even reaches the valves. In this case, the request never reaches the valves as the redirect is done within the connector. Cheers, Matt. -Original Message- From: Cyrille Le Clerc [mailto:clecl...@xebia.fr] Sent: Friday, 18 June 2010 8:30 AM To: Tomcat Users List; Matthew Peterson Subject: Re: HTTP connector to be aware of proxied SSL requests Hello Matt, I think the RemoteIpValve does what you need : it looks at http headers filled in the request by preceding network components (layer 7 load balancer, ssl accelerator, etc) such as 'x-forwarded-for' to get the real ip address and 'x-forwarded-proto' to get the http/https protocol. A concept of internal/trusted incoming proxies is used to decide weither the http headers can be trusted or not. Configuration is detailed in the javadocs : http://tomcat.apache.org/tomcat-6.0-doc/api/org/apache/catalina/valves/RemoteIpValve.html The documentation of RemoteIpValve has been enhanced in Tomcat 7 to integrate the content of the java doc. I wrote a blog post in french to explain how it works with detailed diagrams here : http://blog.xebia.fr/2009/11/13/tomcat-ssl-communications-securisees-et-x-forwarded-proto/ Basically, if you want to trust http header x-forwarded-for and x-forwarded-proto coming from LB/web-server 192.168.0.10 and 192.168.0.11, the valve configuration will look like : Server ... ... Service name=Catalina Connector ... / Engine ... !-- Process X-Forwarded-For to get remote address and X-Forwarded-Proto to identify SSL requests -- Valve className=org.apache.catalina.valves.RemoteIpValve internalProxies=192\.168\.0\.10, 192\.168\.0\.11 protocolHeader=X-Forwarded-Proto / !-- AccessLogValve must be declared after RemoteIpValve to get the remote address and the scheme https/http -- Valve className=org.apache.catalina.valves.AccessLogValve directory=logs pattern=common prefix=access_log. resolveHosts=false suffix=.txt / ... /Host /Engine /Service /Server Please note that you can simplify the configuration omitting 'internalProxies' attribute and rely on the default that trusts all the class A, B C private IP addresses. Hope this helps, Cyrille -- Cyrille Le Clerc clecl...@xebia.fr http://blog.xebia.fr On Thu, Jun 17, 2010 at 2:41 AM, Matt Peterson matt.peter...@une.edu.au wrote: Hi All, We have a hardware load balancer terminating SSL requests before making a plain-text connection with Tomcat. So that all contexts are aware that the request is actually a secure request, we have implemented the RemoteIpValve with a LB injected header. This works well for our apps. However, we have noticed that there is some processing of the request happening within the connector, before the valves are processed. In particular, the redirecting to URLs with a trailing slash. Because this processing is occurring before the valves are processed the Connector still thinks that the original request was a non-secure one, even though it was not. The result is that requests to https://domain.name/context are redirected to http://domain.name/context/ instead of to https://domain.name/context/. This is not major, because our LB then redirects from http://domain.name/context/ to https://domain.name/context/ and all is good (except for the extra redirect). I can't find any documentation on the order of events for the Connector, so I'm not sure what other decisions get made based on the request attributes, but assume there are others. Is there another solution to handling proxied SSL requests so that Catalina as well as our apps are aware that the requests are secure??? One possibility is to have two Connectors (1 using the secure, scheme and serverPort attributes for secure and 1 for non-secure) and have the LB connect to the appropriate Connector depending on the request. But this effectively doubles the amount of config needed to be managed (2nd set of config for LB + 2nd connector), which is considerable when dealing with 6 TC clusters each with their own set of LB config. Should I lodge an enhancement request for the Connector to become aware of proxied SSL requests (perhaps via an injected x-forwarded-proto header, ala WebLogic)? Cheers, Matt. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: how to calculate a memory tomcat
Yup. Have you seen the Bugs Issues forum? http://www.lambdaprobe.org/forum2/forum.jspa?forumID=2start=0 or the Feature Request forum? http://www.lambdaprobe.org/forum2/forum.jspa?forumID=3 There are a lot of people who do think that it is lacking something, but their requests have been falling on deaf ears until the project was forked (for this very reason). While Lambda Probe sits stale and unattended, progress continues in dependent areas (Tomcat, JDBC, etc.) regardless. It did a great job for the era it was developed for, but has slipped behind more recent developments, that's all. Cheers, Matt. -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Wednesday, 16 June 2010 3:03 AM To: Tomcat Users List Subject: Re: how to calculate a memory tomcat -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Matt, On 6/14/2010 6:42 PM, Matthew Peterson wrote: Lambda Probe is stale. It has been forked to Psi Probe which has regular activity: http://code.google.com/p/psi-probe/ Is Lambda Probe stale? It may not have gotten any updates for a while, but is it really lacking anything? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkwXskIACgkQ9CaO5/Lv0PCH1ACfQByPb9MOuaHz0Ia5asvXm5Kb 2IAAoK/vXmq6pGUuFHV1VbSICAspz0In =9Rij -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Setting scheme on catalina Requests
I have discovered that the RemoteIPValve which has been shipped with Tomcat since v6.0.24 also performs the tasks am trying to perform with my valve. I had overlooked it previously due to its name. We are using v6.0.26, so I'll give it a whirl! Cheers, Matt. -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, 15 June 2010 8:06 AM To: Tomcat Users List Subject: Re: Setting scheme on catalina Requests -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Matt, On 6/13/2010 6:03 PM, Matt Peterson wrote: I am trying to develop a valve to modify requests based on a HTTP request header as set by our SSL terminating load balancer. The valve is to watch out for a particular header and when found, call the setSecure(true), setScheme(https) and setServerPort(443) methods so that the receiving servlet is aware that the request is a secure one. Why set the server port? You may end up confusing code that performs redirects and things like that. Can you get away with simply setSecure()/setScheme()? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkwWp88ACgkQ9CaO5/Lv0PCE6QCePtQgxZKDiDgB7GVjRK7HiM2k IA0An3Wnzw4RdIM1IMHw+q4WNFupseiv =EHI5 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Re: how to calculate a memory tomcat
Lambda Probe is stale. It has been forked to Psi Probe which has regular activity: http://code.google.com/p/psi-probe/ -Original Message- From: Myk Bova [mailto:syste...@narod.ru] Sent: Tuesday, 15 June 2010 2:07 AM To: Tomcat Users List Subject: Re: Re: how to calculate a memory tomcat Lambda Probe for Apache Tomcat ? http://www.lambdaprobe.org/d/index.htm 14.06.10, 19:54, Tobias Crefeld t...@cataneo.eu: Am Sun, 13 Jun 2010 22:27:40 +0700 schrieb andy susanto : is there any tool that i can monitor my tomcat ?, because at peak hour You should ask your preferred search engine after JMX. JDK offers some applications like jconsole (old fashioned but usually sufficent) or jvisualvm (looks nicer, less stable) to monitor the JVM. Additional, more tomcat-specific data can be monitored by the tomcat-app lamdba-probe. Regards, Tobias. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- /* ** */ Myk Bova Phone: +380447131381 Cell: +380983225480 ICQ: 157902492 Email: syste...@narod.ru Web: http://www.chantingwolf.narod.ru /* ** */ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Setting scheme on catalina Requests
Never mind. I found out how to do it. For anyone else interested, from a catalina.Request object you need to get the underlying coyote.Request object, access it's Scheme object (type MessageByte) and then set it's String value to https. I still don't understand why the catalina.Request object doesn't implement the convenience method setScheme(String) for this... Cheers, Matt. -Original Message- From: Matt Peterson [mailto:matt.peter...@une.edu.au] Sent: Monday, 14 June 2010 8:04 AM To: users@tomcat.apache.org Subject: Setting scheme on catalina Requests Using Tc 6.0.26, Java 6 on Win XP Pro. I am trying to develop a valve to modify requests based on a HTTP request header as set by our SSL terminating load balancer. The valve is to watch out for a particular header and when found, call the setSecure(true), setScheme(https) and setServerPort(443) methods so that the receiving servlet is aware that the request is a secure one. The setSecure() setServerPort() methods work as expected, but the setScheme() method does not set the scheme. I have looked into the source for catalina.connector.Request and have found the setScheme() method is made of a single line: // Not used. This would explain why the setScheme() method is not setting the scheme as I expect it would. So, how else could I set the scheme of the request to 'https'? There must be a way, because the http connector is able to set it if I use the 'scheme' attribute in the connectors XML config in server.xml. I have tried to find the code which does this, but have not been able to find it. Any help is ppreciated. Cheers, Matt. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Setting scheme on catalina Requests
Hi Mark, I cannot find another reference to the setScheme method by searching the tomcat-users archive (http://marc.info/?l=tomcat-userw=2r=1s=setSchemeq=b). Where else would I find some info on this topic? Cheers, Matt. -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Monday, 14 June 2010 9:58 AM To: Tomcat Users List Subject: Re: Setting scheme on catalina Requests On 13/06/2010 23:40, Matthew Peterson wrote: I still don't understand why the catalina.Request object doesn't implement the convenience method setScheme(String) for this... Try searching the archives. This has bee discussed previously. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
