Re: VERY HIGH TRAFFIC TUNING
On Wed, Jul 9, 2014 at 4:47 AM, Hernán Marsili her...@cmsmedios.com wrote: Hi, Hello Hernán, For the past 4 years we has been working with a 'stable' configuration in which we put APACHE in front of TOMCAT7 (previously Tomcat6) with mod_jk connector. We usually serve high traffic sites with about 7000 to 10.000 concurrent users per box (8gb RAM / 4 vcpu) (50.000 active users total). We are OK with the performance, but sometimes we notice Tomcat stops responding normally while there are at least 2 full CPU left to be consumed (JAVA memory is fine). Hard to tell from here, but dropping performance while still having resources is often an indicator for synchronization issues. You should analyze your thread usage. You could do it with jstack (save multiple stacks in short slots, like every 10 seconds for 2-3 minutes). Check what threads are in what states: - do you have any threads in BLOCKED state? If yes, what they are waiting for? - what are you RUNNABLE threads doing? Are they waiting for something, even not blocked - for example reading the database or reading the incoming request. - is your amount of TIMED_WAITING threads sufficient? If you have non, your thread pool is probably out of threads. This is the configuration we use for the connector: Connector port=8009 protocol=AJP/1.3 address=127.0.0.1 emptySessionPath=true redirectPort=8443 maxThreads=1024 minSpareThreads=32 enableLookups=false request.registerRequests=false / Generally removing apache httpd can increase performance. I assume you have a hardware loadbalancer in front of things, so httpd doesn't do you any good. I have a couple of questions: 1) should we set a particular connector or let Tomcat7 decide? I understand using protocol=AJP/1.3 the auto-switch kicks in. But, for non-SSL high concurrency sites maybe is best to fixed on APR? Warning, flame war is about to begin, but personally I always found that the plain old java connector is the best option for speed. And the simplest to use too. If you insist of having apache httpd in front, you may want to try mod_proxy (or was it mod_proxy_ajp?) instead. 2) how many THREADS can we have? can we go beyond the 1024? Depends on your OS. On modern linuxes sky is the limit. However context switches can kill you too. If you have very fast connections, got for a smaller amount. If you have keepalive and slow connections, remember that every connection can hold 1-2 threads without doing anything at all. But you can go up to 4096 without second thought, if you are really out of threads. However, if your problem are blocked threads or a slow DB, you will make things much much much worse. 3) is there any advantage on using processorCache? 4) We are not defining a CONNECTION TIMEOUT not a KEEP ALIVE. Any advice on this one? The average user session is 7 minutes. If playing with CONNECTION_TIMEOUT, than better on OS level. But again that depends on what is really happening within your application and with the connections. You could monitor it with netstat and see if you have too many CLOSE_WAITs or something. That it's easier to decide what to do. Thanks for the help! Hernán. Leon. P.S. needless to say, but having a monitoring tool like http://www.moskito.org will help either ;-)
Re: Servlet 3.1 asynchronous processing API
Hi,
On Wed, Jul 9, 2014 at 1:18 AM, Marko Asplund marko.aspl...@gmail.com
wrote:
Hi,
I'm just starting out with using the Servlet 3.1 asynchronous processing
API.
I use the API for generating response content, that can potentially be a
slow process, in another thread.
The code appears to be working, but since the specification contains many
caveats, I'd like to try and verify correctness.
There's a couple of parts in particular that I'm wondering about:
- what's the correct way to deliver error response to client? Can I just
cast the response acquired from AsyncContext to HttpServletResponse and use
the normal Servlet API mechanisms?
- does the code contain thread-safety issues in particular related to the
response object and OutputStream?
Here's a simplified version of my code:
@Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
final AsyncContext asyncContext = req.startAsync();
asyncContext.start(new Runnable() {
@Override
public void run() {
HttpServletResponse response = null;
try {
response = (HttpServletResponse)asyncContext;
Maybe it is implementation detail that this cast works but usually you have
to use
http://docs.oracle.com/javaee/6/api/javax/servlet/AsyncContext.html#getResponse()
to get the response
if(hasOperationFailed()) {
response.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
} else {
response.getOutputStream().write(generateResponseContent().getBytes());
}
} catch (IOException e) {
try {
response.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
} catch (IOException e1) {
e1.printStackTrace();
}
}
asyncContext.complete();
}
});
}
Any feedback and pointers would be welcome.
marko
Tomcat (v 6.0.32) write response to wrong client socket
Hello, We got a serious problem on our online banking applications: a user U1 of bank A got to see the data of another user U2 of another bank B. It happened only once, bevor and after that went everything well. The two same online banking applications are running on one instance of tomcat (V.6.0.32) and have different backend path. The logs show that the problem muss be not on the side of web application (frondend), backend and firewall. The senario looks as following: 1. U1 log on the online banking of bank A and U2 log on the online banking of bank B, and the two sessions have been running well a while till 2. later at the exactly same time U1 wanted to see the start page and U2 booking detail 3. U1 got to see the false data of the booking detail of U2. 4. after that the two sessions went further well without any problem In the access log of the tomcat seems everything went well: Access log Bank A (U1): 10.25.4.8 - - F862B9AD5DA9AC4A0D38B46D4E5B0D6C [15/Jun/2014:16:03:41 +0200] HTTP/1.1 GET /finprdcbo/defAccountStartPage.account ?DIRTY=YDEFAULT=1node=STARTSEITE 200 69000 /ebanking/defAccountStatementDetail.account Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:29.0) Gecko/20100101 Firefox/29.0 Access log Bank B (U2): 10.25.4.8 - - 12A6E21F0D6321A95C553B160DBCC9A0 [15/Jun/2014:16:03:40 +0200] HTTP/1.1 POST /finprdzrb/defAccountStatementOfAccount.account 200 45765 /ebanking/defAccountAssetOverview.account Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0 But the reponse of the request for booking detail from U 2 was mistakenly sent to U1. We cann confirm this by looking into the logs of firewall as following: Firewall log of Session U2: Jun 15 16:04:16 Web-Requests Access m:WR-SG-SUMMARY vhost:wwwsec.zrb.clientis.ch:443 (https) POST /ebanking/defAccountStatementOfAccount.account = https://10.25.2.43:5215/finprdzrb/defAccountStatementOfAccount.account , status:n/a , redirection URL:n/a , referer:/ebanking/defAccountAssetOverview.account , mapping:blappl-zrb , request size: 1089 , backend response size: n/a , audit token:308792478954626740 rid:U52nvH8AAAEAAAoy1CIAAAno sid:384490af19eb229b7f0874b6ef0323c8 ip:84.74.211.190 12 Fact : backend response size: n/a means there is no reponse from tomcat for the request of POST /ebanking/defAccountStatementOfAccount.account and a timeout is trigged Firewall log of Session U1: Jun 15 16:03:40 Web-Requests Access SG_child[14145]: m:WR-SG-SUMMARY vhost:wwwsec.oberuzwil.clientis.ch:443 (https) GET /ebanking/defAccountStartPage.account?DIRTY=YDEFAULT=1node=STARTSEITE = https://10.25.2.43:5215/finprdcbo/defAccountStartPage.account , status:200 , redirection URL:n/a , referer:/ebanking/defAccountStatementDetail.account , mapping:blappl-cbo , request size: 614 , backend response size: 45765 , audit token:147310158715306970 request total 287469 , allow/deny filters 3780 , backend responsiveness 208455 , response processing 74282 , ICAP reqmod n/a , ICAP respmod n/a rid:U52nvH8AAAEAAATYu6cAAATi sid:1dad4372b4a2f67d980e6e195aa954fe ip:84.73.20.65 12 Fact: backend response size: 45765 means the reponse of Request of U2 is mistakenly passed to U1. See the same size of the response in the access log Bank B (U2): My questions: 1. Is such a problem (bug) already known? 2. When will the tomcat access log be written? after sent of response or bevor? 3. how could the problem happen on the side of tomcat? 4. wo could the problem be hidden otherwise? Thanks for your help ! Yanchun Yang
Re: HttpServletRequest - getHeaders() vs getCookies()
Hi, I had a look at the documentation and the tomcat source to get a better understanding of what the '|org.apache.catalina.connector.RECYCLE_FACADE' parameter actually does.| I have seen that Tomcat objects like Cookies, Request etc. are designed to be reusable. What I currently do not understand is: In which scenario and what context are they going to be reused? I see there are Endpoints classes (like NIOEndpoint) which are used to process the different requests. This seems to be the most likely entry point into the scenario. Maybe somebody can provide some general outline of how requests and the reusing of the object actually works together? Is there some kind of relation to the IP of an incoming request? In our code we hold a reference to the httpSession outside of the request (in a application scoped map). We use this session object to read the SessionId, to invalidate or to adjust the session timeout from outside a client request. I finally received the logs where the session itself was created: Basically there is some communication ongoing between client and server, then the server receives an unexpected command and decides to invalidate the session because of this. Later another command of the client with the old cookie arrives and is rejected by our server. After this any other request coming from the same IP fails. As mentioned in the logs we see the httpHeader that does contain a cookie with a different value, but the getCookies() method always provides the cookie value of the 'original session'. Around the same time when the original session was still alive I found an error message in the Catalina log: ... org.apache.coyote.AbstractProtocol$AbstractConnectionHandler process SEVERE: null java.lang.IllegalStateException: Calling [asyncPostProcess()] is not valid for a request with Async state [STARTED] at org.apache.coyote.AsyncStateMachine.asyncPostProcess(AsyncStateMachine.java:202) at org.apache.coyote.AbstractProcessor.asyncPostProcess(AbstractProcessor.java:116) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) I am not sure whether this is somewhat related to the problem scenario, but at least this looks like something that should not happen. Regards, Simon Kulessa Am 08.07.2014 11:35, schrieb Simon Kulessa: Hi Konstantin, thanks for you answer. Am 08.07.2014 10:22, schrieb Konstantin Kolinko: 2014-07-08 11:48 GMT+04:00 Simon Kulessa kule...@flexsecure.de: Hi, we are running a server on Apache Tomcat 7.0.29 with usage of the servlet-api 3.0.1. 1. The current version is 7.0.54 We updated to this version now, but since our server is already running at the customer exchanging the tomcat might not be a option for now. Before we can do this, we should at least have an idea about where the problem is coming from. 2. What is 3.0.1. You may use any servlet-api jar when compiling, but at run time you must use the one that comes with Tomcat. Ok. So correct would be we compile against the servlet-api 3.0.1 (maven dependecy groupId=javax.servlet, artifactId=javax.servlet-api) and use the one from tomcat in production. The tomcat version seems to provide a different version 3.0.FR (taken from the manifest). Could this be an issue? From one of our customers we received a bug report that in certain times an IOException (Stream is closed) is thrown while reading the data from a HttpServletRequest. After some investigation we added some logging and now we see the following information: First we logged the headers of the http request (by using HttpServletRequest#getHeaders()) and we see a JSESSIONID cookie with a value of 'A'. After this we use HttpServletRequest#getCookies() to log the cookies as well. Here we only see a JSESSIONID cookie with a value of 'B'. The requested sessionId from the servletRequest has value 'A' as well. My question is: How can this be? How is it possible that getCookies returns a cookie that is not defined in the header of the http request? Where does this cookie come from? And were did the cookie go that was defined in the header? 3. Usually that is caused by bad programming in a web application, when newbie programmers share request object between requests. The unexpected Stream is closed issues are also symptoms of that. An example: http://markmail.org/thread/pxh6bk6saowy3rk6 Currently I see nothing in our code that indicates this, but I will have another look at that, since this seems to be the most likely explanation. 4. I think getHeaders() may return some garbage for Cookie header (as the value may be corrupted when parsing the value into a cookie), but it
Re: VERY HIGH TRAFFIC TUNING
Hi. This kind of question is very difficult to answer reasonably, for anyone that is not you and does not have direct access to your system, to see what happens, when it happens. The general and reasonable answer would be that you need to use some monitoring tools, to find out where exactly the bottleneck is, and then, very carefully, start tuning your system one parameter at a time, to try to improve the situation. The worst thing to do is to start changing multiple things without really knowing what they are doing, because then you will get very confused very quickly. Ah, one more thing : the default parameters of both Apache httpd and Tomcat, are chosen by people who know what they are doing, to cover a majority of reasonable cases. So, changing these parameters without knowing exactly what each parameters does, and how different parameters interact with one another, is always risky. (This being said, the mere fact that you are asking on this list before doing that, is a bonus point for you). All this being said, a couple of notes below : Hernán Marsili wrote: Hi, For the past 4 years we has been working with a 'stable' configuration in which we put APACHE in front of TOMCAT7 (previously Tomcat6) with mod_jk connector. We usually serve high traffic sites with about 7000 to 10.000 concurrent users per box (8gb RAM / 4 vcpu) (50.000 active users total). So, basically, your configuration is fine and has been running reliably and in a stable way for 4 years, including a Tomcat version change. Good. One more reason for only changing things carefully. We are OK with the performance, but sometimes we notice Tomcat stops responding normally while there are at least 2 full CPU left to be consumed (JAVA memory is fine). You may want to indicate a bit more precisely what you mean by sometimes and by stops responding normally. This is the configuration we use for the connector: Connector port=8009 protocol=AJP/1.3 address=127.0.0.1 emptySessionPath=true redirectPort=8443 maxThreads=1024 minSpareThreads=32 enableLookups=false request.registerRequests=false / Note # 1 : you say that you have up to 10,000 concurrent users. Yet, there are only 1024 Threads in Tomcat. Users is not necessarily equal to requests, but let's assume for a moment that they are. Basically, Tomcat will use one Thread to process one HTTP request, from the time the request is received, to the time when the response to that request has been sent back to the user. So, maybe, there are times when your Tomcat is running out of available Threads to process all the requests that come in at some moments ? If that is the case, what will happen is : - The TCP/IP stack on Tomcat will accept the client connection - but this connection will be put in a queue, waiting for a thread to become available (iow : when it finishes the current request that it is processing). If all 1024 threads are currently busy processing requests (or waiting for additional requests from the same client, because of the keep-alive timeout, see below), then it will appear for the client as if Tomcat is not responding normally. I have a couple of questions: 1) should we set a particular connector or let Tomcat7 decide? I understand using protocol=AJP/1.3 the auto-switch kicks in. But, for non-SSL high concurrency sites maybe is best to fixed on APR? I cannot answer that and will wait for someone else more qualified to do that. 2) how many THREADS can we have? can we go beyond the 1024? Yes, there is no limit other than the available memory and the general performance of the machine. That is also a very easy parameter to change, and one that does not have a lot of obscure side-effects. Apart from everything else, I would suggest raising it to e.g. 4096, and see what happens. (As someone else commented however : if the problem is not really in Tomcat, but in some back-end database server, then this will make things worse). 3) is there any advantage on using processorCache? don't know. 4) We are not defining a CONNECTION TIMEOUT not a KEEP ALIVE. Any advice on this one? The average user session is 7 minutes. I do not rememember what the default value is for keep-alive. But this may also be something easy to adjust, and with potentially big effects. If it is currently anywhere larger than some 5 seconds, change it to 5 seconds maximum. The keep-alive logic was introduced at a time when networks were slower, and when setting up a new TCP/IP connection between a client and a server for each new request, was quite expensive. So the idea was : after a first request by a client, let's leave the connection open, to see if this client has more requests to send, over that same connection, within just a few seconds. This way, we avoid closing the connection each time, and have to re-create a connection for each request. (Example : a HTML page, with a lot of img tags in it.). Unfortunately, what happens in a
Re: Servlet 3.1 asynchronous processing API
On 08/07/2014 23:18, Marko Asplund wrote: Hi, I'm just starting out with using the Servlet 3.1 asynchronous processing API. I use the API for generating response content, that can potentially be a slow process, in another thread. The code appears to be working, but since the specification contains many caveats, I'd like to try and verify correctness. There's a couple of parts in particular that I'm wondering about: - what's the correct way to deliver error response to client? Can I just cast the response acquired from AsyncContext to HttpServletResponse and use the normal Servlet API mechanisms? Personally, I'd do a dispatch since that completes the AsyncContext as well. Your approach should work. Which is better is probably a matter of style. - does the code contain thread-safety issues in particular related to the response object and OutputStream? Ensuring thread-safe access of those objects is entirely an application responsibility. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat (v 6.0.32) write response to wrong client socket
On 09/07/2014 08:43, yanchun.y...@finnova.ch wrote: Hello, We got a serious problem on our online banking applications: a user U1 of bank A got to see the data of another user U2 of another bank B. It happened only once, before and after that went everything well. The two same online banking applications are running on one instance of tomcat (V.6.0.32) and have different backend path. That is quite an old version with a number of public security vulnerabilities to be using for a banking application. The logs show that the problem must be not on the side of web application (frontend), backend and firewall. The scenario looks as following: 1. U1 log on the online banking of bank A and U2 log on the online banking of bank B, and the two sessions have been running well a while till 2. later at the exactly same time U1 wanted to see the start page and U2 booking detail 3. U1 got to see the false data of the booking detail of U2. 4. after that the two sessions went further well without any problem In the access log of the tomcat seems everything went well: Access log Bank A (U1): 10.25.4.8 - - F862B9AD5DA9AC4A0D38B46D4E5B0D6C [15/Jun/2014:16:03:41 +0200] HTTP/1.1 GET /finprdcbo/defAccountStartPage.account ?DIRTY=YDEFAULT=1node=STARTSEITE 200 69000 /ebanking/defAccountStatementDetail.account Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:29.0) Gecko/20100101 Firefox/29.0 Access log Bank B (U2): 10.25.4.8 - - 12A6E21F0D6321A95C553B160DBCC9A0 [15/Jun/2014:16:03:40 +0200] HTTP/1.1 POST /finprdzrb/defAccountStatementOfAccount.account 200 45765 /ebanking/defAccountAssetOverview.account Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0 These logs show the requests coming from the same IP address. Since the requests are from different users then there must be something acting as a reverse proxy. Is it the firewall? Is it some other component? But the response of the request for booking detail from U2 was mistakenly sent to U1. We can confirm this by looking into the logs of firewall as following: Firewall log of Session U2: Jun 15 16:04:16 Web-Requests Access m:WR-SG-SUMMARY vhost:wwwsec.zrb.clientis.ch:443 (https) POST /ebanking/defAccountStatementOfAccount.account = https://10.25.2.43:5215/finprdzrb/defAccountStatementOfAccount.account , status:n/a , redirection URL:n/a , referer:/ebanking/defAccountAssetOverview.account , mapping:blappl-zrb , request size: 1089 , backend response size: n/a , audit token:308792478954626740 rid:U52nvH8AAAEAAAoy1CIAAAno sid:384490af19eb229b7f0874b6ef0323c8 ip:84.74.211.190 12 Fact : backend response size: n/a means there is no reponse from tomcat for the request of POST /ebanking/defAccountStatementOfAccount.account and a timeout is trigged Firewall log of Session U1: Jun 15 16:03:40 Web-Requests Access SG_child[14145]: m:WR-SG-SUMMARY vhost:wwwsec.oberuzwil.clientis.ch:443 (https) GET /ebanking/defAccountStartPage.account?DIRTY=YDEFAULT=1node=STARTSEITE = https://10.25.2.43:5215/finprdcbo/defAccountStartPage.account , status:200 , redirection URL:n/a , referer:/ebanking/defAccountStatementDetail.account , mapping:blappl-cbo , request size: 614 , backend response size: 45765 , audit token:147310158715306970 request total 287469 , allow/deny filters 3780 , backend responsiveness 208455 , response processing 74282 , ICAP reqmod n/a , ICAP respmod n/a rid:U52nvH8AAAEAAATYu6cAAATi sid:1dad4372b4a2f67d980e6e195aa954fe ip:84.73.20.65 12 Fact: backend response size: 45765 means the reponse of Request of U2 is mistakenly passed to U1. See the same size of the response in the access log Bank B (U2):s The firewall appears to be transforming the URL. That suggests it is the firewall that is acting as the reverse proxy. (Aside: Transforming the URL like this is doable but does require very careful configuration to ensure everything still works properly) My questions: 1. Is such a problem (bug) already known? Have you looked in the changelog? Mixing up responses would be a security issue so have you looked in http://tomcat.apache.org/security-6.html 2. When will the tomcat access log be written? after sent of response or before? It isn't quite that simple. In Tomcat 6 (it changes in 7.0.x) the access log is written after the application has finished writing the response to the buffer but before the content of that buffer is written to the client. 3. how could the problem happen on the side of tomcat? Depending on exactly how the application works, CVE-2011-3375 might be able to trigger this but given that requires an error to occur it looks unlikely. 4. how could the problem be hidden otherwise? Most errors of this type are eventually traced to application errors. The most frequent error is the application retaining a reference to the response and/or request objects beyond a single request. Mark
Re: Servlet 3.1 asynchronous processing API
On Wed, 09 Jul 2014 07:23 Martin Grigorov wrote:
...
try {
response = (HttpServletResponse)asyncContext;
Maybe it is implementation detail that this cast works but usually you
have
to use
http://docs.oracle.com/javaee/6/api/javax/servlet/AsyncContext.html#getResponse()
to get the response
This was one of the things I was wondering about: calling
AsyncContext#getResponse() returns a ServletResponse instance that doesn't
implement the HTTP protocol specific sendError etc. methods.
What would be the correct way to return an error response (including HTTP
status code) to the client in this case?
marko
Re: Servlet 3.1 asynchronous processing API
On Wed, Jul 9, 2014 at 12:48 PM, Marko Asplund marko.aspl...@gmail.com
wrote:
On Wed, 09 Jul 2014 07:23 Martin Grigorov wrote:
...
try {
response = (HttpServletResponse)asyncContext;
Maybe it is implementation detail that this cast works but usually you
have
to use
http://docs.oracle.com/javaee/6/api/javax/servlet/AsyncContext.html#getResponse()
to get the response
This was one of the things I was wondering about: calling
AsyncContext#getResponse() returns a ServletResponse instance that doesn't
implement the HTTP protocol specific sendError etc. methods.
Casting ServletResponse to HttpServletResponse is OK most of the time. You
can check with instanceof before doing the cast.
But casting AsyncContext to HttpServletResponse is way too brave.
What would be the correct way to return an error response (including HTTP
status code) to the client in this case?
marko
Handshake Failure error !
Hi, We are running Tomcat 6.0.37 and Java JDK 1.6.0_60 We recently upgraded to JDK 1.6.0_75 and recieved below error at several places javax.net.ssl.SSLException: Fatal Alert received: Handshake Failure We debugged and after analysis found that if we remove below 3 ciphers suits from server.xml file TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA SSL_RSA_WITH_3DES_EDE_CBC_SHA The error is no more seen. I need your opinion in order to proceed with the changes. 1.What will be the effect of removing these cipher? 2. Found this link on ciphers http://docs.oracle.com/javase/7/docs/technotes/guides/security/SunProviders.html The cipher codes I mentioned above have been marked as 'X'. Most of the cipher codes mentioned in my server.xml are marked as 'X'. So I am confused as to am I on correct path of removing these problematic cipher from server.xml or not. Can you help in answering the 2 questions above? Thanks for your help in advance. -Utkarsh
erro undeploy tomcat + jsf
como corrigir isso? jul 09, 2014 9:22:01 AM org.apache.catalina.loader. WebappClassLoader clearReferencesThreads GRAVE: The web application [/jsf] appears to have started a thread named [Mojarra-WebResourceMonitor-1-thread-1] but has failed to stop it. This is very likely to create a memory leak.
Re: Tomcat (v 6.0.32) write response to wrong client socket
if Tomcat is fronted by a web server then I would also check the connector. I've seen similar issues in the past with IIS and a faulty AJP connector. switching to nginx with an APR connector resolved it in my case. Igal On 7/9/2014 2:49 AM, Mark Thomas wrote: On 09/07/2014 08:43, yanchun.y...@finnova.ch wrote: Hello, We got a serious problem on our online banking applications: a user U1 of bank A got to see the data of another user U2 of another bank B. It happened only once, before and after that went everything well. The two same online banking applications are running on one instance of tomcat (V.6.0.32) and have different backend path. That is quite an old version with a number of public security vulnerabilities to be using for a banking application. The logs show that the problem must be not on the side of web application (frontend), backend and firewall. The scenario looks as following: 1. U1 log on the online banking of bank A and U2 log on the online banking of bank B, and the two sessions have been running well a while till 2. later at the exactly same time U1 wanted to see the start page and U2 booking detail 3. U1 got to see the false data of the booking detail of U2. 4. after that the two sessions went further well without any problem In the access log of the tomcat seems everything went well: Access log Bank A (U1): 10.25.4.8 - - F862B9AD5DA9AC4A0D38B46D4E5B0D6C [15/Jun/2014:16:03:41 +0200] HTTP/1.1 GET /finprdcbo/defAccountStartPage.account ?DIRTY=YDEFAULT=1node=STARTSEITE 200 69000 /ebanking/defAccountStatementDetail.account Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:29.0) Gecko/20100101 Firefox/29.0 Access log Bank B (U2): 10.25.4.8 - - 12A6E21F0D6321A95C553B160DBCC9A0 [15/Jun/2014:16:03:40 +0200] HTTP/1.1 POST /finprdzrb/defAccountStatementOfAccount.account 200 45765 /ebanking/defAccountAssetOverview.account Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0 These logs show the requests coming from the same IP address. Since the requests are from different users then there must be something acting as a reverse proxy. Is it the firewall? Is it some other component? But the response of the request for booking detail from U2 was mistakenly sent to U1. We can confirm this by looking into the logs of firewall as following: Firewall log of Session U2: Jun 15 16:04:16 Web-Requests Access m:WR-SG-SUMMARY vhost:wwwsec.zrb.clientis.ch:443 (https) POST /ebanking/defAccountStatementOfAccount.account = https://10.25.2.43:5215/finprdzrb/defAccountStatementOfAccount.account , status:n/a , redirection URL:n/a , referer:/ebanking/defAccountAssetOverview.account , mapping:blappl-zrb , request size: 1089 , backend response size: n/a , audit token:308792478954626740 rid:U52nvH8AAAEAAAoy1CIAAAno sid:384490af19eb229b7f0874b6ef0323c8 ip:84.74.211.190 12 Fact : backend response size: n/a means there is no reponse from tomcat for the request of POST /ebanking/defAccountStatementOfAccount.account and a timeout is trigged Firewall log of Session U1: Jun 15 16:03:40 Web-Requests Access SG_child[14145]: m:WR-SG-SUMMARY vhost:wwwsec.oberuzwil.clientis.ch:443 (https) GET /ebanking/defAccountStartPage.account?DIRTY=YDEFAULT=1node=STARTSEITE = https://10.25.2.43:5215/finprdcbo/defAccountStartPage.account , status:200 , redirection URL:n/a , referer:/ebanking/defAccountStatementDetail.account , mapping:blappl-cbo , request size: 614 , backend response size: 45765 , audit token:147310158715306970 request total 287469 , allow/deny filters 3780 , backend responsiveness 208455 , response processing 74282 , ICAP reqmod n/a , ICAP respmod n/a rid:U52nvH8AAAEAAATYu6cAAATi sid:1dad4372b4a2f67d980e6e195aa954fe ip:84.73.20.65 12 Fact: backend response size: 45765 means the reponse of Request of U2 is mistakenly passed to U1. See the same size of the response in the access log Bank B (U2):s The firewall appears to be transforming the URL. That suggests it is the firewall that is acting as the reverse proxy. (Aside: Transforming the URL like this is doable but does require very careful configuration to ensure everything still works properly) My questions: 1. Is such a problem (bug) already known? Have you looked in the changelog? Mixing up responses would be a security issue so have you looked in http://tomcat.apache.org/security-6.html 2. When will the tomcat access log be written? after sent of response or before? It isn't quite that simple. In Tomcat 6 (it changes in 7.0.x) the access log is written after the application has finished writing the response to the buffer but before the content of that buffer is written to the client. 3. how could the problem happen on the side of tomcat? Depending on exactly how the application works, CVE-2011-3375 might be able to trigger this but given that requires an error to occur it looks unlikely. 4. how could the problem be hidden otherwise? Most errors of this
Re: VERY HIGH TRAFFIC TUNING
Thank you for the answer. I will begging experimenting on 1 box with this configuration: Connector port=8009 protocol=AJP/1.3 address=127.0.0.1 emptySessionPath=true redirectPort=8443 maxThreads=2048 minSpareThreads=32 connectionTimeout=2 keepAliveTimeout=1 enableLookups=false request.registerRequests=false / I a couple of questions raises from your reply: 1) a couple of years ago we did a BENCHMARK mod_proxy vs mod_jk and the difference was noticeable in favor of mod_jk. There was any improvement on mod_proxy connector in particular? 2) this is definitely an ignorant question, but I see a lot of connections made to 8080 port instead of 8009. That said, I don't fully understand WHY and also, do I need to also tuned this? I have nothing going directly to tomcat, everything goes trough Apache mod_jk. Connector executor=tomcatThreadPool port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / -- View this message in context: http://tomcat.10.x6.nabble.com/VERY-HIGH-TRAFFIC-TUNING-tp5019838p5019875.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Servlet 3.1 asynchronous processing API
On 09/07/2014 09:16, Mark Thomas wrote: There's a couple of parts in particular that I'm wondering about: - what's the correct way to deliver error response to client? Can I just cast the response acquired from AsyncContext to HttpServletResponse and use the normal Servlet API mechanisms? Personally, I'd do a dispatch since that completes the AsyncContext as well. Your approach should work. Which is better is probably a matter of style. Thanks Mark! Dispatching to a separate error handler might be the cleaner approach. I guess that would involve passing error status and other info as needed to the dispatch target in the request object. So, with the response.sendError approach I would still need to finally invoke asyncContext.complete() in order to complete request processing, right? - does the code contain thread-safety issues in particular related to the response object and OutputStream? Ensuring thread-safe access of those objects is entirely an application responsibility. Are there any potential thread-safety issues in the sample code related to how the application and container threads share the response object? For example if the application touches HttpServletResponse headers, status code or OutputStream, then invokes AsyncContext#complete(), can this result in thread-safety issues with the container thread when the container serializes the response to the client? As an aside, from an application developer point of view, the JAX-RS 2 way of completing asynchronous request processing in both happy path and erroneous scenarios with AsyncResponse#resume() seems quite handy. I realize that these are different abstraction levels and that Servlet API doesn't e.g. include exception mapper concept. marko
Re: Servlet 3.1 asynchronous processing API
On Wed, 09 Jul 2014 09:59 Martin Grigorov wrote: Casting ServletResponse to HttpServletResponse is OK most of the time. You can check with instanceof before doing the cast. But casting AsyncContext to HttpServletResponse is way too brave. Quite true, I'm not usually that courageous on purpose :-) Meant to write asyncContext.getResponse(). And as you suggested, adding the type check using instanceof would obviously make a lot of sense. marko
Re: Servlet 3.1 asynchronous processing API
On 09/07/2014 16:53, Marko Asplund wrote: On 09/07/2014 09:16, Mark Thomas wrote: There's a couple of parts in particular that I'm wondering about: - what's the correct way to deliver error response to client? Can I just cast the response acquired from AsyncContext to HttpServletResponse and use the normal Servlet API mechanisms? Personally, I'd do a dispatch since that completes the AsyncContext as well. Your approach should work. Which is better is probably a matter of style. Thanks Mark! Dispatching to a separate error handler might be the cleaner approach. I guess that would involve passing error status and other info as needed to the dispatch target in the request object. So, with the response.sendError approach I would still need to finally invoke asyncContext.complete() in order to complete request processing, right? Right. - does the code contain thread-safety issues in particular related to the response object and OutputStream? Ensuring thread-safe access of those objects is entirely an application responsibility. Are there any potential thread-safety issues in the sample code related to how the application and container threads share the response object? For example if the application touches HttpServletResponse headers, status code or OutputStream, then invokes AsyncContext#complete(), can this result in thread-safety issues with the container thread when the container serializes the response to the client? I don;t see any. The basic rule is make sure that the application thread doesn't modify any container provided object (request, response, outputstream etc.) after you have called complete(). That is the point where control returns to the container. Mark As an aside, from an application developer point of view, the JAX-RS 2 way of completing asynchronous request processing in both happy path and erroneous scenarios with AsyncResponse#resume() seems quite handy. I realize that these are different abstraction levels and that Servlet API doesn't e.g. include exception mapper concept. marko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Problem with Transfer-Encoding
On 7/8/2014 2:52 AM, Konstantin Kolinko wrote: 2014-07-07 21:07 GMT+04:00 Terence M. Bandoian tere...@tmbsw.com: On 7/5/2014 6:36 PM, André Warnier wrote: I agree with André about the difficulties of debugging character encodings. A couple of things you might check are the character encodings of the page and the form. The character encoding of the page may be set with the Content-type meta tag: meta http-equiv=ontent-type content=text/html;charset=UTF-8/ Setting explicit value for content attribute like that is risky. The value must match the Content-Type HTTP header produced by web server. If they do not match, some browsers ignore both and start guessing the encoding. I usually write it as %@ page contentType=ext/html; charset=UTF-8 % ... META http-equiv=ontent-type content=%=response.getContentType() % Best regards, Konstantin Kolinko Hi, Konstantin- With JSP pages, I use contentType attribute of the page directive as well but didn't think to include it in my reply. Thanks for pointing it out. -Terence Bandoian - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Servlet 3.1 asynchronous processing API
Hi, I did some more experimenting with both error handling approaches. Tomcat 7.0.54 and 8.0.9 seemed to work a bit differently: on 8.0.9 some requests timed out with Exception while processing an asynchronous request error message. I was unable to reproduce this on TC 7.0.54 or Resin 4.0.40. What could be causing this? A bug in my servlet code? The code and results can be found here: https://github.com/marko-asplund/servlet3-async marko
Re: Embedded Tomcat question
Well, thanks for the helpful advice thus far..
So, now that I've added a listener properly, I get this:
java.lang.UnsupportedOperationException: Section 4.4 of the Servlet
3.0 specification does not permit this method to be called from a
ServletContextListener that was not defined in web.xml, a
web-fragment.xml file nor annotated with @WebListener
at
org.apache.catalina.core.StandardContext$NoPluggabilityServletContext.addListener(StandardContext.java:7019)
When I try adding my listener...
@WebListener
public class CdiServletContextListener implements ServletContextListener
{
@Override
public void contextInitialized(ServletContextEvent servletContextEvent)
{
servletContextEvent.getServletContext().addListener(new
CdiCtrlListener());
}
I even tried rigging my class to have WebListener on it.. until I
noticed the method just threw an exception... no questions asked. :-)
On Tue, Jul 1, 2014 at 12:37 PM, Mark Thomas ma...@apache.org wrote:
On 01/07/2014 17:16, John D. Ament wrote:
I looked for the source code, at least on github, there's no tag for
7.0.55 defined (see [1])
Given you are using 7.0.54, why would you look for a tag for 7.0.55?
How do I get access to a StandardContext? CAn I cast the Context object?
Yes. It would have taken less time to try that than it did to type the
question.
Mark
[1]: https://github.com/apache/tomcat
On Mon, Jun 30, 2014 at 4:51 AM, Mark Thomas ma...@apache.org wrote:
On 30/06/2014 01:27, John D. Ament wrote:
I spoke a little too quickly, ServletContext is avilable from ctx
(just didn't search hard enough). Adding the request listener here
though results in:
java.lang.NullPointerException
at
org.apache.catalina.core.ApplicationContext.createListener(ApplicationContext.java:1402)
at
org.apache.catalina.core.ApplicationContext.addListener(ApplicationContext.java:1307)
at
org.apache.catalina.core.ApplicationContextFacade.addListener(ApplicationContextFacade.java:636)
A quick look in the source code will tell you why you are getting an NPE
here.
You can't add a ServletRequestListener directly. There are a couple of
ways to do this. Probably the simplest is to create a
ServletContextListener, add that directly to Tomcat's StandardContext
with addApplicationLifecycleListener() and then from that
ServletContextListener add and request and/or session listeners from the
contextInitialized() event.
You could also do this with a ServletContainerInitializer.
Mark
On Sun, Jun 29, 2014 at 8:21 PM, John D. Ament john.d.am...@gmail.com
wrote:
Hi,
I'm on Tomcat 7.0.54, though I would hope your APIs aren't volatile
enough that your internals should matter..
I tried this:
Wrapper wrapper =
Tomcat.addServlet(ctx,Greeter,GreeterServlet.class.getName());
wrapper.addMapping(/*);
wrapper.getServlet().getServletConfig().getServletContext().addListener(MyListener.class);
This results in a NPE. This makes me think I don't have access to the
context yet. I would actually think it's somewhere in this chain:
tomcat = new Tomcat();
tomcat.setPort(8080);
File base = new File(target/webapp-runner);
if(!base.exists()) {
base.mkdirs();
}
Context ctx = tomcat.addContext(/,base.getAbsolutePath());
however, it doesn't look like Context here is a ServletContext.
On Sun, Jun 29, 2014 at 6:30 PM, Caldarale, Charles R
chuck.caldar...@unisys.com wrote:
From: John D. Ament [mailto:john.d.am...@gmail.com]
Subject: Embedded Tomcat question
Playing around a bit with embedded tomcat. It looks like there are
APIs to add all of the tomcat specific listeners, however how would I
add a ServletRequestListener?
Look in the servlet spec for the version of Tomcat you're using (which
you didn't mention), and the JavaDoc for ServletContext (e.g.,
addListener).
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
