Re: parallel webapp initialization
Alexander Knöller wrote: Hello mailing list. We use a single tomcat instance (soon switching from 5.5.23 to 7.x) with 24 webapps. Each webapp is based on spring and hibernate doing a lot of I/O during initialization. Tomcat is often restarted which causes long downtimes. As far as i can see the tomcat initializes its wepapps sequentially. So despite the fact, that our tomcat-intance runs on a 8 core linux system, tomcat seems to use a single thread to initalize the webapps. I'd love to see such patch as well, we have ~20 webaps with a total hundreds of megabytes of WAR files and full tomcat restart takes like 2 minutes on 24 thread machine with SSD storage subsystem. Some tuning options are needed as well, as webapp initialization can be IO bound on some storage so having more threads can actually make it slower ( once disk heads start moving... ). Good and safe default would be 1 init thread and max bound by cpus available in the system ? Regards, Darius. -- View this message in context: http://old.nabble.com/parallel-webapp-initialization-tp32619136p32620647.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Request params randomly null in servlet(s)
Pid * wrote: What is your server.xml config? (please remove comments, passwords etc post it inline, in the response) Here it is, nothing special ( except usage of RemoteIpValve ). Thanks for looking into this problem. ?xml version='1.0' encoding='utf-8'? Server port=8005 shutdown=SHUTDOWN Listener className=org.apache.catalina.core.AprLifecycleListener SSLEngine=on / Listener className=org.apache.catalina.core.JasperListener / Listener className=org.apache.catalina.core.JreMemoryLeakPreventionListener/ Listener className=org.apache.catalina.mbeans.GlobalResourcesLifecycleListener / Listener className=org.apache.catalina.core.ThreadLocalLeakPreventionListener / Resource name=UserDatabase auth=Container type=org.apache.catalina.UserDatabase description=User database that can be updated and saved factory=org.apache.catalina.users.MemoryUserDatabaseFactory pathname=conf/tomcat-users.xml / /GlobalNamingResources Service name=Catalina Connector port=8080 protocol=org.apache.coyote.http11.Http11NioProtocol connectionTimeout=2 enableLookups=false acceptCount=1000 maxThreads=500 maxPostSize=10485760 / Engine name=Catalina defaultHost=localhost Realm className=org.apache.catalina.realm.LockOutRealm Realm className=org.apache.catalina.realm.UserDatabaseRealm resourceName=UserDatabase/ /Realm Host name=localhost appBase=webapps unpackWARs=true autoDeploy=true Valve className=org.apache.catalina.valves.RemoteIpValve protocolHeader=X-Forwarded-Proto / Valve className=org.apache.catalina.valves.AccessLogValve directory=logs prefix=localhost_access_log. suffix=.txt pattern=%h %l %u %t quot;%rquot; %s %b resolveHosts=false/ /Host /Engine /Service /Server -- View this message in context: http://old.nabble.com/Request-params-randomly-null-in-servlet%28s%29-tp32461421p32471737.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Request params randomly null in servlet(s)
Pid * wrote: Exactly which version of Tomcat are you running? Latest, 7.0.21. -- View this message in context: http://old.nabble.com/Request-params-randomly-null-in-servlet%28s%29-tp32461421p32474158.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Request params randomly null in servlet(s)
Hi,
we have a problem of request params beeing null in servlets where it
shouldn't be ( post or get, in get we can even see that
request.getQueryString() is fine, but params are null).
It is more pronounced when using dedicated thread executor, moving to
threads provided by connector itself (with maxThreads=500 option using NIO
) seems to be lowering counts of errors.
Atm we are seeing 3 modes of failure in our tomcat setup ( nginx reverse
proxy - tomcat with NIO connector )
1)
SEVERE: Error finishing response
java.lang.NullPointerException
and/or
SEVERE: Error processing request
java.lang.NullPointerException
in logs. ( could be some app problem, but still shady, as no stack trace is
given, like if exception object is gone itself??? )
2) Sometimes we get 502 (from nginx, that got upstream error) like Rakesh in
other thread, from servlets that should be fine ( result of NPE in real
strange places ).
3) And most scary ones are the following:
2011 09 13 14:18:20.568 WARN Servlet - error response: error=no action
query: [type=groupsChangedActionv=1315912033175] content type:null
protocol: HTTP/1.0 length:-1 character encoding: null GET
2011 09 13 14:18:20.937 WARN Servlet - error response: error=no action
query: [type=groupsChangedActionv=1315912699092] content type:null
protocol: HTTP/1.0 length:-1 character encoding: null GET
In tomcat access log they are shown as:
127.0.0.1 - - [13/Sep/2011:14:18:20 +0300] GET
/ourapp/servlet/Servlet?type=groupsChangedActionv=1315912033175 HTTP/1.0
200 15
127.0.0.1 - - [13/Sep/2011:14:18:20 +0300] GET
/ourapp/servlet/Servlet?type=groupsChangedActionv=1315912699092 HTTP/1.0
200 15
So as you can see, tomcat got that GET reqest with type and v params like it
gets and processes thousands others.
Servlet code looks like:
private void doProcess(HttpServletRequest request, HttpServletResponse
response) throws IOException {
HttpSession session = request.getSession();
if (session != null) {
String action = request.getParameter(type);
logger.debug(action: + action);
if (action == null) {
printError(request, response,error=no
action);
private void printError(HttpServletRequest request,
HttpServletResponse response, String error) throws IOException {
logger.warn(error response: + error + ip: +
request.getRemoteAddr() + query: [ + request.getQueryString() + ] +
content type: + request.getContentType() + protocol: +
request.getProtocol() + length: + request.getContentLength() +
character encoding: + request.getCharacterEncoding() ++
request.getMethod());
response.setContentType(text/plain);
PrintWriter out = response.getWriter();
out.print(error);
out.flush();
out.close();
}
It happens several times in a day in bursts ( GC collecting something?).
So really puzzling and random error, tomcat got proper request as shown in
access logs (and even request.getQueryString() is valid :) ) , processed
request from same user 5s ago, will process one more 5s later, but somehow
request is NULL??? Now we handle it by retrying, but such random NULL is
fatal for other servlets and services that throw NPEs and it seems that some
tomcat code is affected too with (1).
I'd grateful if some smart guys could look into this problem and willing to
provide any other info.
Only connector running is:
Connector port=8080
protocol=org.apache.coyote.http11.Http11NioProtocol
connectionTimeout=2
enableLookups=false
acceptCount=1000
maxThreads=500
maxPostSize=10485760
/
Also using
Valve className=org.apache.catalina.valves.RemoteIpValve
protocolHeader=X-Forwarded-Proto /
BR,
Darius.
--
View this message in context:
http://old.nabble.com/Request-params-randomly-null-in-servlet%28s%29-tp32461421p32461421.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Request params randomly null in servlet(s)
Pid * wrote:
doProcess(req, res) is not a Servlet API method. What other method(s)
call it?
Usually this type of thing occurs because the request (or response) is
being held as an instance field somewhere in a thread-unsafe way.
It is called from:
public void doGet(HttpServletRequest request, HttpServletResponse
response)
throws ServletException, IOException {
doProcess(request, response);
}
and doPost(...) as well.
I am not quite sure where and what could be holding the request, as this
method is the called directly and request params are already NULL ?
--
View this message in context:
http://old.nabble.com/Request-params-randomly-null-in-servlet%28s%29-tp32461421p32462485.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Request params randomly null in servlet(s)
Pid * wrote:
On 14/09/2011 11:55, Darius D. wrote:
They are in *that* thread.
It is possible to expose either request or response object outside of
the scope of the thread, so a second request sees a modified version of
the same object.
E.g.
public class SomeServlet extends HttpServlet {
private HttpServletRequest request;
protected void doGet(HttpServletRequest rq, HttpServletResponse rs) {
// this is a handy value, let's store it (FAIL!)
this.request = request;
// do something...
process(this.request);
// finish...
}
}
So a different thread entering the method causes the request object to
be updated before the other thread has finished using it. There are
other, more subtle, ways to cause the same effect.
p
Well the problem is that we already looked for that mode of failure, our
servlet class has 0 instance variables... Basically it is a method like i
pasted before, doing HttpSession session = request.getSession(); and if
session is valid (it is!) immediately reading those params with String
action = request.getParameter(type);
Request object is perfectly valid when those params are missing ( session,
query etc are preserved, its just those params we are missing ). Error
happens in bursts in a rate of ~25 / million requests.
--
View this message in context:
http://old.nabble.com/Request-params-randomly-null-in-servlet%28s%29-tp32461421p32464563.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Tomcat unable to send the request. tomcat throwing 502 Gate Way error
(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:354) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Darius, if you require more information, i ca provide you the details. Regards Rakesh On Mon, Sep 12, 2011 at 5:49 PM, Darius D. darius@gmail.com wrote: rakesh k-3 wrote: Hi Christopher thanks for viewing m configuration ..as mentioned we are using http connectors in my configuration not AJP. and here are my configuration details. Apache tomcat configuration (server.xml) Executor name=tomcatThreadPool namePrefix=catalina-exec- maxThreads=150 minSpareThreads=4/ Connector executor=tomcatThreadPool port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Apache httpd configuration Httpd.conf. ProxyPass / http://localhost:8080 route=msf1 Regards Rakesh Hi, can you check your catalina.out to see if there is something like: org.apache.coyote.http11.AbstractHttp11Processor endRequest SEVERE: Error finishing response java.lang.NullPointerException or SEVERE: Error processing request java.lang.NullPointerException without any further details / stack traces ? Darius. -- View this message in context: http://old.nabble.com/Apache-Tomcat-unable-to-send-the-request.-tomcat-throwing-502-Gate-Way-error-tp32248578p32447382.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- View this message in context: http://old.nabble.com/Apache-Tomcat-unable-to-send-the-request.-tomcat-throwing-502-Gate-Way-error-tp32248578p32455189.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Tomcat unable to send the request. tomcat throwing 502 Gate Way error
rakesh k-3 wrote: Hi Christopher thanks for viewing m configuration ..as mentioned we are using http connectors in my configuration not AJP. and here are my configuration details. Apache tomcat configuration (server.xml) Executor name=tomcatThreadPool namePrefix=catalina-exec- maxThreads=150 minSpareThreads=4/ Connector executor=tomcatThreadPool port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Apache httpd configuration Httpd.conf. ProxyPass / http://localhost:8080 route=msf1 Regards Rakesh Hi, can you check your catalina.out to see if there is something like: org.apache.coyote.http11.AbstractHttp11Processor endRequest SEVERE: Error finishing response java.lang.NullPointerException or SEVERE: Error processing request java.lang.NullPointerException without any further details / stack traces ? Darius. -- View this message in context: http://old.nabble.com/Apache-Tomcat-unable-to-send-the-request.-tomcat-throwing-502-Gate-Way-error-tp32248578p32447382.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance for many small requests
Some good advice in this thread already, but given the power of server there should be no problem serving even more requests ( as long as they are small, not bound by CPU or I/O). I'd start looking at JVM GC properties. Turn on gc logging with -Xloggc:/someplace/gclog_tomcat.txt -XX:+PrintGCDetails ( or some verbose / timestamped variant of this ) And see what your gc pauses look like, how frequent young gen / full gc are. With hundreds of requests coming in every sec, you get into situation where garbage collection pause will create a burst of requests once its done ( and there is a risk that those burst objects will get promoted to old gen! ) and cause stampede if server is near limits already. Generally You want your requests to never hit old gen memory in JDK and get collected in some magic parallel young gen GC collector ( -XX:+UseParNewGC, -XX:NewSize=??? -XX:MaxNewSize=??? -XX:SurvivorRatio=??? are worth checking / or even trying G1C in JDK7 if You feel adventurous ). As already was mentioned - finetuning JVM params is good idea, but only after checking GC logs. Only thing i recommend outright is pinning JVM heap with matching -Xmx and -Xms directives - or else JVM will keep allocating/deallocating memory from OS, potentially creating fragmentation issues in long term. And there is a question of 64bit JVM, unless you need Java heap above ~1.5G, 32bit JVM should do just fine, otherwise you are just paying huge tax in memory usage and CPU cache/TLB misses. -XX:+UseCompressedOops can help to remove some of this tax, but in my opinion using 64bit JVM with such a small heap is only needed if performance testing shows gains versus 32bit JVM. -- View this message in context: http://old.nabble.com/Performance-for-many-small-requests-tp32372424p32392622.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Performance for many small requests
n828cl wrote: From: Darius D. [mailto:darius@gmail.com] Subject: Re: Performance for many small requests in my opinion using 64bit JVM with such a small heap is only needed if performance testing shows gains versus 32bit JVM. The main advantage of using a 64-bit JVM is the increased number of registers available in the x86-64 architecture, which can result in vastly reduced memory references. Whether or not that's important to overall performance is highly dependent on the application, of course. Yeah, but I'd err on 32bit JVM side :) Gains from more registers are very specific, but penalty from increased cache/TLB misses is big, and if you start hitting hard page faults ( that would have been avoidable due to lower heap size with 32bit JVM ) - even one of those will erase all gains :) There are reasons why Linux and other stuff is shipped compiled optimized for size, not with some fancy -O666 option. -- View this message in context: http://old.nabble.com/Performance-for-many-small-requests-tp32372424p32392870.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance for many small requests
David Kerber wrote: On 9/3/2011 1:15 PM, Darius D. wrote: n828cl wrote: From: Darius D. [mailto:darius@gmail.com] Subject: Re: Performance for many small requests in my opinion using 64bit JVM with such a small heap is only needed if performance testing shows gains versus 32bit JVM. The main advantage of using a 64-bit JVM is the increased number of registers available in the x86-64 architecture, which can result in vastly reduced memory references. Whether or not that's important to overall performance is highly dependent on the application, of course. Yeah, but I'd err on 32bit JVM side :) Gains from more registers are very specific, but penalty from increased cache/TLB misses is big, and if you start hitting hard page faults ( that would have been avoidable due to lower heap size with 32bit JVM ) - even one of those will erase all gains :) Then why shouldn't I just double my heap size? Wouldn't that eliminate the risk of increased cache misses? I was just using the settings from my 32-bit installation, but have plenty of RAM to allow me to increase the memory settings if that would help. Dave Umm, sorry, it seems that 64 vs 32bit discussion has thrown this thread of track. In your case it probably makes little difference. I'd definately start by profiling and looking at gc logs. As a side note - (CPU)cache/TLB misses have nothing to do with heap size. Too big heap size can be as bad as too low ( by stealing memory from OS that could have been used for file caches and other apps and increasing GC pauses ). -- View this message in context: http://old.nabble.com/Performance-for-many-small-requests-tp32372424p32392937.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Performance for many small requests
Jeff Sturm wrote: The Linux JVM has a nice option -XX:+UseLargePages to help avoid TLB misses on heap accesses. It's good for modest gains, and has the side effect of locking the heap into memory. I tend to use it on systems with large heaps. Not much you can do about cache misses though, besides getting processors with more cache. Yeah, Large(Huge) pages is big, free 10-15% perf if app is memory heavy. And as a bonus you also reduce the size of kernel page translation tables ( works wonders if you are using for example Oracle that is using shared mem for instances, savings can be up to gigabytes on big systems ). Even greater news is that recent Linux kernel versions ( i think 2.6.38 and above ) have so called Transparent Huge Pages - that basically enable them with some magic for all processes, giving 95% of HugePages benefits without LargePages penalties (heap locking/pinning and app specific config). So users of virtualisation, databases and JVMs with large heaps can rejoice :) -- View this message in context: http://old.nabble.com/Performance-for-many-small-requests-tp32372424p32393854.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: APR connector pollTime defaults are strange in tomcat6/7
My goal with this thread was to rise awareness with APR connector poll time defaults, as some users will not really bother investigating why their servers have such high context switches / timer interrupts. There is no problem here as Tomcat is working fine with defaults. There should be no harm by setting it to 100 microseconds, as NIO connectors are using 1000ms as default selectorTimeout ( same epoll inside ) and working just fine. Christopher Schultz-2 wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Darius, On 7/18/2011 2:23 PM, Darius D. wrote: Does Tomcat APR really needs pollTime set so low by default? I thought timeout is meant for some sort of book keeping, where is all connections in FD set are idle, no events come for timeout period - you force timeout and do bookkeeping - on a busy system you will get events anyway cause of socket traffic. Also connection timeout is 60s by default, so ending connection @ 2ms precision is not enhancing latency in any way. Seems like a reasonable question. P.S. There exists perfect workaround in latest Tomcat7, using protocol=org.apache.coyote.http11.Http11NioProtocol and protocol=org.apache.coyote.ajp.AjpNioProtocol for AJP will do away with all unneeded context switches. Yes, switching from APR connector to another one certainly does alleviate any issues you are experiencing by using the APR connector. This isn't really a workaround. :) On the other hand, a better workaround would be to set these values appropriately for your environment. What's stopping you from setting the pollTime to, as you suggest, 10 microseconds? That isn't really a workaround, either: it's proper configuration. It's probably worth discussing what the defaults should be, but there's a perfectly reasonable course of action for you at this point: change the configuration. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4tw8UACgkQ9CaO5/Lv0PDJ2ACeNAYeMDPWDw9jyjtXz2J82O9z 5b0An0a1E4LPyrIVcREaBqt+deRvVsOa =bJY5 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- View this message in context: http://old.nabble.com/APR-connector-pollTime-defaults-are-strange-in-tomcat6-7-tp32085364p32173790.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: APR connector pollTime defaults are strange in tomcat6/7
Darius D. wrote: Does Tomcat APR really needs pollTime set so low by default? I thought timeout is meant for some sort of book keeping, where is all connections in FD set are idle, no events come for timeout period - you force timeout and do bookkeeping - on a busy system you will get events anyway cause of socket traffic. Also connection timeout is 60s by default, so ending connection @ 2ms precision is not enhancing latency in any way. I think defaults should be increased to something reasonable like 100ms (pollTime =10) to avoid unneeded wakeups (and wakeups are bad, cause they cause context switch, and context switches pollute caches, TLB buffers and on modern servers burn electricity by forcing CPUs from low C states ) I guess there is no interest in efficiency and reducing overhead with APR connectors? Overhead is quite substantial. Consider the following - on a lightly loaded system we were seeing ~1.8k timer interrups and context switches with Linux 2.6.39 kernel and latest Tomcat 7 + 1.20 TCNative + APR. And its easy to see where from they are coming - 3 connector (AJP 8009, HTTP, HTTPS) , all APR, all 2000 microseconds PollTime. So we were getting ~500x3 context switches from all those epoll_wait(...,2ms) calls. And they were just burning CPU and polluting caches. After switching to NIO connectors on same system and same load CS and interrupts are down to ~600. ( note that to reproduce this you need a system with NO_HZ kernel and HPET to actually get a epoll_wait timeout of 2000us instead of ~1/HZ (10ms on 100HZ kernel ) minimum on normal kernels ) I have attached screenshot from munin irq stats display. http://old.nabble.com/file/p32115035/irqstats-week.png irqstats-week.png So results are pretty obviuos. -- View this message in context: http://old.nabble.com/APR-connector-pollTime-defaults-are-strange-in-tomcat6-7-tp32085364p32115035.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
APR connector pollTime defaults are strange in tomcat6/7
From documentation: TC6: Duration of a poll call. Lowering this value will slightly decrease latency of connections being kept alive in some cases, but will use more CPU as more poll calls are being made. The default value is 2000 (5ms). TC7: Duration of a poll call in microseconds. Lowering this value will slightly decrease latency of connections being kept alive in some cases , but will use more CPU as more poll calls are being made. The default value is 2000 (2ms). TC6 APR connector defaults are also microseconds. But it gives not 5ms, but some arbitrary value that depends on kernel configuration. On distribution default kernels (debian, redhat...) with 100HZ configs ( very common on servers ) it gives epoll time of 10ms ( sounds reasonable, but... ). Now where trouble starts is on kernels with NO_HZ and HPET timers - it actually gives epoll time of 2ms. The problem is that on reasonably loaded servers tomcat java processes start to dominate wake up reasons and timer interrupt reasons - waking up each thread with APR connector ~480 times per second. pidstat -t -w -C java 1 will show those threads and ~480 context switches they are causing. and you can confirm the reason for those wakeups with: gdb -batch -ex bt -p 4056 warning: process 4056 is a cloned process [Thread debugging using libthread_db enabled] 0x7f652a096623 in epoll_wait () from /lib/libc.so.6 #0 0x7f652a096623 in epoll_wait () from /lib/libc.so.6 #1 0x7f6521147ca3 in ?? () from /usr/lib/libapr-1.so.0 #2 0x7f6521146908 in apr_pollset_poll () from /usr/lib/libapr-1.so.0 #3 0x7f652196b2b3 in Java_org_apache_tomcat_jni_Poll_poll (e=0x40fbf9c8, o=value optimized out, pollset=1092883016, timeout=2000, set=0x7f651b622750, remove=1 '\001') at src/poll.c:311 Also if you do strace -r -p APRconnectorpid, you will see that there is a mass of epool_wait calls going on, most of them each doing absolutely nothing. Does Tomcat APR really needs pollTime set so low by default? I thought timeout is meant for some sort of book keeping, where is all connections in FD set are idle, no events come for timeout period - you force timeout and do bookkeeping - on a busy system you will get events anyway cause of socket traffic. Also connection timeout is 60s by default, so ending connection @ 2ms precision is not enhancing latency in any way. I think defaults should be increased to something reasonable like 100ms (pollTime =10) to avoid unneeded wakeups (and wakeups are bad, cause they cause context switch, and context switches pollute caches, TLB buffers and on modern servers burn electricity by forcing CPUs from low C states ) P.S. There exists perfect workaround in latest Tomcat7, using protocol=org.apache.coyote.http11.Http11NioProtocol and protocol=org.apache.coyote.ajp.AjpNioProtocol for AJP will do away with all unneeded context switches. -- View this message in context: http://old.nabble.com/APR-connector-pollTime-defaults-are-strange-in-tomcat6-7-tp32085364p32085364.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
