On 22/12/2016 17:50, Greg Huber wrote: > Chris, > > Thanks, I have been running this setup for ages, 8 plus years and various > tomcat versions. It has never failed on <= 8.0.x I had to stop/start a > couple of times to get it to fail and had to run a parallel maven build to > slow things down. > > Do not read much on dumps, but what would this -locked be?
It is perfectly normal. Your problem is with the connection pool. Note that when Tomcat switched form DBCP1 to DBCP2, some of the names of the attributes changed. You might not be using the settings you think you are using. Please post your resource config. Mark > > at > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1437) > at > org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) > - locked <0x00000000f1c54ed8> (a > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.lang.Thread.run(Thread.java:745) > > Cheers Greg > > > On 22 December 2016 at 16:24, Christopher Schultz < > ch...@christopherschultz.net> wrote: > > Greg, > > On 12/22/16 5:18 AM, Greg Huber wrote: >>>> Here is the first dump and I cannot send all three. Its my prod >>>> server which is locked up at the moment. I need to get it back on >>>> line so is this enough info? I can emile the whole file which has >>>> the three dumps if needed. >>>> >>>> [snip] >>>> >>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed >>>> mode): >>>> >>>> "ajp-nio-8009-exec-26" #61 daemon prio=5 os_prio=0 >>>> tid=0x000000000cb10000 nid=0x1960 in Object.wait() >>>> [0x00002ba658a4b000] java.lang.Thread.State: WAITING (on object >>>> monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj > ectPool.java:1104) >>>> >>>> > - locked <0x00000000f2879c10> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > Waiting on a database connection. > > >>>> "ajp-nio-8009-exec-25" #60 daemon prio=5 os_prio=0 >>>> tid=0x000000000c74f800 nid=0x195d in Object.wait() >>>> [0x00002ba658a0a000] java.lang.Thread.State: WAITING (on object >>>> monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj > ectPool.java:1104) >>>> >>>> > - locked <0x00000000f2852128> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > Another thread waiting for a db connection. > >>>> "ajp-nio-8009-exec-24" #59 daemon prio=5 os_prio=0 >>>> tid=0x000000000dd3b000 nid=0x1959 in Object.wait() >>>> [0x00002ba6589c9000] java.lang.Thread.State: WAITING (on object >>>> monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj > ectPool.java:1104) >>>> >>>> > - locked <0x00000000f282a818> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > I'm starting to see a pattern, here. > >>>> "ajp-nio-8009-exec-23" #58 daemon prio=5 os_prio=0 >>>> tid=0x000000000cbe6000 nid=0x1953 in Object.wait() >>>> [0x00002ba658988000] java.lang.Thread.State: WAITING (on object >>>> monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj > ectPool.java:1104) >>>> >>>> > - locked <0x00000000f28030f8> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > Hmm. Could it be... > >>>> "ajp-nio-8009-exec-22" #57 daemon prio=5 os_prio=0 >>>> tid=0x000000000e797800 nid=0x194f in Object.wait() >>>> [0x00002ba659472000] java.lang.Thread.State: WAITING (on object >>>> monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj > ectPool.java:1104) >>>> >>>> > - locked <0x00000000f27daf28> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > ... just possibly ... > >>>> "ajp-nio-8009-exec-21" #56 daemon prio=5 os_prio=0 >>>> tid=0x000000000e874000 nid=0x194d in Object.wait() >>>> [0x00002ba656c67000] java.lang.Thread.State: WAITING (on object >>>> monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj > ectPool.java:1104) >>>> >>>> > - locked <0x00000000f27b3d20> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > ... that you have run out of ... > >>>> "QuartzScheduler_scheduler-NON_CLUSTERED_MisfireHandler" #55 >>>> prio=5 os_prio=0 tid=0x000000000cbe5000 nid=0x194b waiting on >>>> condition [0x00002ba658c50000] java.lang.Thread.State: >>>> TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) >>>> at >>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobSto > reSupport.java:3992) > > (Quartz >>>> > waiting around for some event. Ignore this one in erms of my > overarching narrative.) > >>>> "ajp-nio-8009-exec-20" #54 daemon prio=5 os_prio=0 >>>> tid=0x000000000e44f000 nid=0x194a in Object.wait() >>>> [0x00002ba658948000] java.lang.Thread.State: WAITING (on object >>>> monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj > ectPool.java:1104) >>>> >>>> > - locked <0x00000000f1f541b8> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > Database connections? > >>>> "ajp-nio-8009-exec-19" #53 daemon prio=5 os_prio=0 >>>> tid=0x000000000cc2e800 nid=0x1940 in Object.wait() >>>> [0x00002ba658908000] java.lang.Thread.State: RUNNABLE at >>>> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36) > > Ooh! >>>> > This one is doing something different. I suspect any ms it will > ask for another db connection. > >>>> "ajp-nio-8009-exec-18" #52 daemon prio=5 os_prio=0 >>>> tid=0x000000000cc7b800 nid=0x1933 in Object.wait() >>>> [0x00002ba6588c6000] java.lang.Thread.State: WAITING (on object >>>> monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj > ectPool.java:1104) >>>> >>>> > - locked <0x00000000f1d766c0> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > That's more like it. > >>>> "ajp-nio-8009-exec-17" #51 daemon prio=5 os_prio=0 >>>> tid=0x000000000cb83000 nid=0x1930 in Object.wait() >>>> [0x00002ba658885000] java.lang.Thread.State: WAITING (on object >>>> monitor) at java.lang.Object.wait(Native Method) at >>>> java.lang.Object.wait(Object.java:502) at >>>> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj > ectPool.java:1104) >>>> >>>> > - locked <0x00000000f1d4e918> (a >>>> org.apache.commons.pool.impl.GenericObjectPool$Latch) > > I'm reminded of a Far Side cartoon. > [http://grahammercer.com.au/humour/GodMakesTheSnake.jpg] > >>>> "ajp-nio-8009-exec-16" #50 daemon prio=5 os_prio=0 >>>> tid=0x000000000ea8b000 nid=0x192e in Object.wait() >>>> [0x00002ba658ed7000] java.lang.Thread.State: RUNNABLE at >>>> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36) > > Another >>>> > thread caught in the filter. Filters are usually quick, so I > think this is a transient state. Probably about to request a database > connection. > >>>> "ajp-nio-8009-exec-15" #49 daemon prio=5 os_prio=0 >>>> tid=0x000000000c92a000 nid=0x1928 in Object.wait() >>>> [0x00002ba65452d000] java.lang.Thread.State: RUNNABLE at >>>> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36) > > Here, >>>> > too. > > > Okay, I'm done reading the thread dump. > > Have a look at your database pool's size and "active" counts[1]. You > may find that your application is locked-up waiting on db connections. > you may have a resource leak (db connections). > > -chris > > [1] > http://people.apache.org/~schultz/ApacheCon%20NA%202016/Monitoring%20Apa > che%20Tomcat%20with%20JMX.pdf, > slide 16 >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
