-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Greg,
On 12/22/16 5:18 AM, Greg Huber wrote: > Here is the first dump and I cannot send all three. Its my prod > server which is locked up at the moment. I need to get it back on > line so is this enough info? I can emile the whole file which has > the three dumps if needed. > > [snip] > > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed > mode): > > "ajp-nio-8009-exec-26" #61 daemon prio=5 os_prio=0 > tid=0x000000000cb10000 nid=0x1960 in Object.wait() > [0x00002ba658a4b000] java.lang.Thread.State: WAITING (on object > monitor) at java.lang.Object.wait(Native Method) at > java.lang.Object.wait(Object.java:502) at > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj ectPool.java:1104) > > - - locked <0x00000000f2879c10> (a > org.apache.commons.pool.impl.GenericObjectPool$Latch) Waiting on a database connection. > "ajp-nio-8009-exec-25" #60 daemon prio=5 os_prio=0 > tid=0x000000000c74f800 nid=0x195d in Object.wait() > [0x00002ba658a0a000] java.lang.Thread.State: WAITING (on object > monitor) at java.lang.Object.wait(Native Method) at > java.lang.Object.wait(Object.java:502) at > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj ectPool.java:1104) > > - - locked <0x00000000f2852128> (a > org.apache.commons.pool.impl.GenericObjectPool$Latch) Another thread waiting for a db connection. > "ajp-nio-8009-exec-24" #59 daemon prio=5 os_prio=0 > tid=0x000000000dd3b000 nid=0x1959 in Object.wait() > [0x00002ba6589c9000] java.lang.Thread.State: WAITING (on object > monitor) at java.lang.Object.wait(Native Method) at > java.lang.Object.wait(Object.java:502) at > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj ectPool.java:1104) > > - - locked <0x00000000f282a818> (a > org.apache.commons.pool.impl.GenericObjectPool$Latch) I'm starting to see a pattern, here. > "ajp-nio-8009-exec-23" #58 daemon prio=5 os_prio=0 > tid=0x000000000cbe6000 nid=0x1953 in Object.wait() > [0x00002ba658988000] java.lang.Thread.State: WAITING (on object > monitor) at java.lang.Object.wait(Native Method) at > java.lang.Object.wait(Object.java:502) at > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj ectPool.java:1104) > > - - locked <0x00000000f28030f8> (a > org.apache.commons.pool.impl.GenericObjectPool$Latch) Hmm. Could it be... > "ajp-nio-8009-exec-22" #57 daemon prio=5 os_prio=0 > tid=0x000000000e797800 nid=0x194f in Object.wait() > [0x00002ba659472000] java.lang.Thread.State: WAITING (on object > monitor) at java.lang.Object.wait(Native Method) at > java.lang.Object.wait(Object.java:502) at > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj ectPool.java:1104) > > - - locked <0x00000000f27daf28> (a > org.apache.commons.pool.impl.GenericObjectPool$Latch) ... just possibly ... > "ajp-nio-8009-exec-21" #56 daemon prio=5 os_prio=0 > tid=0x000000000e874000 nid=0x194d in Object.wait() > [0x00002ba656c67000] java.lang.Thread.State: WAITING (on object > monitor) at java.lang.Object.wait(Native Method) at > java.lang.Object.wait(Object.java:502) at > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj ectPool.java:1104) > > - - locked <0x00000000f27b3d20> (a > org.apache.commons.pool.impl.GenericObjectPool$Latch) ... that you have run out of ... > "QuartzScheduler_scheduler-NON_CLUSTERED_MisfireHandler" #55 > prio=5 os_prio=0 tid=0x000000000cbe5000 nid=0x194b waiting on > condition [0x00002ba658c50000] java.lang.Thread.State: > TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) > at > org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobSto reSupport.java:3992) (Quartz > waiting around for some event. Ignore this one in erms of my overarching narrative.) > "ajp-nio-8009-exec-20" #54 daemon prio=5 os_prio=0 > tid=0x000000000e44f000 nid=0x194a in Object.wait() > [0x00002ba658948000] java.lang.Thread.State: WAITING (on object > monitor) at java.lang.Object.wait(Native Method) at > java.lang.Object.wait(Object.java:502) at > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj ectPool.java:1104) > > - - locked <0x00000000f1f541b8> (a > org.apache.commons.pool.impl.GenericObjectPool$Latch) Database connections? > "ajp-nio-8009-exec-19" #53 daemon prio=5 os_prio=0 > tid=0x000000000cc2e800 nid=0x1940 in Object.wait() > [0x00002ba658908000] java.lang.Thread.State: RUNNABLE at > org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36) Ooh! > This one is doing something different. I suspect any ms it will ask for another db connection. > "ajp-nio-8009-exec-18" #52 daemon prio=5 os_prio=0 > tid=0x000000000cc7b800 nid=0x1933 in Object.wait() > [0x00002ba6588c6000] java.lang.Thread.State: WAITING (on object > monitor) at java.lang.Object.wait(Native Method) at > java.lang.Object.wait(Object.java:502) at > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj ectPool.java:1104) > > - - locked <0x00000000f1d766c0> (a > org.apache.commons.pool.impl.GenericObjectPool$Latch) That's more like it. > "ajp-nio-8009-exec-17" #51 daemon prio=5 os_prio=0 > tid=0x000000000cb83000 nid=0x1930 in Object.wait() > [0x00002ba658885000] java.lang.Thread.State: WAITING (on object > monitor) at java.lang.Object.wait(Native Method) at > java.lang.Object.wait(Object.java:502) at > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj ectPool.java:1104) > > - - locked <0x00000000f1d4e918> (a > org.apache.commons.pool.impl.GenericObjectPool$Latch) I'm reminded of a Far Side cartoon. [http://grahammercer.com.au/humour/GodMakesTheSnake.jpg] > "ajp-nio-8009-exec-16" #50 daemon prio=5 os_prio=0 > tid=0x000000000ea8b000 nid=0x192e in Object.wait() > [0x00002ba658ed7000] java.lang.Thread.State: RUNNABLE at > org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36) Another > thread caught in the filter. Filters are usually quick, so I think this is a transient state. Probably about to request a database connection. > "ajp-nio-8009-exec-15" #49 daemon prio=5 os_prio=0 > tid=0x000000000c92a000 nid=0x1928 in Object.wait() > [0x00002ba65452d000] java.lang.Thread.State: RUNNABLE at > org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36) Here, > too. Okay, I'm done reading the thread dump. Have a look at your database pool's size and "active" counts[1]. You may find that your application is locked-up waiting on db connections. you may have a resource leak (db connections). - -chris [1] http://people.apache.org/~schultz/ApacheCon%20NA%202016/Monitoring%20Apa che%20Tomcat%20with%20JMX.pdf, slide 16 -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJYW/4vAAoJEBzwKT+lPKRYj8QP/jB4YkmoyLwS6z1RZ4XnV9Gw 43b2aoSdhDihonAao+Zu23+PQCELkzhv06hA2xZ33fW+HFem7cMw3isVh1NTxtVO 1fKsS4ZVftMnjHrFSMN9dzWnClf2vnDgN78to/tsEJGkinP7wJL55qZkdETJF4FO L3vMabMT1LeOUo8bdJfYKJXnA2vrjoU7W8vLP883LZy3YMUCdVclFOG10v/kr7vB oSWtSXBlq83fFJasXCSXEH7YvLteY1+jqrOx87JTeKgnYIkBF1KujMKuqw0ADRig brYegISWAdjs9ovyeTh4PKyWc29lNp7jwRcEMOwh1XF7WgV1msW3awqdIdPnNrtt 1kKLNXJv/aKILwMRCWQyR9/2jCsQygOE3Q5qXAjkjMe8qfORZYDdPQlZzpEuGt3U IMrRbpUN+jhMLKtfs/PWsuCker5rdGIIQu/SkCiYqpSDPiRpXLTteOW82SjxvLN/ QrasunCCz3N3Y0XEFJbkuE49lNYRruYuhEpmxXYuB+Xv6GOQzic9FYfztwq9z41S AvOjCiTCSacpScPoB9k3q8YGdxZniKQG6pgm0p+NoQYTO8mid/gznftr35GO+Dw8 c1zREZd5qv858CON1ELthxTFSn43Bte7bxfFl9ZvZ8DOWXEYi8P7fDJcuoBDoxuK Ag1tPGX3fMKaeC5Bwyw9 =ogwm -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
