Re: Upgrade to 8.5.8/9

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Greg,

On 12/27/16 4:58 AM, Greg Huber wrote:
> Since switching to commons-dbcp2 now seems to start OK (so far)
> which prior would have definitely hung.  Fingers crossed.
> 
>> You get complaints about the APR library not being installed?
>> Yeah, just comment-out the APRLifecycleListener in
>> conf/server.xml.
> 
> So using the native connector won't improve the performance?  I use
> https but through Apache.  The docs say:
> 
> These features allows making Tomcat a general purpose webserver,
> will enable much better integration with other native web
> technologies, and overall make Java much more viable as a full
> fledged webserver platform rather than simply a backend focused
> technology.
> 
> So this does not apply when using mod_jk?  It would save having to
> keep compiling the tomcat-native also.

mod_jk doesn't use TLS, and AJP13 expects to use long-lived
connections from the reverse proxies, so keep-alive behavior with
regard to threading isn't really an issue.

So no, I don't believe the use of the native connector will give you
any discernible difference in performance.

- -chris

> On 19 December 2016 at 18:11, Christopher Schultz < 
> ch...@christopherschultz.net> wrote:
> 
> Greg,
> 
> On 12/19/16 4:47 AM, Greg Huber wrote:
 I am currently running tomcat 8.0.32 and have tried to
 upgrade to 8.5.8 and 8.5.9 without success.  I use mod_jk to
 connect apache to tomcat running on centos 5.11 with Apache
 2.2.3.
 
 I have installed openSSL 1.0.2.j to compile the native 
 tomcat-native-1.2.10 using apr-1.5.2 and
 tomcat-connectors-1.2.42 for the mod_jk.
> 
> If you are using httpd + mod_jk, why are you bothering with APR
> and tcnative on the Tomcat side?
> 
 The problem is that tomcat seems to start but fails sometimes
 to connect correctly to apache and causes apache to hang.  If
 I stop tomcat, apache starts working again.  I have been
 through the logs with little indication of the problem.  If I
 revet back to 8.0.32 everything works OK.
> 
> What if you use the NIO connector instead of APR?
> 
 All I can find is this in the mod_jk.log:
 
 
 [Sat Dec 17 06:11:48 2016][3212:47280261211024] [info] 
 init_jk::mod_jk.c (3595): mod_jk/1.2.42 initialized [Sat Dec
 17 06:11:50 2016][3254:47280261211024] [info] 
 jk_open_socket::jk_connect.c (817): connect to
 127.0.0.1:8009 failed (errno=111) [Sat Dec 17 06:11:50
 2016][3254:47280261211024] [info]
 ajp_connect_to_endpoint::jk_ajp_common.c (1068): (worker1) 
 Failed opening socket to (127.0.0.1:8009) (errno=111) [Sat
 Dec 17 06:11:50 2016][3254:47280261211024] [error] 
 ajp_send_request::jk_ajp_common.c (1728): (worker1)
 connecting to backend failed. Tomcat is probably not started
 or is listening on the wrong port (errno=111) [Sat Dec 17
 06:11:50 2016][3254:47280261211024] [info]
 ajp_service::jk_ajp_common.c (2778): (worker1) sending
 request to tomcat failed (recoverable), because of error
 during request sending (attempt=1) [Sat Dec 17 06:11:50
 2016][3254:47280261211024] [info] 
 jk_open_socket::jk_connect.c (817): connect to
 127.0.0.1:8009 failed (errno=111) [Sat Dec 17 06:11:50
 2016][3254:47280261211024] [info]
 ajp_connect_to_endpoint::jk_ajp_common.c (1068): (worker1) 
 Failed opening socket to (127.0.0.1:8009) (errno=111) [Sat
 Dec 17 06:11:50 2016][3254:47280261211024] [error] 
 ajp_send_request::jk_ajp_common.c (1728): (worker1)
 connecting to backend failed. Tomcat is probably not started
 or is listening on the wrong port (errno=111) [Sat Dec 17
 06:11:50 2016][3254:47280261211024] [info]
 ajp_service::jk_ajp_common.c (2778): (worker1) sending
 request to tomcat failed (recoverable), because of error
 during request sending (attempt=2) [Sat Dec 17 06:11:50
 2016][3254:47280261211024] [error] 
 ajp_service::jk_ajp_common.c (2799): (worker1) connecting to
 tomcat failed (rc=-3, errors=1, client_errors=0). [Sat Dec 17
 06:11:50 2016][3254:47280261211024] [info]
 jk_handler::mod_jk.c (2995): Service error=-3 for
 worker=worker1
 
 Any ideas on what I should check?
> 
> Can you post your  configuration?
> 
> Any errors in the log files (specifically logs/catalina.out or, if
> you are using jsvc or Windows services,
> stdout.log/stderr.log/etc.)?
> 
> -chris
>> 
>> -
>>
>> 
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
>> 
> 
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYYnI3AAoJEBzwKT+lPKRYRD4P/2lt6SMqbUk4rKnQN7mI648f
zMbpP2UqblFv2b5DWm0hcWgFTrzuuzbMTJRqFDXvMYUYZ43HqYC2m7X/ECwyZSiF

Re: Upgrade to 8.5.8/9

[Greg Huber]

Since switching to commons-dbcp2 now seems to start OK (so far) which prior
would have definitely hung.  Fingers crossed.

>You get complaints about the APR library not being
> installed? Yeah, just comment-out the
> APRLifecycleListener in conf/server.xml.

So using the native connector won't improve the performance?  I use https
but through Apache.  The docs say:

These features allows making Tomcat a general purpose webserver, will
enable much better integration with other native web technologies, and
overall make Java much more viable as a full fledged webserver platform
rather than simply a backend focused technology.

So this does not apply when using mod_jk?  It would save having to keep
compiling the tomcat-native also.

Cheers Greg


On 19 December 2016 at 18:11, Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Greg,
>
> On 12/19/16 4:47 AM, Greg Huber wrote:
> > I am currently running tomcat 8.0.32 and have tried to upgrade to
> > 8.5.8 and 8.5.9 without success.  I use mod_jk to connect apache to
> > tomcat running on centos 5.11 with Apache 2.2.3.
> >
> > I have installed openSSL 1.0.2.j to compile the native
> > tomcat-native-1.2.10 using apr-1.5.2 and tomcat-connectors-1.2.42
> > for the mod_jk.
>
> If you are using httpd + mod_jk, why are you bothering with APR and
> tcnative on the Tomcat side?
>
> > The problem is that tomcat seems to start but fails sometimes to
> > connect correctly to apache and causes apache to hang.  If I stop
> > tomcat, apache starts working again.  I have been through the logs
> > with little indication of the problem.  If I revet back to 8.0.32
> > everything works OK.
>
> What if you use the NIO connector instead of APR?
>
> > All I can find is this in the mod_jk.log:
> >
> >
> > [Sat Dec 17 06:11:48 2016][3212:47280261211024] [info]
> > init_jk::mod_jk.c (3595): mod_jk/1.2.42 initialized [Sat Dec 17
> > 06:11:50 2016][3254:47280261211024] [info]
> > jk_open_socket::jk_connect.c (817): connect to 127.0.0.1:8009
> > failed (errno=111) [Sat Dec 17 06:11:50 2016][3254:47280261211024]
> > [info] ajp_connect_to_endpoint::jk_ajp_common.c (1068): (worker1)
> > Failed opening socket to (127.0.0.1:8009) (errno=111) [Sat Dec 17
> > 06:11:50 2016][3254:47280261211024] [error]
> > ajp_send_request::jk_ajp_common.c (1728): (worker1) connecting to
> > backend failed. Tomcat is probably not started or is listening on
> > the wrong port (errno=111) [Sat Dec 17 06:11:50
> > 2016][3254:47280261211024] [info] ajp_service::jk_ajp_common.c
> > (2778): (worker1) sending request to tomcat failed (recoverable),
> > because of error during request sending (attempt=1) [Sat Dec 17
> > 06:11:50 2016][3254:47280261211024] [info]
> > jk_open_socket::jk_connect.c (817): connect to 127.0.0.1:8009
> > failed (errno=111) [Sat Dec 17 06:11:50 2016][3254:47280261211024]
> > [info] ajp_connect_to_endpoint::jk_ajp_common.c (1068): (worker1)
> > Failed opening socket to (127.0.0.1:8009) (errno=111) [Sat Dec 17
> > 06:11:50 2016][3254:47280261211024] [error]
> > ajp_send_request::jk_ajp_common.c (1728): (worker1) connecting to
> > backend failed. Tomcat is probably not started or is listening on
> > the wrong port (errno=111) [Sat Dec 17 06:11:50
> > 2016][3254:47280261211024] [info] ajp_service::jk_ajp_common.c
> > (2778): (worker1) sending request to tomcat failed (recoverable),
> > because of error during request sending (attempt=2) [Sat Dec 17
> > 06:11:50 2016][3254:47280261211024] [error]
> > ajp_service::jk_ajp_common.c (2799): (worker1) connecting to tomcat
> > failed (rc=-3, errors=1, client_errors=0). [Sat Dec 17 06:11:50
> > 2016][3254:47280261211024] [info] jk_handler::mod_jk.c (2995):
> > Service error=-3 for worker=worker1
> >
> > Any ideas on what I should check?
>
> Can you post your  configuration?
>
> Any errors in the log files (specifically logs/catalina.out or, if you
> are using jsvc or Windows services, stdout.log/stderr.log/etc.)?
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJYWCLGAAoJEBzwKT+lPKRYzqcP/R34lxIW90ZPn7SQ5SBsbKN8
> rvYD+4k7a3EGEnPUhfJaajOMbpTXdPTHyltHOID9CpRRFIGrb+/kcKmaLbllYIqq
> NUDvWSDUHK0yELaSjb7+Y980mygqXV3j8wAardcEnZ/wG143ZZozhpQtKjhpLmsb
> XMe0HiC9/4TeqCtyZ5IWRdq/UOotnU0m1QMjgGYQPztGwQlfvNYDZ//8apiYkCKY
> qUoblyNM1RiH4H6tiX+7ZEfk3exvArw84adcWULYiQNpHMwvsRlfMi1aq8Xcty5+
> xhB4SitKoEB6CY/oLwRjWYAUsQIov2trGoqbhxIiQcNpHxoKslLabvzdjmjMPMj0
> kbMiGpOFfK4ULqyk1255HNxrGL97lnA2nfIz4q23XUU4AYjQkvWHVe94zbUGzdHM
> 6WZWLn8aZcugC5ltENAp7bCuye2KzpbA4BRbaoGBgiGRQaGSx+0poTubLODSNRFl
> cJv+KyoCS+NX4Lz0gK3sIhHr7xRQno5tYmUKNjIn3/v3Rt5zhjfXCvoElQ+MIlEW
> iQfkO6AMw+uWlNnd6Iuya9do4bYs/7oN4NiySy6yuz9v96JC6qjZvOtfj4nViNnm
> bTRNDreU8IyBR0/vkgKTNTF7KfebJ6ccxRUuFeqqK3rUikBVDW1SsKGskuQK+Htr
> I3y6BDq4qz2VjDdPTXL0
> =XGBI
> -END PGP SIGNATURE-
>
> 

Re: Upgrade to 8.5.8/9

[Mark Thomas]

On 24/12/2016 13:33, Greg Huber wrote:
> We use spring for startup only,  struts2 and hibernate for persistance.

Thanks. I was expecting to see some pool size settings there but I
didn't. In which the the default will be used which is unchanged from 8.

Looking at the thread dump, you have a few threads at:
java.lang.Thread.State: RUNNABLE
 at org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36)
 etc.

and the rest stuck at:
java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:502)
 at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1104)
- locked <0xf1f541b8> (a
org.apache.commons.pool.impl.GenericObjectPool$Latch)
 at
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)

Note that is Commons DBCP 1.x.

That looks like a connection leak to me.

The standard debug approach would be to enable abandoned connection logging.

Mark



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Upgrade to 8.5.8/9

[Greg Huber]

We use spring for startup only,  struts2 and hibernate for persistance.

jdbc.properties

jdbc.driverClassName=org.mariadb.jdbc.Driver
jdbc.url=jdbc:mysql://localhost:3306/eventsdb?autoReconnect=trueuseUnicode=truecharacterEncoding=utf-8mysqlEncoding=utf8
hibernate.dialect=org.hibernate.dialect.MySQL5Dialect
jndi.value=java:comp/env/jdbc/eventsdbxa

hibernate.cfg



org.events.business.hibernate.ThreadLocalSessionContextNoAutoClose
false
true
false
 org.hibernate.transaction.JDBCTransactionFactory
true
org.hibernate.cache.EhCacheProvider
org.hibernate.cache.ehcache.EhCacheRegionFactory
false
false
true




hibernate.properties

file is empty


spring.cfg





${jdbc.driverClassName}


${jdbc.url}


${jdbc.username}


${jdbc.password}




${jdbc.driverClassName}


${quartz.jdbc.url}


${jdbc.username}


${jdbc.password}








${hibernate.dialect}











org.events.pojos







classpath:hibernate.cfg.xml



















On 24 December 2016 at 12:07, Mark Thomas  wrote:

> On 24/12/2016 08:49, Greg Huber wrote:
> > ...Well its not hibernate, will look at switching to commons-dbcp2 if
> > possible.
>
> Again, please post your resource config.
>
> Mark
>
>
> >
> > Cheers
> >
> > On 22 December 2016 at 20:26, Mark Thomas  wrote:
> >
> >> On 22/12/2016 17:50, Greg Huber wrote:
> >>> Chris,
> >>>
> >>> Thanks, I have been running this setup for ages, 8 plus years and
> various
> >>> tomcat versions.  It has never failed on  <= 8.0.x  I had to
> stop/start a
> >>> couple of times to get it to fail and had to run a parallel maven build
> >> to
> >>> slow things down.
> >>>
> >>> Do not read much on dumps, but what would this -locked be?
> >>
> >> It is perfectly normal.
> >>
> >> Your problem is with the connection pool.
> >>
> >> Note that when Tomcat switched form DBCP1 to DBCP2, some of the names of
> >> the attributes changed. You might not be using the settings you think
> >> you are using. Please post your resource config.
> >>
> >> Mark
> >>
> >>>
> >>> at
> >>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.
> >> doRun(NioEndpoint.java:1437)
> >>> at
> >>> org.apache.tomcat.util.net.SocketProcessorBase.run(
> >> SocketProcessorBase.java:49)
> >>> - locked <0xf1c54ed8> (a
> >>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
> >>> at
> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(
> >> ThreadPoolExecutor.java:1142)
> >>> at
> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> >> ThreadPoolExecutor.java:617)
> >>> at
> >>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(
> >> TaskThread.java:61)
> >>> at java.lang.Thread.run(Thread.java:745)
> >>>
> >>> Cheers Greg
> >>>
> >>>
> >>> On 22 December 2016 at 16:24, Christopher Schultz <
> >>> ch...@christopherschultz.net> wrote:
> >>>
> >>> Greg,
> >>>
> >>> On 12/22/16 5:18 AM, Greg Huber wrote:
> >> Here is the first dump and I cannot send all three.  Its my prod
> >> server which is locked up at the moment.  I need to get it back on
> >> line so is this enough info?  I can emile the whole file which has
> >> the three dumps if needed.
> >>
> >> [snip]
> >>
> >> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
> >> mode):
> >>
> >> "ajp-nio-8009-exec-26" #61 daemon prio=5 os_prio=0
> >> tid=0x0cb1 nid=0x1960 in Object.wait()
> >> [0x2ba658a4b000] java.lang.Thread.State: WAITING (on object
> >> monitor) at java.lang.Object.wait(Native Method) at
> >> java.lang.Object.wait(Object.java:502) at
> >> org.apache.commons.pool.impl.GenericObjectPool.
> >> borrowObject(GenericObj
> >>> ectPool.java:1104)
> >>
> >>
> >>> - locked <0xf2879c10> (a
> >> org.apache.commons.pool.impl.GenericObjectPool$Latch)
> >>>
> >>> Waiting on a database connection.
> >>>
> >>>
> >> "ajp-nio-8009-exec-25" #60 daemon prio=5 os_prio=0
> >> tid=0x0c74f800 nid=0x195d in Object.wait()
> >> [0x2ba658a0a000] 

Re: Upgrade to 8.5.8/9

[Mark Thomas]

On 24/12/2016 08:49, Greg Huber wrote:
> ...Well its not hibernate, will look at switching to commons-dbcp2 if
> possible.

Again, please post your resource config.

Mark


> 
> Cheers
> 
> On 22 December 2016 at 20:26, Mark Thomas  wrote:
> 
>> On 22/12/2016 17:50, Greg Huber wrote:
>>> Chris,
>>>
>>> Thanks, I have been running this setup for ages, 8 plus years and various
>>> tomcat versions.  It has never failed on  <= 8.0.x  I had to stop/start a
>>> couple of times to get it to fail and had to run a parallel maven build
>> to
>>> slow things down.
>>>
>>> Do not read much on dumps, but what would this -locked be?
>>
>> It is perfectly normal.
>>
>> Your problem is with the connection pool.
>>
>> Note that when Tomcat switched form DBCP1 to DBCP2, some of the names of
>> the attributes changed. You might not be using the settings you think
>> you are using. Please post your resource config.
>>
>> Mark
>>
>>>
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.
>> doRun(NioEndpoint.java:1437)
>>> at
>>> org.apache.tomcat.util.net.SocketProcessorBase.run(
>> SocketProcessorBase.java:49)
>>> - locked <0xf1c54ed8> (a
>>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1142)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:617)
>>> at
>>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(
>> TaskThread.java:61)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Cheers Greg
>>>
>>>
>>> On 22 December 2016 at 16:24, Christopher Schultz <
>>> ch...@christopherschultz.net> wrote:
>>>
>>> Greg,
>>>
>>> On 12/22/16 5:18 AM, Greg Huber wrote:
>> Here is the first dump and I cannot send all three.  Its my prod
>> server which is locked up at the moment.  I need to get it back on
>> line so is this enough info?  I can emile the whole file which has
>> the three dumps if needed.
>>
>> [snip]
>>
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>> mode):
>>
>> "ajp-nio-8009-exec-26" #61 daemon prio=5 os_prio=0
>> tid=0x0cb1 nid=0x1960 in Object.wait()
>> [0x2ba658a4b000] java.lang.Thread.State: WAITING (on object
>> monitor) at java.lang.Object.wait(Native Method) at
>> java.lang.Object.wait(Object.java:502) at
>> org.apache.commons.pool.impl.GenericObjectPool.
>> borrowObject(GenericObj
>>> ectPool.java:1104)
>>
>>
>>> - locked <0xf2879c10> (a
>> org.apache.commons.pool.impl.GenericObjectPool$Latch)
>>>
>>> Waiting on a database connection.
>>>
>>>
>> "ajp-nio-8009-exec-25" #60 daemon prio=5 os_prio=0
>> tid=0x0c74f800 nid=0x195d in Object.wait()
>> [0x2ba658a0a000] java.lang.Thread.State: WAITING (on object
>> monitor) at java.lang.Object.wait(Native Method) at
>> java.lang.Object.wait(Object.java:502) at
>> org.apache.commons.pool.impl.GenericObjectPool.
>> borrowObject(GenericObj
>>> ectPool.java:1104)
>>
>>
>>> - locked <0xf2852128> (a
>> org.apache.commons.pool.impl.GenericObjectPool$Latch)
>>>
>>> Another thread waiting for a db connection.
>>>
>> "ajp-nio-8009-exec-24" #59 daemon prio=5 os_prio=0
>> tid=0x0dd3b000 nid=0x1959 in Object.wait()
>> [0x2ba6589c9000] java.lang.Thread.State: WAITING (on object
>> monitor) at java.lang.Object.wait(Native Method) at
>> java.lang.Object.wait(Object.java:502) at
>> org.apache.commons.pool.impl.GenericObjectPool.
>> borrowObject(GenericObj
>>> ectPool.java:1104)
>>
>>
>>> - locked <0xf282a818> (a
>> org.apache.commons.pool.impl.GenericObjectPool$Latch)
>>>
>>> I'm starting to see a pattern, here.
>>>
>> "ajp-nio-8009-exec-23" #58 daemon prio=5 os_prio=0
>> tid=0x0cbe6000 nid=0x1953 in Object.wait()
>> [0x2ba658988000] java.lang.Thread.State: WAITING (on object
>> monitor) at java.lang.Object.wait(Native Method) at
>> java.lang.Object.wait(Object.java:502) at
>> org.apache.commons.pool.impl.GenericObjectPool.
>> borrowObject(GenericObj
>>> ectPool.java:1104)
>>
>>
>>> - locked <0xf28030f8> (a
>> org.apache.commons.pool.impl.GenericObjectPool$Latch)
>>>
>>> Hmm. Could it be...
>>>
>> "ajp-nio-8009-exec-22" #57 daemon prio=5 os_prio=0
>> tid=0x0e797800 nid=0x194f in Object.wait()
>> [0x2ba659472000] java.lang.Thread.State: WAITING (on object
>> monitor) at java.lang.Object.wait(Native Method) at
>> java.lang.Object.wait(Object.java:502) at
>> org.apache.commons.pool.impl.GenericObjectPool.
>> borrowObject(GenericObj
>>> ectPool.java:1104)
>>
>>
>>> - locked <0xf27daf28> (a
>> org.apache.commons.pool.impl.GenericObjectPool$Latch)
>>>
>>> ... just possibly ...
>>>
>> "ajp-nio-8009-exec-21" #56 daemon prio=5 os_prio=0

Re: Upgrade to 8.5.8/9

[Greg Huber]

This is a dump of it working:

for

ajp-nio-8009-exec,  I do not see any

- locked <0xf0a3f6b8> (a
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)

from the not working dump. possibly a result of stuff higher up in the
chain?

Cheers Greg

Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):

"pool-4-thread-5" #54 prio=5 os_prio=0 tid=0x0a6e5800 nid=0x2dfc
waiting on condition [0x2aef09792000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0xd233fca8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"pool-4-thread-4" #53 prio=5 os_prio=0 tid=0x0a834000 nid=0x2dc5
waiting on condition [0x2aef09814000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0xd233fca8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"QuartzScheduler_scheduler-NON_CLUSTERED_MisfireHandler" #52 prio=5
os_prio=0 tid=0x0a692800 nid=0x2d67 waiting on condition
[0x2aef0a1ed000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3992)

"pool-3-thread-1" #51 prio=5 os_prio=0 tid=0x0a49a000 nid=0x2d66
waiting on condition [0x2aef09855000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0xd21d62c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"ajp-nio-8009-exec-10" #45 daemon prio=5 os_prio=0 tid=0x0b45c800
nid=0x2d40 waiting on condition [0x2aef09ea8000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0xd2cdb7a8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

"ajp-nio-8009-exec-9" #44 daemon prio=5 os_prio=0 tid=0x0b45c000
nid=0x2d3f waiting on condition [0x2aef09e67000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0xd2cdb7a8> (a

Re: Upgrade to 8.5.8/9

[Greg Huber]

Mark/Chris,

Upgraded to dbcp2 with no luck, and get this in the dumps, repeating
pattern.


"ajp-nio-8009-exec-28" #60 daemon prio=5 os_prio=0 tid=0x2b28245c3800
nid=0x2a02 waiting on condition [0x2b28316aa000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0xd1026878> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:583)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at
org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
at
org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533)
at
org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at
org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
at
org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:99)
at
org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:129)
at
org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:247)
at
org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:254)
at
org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:203)
at
org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:56)
at
org.hibernate.internal.AbstractSharedSessionContract.beginTransaction(AbstractSharedSessionContract.java:401)
at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:355)
at com.sun.proxy.$Proxy65.beginTransaction(Unknown Source)
at
org.events.business.hibernate.HibernatePersistenceStrategy.getSession(HibernatePersistenceStrategy.java:99)
at
org.events.business.hibernate.HibernateWebsiteManagerImpl.getWebsiteDetailByHandle(HibernateWebsiteManagerImpl.java:319)
at
org.events.business.WebsiteManagerImpl.getWebsiteByHandle(WebsiteManagerImpl.java:252)
at
org.events.business.WebsiteManagerImpl.checkWebsiteByHandle(WebsiteManagerImpl.java:170)
at
org.events.ui.rendering.WebsiteRequestMapper.isWebsite(WebsiteRequestMapper.java:389)
at
org.events.ui.rendering.WebsiteRequestMapper.handleRequest(WebsiteRequestMapper.java:118)
at
org.events.ui.rendering.filters.RequestMappingFilter.doFilter(RequestMappingFilter.java:107)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at
org.events.ui.rendering.filters.DeviceResolverRequestFilter.doFilterInternal(DeviceResolverRequestFilter.java:62)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.events.ui.core.filters.InitFilter.doFilter(InitFilter.java:57)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at
org.events.ui.core.filters.PersistenceSessionFilter.doFilter(PersistenceSessionFilter.java:36)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at

Re: Upgrade to 8.5.8/9

[Greg Huber]

...Well its not hibernate, will look at switching to commons-dbcp2 if
possible.

Cheers

On 22 December 2016 at 20:26, Mark Thomas  wrote:

> On 22/12/2016 17:50, Greg Huber wrote:
> > Chris,
> >
> > Thanks, I have been running this setup for ages, 8 plus years and various
> > tomcat versions.  It has never failed on  <= 8.0.x  I had to stop/start a
> > couple of times to get it to fail and had to run a parallel maven build
> to
> > slow things down.
> >
> > Do not read much on dumps, but what would this -locked be?
>
> It is perfectly normal.
>
> Your problem is with the connection pool.
>
> Note that when Tomcat switched form DBCP1 to DBCP2, some of the names of
> the attributes changed. You might not be using the settings you think
> you are using. Please post your resource config.
>
> Mark
>
> >
> > at
> > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.
> doRun(NioEndpoint.java:1437)
> > at
> > org.apache.tomcat.util.net.SocketProcessorBase.run(
> SocketProcessorBase.java:49)
> > - locked <0xf1c54ed8> (a
> > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> > at
> > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(
> TaskThread.java:61)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > Cheers Greg
> >
> >
> > On 22 December 2016 at 16:24, Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> > Greg,
> >
> > On 12/22/16 5:18 AM, Greg Huber wrote:
>  Here is the first dump and I cannot send all three.  Its my prod
>  server which is locked up at the moment.  I need to get it back on
>  line so is this enough info?  I can emile the whole file which has
>  the three dumps if needed.
> 
>  [snip]
> 
>  Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>  mode):
> 
>  "ajp-nio-8009-exec-26" #61 daemon prio=5 os_prio=0
>  tid=0x0cb1 nid=0x1960 in Object.wait()
>  [0x2ba658a4b000] java.lang.Thread.State: WAITING (on object
>  monitor) at java.lang.Object.wait(Native Method) at
>  java.lang.Object.wait(Object.java:502) at
>  org.apache.commons.pool.impl.GenericObjectPool.
> borrowObject(GenericObj
> > ectPool.java:1104)
> 
> 
> > - locked <0xf2879c10> (a
>  org.apache.commons.pool.impl.GenericObjectPool$Latch)
> >
> > Waiting on a database connection.
> >
> >
>  "ajp-nio-8009-exec-25" #60 daemon prio=5 os_prio=0
>  tid=0x0c74f800 nid=0x195d in Object.wait()
>  [0x2ba658a0a000] java.lang.Thread.State: WAITING (on object
>  monitor) at java.lang.Object.wait(Native Method) at
>  java.lang.Object.wait(Object.java:502) at
>  org.apache.commons.pool.impl.GenericObjectPool.
> borrowObject(GenericObj
> > ectPool.java:1104)
> 
> 
> > - locked <0xf2852128> (a
>  org.apache.commons.pool.impl.GenericObjectPool$Latch)
> >
> > Another thread waiting for a db connection.
> >
>  "ajp-nio-8009-exec-24" #59 daemon prio=5 os_prio=0
>  tid=0x0dd3b000 nid=0x1959 in Object.wait()
>  [0x2ba6589c9000] java.lang.Thread.State: WAITING (on object
>  monitor) at java.lang.Object.wait(Native Method) at
>  java.lang.Object.wait(Object.java:502) at
>  org.apache.commons.pool.impl.GenericObjectPool.
> borrowObject(GenericObj
> > ectPool.java:1104)
> 
> 
> > - locked <0xf282a818> (a
>  org.apache.commons.pool.impl.GenericObjectPool$Latch)
> >
> > I'm starting to see a pattern, here.
> >
>  "ajp-nio-8009-exec-23" #58 daemon prio=5 os_prio=0
>  tid=0x0cbe6000 nid=0x1953 in Object.wait()
>  [0x2ba658988000] java.lang.Thread.State: WAITING (on object
>  monitor) at java.lang.Object.wait(Native Method) at
>  java.lang.Object.wait(Object.java:502) at
>  org.apache.commons.pool.impl.GenericObjectPool.
> borrowObject(GenericObj
> > ectPool.java:1104)
> 
> 
> > - locked <0xf28030f8> (a
>  org.apache.commons.pool.impl.GenericObjectPool$Latch)
> >
> > Hmm. Could it be...
> >
>  "ajp-nio-8009-exec-22" #57 daemon prio=5 os_prio=0
>  tid=0x0e797800 nid=0x194f in Object.wait()
>  [0x2ba659472000] java.lang.Thread.State: WAITING (on object
>  monitor) at java.lang.Object.wait(Native Method) at
>  java.lang.Object.wait(Object.java:502) at
>  org.apache.commons.pool.impl.GenericObjectPool.
> borrowObject(GenericObj
> > ectPool.java:1104)
> 
> 
> > - locked <0xf27daf28> (a
>  org.apache.commons.pool.impl.GenericObjectPool$Latch)
> >
> > ... just possibly ...
> >
>  "ajp-nio-8009-exec-21" #56 daemon prio=5 os_prio=0
>  tid=0x0e874000 nid=0x194d in Object.wait()
>  [0x2ba656c67000] java.lang.Thread.State: WAITING (on 

Re: Upgrade to 8.5.8/9

[Greg Huber]

Mark/Chris,

Taking your advice I looked at recent changes to my pom and noticed that
hibernate updated to 5.2.5 from 5.2.4 on December 12th.  So, I switched
back to 5.2.4 and have rebooted and restarted tomcat several times without
the lockup.  I have looked through the hibernate changes but cannot see any
connection pool related mods.  Maybe I will wait a couple of days and if
its fixed, log a hibernate issue or try 5.2.6.

Thanks for your help!

Cheers Greg

On 22 December 2016 at 20:26, Mark Thomas  wrote:

> On 22/12/2016 17:50, Greg Huber wrote:
> > Chris,
> >
> > Thanks, I have been running this setup for ages, 8 plus years and various
> > tomcat versions.  It has never failed on  <= 8.0.x  I had to stop/start a
> > couple of times to get it to fail and had to run a parallel maven build
> to
> > slow things down.
> >
> > Do not read much on dumps, but what would this -locked be?
>
> It is perfectly normal.
>
> Your problem is with the connection pool.
>
> Note that when Tomcat switched form DBCP1 to DBCP2, some of the names of
> the attributes changed. You might not be using the settings you think
> you are using. Please post your resource config.
>
> Mark
>
> >
> > at
> > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.
> doRun(NioEndpoint.java:1437)
> > at
> > org.apache.tomcat.util.net.SocketProcessorBase.run(
> SocketProcessorBase.java:49)
> > - locked <0xf1c54ed8> (a
> > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> > at
> > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(
> TaskThread.java:61)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > Cheers Greg
> >
> >
> > On 22 December 2016 at 16:24, Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> > Greg,
> >
> > On 12/22/16 5:18 AM, Greg Huber wrote:
>  Here is the first dump and I cannot send all three.  Its my prod
>  server which is locked up at the moment.  I need to get it back on
>  line so is this enough info?  I can emile the whole file which has
>  the three dumps if needed.
> 
>  [snip]
> 
>  Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
>  mode):
> 
>  "ajp-nio-8009-exec-26" #61 daemon prio=5 os_prio=0
>  tid=0x0cb1 nid=0x1960 in Object.wait()
>  [0x2ba658a4b000] java.lang.Thread.State: WAITING (on object
>  monitor) at java.lang.Object.wait(Native Method) at
>  java.lang.Object.wait(Object.java:502) at
>  org.apache.commons.pool.impl.GenericObjectPool.
> borrowObject(GenericObj
> > ectPool.java:1104)
> 
> 
> > - locked <0xf2879c10> (a
>  org.apache.commons.pool.impl.GenericObjectPool$Latch)
> >
> > Waiting on a database connection.
> >
> >
>  "ajp-nio-8009-exec-25" #60 daemon prio=5 os_prio=0
>  tid=0x0c74f800 nid=0x195d in Object.wait()
>  [0x2ba658a0a000] java.lang.Thread.State: WAITING (on object
>  monitor) at java.lang.Object.wait(Native Method) at
>  java.lang.Object.wait(Object.java:502) at
>  org.apache.commons.pool.impl.GenericObjectPool.
> borrowObject(GenericObj
> > ectPool.java:1104)
> 
> 
> > - locked <0xf2852128> (a
>  org.apache.commons.pool.impl.GenericObjectPool$Latch)
> >
> > Another thread waiting for a db connection.
> >
>  "ajp-nio-8009-exec-24" #59 daemon prio=5 os_prio=0
>  tid=0x0dd3b000 nid=0x1959 in Object.wait()
>  [0x2ba6589c9000] java.lang.Thread.State: WAITING (on object
>  monitor) at java.lang.Object.wait(Native Method) at
>  java.lang.Object.wait(Object.java:502) at
>  org.apache.commons.pool.impl.GenericObjectPool.
> borrowObject(GenericObj
> > ectPool.java:1104)
> 
> 
> > - locked <0xf282a818> (a
>  org.apache.commons.pool.impl.GenericObjectPool$Latch)
> >
> > I'm starting to see a pattern, here.
> >
>  "ajp-nio-8009-exec-23" #58 daemon prio=5 os_prio=0
>  tid=0x0cbe6000 nid=0x1953 in Object.wait()
>  [0x2ba658988000] java.lang.Thread.State: WAITING (on object
>  monitor) at java.lang.Object.wait(Native Method) at
>  java.lang.Object.wait(Object.java:502) at
>  org.apache.commons.pool.impl.GenericObjectPool.
> borrowObject(GenericObj
> > ectPool.java:1104)
> 
> 
> > - locked <0xf28030f8> (a
>  org.apache.commons.pool.impl.GenericObjectPool$Latch)
> >
> > Hmm. Could it be...
> >
>  "ajp-nio-8009-exec-22" #57 daemon prio=5 os_prio=0
>  tid=0x0e797800 nid=0x194f in Object.wait()
>  [0x2ba659472000] java.lang.Thread.State: WAITING (on object
>  monitor) at java.lang.Object.wait(Native Method) at
>  java.lang.Object.wait(Object.java:502) at
>  org.apache.commons.pool.impl.GenericObjectPool.
> 

Re: Upgrade to 8.5.8/9

[Mark Thomas]

On 22/12/2016 17:50, Greg Huber wrote:
> Chris,
> 
> Thanks, I have been running this setup for ages, 8 plus years and various
> tomcat versions.  It has never failed on  <= 8.0.x  I had to stop/start a
> couple of times to get it to fail and had to run a parallel maven build to
> slow things down.
> 
> Do not read much on dumps, but what would this -locked be?

It is perfectly normal.

Your problem is with the connection pool.

Note that when Tomcat switched form DBCP1 to DBCP2, some of the names of
the attributes changed. You might not be using the settings you think
you are using. Please post your resource config.

Mark

> 
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1437)
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> - locked <0xf1c54ed8> (a
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:745)
> 
> Cheers Greg
> 
> 
> On 22 December 2016 at 16:24, Christopher Schultz <
> ch...@christopherschultz.net> wrote:
> 
> Greg,
> 
> On 12/22/16 5:18 AM, Greg Huber wrote:
 Here is the first dump and I cannot send all three.  Its my prod
 server which is locked up at the moment.  I need to get it back on
 line so is this enough info?  I can emile the whole file which has
 the three dumps if needed.

 [snip]

 Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
 mode):

 "ajp-nio-8009-exec-26" #61 daemon prio=5 os_prio=0
 tid=0x0cb1 nid=0x1960 in Object.wait()
 [0x2ba658a4b000] java.lang.Thread.State: WAITING (on object
 monitor) at java.lang.Object.wait(Native Method) at
 java.lang.Object.wait(Object.java:502) at
 org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)


> - locked <0xf2879c10> (a
 org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> Waiting on a database connection.
> 
> 
 "ajp-nio-8009-exec-25" #60 daemon prio=5 os_prio=0
 tid=0x0c74f800 nid=0x195d in Object.wait()
 [0x2ba658a0a000] java.lang.Thread.State: WAITING (on object
 monitor) at java.lang.Object.wait(Native Method) at
 java.lang.Object.wait(Object.java:502) at
 org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)


> - locked <0xf2852128> (a
 org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> Another thread waiting for a db connection.
> 
 "ajp-nio-8009-exec-24" #59 daemon prio=5 os_prio=0
 tid=0x0dd3b000 nid=0x1959 in Object.wait()
 [0x2ba6589c9000] java.lang.Thread.State: WAITING (on object
 monitor) at java.lang.Object.wait(Native Method) at
 java.lang.Object.wait(Object.java:502) at
 org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)


> - locked <0xf282a818> (a
 org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> I'm starting to see a pattern, here.
> 
 "ajp-nio-8009-exec-23" #58 daemon prio=5 os_prio=0
 tid=0x0cbe6000 nid=0x1953 in Object.wait()
 [0x2ba658988000] java.lang.Thread.State: WAITING (on object
 monitor) at java.lang.Object.wait(Native Method) at
 java.lang.Object.wait(Object.java:502) at
 org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)


> - locked <0xf28030f8> (a
 org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> Hmm. Could it be...
> 
 "ajp-nio-8009-exec-22" #57 daemon prio=5 os_prio=0
 tid=0x0e797800 nid=0x194f in Object.wait()
 [0x2ba659472000] java.lang.Thread.State: WAITING (on object
 monitor) at java.lang.Object.wait(Native Method) at
 java.lang.Object.wait(Object.java:502) at
 org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)


> - locked <0xf27daf28> (a
 org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> ... just possibly ...
> 
 "ajp-nio-8009-exec-21" #56 daemon prio=5 os_prio=0
 tid=0x0e874000 nid=0x194d in Object.wait()
 [0x2ba656c67000] java.lang.Thread.State: WAITING (on object
 monitor) at java.lang.Object.wait(Native Method) at
 java.lang.Object.wait(Object.java:502) at
 org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)


> - locked <0xf27b3d20> (a
 org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> ... that you have run out of ...
> 
 "QuartzScheduler_scheduler-NON_CLUSTERED_MisfireHandler" #55
 

Re: Upgrade to 8.5.8/9

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Greg,

On 12/22/16 12:50 PM, Greg Huber wrote:
> Thanks, I have been running this setup for ages, 8 plus years and
> various tomcat versions.  It has never failed on  <= 8.0.x

Interesting. Usually, things don't just stop working like that.

Did you change anything besides replacing Tomcat 8.0.x with 8.5.x?

> I had to stop/start a couple of times to get it to fail and had to 
> run a parallel maven build to slow things down.

Does that "slow down" cause your database to slow-down as well?
Long-running queries can pile-up on top of one another sometimes.

In dev, I always run with pool size=1 so I can catch problem early. Is
it possible that you have a very small pool size and even a little bit
of load can bring everything to a crawl?

> Do not read much on dumps, but what would this -locked be?
> 
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoi
nt.java:1437)
>
> 
at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase
.java:49)
>
> 
- - locked <0xf1c54ed8> (a
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
ava:1142)
>
> 
at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
java:617)
>
> 
at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThr
ead.java:61)
>
> 
at java.lang.Thread.run(Thread.java:745)

"Locked" means that the thread holds a lock on a monitor. Basically,
the thread has synchronized on something and it's telling you what the
object id is (I think that's just a pointer, actually) and its type
(an NioSocketWrapper).

You'll see the same thing for the datasource: a thread has locked the
data source's latch object while it waits for a connection to become
available.

- -chris

> On 22 December 2016 at 16:24, Christopher Schultz < 
> ch...@christopherschultz.net> wrote:
> 
> Greg,
> 
> On 12/22/16 5:18 AM, Greg Huber wrote:
 Here is the first dump and I cannot send all three.  Its my
 prod server which is locked up at the moment.  I need to get
 it back on line so is this enough info?  I can emile the
 whole file which has the three dumps if needed.
 
 [snip]
 
 Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25
 mixed mode):
 
 "ajp-nio-8009-exec-26" #61 daemon prio=5 os_prio=0 
 tid=0x0cb1 nid=0x1960 in Object.wait() 
 [0x2ba658a4b000] java.lang.Thread.State: WAITING (on
 object monitor) at java.lang.Object.wait(Native Method) at 
 java.lang.Object.wait(Object.java:502) at 
 org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic
Obj
>
 
ectPool.java:1104)
 
 
> - locked <0xf2879c10> (a
 org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> Waiting on a database connection.
> 
> 
 "ajp-nio-8009-exec-25" #60 daemon prio=5 os_prio=0 
 tid=0x0c74f800 nid=0x195d in Object.wait() 
 [0x2ba658a0a000] java.lang.Thread.State: WAITING (on
 object monitor) at java.lang.Object.wait(Native Method) at 
 java.lang.Object.wait(Object.java:502) at 
 org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic
Obj
>
 
ectPool.java:1104)
 
 
> - locked <0xf2852128> (a
 org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> Another thread waiting for a db connection.
> 
 "ajp-nio-8009-exec-24" #59 daemon prio=5 os_prio=0 
 tid=0x0dd3b000 nid=0x1959 in Object.wait() 
 [0x2ba6589c9000] java.lang.Thread.State: WAITING (on
 object monitor) at java.lang.Object.wait(Native Method) at 
 java.lang.Object.wait(Object.java:502) at 
 org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic
Obj
>
 
ectPool.java:1104)
 
 
> - locked <0xf282a818> (a
 org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> I'm starting to see a pattern, here.
> 
 "ajp-nio-8009-exec-23" #58 daemon prio=5 os_prio=0 
 tid=0x0cbe6000 nid=0x1953 in Object.wait() 
 [0x2ba658988000] java.lang.Thread.State: WAITING (on
 object monitor) at java.lang.Object.wait(Native Method) at 
 java.lang.Object.wait(Object.java:502) at 
 org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic
Obj
>
 
ectPool.java:1104)
 
 
> - locked <0xf28030f8> (a
 org.apache.commons.pool.impl.GenericObjectPool$Latch)
> 
> Hmm. Could it be...
> 
 "ajp-nio-8009-exec-22" #57 daemon prio=5 os_prio=0 
 tid=0x0e797800 nid=0x194f in Object.wait() 
 [0x2ba659472000] java.lang.Thread.State: WAITING (on
 object monitor) at java.lang.Object.wait(Native Method) at 
 java.lang.Object.wait(Object.java:502) at 
 org.apache.commons.pool.impl.GenericObjectPool.borrowObject(Generic
Obj
>
 
ectPool.java:1104)
 
 
> - locked <0xf27daf28> (a
 

Re: Upgrade to 8.5.8/9

[Greg Huber]

Chris,

Thanks, I have been running this setup for ages, 8 plus years and various
tomcat versions.  It has never failed on  <= 8.0.x  I had to stop/start a
couple of times to get it to fail and had to run a parallel maven build to
slow things down.

Do not read much on dumps, but what would this -locked be?

at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1437)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0xf1c54ed8> (a
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

Cheers Greg


On 22 December 2016 at 16:24, Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Greg,
>
> On 12/22/16 5:18 AM, Greg Huber wrote:
> > Here is the first dump and I cannot send all three.  Its my prod
> > server which is locked up at the moment.  I need to get it back on
> > line so is this enough info?  I can emile the whole file which has
> > the three dumps if needed.
> >
> > [snip]
> >
> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
> > mode):
> >
> > "ajp-nio-8009-exec-26" #61 daemon prio=5 os_prio=0
> > tid=0x0cb1 nid=0x1960 in Object.wait()
> > [0x2ba658a4b000] java.lang.Thread.State: WAITING (on object
> > monitor) at java.lang.Object.wait(Native Method) at
> > java.lang.Object.wait(Object.java:502) at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)
> >
> >
> - - locked <0xf2879c10> (a
> > org.apache.commons.pool.impl.GenericObjectPool$Latch)
>
> Waiting on a database connection.
>
>
> > "ajp-nio-8009-exec-25" #60 daemon prio=5 os_prio=0
> > tid=0x0c74f800 nid=0x195d in Object.wait()
> > [0x2ba658a0a000] java.lang.Thread.State: WAITING (on object
> > monitor) at java.lang.Object.wait(Native Method) at
> > java.lang.Object.wait(Object.java:502) at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)
> >
> >
> - - locked <0xf2852128> (a
> > org.apache.commons.pool.impl.GenericObjectPool$Latch)
>
> Another thread waiting for a db connection.
>
> > "ajp-nio-8009-exec-24" #59 daemon prio=5 os_prio=0
> > tid=0x0dd3b000 nid=0x1959 in Object.wait()
> > [0x2ba6589c9000] java.lang.Thread.State: WAITING (on object
> > monitor) at java.lang.Object.wait(Native Method) at
> > java.lang.Object.wait(Object.java:502) at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)
> >
> >
> - - locked <0xf282a818> (a
> > org.apache.commons.pool.impl.GenericObjectPool$Latch)
>
> I'm starting to see a pattern, here.
>
> > "ajp-nio-8009-exec-23" #58 daemon prio=5 os_prio=0
> > tid=0x0cbe6000 nid=0x1953 in Object.wait()
> > [0x2ba658988000] java.lang.Thread.State: WAITING (on object
> > monitor) at java.lang.Object.wait(Native Method) at
> > java.lang.Object.wait(Object.java:502) at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)
> >
> >
> - - locked <0xf28030f8> (a
> > org.apache.commons.pool.impl.GenericObjectPool$Latch)
>
> Hmm. Could it be...
>
> > "ajp-nio-8009-exec-22" #57 daemon prio=5 os_prio=0
> > tid=0x0e797800 nid=0x194f in Object.wait()
> > [0x2ba659472000] java.lang.Thread.State: WAITING (on object
> > monitor) at java.lang.Object.wait(Native Method) at
> > java.lang.Object.wait(Object.java:502) at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)
> >
> >
> - - locked <0xf27daf28> (a
> > org.apache.commons.pool.impl.GenericObjectPool$Latch)
>
> ... just possibly ...
>
> > "ajp-nio-8009-exec-21" #56 daemon prio=5 os_prio=0
> > tid=0x0e874000 nid=0x194d in Object.wait()
> > [0x2ba656c67000] java.lang.Thread.State: WAITING (on object
> > monitor) at java.lang.Object.wait(Native Method) at
> > java.lang.Object.wait(Object.java:502) at
> > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1104)
> >
> >
> - - locked <0xf27b3d20> (a
> > org.apache.commons.pool.impl.GenericObjectPool$Latch)
>
> ... that you have run out of ...
>
> > "QuartzScheduler_scheduler-NON_CLUSTERED_MisfireHandler" #55
> > prio=5 os_prio=0 tid=0x0cbe5000 nid=0x194b waiting on
> > condition [0x2ba658c5] java.lang.Thread.State:
> > TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method)
> > at
> > org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobSto
> reSupport.java:3992)
>
> (Quartz
> >
> waiting around for some event. Ignore this one in erms of my
> 

Re: Upgrade to 8.5.8/9

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Greg,

On 12/22/16 5:18 AM, Greg Huber wrote:
> Here is the first dump and I cannot send all three.  Its my prod
> server which is locked up at the moment.  I need to get it back on
> line so is this enough info?  I can emile the whole file which has
> the three dumps if needed.
> 
> [snip]
> 
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed
> mode):
> 
> "ajp-nio-8009-exec-26" #61 daemon prio=5 os_prio=0
> tid=0x0cb1 nid=0x1960 in Object.wait()
> [0x2ba658a4b000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1104)
>
> 
- - locked <0xf2879c10> (a
> org.apache.commons.pool.impl.GenericObjectPool$Latch)

Waiting on a database connection.


> "ajp-nio-8009-exec-25" #60 daemon prio=5 os_prio=0
> tid=0x0c74f800 nid=0x195d in Object.wait()
> [0x2ba658a0a000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1104)
>
> 
- - locked <0xf2852128> (a
> org.apache.commons.pool.impl.GenericObjectPool$Latch)

Another thread waiting for a db connection.

> "ajp-nio-8009-exec-24" #59 daemon prio=5 os_prio=0
> tid=0x0dd3b000 nid=0x1959 in Object.wait()
> [0x2ba6589c9000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1104)
>
> 
- - locked <0xf282a818> (a
> org.apache.commons.pool.impl.GenericObjectPool$Latch)

I'm starting to see a pattern, here.

> "ajp-nio-8009-exec-23" #58 daemon prio=5 os_prio=0
> tid=0x0cbe6000 nid=0x1953 in Object.wait()
> [0x2ba658988000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1104)
>
> 
- - locked <0xf28030f8> (a
> org.apache.commons.pool.impl.GenericObjectPool$Latch)

Hmm. Could it be...

> "ajp-nio-8009-exec-22" #57 daemon prio=5 os_prio=0
> tid=0x0e797800 nid=0x194f in Object.wait()
> [0x2ba659472000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1104)
>
> 
- - locked <0xf27daf28> (a
> org.apache.commons.pool.impl.GenericObjectPool$Latch)

... just possibly ...

> "ajp-nio-8009-exec-21" #56 daemon prio=5 os_prio=0
> tid=0x0e874000 nid=0x194d in Object.wait()
> [0x2ba656c67000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1104)
>
> 
- - locked <0xf27b3d20> (a
> org.apache.commons.pool.impl.GenericObjectPool$Latch)

... that you have run out of ...

> "QuartzScheduler_scheduler-NON_CLUSTERED_MisfireHandler" #55
> prio=5 os_prio=0 tid=0x0cbe5000 nid=0x194b waiting on
> condition [0x2ba658c5] java.lang.Thread.State:
> TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) 
> at 
> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobSto
reSupport.java:3992)

(Quartz
> 
waiting around for some event. Ignore this one in erms of my
overarching narrative.)

> "ajp-nio-8009-exec-20" #54 daemon prio=5 os_prio=0
> tid=0x0e44f000 nid=0x194a in Object.wait()
> [0x2ba658948000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1104)
>
> 
- - locked <0xf1f541b8> (a
> org.apache.commons.pool.impl.GenericObjectPool$Latch)

Database connections?

> "ajp-nio-8009-exec-19" #53 daemon prio=5 os_prio=0
> tid=0x0cc2e800 nid=0x1940 in Object.wait()
> [0x2ba658908000] java.lang.Thread.State: RUNNABLE at
> org.events.ui.core.filters.IPBanFilter.doFilter(IPBanFilter.java:36)

Ooh!
> 
This one is doing something different. I suspect any ms it will
ask for another db connection.

> "ajp-nio-8009-exec-18" #52 daemon prio=5 os_prio=0
> tid=0x0cc7b800 nid=0x1933 in Object.wait()
> [0x2ba6588c6000] java.lang.Thread.State: WAITING (on object
> monitor) at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at 
> 

Re: Upgrade to 8.5.8/9

[Greg Huber]

Mark,

Its locked up now and I have the dump but it wont let me email it.  Can I
email the whole file somewhere?

Cheers Greg

On 22 December 2016 at 09:17, Mark Thomas  wrote:

> On 22/12/2016 08:55, Greg Huber wrote:
> > I installed exactly the same versions on my newer centos 7 server and it
> > does not lock up.
> >
> > I have seem this problem before as the production box is old and very
> > slow.  There may have been some changes to the connector stuff for a
> > map/list where it is trying to add and remove at the same time causing a
> > dead lock (eg use a CopyOnWriteArrayList etc instead). This would explain
> > why it is erratic and happens on startup (where system is going to be
> slow).
> >
> > Would it be better to log an issue for this?
>
> Unless you can provide reliable steps to reproduce, an issue based on
> the information provided so far is likely to get resolved as invalid.
>
> When the lock up occurs, take 3 thread dumps ~10s apart. That should
> tell you a) any deadlocks, b) what each of the threads is doing and c)
> identify which threads should be moving that aren't.
>
> If you need help analysing the thread dumps, post them (or links to
> them) here.
>
> Mark
>
> >
> > Cheers Greg
> >
> >
> >
> > On 20 December 2016 at 14:10, Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> > Greg,
> >
> > On 12/20/16 3:21 AM, Greg Huber wrote:
> > If you are using httpd + mod_jk, why are you bothering with APR
> > and tcnative on the Tomcat side?
> 
>  It complains in the log file so I do what it says.  If I don't need
>  it it saves me having to compile it all the time.
> >
> > You get complaints about the APR library not being installed? Yeah,
> > just comment-out the APRLifecycleListener in conf/server.xml.
> >
> > What if you use the NIO connector instead of APR?
> 
>  Have always run tomcat with apache and mod_jk has always worked. I
>  could look at NIO if all else fails.
> >
> > No matter what kind of protocol you use (HTTP, AJP13), you can still
> > choose the I/O strategy (APR, NIO, NIO2). The pure-Java blocking I/O
> > connector (BIO) was removed in 8.5.x and 9.0.x, so that's no longer an
> > option.
> >
> > Can you post your  configuration?
> 
>  I do not modify anything here.
> 
>  My setup:
> 
>  workers.properties:
> 
>  worker.list=worker1 worker.worker1.type=ajp13
>  worker.worker1.host=localhost worker.worker1.port=8009
>  worker.worker1.lbfactor=50 worker.worker1.socket_keepalive=1
> 
>  tomcat.conf
> 
>  ... JkWorkersFile "/etc/httpd/conf.d/workers.properties" # Set the
>  log file JkLogFile "/etc/httpd/logs/mod_jk.log" JkShmFile
>  "/etc/httpd/logs/jk-runtime-status.log" JkLogLevel info
>  JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"  
>  ... JkMount  / worker1 JkMount  /* worker1 
> >
> > That looks okay to me, though it doesn't look like you actually need
> > httpd in the mix, here. I'll just assume that you are fronting Tomcat
> > for a good reason and not "because that's how it's supposed to be done."
> >
>  setenv.sh :
> 
>  export CATALINA_OPTS="$CATALINA_OPTS -Xms256m" export
>  CATALINA_OPTS="$CATALINA_OPTS -Xmx768m"
> >
> > Off-topic hint: if you are going to give your JVM 768M, you may as
> > well do it all at once and set -Xms == -Xmx.
> >
>  export CATALINA_OPTS="$CATALINA_OPTS -Xss256k"
> 
>  export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParallelGC"
> 
>  export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=1500"
> 
>  export CATALINA_OPTS="$CATALINA_OPTS -XX:GCTimeRatio=9"
> 
>  export CATALINA_OPTS="$CATALINA_OPTS -server"
> 
>  export CATALINA_OPTS="$CATALINA_OPTS -XX:+DisableExplicitGC"
> 
> 
>  Not sure where these exports came from but they seem to work up to
>  8.5.x
> >
> > They control the JVM, not Tomcat. If you are administering this Tomcat
> > instance, I urge you to learn more about what they do. Mysteries don't
> > help administrators do their jobs.
> >
>  The last start/stop catalina.out :
> 
>  17-Dec-2016 15:21:51.022 INFO [main]
>  org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
>  APR based Apache Tomcat Native library 1.2.10 using APR version
>  1.5.2. 17-Dec-2016 15:21:51.022 INFO [main]
>  org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
>  capabilities: IPv6 [true], sendfile [true], accept filters [false],
>  random [true]. 17-Dec-2016 15:21:51.023 INFO [main]
>  org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
>  APR/OpenSSL configuration: useAprConnector [false], useOpenSSL
>  [true] 17-Dec-2016 15:21:51.036 INFO [main]
>  org.apache.catalina.core.AprLifecycleListener.initializeSSL
>  OpenSSL successfully initialized (OpenSSL 1.0.2j  26 Sep 2016)
>  17-Dec-2016 15:21:52.967 INFO [main]
>  

Re: Upgrade to 8.5.8/9

[Greg Huber]

OK, will see if I can do the dumps.  Think i did something similar a while
back when tomcat got into a random loop/high cpu a while back.

Cheers Greg.

On 22 December 2016 at 09:17, Mark Thomas  wrote:

> On 22/12/2016 08:55, Greg Huber wrote:
> > I installed exactly the same versions on my newer centos 7 server and it
> > does not lock up.
> >
> > I have seem this problem before as the production box is old and very
> > slow.  There may have been some changes to the connector stuff for a
> > map/list where it is trying to add and remove at the same time causing a
> > dead lock (eg use a CopyOnWriteArrayList etc instead). This would explain
> > why it is erratic and happens on startup (where system is going to be
> slow).
> >
> > Would it be better to log an issue for this?
>
> Unless you can provide reliable steps to reproduce, an issue based on
> the information provided so far is likely to get resolved as invalid.
>
> When the lock up occurs, take 3 thread dumps ~10s apart. That should
> tell you a) any deadlocks, b) what each of the threads is doing and c)
> identify which threads should be moving that aren't.
>
> If you need help analysing the thread dumps, post them (or links to
> them) here.
>
> Mark
>
> >
> > Cheers Greg
> >
> >
> >
> > On 20 December 2016 at 14:10, Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> > Greg,
> >
> > On 12/20/16 3:21 AM, Greg Huber wrote:
> > If you are using httpd + mod_jk, why are you bothering with APR
> > and tcnative on the Tomcat side?
> 
>  It complains in the log file so I do what it says.  If I don't need
>  it it saves me having to compile it all the time.
> >
> > You get complaints about the APR library not being installed? Yeah,
> > just comment-out the APRLifecycleListener in conf/server.xml.
> >
> > What if you use the NIO connector instead of APR?
> 
>  Have always run tomcat with apache and mod_jk has always worked. I
>  could look at NIO if all else fails.
> >
> > No matter what kind of protocol you use (HTTP, AJP13), you can still
> > choose the I/O strategy (APR, NIO, NIO2). The pure-Java blocking I/O
> > connector (BIO) was removed in 8.5.x and 9.0.x, so that's no longer an
> > option.
> >
> > Can you post your  configuration?
> 
>  I do not modify anything here.
> 
>  My setup:
> 
>  workers.properties:
> 
>  worker.list=worker1 worker.worker1.type=ajp13
>  worker.worker1.host=localhost worker.worker1.port=8009
>  worker.worker1.lbfactor=50 worker.worker1.socket_keepalive=1
> 
>  tomcat.conf
> 
>  ... JkWorkersFile "/etc/httpd/conf.d/workers.properties" # Set the
>  log file JkLogFile "/etc/httpd/logs/mod_jk.log" JkShmFile
>  "/etc/httpd/logs/jk-runtime-status.log" JkLogLevel info
>  JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"  
>  ... JkMount  / worker1 JkMount  /* worker1 
> >
> > That looks okay to me, though it doesn't look like you actually need
> > httpd in the mix, here. I'll just assume that you are fronting Tomcat
> > for a good reason and not "because that's how it's supposed to be done."
> >
>  setenv.sh :
> 
>  export CATALINA_OPTS="$CATALINA_OPTS -Xms256m" export
>  CATALINA_OPTS="$CATALINA_OPTS -Xmx768m"
> >
> > Off-topic hint: if you are going to give your JVM 768M, you may as
> > well do it all at once and set -Xms == -Xmx.
> >
>  export CATALINA_OPTS="$CATALINA_OPTS -Xss256k"
> 
>  export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParallelGC"
> 
>  export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=1500"
> 
>  export CATALINA_OPTS="$CATALINA_OPTS -XX:GCTimeRatio=9"
> 
>  export CATALINA_OPTS="$CATALINA_OPTS -server"
> 
>  export CATALINA_OPTS="$CATALINA_OPTS -XX:+DisableExplicitGC"
> 
> 
>  Not sure where these exports came from but they seem to work up to
>  8.5.x
> >
> > They control the JVM, not Tomcat. If you are administering this Tomcat
> > instance, I urge you to learn more about what they do. Mysteries don't
> > help administrators do their jobs.
> >
>  The last start/stop catalina.out :
> 
>  17-Dec-2016 15:21:51.022 INFO [main]
>  org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
>  APR based Apache Tomcat Native library 1.2.10 using APR version
>  1.5.2. 17-Dec-2016 15:21:51.022 INFO [main]
>  org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
>  capabilities: IPv6 [true], sendfile [true], accept filters [false],
>  random [true]. 17-Dec-2016 15:21:51.023 INFO [main]
>  org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
>  APR/OpenSSL configuration: useAprConnector [false], useOpenSSL
>  [true] 17-Dec-2016 15:21:51.036 INFO [main]
>  org.apache.catalina.core.AprLifecycleListener.initializeSSL
>  OpenSSL successfully initialized (OpenSSL 1.0.2j  26 Sep 2016)
>  17-Dec-2016 15:21:52.967 INFO 

Re: Upgrade to 8.5.8/9

[Mark Thomas]

On 22/12/2016 08:55, Greg Huber wrote:
> I installed exactly the same versions on my newer centos 7 server and it
> does not lock up.
> 
> I have seem this problem before as the production box is old and very
> slow.  There may have been some changes to the connector stuff for a
> map/list where it is trying to add and remove at the same time causing a
> dead lock (eg use a CopyOnWriteArrayList etc instead). This would explain
> why it is erratic and happens on startup (where system is going to be slow).
> 
> Would it be better to log an issue for this?

Unless you can provide reliable steps to reproduce, an issue based on
the information provided so far is likely to get resolved as invalid.

When the lock up occurs, take 3 thread dumps ~10s apart. That should
tell you a) any deadlocks, b) what each of the threads is doing and c)
identify which threads should be moving that aren't.

If you need help analysing the thread dumps, post them (or links to
them) here.

Mark

> 
> Cheers Greg
> 
> 
> 
> On 20 December 2016 at 14:10, Christopher Schultz <
> ch...@christopherschultz.net> wrote:
> 
> Greg,
> 
> On 12/20/16 3:21 AM, Greg Huber wrote:
> If you are using httpd + mod_jk, why are you bothering with APR
> and tcnative on the Tomcat side?

 It complains in the log file so I do what it says.  If I don't need
 it it saves me having to compile it all the time.
> 
> You get complaints about the APR library not being installed? Yeah,
> just comment-out the APRLifecycleListener in conf/server.xml.
> 
> What if you use the NIO connector instead of APR?

 Have always run tomcat with apache and mod_jk has always worked. I
 could look at NIO if all else fails.
> 
> No matter what kind of protocol you use (HTTP, AJP13), you can still
> choose the I/O strategy (APR, NIO, NIO2). The pure-Java blocking I/O
> connector (BIO) was removed in 8.5.x and 9.0.x, so that's no longer an
> option.
> 
> Can you post your  configuration?

 I do not modify anything here.

 My setup:

 workers.properties:

 worker.list=worker1 worker.worker1.type=ajp13
 worker.worker1.host=localhost worker.worker1.port=8009
 worker.worker1.lbfactor=50 worker.worker1.socket_keepalive=1

 tomcat.conf

 ... JkWorkersFile "/etc/httpd/conf.d/workers.properties" # Set the
 log file JkLogFile "/etc/httpd/logs/mod_jk.log" JkShmFile
 "/etc/httpd/logs/jk-runtime-status.log" JkLogLevel info
 JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"  
 ... JkMount  / worker1 JkMount  /* worker1 
> 
> That looks okay to me, though it doesn't look like you actually need
> httpd in the mix, here. I'll just assume that you are fronting Tomcat
> for a good reason and not "because that's how it's supposed to be done."
> 
 setenv.sh :

 export CATALINA_OPTS="$CATALINA_OPTS -Xms256m" export
 CATALINA_OPTS="$CATALINA_OPTS -Xmx768m"
> 
> Off-topic hint: if you are going to give your JVM 768M, you may as
> well do it all at once and set -Xms == -Xmx.
> 
 export CATALINA_OPTS="$CATALINA_OPTS -Xss256k"

 export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParallelGC"

 export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=1500"

 export CATALINA_OPTS="$CATALINA_OPTS -XX:GCTimeRatio=9"

 export CATALINA_OPTS="$CATALINA_OPTS -server"

 export CATALINA_OPTS="$CATALINA_OPTS -XX:+DisableExplicitGC"


 Not sure where these exports came from but they seem to work up to
 8.5.x
> 
> They control the JVM, not Tomcat. If you are administering this Tomcat
> instance, I urge you to learn more about what they do. Mysteries don't
> help administrators do their jobs.
> 
 The last start/stop catalina.out :

 17-Dec-2016 15:21:51.022 INFO [main]
 org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
 APR based Apache Tomcat Native library 1.2.10 using APR version
 1.5.2. 17-Dec-2016 15:21:51.022 INFO [main]
 org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
 capabilities: IPv6 [true], sendfile [true], accept filters [false],
 random [true]. 17-Dec-2016 15:21:51.023 INFO [main]
 org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
 APR/OpenSSL configuration: useAprConnector [false], useOpenSSL
 [true] 17-Dec-2016 15:21:51.036 INFO [main]
 org.apache.catalina.core.AprLifecycleListener.initializeSSL
 OpenSSL successfully initialized (OpenSSL 1.0.2j  26 Sep 2016)
 17-Dec-2016 15:21:52.967 INFO [main]
 org.apache.coyote.AbstractProtocol.init Initializing
 ProtocolHandler ["http-nio-8080"] 17-Dec-2016 15:21:53.124 INFO
 [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector
 Using a shared selector for servlet write/read 17-Dec-2016
 15:21:53.160 INFO [main] org.apache.coyote.AbstractProtocol.init
 Initializing ProtocolHandler ["ajp-nio-8009"]
> 
> So, you've got an HTTP connector on port 8080 and 

Re: Upgrade to 8.5.8/9

[Greg Huber]

I installed exactly the same versions on my newer centos 7 server and it
does not lock up.

I have seem this problem before as the production box is old and very
slow.  There may have been some changes to the connector stuff for a
map/list where it is trying to add and remove at the same time causing a
dead lock (eg use a CopyOnWriteArrayList etc instead). This would explain
why it is erratic and happens on startup (where system is going to be slow).

Would it be better to log an issue for this?

Cheers Greg



On 20 December 2016 at 14:10, Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Greg,
>
> On 12/20/16 3:21 AM, Greg Huber wrote:
> >> If you are using httpd + mod_jk, why are you bothering with APR
> >> and tcnative on the Tomcat side?
> >
> > It complains in the log file so I do what it says.  If I don't need
> > it it saves me having to compile it all the time.
>
> You get complaints about the APR library not being installed? Yeah,
> just comment-out the APRLifecycleListener in conf/server.xml.
>
> >> What if you use the NIO connector instead of APR?
> >
> > Have always run tomcat with apache and mod_jk has always worked. I
> > could look at NIO if all else fails.
>
> No matter what kind of protocol you use (HTTP, AJP13), you can still
> choose the I/O strategy (APR, NIO, NIO2). The pure-Java blocking I/O
> connector (BIO) was removed in 8.5.x and 9.0.x, so that's no longer an
> option.
>
> >> Can you post your  configuration?
> >
> > I do not modify anything here.
> >
> > My setup:
> >
> > workers.properties:
> >
> > worker.list=worker1 worker.worker1.type=ajp13
> > worker.worker1.host=localhost worker.worker1.port=8009
> > worker.worker1.lbfactor=50 worker.worker1.socket_keepalive=1
> >
> > tomcat.conf
> >
> > ... JkWorkersFile "/etc/httpd/conf.d/workers.properties" # Set the
> > log file JkLogFile "/etc/httpd/logs/mod_jk.log" JkShmFile
> > "/etc/httpd/logs/jk-runtime-status.log" JkLogLevel info
> > JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"  
> > ... JkMount  / worker1 JkMount  /* worker1 
>
> That looks okay to me, though it doesn't look like you actually need
> httpd in the mix, here. I'll just assume that you are fronting Tomcat
> for a good reason and not "because that's how it's supposed to be done."
>
> > setenv.sh :
> >
> > export CATALINA_OPTS="$CATALINA_OPTS -Xms256m" export
> > CATALINA_OPTS="$CATALINA_OPTS -Xmx768m"
>
> Off-topic hint: if you are going to give your JVM 768M, you may as
> well do it all at once and set -Xms == -Xmx.
>
> > export CATALINA_OPTS="$CATALINA_OPTS -Xss256k"
> >
> > export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParallelGC"
> >
> > export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=1500"
> >
> > export CATALINA_OPTS="$CATALINA_OPTS -XX:GCTimeRatio=9"
> >
> > export CATALINA_OPTS="$CATALINA_OPTS -server"
> >
> > export CATALINA_OPTS="$CATALINA_OPTS -XX:+DisableExplicitGC"
> >
> >
> > Not sure where these exports came from but they seem to work up to
> > 8.5.x
>
> They control the JVM, not Tomcat. If you are administering this Tomcat
> instance, I urge you to learn more about what they do. Mysteries don't
> help administrators do their jobs.
>
> > The last start/stop catalina.out :
> >
> > 17-Dec-2016 15:21:51.022 INFO [main]
> > org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
> > APR based Apache Tomcat Native library 1.2.10 using APR version
> > 1.5.2. 17-Dec-2016 15:21:51.022 INFO [main]
> > org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
> > capabilities: IPv6 [true], sendfile [true], accept filters [false],
> > random [true]. 17-Dec-2016 15:21:51.023 INFO [main]
> > org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
> > APR/OpenSSL configuration: useAprConnector [false], useOpenSSL
> > [true] 17-Dec-2016 15:21:51.036 INFO [main]
> > org.apache.catalina.core.AprLifecycleListener.initializeSSL
> > OpenSSL successfully initialized (OpenSSL 1.0.2j  26 Sep 2016)
> > 17-Dec-2016 15:21:52.967 INFO [main]
> > org.apache.coyote.AbstractProtocol.init Initializing
> > ProtocolHandler ["http-nio-8080"] 17-Dec-2016 15:21:53.124 INFO
> > [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector
> > Using a shared selector for servlet write/read 17-Dec-2016
> > 15:21:53.160 INFO [main] org.apache.coyote.AbstractProtocol.init
> > Initializing ProtocolHandler ["ajp-nio-8009"]
>
> So, you've got an HTTP connector on port 8080 and an AJP connector on
> port 8009. You aren't using TLS at all, so you can definitely let the
> APR LifecycleListener go and stop re-compiling libtcnative for each
> release. It looks like you also aren't using your connector on port
> 8080, so you can disable that, too. You likely have JVM threads that
> are sitting idle all the time.
>
> Oh, and you're already using NIO. :)
>
> > 17-Dec-2016 15:23:08.320 INFO [main]
> > org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler
> > [http-nio-8080] 17-Dec-2016 

Re: Upgrade to 8.5.8/9

[Greg Huber]

Chris,

>After Tomcat is started, can you run this command on >the server running
both httpd and Tomcat?

>$ sudo netstat -plan | grep 8009

The problem is my prod server (centos 5), so I will switch back to 8.5.9
and when it hangs I can run the command.  Also I do have a backup server
(centos 7) which I will also update to 8.5.9 and see if I get the same
issue which will allow more testing also.


Cheers. Greg

On 20 December 2016 at 14:10, Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Greg,
>
> On 12/20/16 3:21 AM, Greg Huber wrote:
> >> If you are using httpd + mod_jk, why are you bothering with APR
> >> and tcnative on the Tomcat side?
> >
> > It complains in the log file so I do what it says.  If I don't need
> > it it saves me having to compile it all the time.
>
> You get complaints about the APR library not being installed? Yeah,
> just comment-out the APRLifecycleListener in conf/server.xml.
>
> >> What if you use the NIO connector instead of APR?
> >
> > Have always run tomcat with apache and mod_jk has always worked. I
> > could look at NIO if all else fails.
>
> No matter what kind of protocol you use (HTTP, AJP13), you can still
> choose the I/O strategy (APR, NIO, NIO2). The pure-Java blocking I/O
> connector (BIO) was removed in 8.5.x and 9.0.x, so that's no longer an
> option.
>
> >> Can you post your  configuration?
> >
> > I do not modify anything here.
> >
> > My setup:
> >
> > workers.properties:
> >
> > worker.list=worker1 worker.worker1.type=ajp13
> > worker.worker1.host=localhost worker.worker1.port=8009
> > worker.worker1.lbfactor=50 worker.worker1.socket_keepalive=1
> >
> > tomcat.conf
> >
> > ... JkWorkersFile "/etc/httpd/conf.d/workers.properties" # Set the
> > log file JkLogFile "/etc/httpd/logs/mod_jk.log" JkShmFile
> > "/etc/httpd/logs/jk-runtime-status.log" JkLogLevel info
> > JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"  
> > ... JkMount  / worker1 JkMount  /* worker1 
>
> That looks okay to me, though it doesn't look like you actually need
> httpd in the mix, here. I'll just assume that you are fronting Tomcat
> for a good reason and not "because that's how it's supposed to be done."
>
> > setenv.sh :
> >
> > export CATALINA_OPTS="$CATALINA_OPTS -Xms256m" export
> > CATALINA_OPTS="$CATALINA_OPTS -Xmx768m"
>
> Off-topic hint: if you are going to give your JVM 768M, you may as
> well do it all at once and set -Xms == -Xmx.
>
> > export CATALINA_OPTS="$CATALINA_OPTS -Xss256k"
> >
> > export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParallelGC"
> >
> > export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=1500"
> >
> > export CATALINA_OPTS="$CATALINA_OPTS -XX:GCTimeRatio=9"
> >
> > export CATALINA_OPTS="$CATALINA_OPTS -server"
> >
> > export CATALINA_OPTS="$CATALINA_OPTS -XX:+DisableExplicitGC"
> >
> >
> > Not sure where these exports came from but they seem to work up to
> > 8.5.x
>
> They control the JVM, not Tomcat. If you are administering this Tomcat
> instance, I urge you to learn more about what they do. Mysteries don't
> help administrators do their jobs.
>
> > The last start/stop catalina.out :
> >
> > 17-Dec-2016 15:21:51.022 INFO [main]
> > org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
> > APR based Apache Tomcat Native library 1.2.10 using APR version
> > 1.5.2. 17-Dec-2016 15:21:51.022 INFO [main]
> > org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
> > capabilities: IPv6 [true], sendfile [true], accept filters [false],
> > random [true]. 17-Dec-2016 15:21:51.023 INFO [main]
> > org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
> > APR/OpenSSL configuration: useAprConnector [false], useOpenSSL
> > [true] 17-Dec-2016 15:21:51.036 INFO [main]
> > org.apache.catalina.core.AprLifecycleListener.initializeSSL
> > OpenSSL successfully initialized (OpenSSL 1.0.2j  26 Sep 2016)
> > 17-Dec-2016 15:21:52.967 INFO [main]
> > org.apache.coyote.AbstractProtocol.init Initializing
> > ProtocolHandler ["http-nio-8080"] 17-Dec-2016 15:21:53.124 INFO
> > [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector
> > Using a shared selector for servlet write/read 17-Dec-2016
> > 15:21:53.160 INFO [main] org.apache.coyote.AbstractProtocol.init
> > Initializing ProtocolHandler ["ajp-nio-8009"]
>
> So, you've got an HTTP connector on port 8080 and an AJP connector on
> port 8009. You aren't using TLS at all, so you can definitely let the
> APR LifecycleListener go and stop re-compiling libtcnative for each
> release. It looks like you also aren't using your connector on port
> 8080, so you can disable that, too. You likely have JVM threads that
> are sitting idle all the time.
>
> Oh, and you're already using NIO. :)
>
> > 17-Dec-2016 15:23:08.320 INFO [main]
> > org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler
> > [http-nio-8080] 17-Dec-2016 15:23:08.341 INFO [main]
> > org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler
> > 

Re: Upgrade to 8.5.8/9

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Greg,

On 12/20/16 3:21 AM, Greg Huber wrote:
>> If you are using httpd + mod_jk, why are you bothering with APR
>> and tcnative on the Tomcat side?
> 
> It complains in the log file so I do what it says.  If I don't need
> it it saves me having to compile it all the time.

You get complaints about the APR library not being installed? Yeah,
just comment-out the APRLifecycleListener in conf/server.xml.

>> What if you use the NIO connector instead of APR?
> 
> Have always run tomcat with apache and mod_jk has always worked. I
> could look at NIO if all else fails.

No matter what kind of protocol you use (HTTP, AJP13), you can still
choose the I/O strategy (APR, NIO, NIO2). The pure-Java blocking I/O
connector (BIO) was removed in 8.5.x and 9.0.x, so that's no longer an
option.

>> Can you post your  configuration?
> 
> I do not modify anything here.
> 
> My setup:
> 
> workers.properties:
> 
> worker.list=worker1 worker.worker1.type=ajp13 
> worker.worker1.host=localhost worker.worker1.port=8009 
> worker.worker1.lbfactor=50 worker.worker1.socket_keepalive=1
> 
> tomcat.conf
> 
> ... JkWorkersFile "/etc/httpd/conf.d/workers.properties" # Set the
> log file JkLogFile "/etc/httpd/logs/mod_jk.log" JkShmFile
> "/etc/httpd/logs/jk-runtime-status.log" JkLogLevel info 
> JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"   
> ... JkMount  / worker1 JkMount  /* worker1 

That looks okay to me, though it doesn't look like you actually need
httpd in the mix, here. I'll just assume that you are fronting Tomcat
for a good reason and not "because that's how it's supposed to be done."

> setenv.sh :
> 
> export CATALINA_OPTS="$CATALINA_OPTS -Xms256m" export
> CATALINA_OPTS="$CATALINA_OPTS -Xmx768m"

Off-topic hint: if you are going to give your JVM 768M, you may as
well do it all at once and set -Xms == -Xmx.

> export CATALINA_OPTS="$CATALINA_OPTS -Xss256k"
> 
> export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParallelGC"
> 
> export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=1500"
> 
> export CATALINA_OPTS="$CATALINA_OPTS -XX:GCTimeRatio=9"
> 
> export CATALINA_OPTS="$CATALINA_OPTS -server"
> 
> export CATALINA_OPTS="$CATALINA_OPTS -XX:+DisableExplicitGC"
> 
> 
> Not sure where these exports came from but they seem to work up to
> 8.5.x

They control the JVM, not Tomcat. If you are administering this Tomcat
instance, I urge you to learn more about what they do. Mysteries don't
help administrators do their jobs.

> The last start/stop catalina.out :
> 
> 17-Dec-2016 15:21:51.022 INFO [main] 
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
> APR based Apache Tomcat Native library 1.2.10 using APR version
> 1.5.2. 17-Dec-2016 15:21:51.022 INFO [main] 
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR 
> capabilities: IPv6 [true], sendfile [true], accept filters [false],
> random [true]. 17-Dec-2016 15:21:51.023 INFO [main] 
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
> APR/OpenSSL configuration: useAprConnector [false], useOpenSSL
> [true] 17-Dec-2016 15:21:51.036 INFO [main] 
> org.apache.catalina.core.AprLifecycleListener.initializeSSL
> OpenSSL successfully initialized (OpenSSL 1.0.2j  26 Sep 2016) 
> 17-Dec-2016 15:21:52.967 INFO [main] 
> org.apache.coyote.AbstractProtocol.init Initializing
> ProtocolHandler ["http-nio-8080"] 17-Dec-2016 15:21:53.124 INFO
> [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector
> Using a shared selector for servlet write/read 17-Dec-2016
> 15:21:53.160 INFO [main] org.apache.coyote.AbstractProtocol.init
> Initializing ProtocolHandler ["ajp-nio-8009"]

So, you've got an HTTP connector on port 8080 and an AJP connector on
port 8009. You aren't using TLS at all, so you can definitely let the
APR LifecycleListener go and stop re-compiling libtcnative for each
release. It looks like you also aren't using your connector on port
8080, so you can disable that, too. You likely have JVM threads that
are sitting idle all the time.

Oh, and you're already using NIO. :)

> 17-Dec-2016 15:23:08.320 INFO [main] 
> org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler 
> [http-nio-8080] 17-Dec-2016 15:23:08.341 INFO [main] 
> org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler 
> [ajp-nio-8009]

Looks like these guys started-up okay.

> 17-Dec-2016 15:23:08.342 INFO [main] 
> org.apache.catalina.startup.Catalina.start Server startup in 75173
> ms

Okay.

> 17-Dec-2016 15:59:45.589 INFO [Thread-7] 
> org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler 
> ["http-nio-8080"] 17-Dec-2016 15:59:45.805 INFO [Thread-7] 
> org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler 
> ["ajp-nio-8009"]

A few minutes later, you stop the server.

I don't see any problems, here, except that you are saying you are
getting connection failures.

After Tomcat is started, can you run this command on the server
running both httpd and Tomcat?

$ sudo 

Re: Upgrade to 8.5.8/9

[Greg Huber]

> If you are using httpd + mod_jk, why are you
> bothering with APR and tcnative on the Tomcat
> side?

It complains in the log file so I do what it says.  If I don't need it it
saves me having to compile it all the time.

> What if you use the NIO connector instead of APR?

Have always run tomcat with apache and mod_jk has always worked. I could
look at NIO if all else fails.

> Can you post your  configuration?

I do not modify anything here.

My setup:

workers.properties:

worker.list=worker1
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=8009
worker.worker1.lbfactor=50
worker.worker1.socket_keepalive=1

tomcat.conf

...
JkWorkersFile "/etc/httpd/conf.d/workers.properties"
# Set the log file
JkLogFile "/etc/httpd/logs/mod_jk.log"
JkShmFile "/etc/httpd/logs/jk-runtime-status.log"
JkLogLevel info
JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"


...
JkMount  / worker1
JkMount  /* worker1



setenv.sh :

export CATALINA_OPTS="$CATALINA_OPTS -Xms256m"
export CATALINA_OPTS="$CATALINA_OPTS -Xmx768m"

export CATALINA_OPTS="$CATALINA_OPTS -Xss256k"

export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParallelGC"

export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=1500"

export CATALINA_OPTS="$CATALINA_OPTS -XX:GCTimeRatio=9"

export CATALINA_OPTS="$CATALINA_OPTS -server"

export CATALINA_OPTS="$CATALINA_OPTS -XX:+DisableExplicitGC"


Not sure where these exports came from but they seem to work up to 8.5.x


The last start/stop catalina.out :

17-Dec-2016 15:21:51.007 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Server
version:Apache Tomcat/8.5.8
17-Dec-2016 15:21:51.012 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Server
built:  Nov 3 2016 21:14:13 UTC
17-Dec-2016 15:21:51.012 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Server
number: 8.5.8.0
17-Dec-2016 15:21:51.012 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log OS
Name:   Linux
17-Dec-2016 15:21:51.013 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log OS
Version:2.6.18-416.el5
17-Dec-2016 15:21:51.013 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log
Architecture:  amd64
17-Dec-2016 15:21:51.013 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Java
Home: /opt/oraclejava1.8/jdk1.8.0_40/jre
17-Dec-2016 15:21:51.014 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log JVM
Version:   1.8.0_40-b26
17-Dec-2016 15:21:51.014 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log JVM
Vendor:Oracle Corporation
17-Dec-2016 15:21:51.014 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log
CATALINA_BASE: /opt/apache-tomcat/apache-tomcat-8.5.8
17-Dec-2016 15:21:51.015 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log
CATALINA_HOME: /opt/apache-tomcat/apache-tomcat-8.5.8
17-Dec-2016 15:21:51.017 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Devents.nodename=VM0
17-Dec-2016 15:21:51.017 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Devents.activemq.portZ=61617
17-Dec-2016 15:21:51.018 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Duser.timezone=Europe/London
17-Dec-2016 15:21:51.018 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
17-Dec-2016 15:21:51.018 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument:
-Djava.util.logging.config.file=/opt/apache-tomcat/apache-tomcat-8.5.8/conf/logging.properties
17-Dec-2016 15:21:51.019 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Xms256m
17-Dec-2016 15:21:51.019 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Xmx768m
17-Dec-2016 15:21:51.019 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Xss256k
17-Dec-2016 15:21:51.020 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -XX:+UseParallelGC
17-Dec-2016 15:21:51.020 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -XX:MaxGCPauseMillis=1500
17-Dec-2016 15:21:51.020 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -XX:GCTimeRatio=9
17-Dec-2016 15:21:51.021 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -XX:+DisableExplicitGC
17-Dec-2016 15:21:51.021 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djava.endorsed.dirs=
17-Dec-2016 15:21:51.021 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dcatalina.base=/opt/apache-tomcat/apache-tomcat-8.5.8

Re: Upgrade to 8.5.8/9

[Greg Huber]

>Are you sure that your new instance has an adequate >number of threads
available for httpd to proxy to?

How would I check the threads for httpd?  It normally hangs on startup so
there should be enough system threads.  Would a GC be required when the
system has just started?

I have run my startup scripts years so maybe tomcat 8.5.x needs something
extra, memory? (see setenv.sh below)  I have reverted back to 8.0.x and it
starts and runs OK with no issues.

My setup:

workers.properties:

worker.list=worker1
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=8009
worker.worker1.lbfactor=50
worker.worker1.socket_keepalive=1

tomcat.conf

...
JkWorkersFile "/etc/httpd/conf.d/workers.properties"
# Set the log file
JkLogFile "/etc/httpd/logs/mod_jk.log"
JkShmFile "/etc/httpd/logs/jk-runtime-status.log"
JkLogLevel info
JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"


...
JkMount  / worker1
JkMount  /* worker1



setenv.sh :

export CATALINA_OPTS="$CATALINA_OPTS -Xms256m"
export CATALINA_OPTS="$CATALINA_OPTS -Xmx768m"

export CATALINA_OPTS="$CATALINA_OPTS -Xss256k"

export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParallelGC"

export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=1500"

export CATALINA_OPTS="$CATALINA_OPTS -XX:GCTimeRatio=9"

export CATALINA_OPTS="$CATALINA_OPTS -server"

export CATALINA_OPTS="$CATALINA_OPTS -XX:+DisableExplicitGC"


Not sure where these exports came from but they seem to work up to 8.5.x

Cheers Greg.



On 19 December 2016 at 17:55, Coty Sutherland  wrote:

> Hm, errno=111 is a connection refusal. Are you sure that your new
> instance has an adequate number of threads available for httpd to
> proxy to? Do you see any errors in your tomcat logging? If you do have
> sufficient threads in the pool, then maybe there is something (like GC
> pauses) hanging your requests that wasn't before and therefore
> exhausting the pool causing rejections.
>
> On Mon, Dec 19, 2016 at 4:47 AM, Greg Huber  wrote:
> > Hello,
> >
> > I am currently running tomcat 8.0.32 and have tried to upgrade to 8.5.8
> and
> > 8.5.9 without success.  I use mod_jk to connect apache to tomcat running
> on
> > centos 5.11 with Apache 2.2.3.
> >
> > I have installed openSSL 1.0.2.j to compile the native
> tomcat-native-1.2.10
> > using apr-1.5.2 and tomcat-connectors-1.2.42 for the mod_jk.
> >
> > The problem is that tomcat seems to start but fails sometimes to connect
> > correctly to apache and causes apache to hang.  If I stop tomcat, apache
> > starts working again.  I have been through the logs with little
> indication
> > of the problem.  If I revet back to 8.0.32 everything works OK.
> >
> >
> > All I can find is this in the mod_jk.log:
> >
> >
> > [Sat Dec 17 06:11:48 2016][3212:47280261211024] [info] init_jk::mod_jk.c
> > (3595): mod_jk/1.2.42 initialized
> > [Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
> > jk_open_socket::jk_connect.c (817): connect to 127.0.0.1:8009 failed
> > (errno=111)
> > [Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
> > ajp_connect_to_endpoint::jk_ajp_common.c (1068): (worker1) Failed
> opening
> > socket to (127.0.0.1:8009) (errno=111)
> > [Sat Dec 17 06:11:50 2016][3254:47280261211024] [error]
> > ajp_send_request::jk_ajp_common.c (1728): (worker1) connecting to
> backend
> > failed. Tomcat is probably not started or is listening on the wrong port
> > (errno=111)
> > [Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
> > ajp_service::jk_ajp_common.c (2778): (worker1) sending request to tomcat
> > failed (recoverable), because of error during request sending (attempt=1)
> > [Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
> > jk_open_socket::jk_connect.c (817): connect to 127.0.0.1:8009 failed
> > (errno=111)
> > [Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
> > ajp_connect_to_endpoint::jk_ajp_common.c (1068): (worker1) Failed
> opening
> > socket to (127.0.0.1:8009) (errno=111)
> > [Sat Dec 17 06:11:50 2016][3254:47280261211024] [error]
> > ajp_send_request::jk_ajp_common.c (1728): (worker1) connecting to
> backend
> > failed. Tomcat is probably not started or is listening on the wrong port
> > (errno=111)
> > [Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
> > ajp_service::jk_ajp_common.c (2778): (worker1) sending request to tomcat
> > failed (recoverable), because of error during request sending (attempt=2)
> > [Sat Dec 17 06:11:50 2016][3254:47280261211024] [error]
> > ajp_service::jk_ajp_common.c (2799): (worker1) connecting to tomcat
> failed
> > (rc=-3, errors=1, client_errors=0).
> > [Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
> jk_handler::mod_jk.c
> > (2995): Service error=-3 for worker=worker1
> >
> > Any ideas on what I should check?
> >
> > Cheers
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Upgrade to 8.5.8/9

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Greg,

On 12/19/16 4:47 AM, Greg Huber wrote:
> I am currently running tomcat 8.0.32 and have tried to upgrade to
> 8.5.8 and 8.5.9 without success.  I use mod_jk to connect apache to
> tomcat running on centos 5.11 with Apache 2.2.3.
> 
> I have installed openSSL 1.0.2.j to compile the native
> tomcat-native-1.2.10 using apr-1.5.2 and tomcat-connectors-1.2.42
> for the mod_jk.

If you are using httpd + mod_jk, why are you bothering with APR and
tcnative on the Tomcat side?

> The problem is that tomcat seems to start but fails sometimes to
> connect correctly to apache and causes apache to hang.  If I stop
> tomcat, apache starts working again.  I have been through the logs
> with little indication of the problem.  If I revet back to 8.0.32
> everything works OK.

What if you use the NIO connector instead of APR?

> All I can find is this in the mod_jk.log:
> 
> 
> [Sat Dec 17 06:11:48 2016][3212:47280261211024] [info]
> init_jk::mod_jk.c (3595): mod_jk/1.2.42 initialized [Sat Dec 17
> 06:11:50 2016][3254:47280261211024] [info] 
> jk_open_socket::jk_connect.c (817): connect to 127.0.0.1:8009
> failed (errno=111) [Sat Dec 17 06:11:50 2016][3254:47280261211024]
> [info] ajp_connect_to_endpoint::jk_ajp_common.c (1068): (worker1)
> Failed opening socket to (127.0.0.1:8009) (errno=111) [Sat Dec 17
> 06:11:50 2016][3254:47280261211024] [error] 
> ajp_send_request::jk_ajp_common.c (1728): (worker1) connecting to
> backend failed. Tomcat is probably not started or is listening on
> the wrong port (errno=111) [Sat Dec 17 06:11:50
> 2016][3254:47280261211024] [info] ajp_service::jk_ajp_common.c
> (2778): (worker1) sending request to tomcat failed (recoverable),
> because of error during request sending (attempt=1) [Sat Dec 17
> 06:11:50 2016][3254:47280261211024] [info] 
> jk_open_socket::jk_connect.c (817): connect to 127.0.0.1:8009
> failed (errno=111) [Sat Dec 17 06:11:50 2016][3254:47280261211024]
> [info] ajp_connect_to_endpoint::jk_ajp_common.c (1068): (worker1)
> Failed opening socket to (127.0.0.1:8009) (errno=111) [Sat Dec 17
> 06:11:50 2016][3254:47280261211024] [error] 
> ajp_send_request::jk_ajp_common.c (1728): (worker1) connecting to
> backend failed. Tomcat is probably not started or is listening on
> the wrong port (errno=111) [Sat Dec 17 06:11:50
> 2016][3254:47280261211024] [info] ajp_service::jk_ajp_common.c
> (2778): (worker1) sending request to tomcat failed (recoverable),
> because of error during request sending (attempt=2) [Sat Dec 17
> 06:11:50 2016][3254:47280261211024] [error] 
> ajp_service::jk_ajp_common.c (2799): (worker1) connecting to tomcat
> failed (rc=-3, errors=1, client_errors=0). [Sat Dec 17 06:11:50
> 2016][3254:47280261211024] [info] jk_handler::mod_jk.c (2995):
> Service error=-3 for worker=worker1
> 
> Any ideas on what I should check?

Can you post your  configuration?

Any errors in the log files (specifically logs/catalina.out or, if you
are using jsvc or Windows services, stdout.log/stderr.log/etc.)?

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYWCLGAAoJEBzwKT+lPKRYzqcP/R34lxIW90ZPn7SQ5SBsbKN8
rvYD+4k7a3EGEnPUhfJaajOMbpTXdPTHyltHOID9CpRRFIGrb+/kcKmaLbllYIqq
NUDvWSDUHK0yELaSjb7+Y980mygqXV3j8wAardcEnZ/wG143ZZozhpQtKjhpLmsb
XMe0HiC9/4TeqCtyZ5IWRdq/UOotnU0m1QMjgGYQPztGwQlfvNYDZ//8apiYkCKY
qUoblyNM1RiH4H6tiX+7ZEfk3exvArw84adcWULYiQNpHMwvsRlfMi1aq8Xcty5+
xhB4SitKoEB6CY/oLwRjWYAUsQIov2trGoqbhxIiQcNpHxoKslLabvzdjmjMPMj0
kbMiGpOFfK4ULqyk1255HNxrGL97lnA2nfIz4q23XUU4AYjQkvWHVe94zbUGzdHM
6WZWLn8aZcugC5ltENAp7bCuye2KzpbA4BRbaoGBgiGRQaGSx+0poTubLODSNRFl
cJv+KyoCS+NX4Lz0gK3sIhHr7xRQno5tYmUKNjIn3/v3Rt5zhjfXCvoElQ+MIlEW
iQfkO6AMw+uWlNnd6Iuya9do4bYs/7oN4NiySy6yuz9v96JC6qjZvOtfj4nViNnm
bTRNDreU8IyBR0/vkgKTNTF7KfebJ6ccxRUuFeqqK3rUikBVDW1SsKGskuQK+Htr
I3y6BDq4qz2VjDdPTXL0
=XGBI
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Upgrade to 8.5.8/9

[Coty Sutherland]

Hm, errno=111 is a connection refusal. Are you sure that your new
instance has an adequate number of threads available for httpd to
proxy to? Do you see any errors in your tomcat logging? If you do have
sufficient threads in the pool, then maybe there is something (like GC
pauses) hanging your requests that wasn't before and therefore
exhausting the pool causing rejections.

On Mon, Dec 19, 2016 at 4:47 AM, Greg Huber  wrote:
> Hello,
>
> I am currently running tomcat 8.0.32 and have tried to upgrade to 8.5.8 and
> 8.5.9 without success.  I use mod_jk to connect apache to tomcat running on
> centos 5.11 with Apache 2.2.3.
>
> I have installed openSSL 1.0.2.j to compile the native tomcat-native-1.2.10
> using apr-1.5.2 and tomcat-connectors-1.2.42 for the mod_jk.
>
> The problem is that tomcat seems to start but fails sometimes to connect
> correctly to apache and causes apache to hang.  If I stop tomcat, apache
> starts working again.  I have been through the logs with little indication
> of the problem.  If I revet back to 8.0.32 everything works OK.
>
>
> All I can find is this in the mod_jk.log:
>
>
> [Sat Dec 17 06:11:48 2016][3212:47280261211024] [info] init_jk::mod_jk.c
> (3595): mod_jk/1.2.42 initialized
> [Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
> jk_open_socket::jk_connect.c (817): connect to 127.0.0.1:8009 failed
> (errno=111)
> [Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
> ajp_connect_to_endpoint::jk_ajp_common.c (1068): (worker1) Failed opening
> socket to (127.0.0.1:8009) (errno=111)
> [Sat Dec 17 06:11:50 2016][3254:47280261211024] [error]
> ajp_send_request::jk_ajp_common.c (1728): (worker1) connecting to backend
> failed. Tomcat is probably not started or is listening on the wrong port
> (errno=111)
> [Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
> ajp_service::jk_ajp_common.c (2778): (worker1) sending request to tomcat
> failed (recoverable), because of error during request sending (attempt=1)
> [Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
> jk_open_socket::jk_connect.c (817): connect to 127.0.0.1:8009 failed
> (errno=111)
> [Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
> ajp_connect_to_endpoint::jk_ajp_common.c (1068): (worker1) Failed opening
> socket to (127.0.0.1:8009) (errno=111)
> [Sat Dec 17 06:11:50 2016][3254:47280261211024] [error]
> ajp_send_request::jk_ajp_common.c (1728): (worker1) connecting to backend
> failed. Tomcat is probably not started or is listening on the wrong port
> (errno=111)
> [Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
> ajp_service::jk_ajp_common.c (2778): (worker1) sending request to tomcat
> failed (recoverable), because of error during request sending (attempt=2)
> [Sat Dec 17 06:11:50 2016][3254:47280261211024] [error]
> ajp_service::jk_ajp_common.c (2799): (worker1) connecting to tomcat failed
> (rc=-3, errors=1, client_errors=0).
> [Sat Dec 17 06:11:50 2016][3254:47280261211024] [info] jk_handler::mod_jk.c
> (2995): Service error=-3 for worker=worker1
>
> Any ideas on what I should check?
>
> Cheers

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Upgrade to 8.5.8/9

[Greg Huber]

Hello,

I am currently running tomcat 8.0.32 and have tried to upgrade to 8.5.8 and
8.5.9 without success.  I use mod_jk to connect apache to tomcat running on
centos 5.11 with Apache 2.2.3.

I have installed openSSL 1.0.2.j to compile the native tomcat-native-1.2.10
using apr-1.5.2 and tomcat-connectors-1.2.42 for the mod_jk.

The problem is that tomcat seems to start but fails sometimes to connect
correctly to apache and causes apache to hang.  If I stop tomcat, apache
starts working again.  I have been through the logs with little indication
of the problem.  If I revet back to 8.0.32 everything works OK.


All I can find is this in the mod_jk.log:


[Sat Dec 17 06:11:48 2016][3212:47280261211024] [info] init_jk::mod_jk.c
(3595): mod_jk/1.2.42 initialized
[Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
jk_open_socket::jk_connect.c (817): connect to 127.0.0.1:8009 failed
(errno=111)
[Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
ajp_connect_to_endpoint::jk_ajp_common.c (1068): (worker1) Failed opening
socket to (127.0.0.1:8009) (errno=111)
[Sat Dec 17 06:11:50 2016][3254:47280261211024] [error]
ajp_send_request::jk_ajp_common.c (1728): (worker1) connecting to backend
failed. Tomcat is probably not started or is listening on the wrong port
(errno=111)
[Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
ajp_service::jk_ajp_common.c (2778): (worker1) sending request to tomcat
failed (recoverable), because of error during request sending (attempt=1)
[Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
jk_open_socket::jk_connect.c (817): connect to 127.0.0.1:8009 failed
(errno=111)
[Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
ajp_connect_to_endpoint::jk_ajp_common.c (1068): (worker1) Failed opening
socket to (127.0.0.1:8009) (errno=111)
[Sat Dec 17 06:11:50 2016][3254:47280261211024] [error]
ajp_send_request::jk_ajp_common.c (1728): (worker1) connecting to backend
failed. Tomcat is probably not started or is listening on the wrong port
(errno=111)
[Sat Dec 17 06:11:50 2016][3254:47280261211024] [info]
ajp_service::jk_ajp_common.c (2778): (worker1) sending request to tomcat
failed (recoverable), because of error during request sending (attempt=2)
[Sat Dec 17 06:11:50 2016][3254:47280261211024] [error]
ajp_service::jk_ajp_common.c (2799): (worker1) connecting to tomcat failed
(rc=-3, errors=1, client_errors=0).
[Sat Dec 17 06:11:50 2016][3254:47280261211024] [info] jk_handler::mod_jk.c
(2995): Service error=-3 for worker=worker1

Any ideas on what I should check?

Cheers