Update:
Started using AWS Application Load Balancer, with SSL offload. This has
reduced CPU utilization by 5 times!
And most importantly, wrt this mail chain, num of connections looks stable
now, with same load. Can see some Client TLS Negotiations Errors in LB's
monitoring console. Num of errors for some period of time are more or less
matching with num of connections increased previously.
Sandeep
On Fri, Jun 23, 2017 at 2:34 PM, Sandeep Dhameshia <
sandeep.dhames...@gmail.com> wrote:
> One more doubt, does it log anything when max connections limit is reached
> and it starts dropping requests? This happened few days back, and since it
> is a production server I did not have time to see num of file descriptors
> for tomcat process and response code to client. Nothing was logged,
> restarted tomcat and it started accepting client requests again.
>
> Have created a cron job after this incident, which restarts the server
> when it is reaching the limit.
>
> regards
>
> On Wed, Jun 21, 2017 at 9:41 AM, Sandeep Dhameshia <
> sandeep.dhames...@gmail.com> wrote:
>
>> Thanks for your reply Mark,
>>
>> *log msg*:
>>
>> Jun 08, 2017 10:13:07 AM org.apache.tomcat.websocket.se
>> rver.WsRemoteEndpointImplServer doClose
>> INFO: Failed to close the ServletOutputStream connection cleanly
>> java.io.IOException: Broken pipe
>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>> at sun.nio.ch.IOUtil.write(IOUtil.java:51)
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:492)
>> at org.apache.tomcat.util.net.SecureNioChannel.flush(SecureNioC
>> hannel.java:141)
>> at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioC
>> hannel.java:385)
>> at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioC
>> hannel.java:413)
>> at org.apache.coyote.http11.upgrade.NioServletOutputStream.doCl
>> ose(NioServletOutputStream.java:138)
>> at org.apache.coyote.http11.upgrade.AbstractServletOutputStream
>> .close(AbstractServletOutputStream.java:129)
>> at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServe
>> r.doClose(WsRemoteEndpointImplServer.java:138)
>> at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.close(W
>> sRemoteEndpointImplBase.java:696)
>> at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServe
>> r.onWritePossible(WsRemoteEndpointImplServer.java:113)
>> at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServe
>> r.doWrite(WsRemoteEndpointImplServer.java:81)
>> at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMe
>> ssagePart(WsRemoteEndpointImplBase.java:456)
>> at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMe
>> ssage(WsRemoteEndpointImplBase.java:344)
>> at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMe
>> ssageBlock(WsRemoteEndpointImplBase.java:276)
>> at org.apache.tomcat.websocket.WsSession.sendCloseMessage(WsSes
>> sion.java:559)
>> at org.apache.tomcat.websocket.WsSession.doClose(WsSession.java:465)
>> at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.onEr
>> ror(WsHttpUpgradeHandler.java:162)
>> at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.acce
>> ss$300(WsHttpUpgradeHandler.java:48)
>> at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler$WsRe
>> adListener.onError(WsHttpUpgradeHandler.java:230)
>> at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler$WsRe
>> adListener.onDataAvailable(WsHttpUpgradeHandler.java:213)
>> at org.apache.coyote.http11.upgrade.AbstractServletInputStream.
>> onDataAvailable(AbstractServletInputStream.java:203)
>> at org.apache.coyote.http11.upgrade.AbstractProcessor.upgradeDi
>> spatch(AbstractProcessor.java:93)
>> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler
>> .process(AbstractProtocol.java:623)
>> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
>> (NioEndpoint.java:1749)
>> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(
>> NioEndpoint.java:1708)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:615)
>> at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.
>> run(TaskThread.java:61)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> *Connector*:
>>
>> <Connector
>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>> port="8443" maxThreads="200"
>> scheme="https" secure="true" SSLEnabled="true"
>> keystoreFile="${catalina.base}/conf/.keystore"
>> keystorePass="dummy"
>> clientAuth="false" sslProtocol="TLS"
>> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA
>> _WITH_AES_128_CBC_SHA,
>> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES
>> _256_CBC_SHA,TLS_ECDHE_RSA_WITH_RC4_128_SHA,
>> TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_
>> SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,
>> TLS_RSA_WITH_AES_256_CBC_SHA,SSL_RSA_WITH_RC4_128_SHA"
>> />
>>
>> regards
>>
>> On Wed, Jun 21, 2017 at 12:01 AM, Mark Thomas <ma...@apache.org> wrote:
>>
>>> On 20/06/2017 18:48, Sandeep Dhameshia wrote:
>>> > Hello,
>>> >
>>> > I don't know about any format in which I should write this mail, so
>>> please
>>> > forgive me if there's any format to follow.
>>> >
>>> > I am using v8.0.43 in latest Amazon Linux AMI(2017.03), on m4.large
>>> > instance. I have deployed modified example Chat application.
>>> >
>>> > Everything works fine as expected, but I feel websocket connections
>>> are not
>>> > getting closed in some instances. I could see INFO logs from
>>> > org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer, wrt
>>> > connection not getting closed cleanly.
>>>
>>> Example log messages please.
>>>
>>> Connector configuration please.
>>>
>>> Mark
>>>
>>>
>>> >
>>> > I've increased limit for no of files to 65536, both hard and soft
>>> limits.
>>> > Num of connections for NIO connector is set to default.
>>> >
>>> > Num of files(sockets) opened are increasing, can see it with "ls
>>> > /proc/PID/fd | wc -l".
>>> >
>>> > Am I missing any config? I understand some clients are not closing
>>> > connection properly, but is there any way to handle this properly on
>>> server
>>> > side?
>>> >
>>> > best regards.
>>> >
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>>
>>
>
