Beware of the AWS ALB, It has a nasty bug where it terminates a chunk of
connections even if they are active. This happens after 10-12 hrs. and
related to the autoscaling of ALB.
On Thu, Jun 29, 2017 at 5:40 PM, Sandeep Dhameshia <
sandeep.dhames...@gmail.com> wrote:
> Update:
>
> Started using AWS Application Load Balancer, with SSL offload. This has
> reduced CPU utilization by 5 times!
>
> And most importantly, wrt this mail chain, num of connections looks stable
> now, with same load. Can see some Client TLS Negotiations Errors in LB's
> monitoring console. Num of errors for some period of time are more or less
> matching with num of connections increased previously.
>
> Sandeep
>
> On Fri, Jun 23, 2017 at 2:34 PM, Sandeep Dhameshia <
> sandeep.dhames...@gmail.com> wrote:
>
> > One more doubt, does it log anything when max connections limit is
> reached
> > and it starts dropping requests? This happened few days back, and since
> it
> > is a production server I did not have time to see num of file descriptors
> > for tomcat process and response code to client. Nothing was logged,
> > restarted tomcat and it started accepting client requests again.
> >
> > Have created a cron job after this incident, which restarts the server
> > when it is reaching the limit.
> >
> > regards
> >
> > On Wed, Jun 21, 2017 at 9:41 AM, Sandeep Dhameshia <
> > sandeep.dhames...@gmail.com> wrote:
> >
> >> Thanks for your reply Mark,
> >>
> >> *log msg*:
> >>
> >> Jun 08, 2017 10:13:07 AM org.apache.tomcat.websocket.se
> >> rver.WsRemoteEndpointImplServer doClose
> >> INFO: Failed to close the ServletOutputStream connection cleanly
> >> java.io.IOException: Broken pipe
> >> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> >> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> >> at sun.nio.ch.IOUtil.write(IOUtil.java:51)
> >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:492)
> >> at org.apache.tomcat.util.net.SecureNioChannel.flush(SecureNioC
> >> hannel.java:141)
> >> at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioC
> >> hannel.java:385)
> >> at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioC
> >> hannel.java:413)
> >> at org.apache.coyote.http11.upgrade.NioServletOutputStream.doCl
> >> ose(NioServletOutputStream.java:138)
> >> at org.apache.coyote.http11.upgrade.AbstractServletOutputStream
> >> .close(AbstractServletOutputStream.java:129)
> >> at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServe
> >> r.doClose(WsRemoteEndpointImplServer.java:138)
> >> at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.close(W
> >> sRemoteEndpointImplBase.java:696)
> >> at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServe
> >> r.onWritePossible(WsRemoteEndpointImplServer.java:113)
> >> at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServe
> >> r.doWrite(WsRemoteEndpointImplServer.java:81)
> >> at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMe
> >> ssagePart(WsRemoteEndpointImplBase.java:456)
> >> at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMe
> >> ssage(WsRemoteEndpointImplBase.java:344)
> >> at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMe
> >> ssageBlock(WsRemoteEndpointImplBase.java:276)
> >> at org.apache.tomcat.websocket.WsSession.sendCloseMessage(WsSes
> >> sion.java:559)
> >> at org.apache.tomcat.websocket.WsSession.doClose(WsSession.java:465)
> >> at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.onEr
> >> ror(WsHttpUpgradeHandler.java:162)
> >> at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.acce
> >> ss$300(WsHttpUpgradeHandler.java:48)
> >> at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler$WsRe
> >> adListener.onError(WsHttpUpgradeHandler.java:230)
> >> at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler$WsRe
> >> adListener.onDataAvailable(WsHttpUpgradeHandler.java:213)
> >> at org.apache.coyote.http11.upgrade.AbstractServletInputStream.
> >> onDataAvailable(AbstractServletInputStream.java:203)
> >> at org.apache.coyote.http11.upgrade.AbstractProcessor.upgradeDi
> >> spatch(AbstractProcessor.java:93)
> >> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler
> >> .process(AbstractProtocol.java:623)
> >> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
> >> (NioEndpoint.java:1749)
> >> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(
> >> NioEndpoint.java:1708)
> >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> >> Executor.java:1145)
> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> >> lExecutor.java:615)
> >> at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.
> >> run(TaskThread.java:61)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >> *Connector*:
> >>
> >> <Connector
> >> protocol="org.apache.coyote.http11.Http11NioProtocol"
> >> port="8443" maxThreads="200"
> >> scheme="https" secure="true" SSLEnabled="true"
> >> keystoreFile="${catalina.base}/conf/.keystore"
> >> keystorePass="dummy"
> >> clientAuth="false" sslProtocol="TLS"
> >> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA
> >> _WITH_AES_128_CBC_SHA,
> >> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES
> >> _256_CBC_SHA,TLS_ECDHE_RSA_WITH_RC4_128_SHA,
> >> TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_
> >> SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,
> >> TLS_RSA_WITH_AES_256_CBC_SHA,SSL_RSA_WITH_RC4_128_SHA"
> >> />
> >>
> >> regards
> >>
> >> On Wed, Jun 21, 2017 at 12:01 AM, Mark Thomas <ma...@apache.org> wrote:
> >>
> >>> On 20/06/2017 18:48, Sandeep Dhameshia wrote:
> >>> > Hello,
> >>> >
> >>> > I don't know about any format in which I should write this mail, so
> >>> please
> >>> > forgive me if there's any format to follow.
> >>> >
> >>> > I am using v8.0.43 in latest Amazon Linux AMI(2017.03), on m4.large
> >>> > instance. I have deployed modified example Chat application.
> >>> >
> >>> > Everything works fine as expected, but I feel websocket connections
> >>> are not
> >>> > getting closed in some instances. I could see INFO logs from
> >>> > org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer, wrt
> >>> > connection not getting closed cleanly.
> >>>
> >>> Example log messages please.
> >>>
> >>> Connector configuration please.
> >>>
> >>> Mark
> >>>
> >>>
> >>> >
> >>> > I've increased limit for no of files to 65536, both hard and soft
> >>> limits.
> >>> > Num of connections for NIO connector is set to default.
> >>> >
> >>> > Num of files(sockets) opened are increasing, can see it with "ls
> >>> > /proc/PID/fd | wc -l".
> >>> >
> >>> > Am I missing any config? I understand some clients are not closing
> >>> > connection properly, but is there any way to handle this properly on
> >>> server
> >>> > side?
> >>> >
> >>> > best regards.
> >>> >
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>>
> >>>
> >>
> >
>
--
Regards
Anurag
