Re: seeing RemoteEndpoint.Basic BINARY_FULL_WRITING errors under load in Tomcat 7.0.48-dev (trunk)
Mark Thomas wrote: On 09/12/2013 22:16, Bob DeRemer wrote: It appears that we are getting the following IOException: java.io.IOException: java.util.concurrent.TimeoutException immediately before we got the BINARY_FULL_WRITING errors. Would this TimeoutException be caused because the underlying socket did not receive an ACK for the sent websocket message, or could something else cause this when trying to write to the websocket? My experience debugging IO issues with WebSocket is that a write is reported to Tomcat as complete once it has been written to the TCP output buffer. Tomcat has no visibility what happens after that point. All you can really be sure of when you see the TimeoutException is that Tomcat has not been able to write the data it was trying to write to the TCP output buffer within the time it was given to complete the write. Possible causes are: - client is reading data more slowly than Tomcat is writing it - client has silently dropped the connection - something else between Tomcat and the client has silently dropped the connection (gateway, firewall,..) - the machine where Tomcat is running is is overloaded it can't write the data fast enough My money would be on the first two options rather than the third. HTH, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: seeing RemoteEndpoint.Basic BINARY_FULL_WRITING errors under load in Tomcat 7.0.48-dev (trunk)
On 09/12/2013 22:16, Bob DeRemer wrote: > It appears that we are getting the following IOException: > > java.io.IOException: java.util.concurrent.TimeoutException > immediately before we got the BINARY_FULL_WRITING errors. > > Would this TimeoutException be caused because the underlying socket > did not receive an ACK for the sent websocket message, or could > something else cause this when trying to write to the websocket? My experience debugging IO issues with WebSocket is that a write is reported to Tomcat as complete once it has been written to the TCP output buffer. Tomcat has no visibility what happens after that point. All you can really be sure of when you see the TimeoutException is that Tomcat has not been able to write the data it was trying to write to the TCP output buffer within the time it was given to complete the write. Possible causes are: - client is reading data more slowly than Tomcat is writing it - client has silently dropped the connection - the machine where Tomcat is running is is overloaded it can't write the data fast enough My money would be on the first two options rather than the third. HTH, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: seeing RemoteEndpoint.Basic BINARY_FULL_WRITING errors under load in Tomcat 7.0.48-dev (trunk)
> -Original Message- > From: Mark Thomas [mailto:ma...@apache.org] > Sent: Wednesday, November 27, 2013 12:59 PM > To: Tomcat Users List > Subject: Re: seeing RemoteEndpoint.Basic BINARY_FULL_WRITING errors under > load in Tomcat 7.0.48-dev (trunk) > > On 27/11/2013 17:28, Bob DeRemer wrote: > > When testing our WebSocket application under heavy load we eventually get > the following exception from the JSR-356 ServerEndpoint's > RemoteEndpoint.Basic when trying to write our binary response message: > > > > java.lang.IllegalStateException: The remote endpoint was in state > [BINARY_FULL_WRITING] which is an invalid state for called method > >at > org.apache.tomcat.websocket.WsRemoteEndpointImplBase$StateMachine.chec > kState(WsRemoteEndpointImplBase.java:1048) ~[tomcat7- > websocket.jar:7.0.48-dev] > >at > org.apache.tomcat.websocket.WsRemoteEndpointImplBase$StateMachine.bina > ryStart(WsRemoteEndpointImplBase.java:1001) ~[tomcat7- > websocket.jar:7.0.48-dev] > >at > org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendBytes(WsRemo > teEndpointImplBase.java:118) ~[tomcat7-websocket.jar:7.0.48-dev] > >at > org.apache.tomcat.websocket.WsRemoteEndpointBasic.sendBinary(WsRemote > EndpointBasic.java:43) ~[tomcat7-websocket.jar:7.0.48-dev] > >at > com.thingworx.communications.server.connection.Jsr356ServerConnection.sen > dBinaryMessage(Jsr356ServerConnection.java:240) ~[thingworx-server- > common.jar:na] > >at > com.thingworx.communications.common.endpoints.CommunicationEndpoint.se > ndBinaryMessage(CommunicationEndpoint.java:321) ~[thingworx-server- > common.jar:na] > >at > com.thingworx.communications.common.endpoints.CommunicationEndpoint.se > ndMessage(CommunicationEndpoint.java:303) ~[thingworx-server- > common.jar:na] > >at > com.thingworx.apiserver.endpoints.RoutingServerEndpoint.sendMessage(Routi > ngServerEndpoint.java:24) ~[thingworx-apiserver.jar:na] > >at > com.thingworx.apiserver.routing.OutboundMessageRoutingTask.run(Outbound > MessageRoutingTask.java:79) ~[thingworx-apiserver.jar:na] > >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1 > 145) [na:1.7.0_45] > >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: > 615) [na:1.7.0_45] > >at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > > > > The Tomcat build is 7.0.48-dev (from today, 11/27) and we're running on > Ubuntu Server 13.10 in EC2 (16 vCPUs). The client created 1000 concurrent > websockets, and is using 100 threads to send 100 concurrent requests every 3 > ms. The server receives the messages, processes them, then sends the > response back to the appropriate client. Many response are sent successfully, > but after some period of time, we start receiving the above errors when > attempting to send the responses back. > > > > Is it possible that we could be filling up the TCP buffers (or something > > related) > and this would cause the above exception? > > What can cause the above exception is if the previous message on that > endpoint failed with an IOException. The nature of WebSocket is such that once > this happens the only option is to close the connection and start a new one. > It appears that we are getting the following IOException: java.io.IOException: java.util.concurrent.TimeoutException immediately before we got the BINARY_FULL_WRITING errors. Would this TimeoutException be caused because the underlying socket did not receive an ACK for the sent websocket message, or could something else cause this when trying to write to the websocket? Thx - bob > > Please note: we lock around a specific ServerEndpoint's > RemoteEndpoint.Basic - to make sure we aren't trying to send more than 1 > message at a time. > > > > Please let me know if this is a bug, or if you need additional information. > > A bug is always possible but at this point it looks like some error handling > is > missing in the app. > > Mark > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: seeing RemoteEndpoint.Basic BINARY_FULL_WRITING errors under load in Tomcat 7.0.48-dev (trunk)
On 27/11/2013 17:28, Bob DeRemer wrote: > When testing our WebSocket application under heavy load we eventually get the > following exception from the JSR-356 ServerEndpoint's RemoteEndpoint.Basic > when trying to write our binary response message: > > java.lang.IllegalStateException: The remote endpoint was in state > [BINARY_FULL_WRITING] which is an invalid state for called method >at > org.apache.tomcat.websocket.WsRemoteEndpointImplBase$StateMachine.checkState(WsRemoteEndpointImplBase.java:1048) > ~[tomcat7-websocket.jar:7.0.48-dev] >at > org.apache.tomcat.websocket.WsRemoteEndpointImplBase$StateMachine.binaryStart(WsRemoteEndpointImplBase.java:1001) > ~[tomcat7-websocket.jar:7.0.48-dev] >at > org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendBytes(WsRemoteEndpointImplBase.java:118) > ~[tomcat7-websocket.jar:7.0.48-dev] >at > org.apache.tomcat.websocket.WsRemoteEndpointBasic.sendBinary(WsRemoteEndpointBasic.java:43) > ~[tomcat7-websocket.jar:7.0.48-dev] >at > com.thingworx.communications.server.connection.Jsr356ServerConnection.sendBinaryMessage(Jsr356ServerConnection.java:240) > ~[thingworx-server-common.jar:na] >at > com.thingworx.communications.common.endpoints.CommunicationEndpoint.sendBinaryMessage(CommunicationEndpoint.java:321) > ~[thingworx-server-common.jar:na] >at > com.thingworx.communications.common.endpoints.CommunicationEndpoint.sendMessage(CommunicationEndpoint.java:303) > ~[thingworx-server-common.jar:na] >at > com.thingworx.apiserver.endpoints.RoutingServerEndpoint.sendMessage(RoutingServerEndpoint.java:24) > ~[thingworx-apiserver.jar:na] >at > com.thingworx.apiserver.routing.OutboundMessageRoutingTask.run(OutboundMessageRoutingTask.java:79) > ~[thingworx-apiserver.jar:na] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_45] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_45] >at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > > The Tomcat build is 7.0.48-dev (from today, 11/27) and we're running on > Ubuntu Server 13.10 in EC2 (16 vCPUs). The client created 1000 concurrent > websockets, and is using 100 threads to send 100 concurrent requests every 3 > ms. The server receives the messages, processes them, then sends the > response back to the appropriate client. Many response are sent > successfully, but after some period of time, we start receiving the above > errors when attempting to send the responses back. > > Is it possible that we could be filling up the TCP buffers (or something > related) and this would cause the above exception? What can cause the above exception is if the previous message on that endpoint failed with an IOException. The nature of WebSocket is such that once this happens the only option is to close the connection and start a new one. > Please note: we lock around a specific ServerEndpoint's RemoteEndpoint.Basic > - to make sure we aren't trying to send more than 1 message at a time. > > Please let me know if this is a bug, or if you need additional information. A bug is always possible but at this point it looks like some error handling is missing in the app. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
