https://bz.apache.org/bugzilla/show_bug.cgi?id=66508
Venkata Manda <mvssrmur...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|FIXED |---
Status|RESOLVED |REOPENED
--- Comment #8 from Venkata Manda <mvssrmur...@gmail.com> ---
I tested with the changes made and the issue is still happening.
Event 3 thread that is trying to write on a socket that was asked to close by
the client as the server didn't respond (Event 1) is failing due to a broken
pipe (IO Exception). As it is using nonblocking write, the exception is not
happening on the Event 3 thread, happening on the NIO2 thread (example thread
name: http-nio2-8082-exec-2).
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at
sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:694)
at
sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:383)
at
sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:400)
at org.apache.tomcat.util.net.Nio2Channel.write(Nio2Channel.java:183)
at
org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper.flushNonBlockingInternal(Nio2Endpoint.java:1285)
at
org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper.writeNonBlockingInternal(Nio2Endpoint.java:1187)
at
org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper.writeNonBlocking(Nio2Endpoint.java:1150)
at
org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:486)
at
org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.onWritePossible(WsRemoteEndpointImplServer.java:148)
at
org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doWrite(WsRemoteEndpointImplServer.java:83)
at
org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:556)
at
org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessage(WsRemoteEndpointImplBase.java:422)
at
org.apache.tomcat.websocket.WsRemoteEndpointImplBase$TextMessageSendHandler.write(WsRemoteEndpointImplBase.java:892)
at
org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendStringByCompletion(WsRemoteEndpointImplBase.java:216)
at
org.apache.tomcat.websocket.WsRemoteEndpointAsync.sendText(WsRemoteEndpointAsync.java:47)
at
org.cometd.websocket.server.WebSocketTransport.send(WebSocketTransport.java:118)
at
org.cometd.websocket.server.WebSocketTransport.send(WebSocketTransport.java:52)
at
org.cometd.websocket.server.common.AbstractWebSocketTransport$AbstractWebSocketScheduler.send(AbstractWebSocketTransport.java:196)
at
org.cometd.websocket.server.common.AbstractWebSocketTransport$AbstractWebSocketScheduler$Flusher.process(AbstractWebSocketTransport.java:627)
at
org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at
org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
at
org.cometd.websocket.server.common.AbstractWebSocketTransport$AbstractWebSocketScheduler.send(AbstractWebSocketTransport.java:420)
at
org.cometd.websocket.server.common.AbstractWebSocketTransport$AbstractWebSocketScheduler.send(AbstractWebSocketTransport.java:522)
at
org.cometd.websocket.server.common.AbstractWebSocketTransport$AbstractWebSocketScheduler.schedule(AbstractWebSocketTransport.java:500)
at
org.cometd.websocket.server.WebSocketTransport$WebSocketScheduler$1.schedule(WebSocketTransport.java:154)
at
org.cometd.websocket.server.common.AbstractWebSocketTransport$AbstractWebSocketScheduler$MetaConnectReplyTask.run(AbstractWebSocketTransport.java:548)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
When this exception is thrown, Nio2Endpoint.writeCompletionHanlder.onFailed
method is called
(https://github.com/apache/tomcat/blob/8.5.x/java/org/apache/tomcat/util/net/Nio2Endpoint.java#L707).
It is releasing writePending lock and is trying to send SocketEvent.ERROR.
endpoint.processSocket(Nio2SocketWrapper.this, SocketEvent.ERROR, true);
By sending the above event, I am guessing it is expecting to release the
messagePartInProgress semaphore waiting to acquire by the Event 1 thread (the
thread trying to consume the close message sent by the client).
https://github.com/apache/tomcat/blob/8.5.x/java/org/apache/tomcat/websocket/WsRemoteEndpointImplBase.java#L292
In my observation below, the call is returning true, so it is not calling the
close method.
https://github.com/apache/tomcat/blob/main/java/org/apache/tomcat/util/net/Nio2Endpoint.java#L702
The above call spins off another thread to fire a socket event.
https://github.com/apache/tomcat/blob/8.5.x/java/org/apache/tomcat/util/net/SocketProcessorBase.java#L46
In my observation, after 20 seconds (the same time the Event 1 thread waiting
to acquire the semaphore) the thread is entering inside SocketProcessorBase.run
method's synchronized block, and the socketWrapper.isClosed() is true and is
returning, without running doRun method.
Due to asynchronous write by the Event 3 thread, even if it gets an IO
Exception (Broken Pipe) it is not releasing the messagePartInProgress semaphore
because the thread that acquired the lock and the thread that is executing the
write is different. Even when the writer encounters a broken pipe it is not
closing socketWrapper.
In my observation, the Event 3 thread is hitting the below line, as the write
is not complete it does not call any handlers to release messagePartInProgress
semaphore.
https://github.com/apache/tomcat/blob/8.5.x/java/org/apache/tomcat/websocket/server/WsRemoteEndpointImplServer.java#L204
Please let me know if any details are needed.
Thank you
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
