[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 --- Comment #19 from Matt M --- This is looking like a case where the remote endpoint is not able to respond with its own close message. We have seen 2 types of exception recorded in the Nio2SocketWrapper present in the heap: AsynchronousCloseException IOException - connection reset by peer -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 --- Comment #18 from Matt M --- After a recent upgrade from 9.0.63 to 9.0.75 (with the 2 websocket fixes from 9.0.77 patched in) I am seeing some occurrences of the WsSession Object stuck on the heap with state OUTPUT_CLOSED in WsHttpUpgradeHandler under Http11Nio2Protocol.waitingProcessors. We are in the process of identifying the steps to reproduce but these occurrences have been rare so far. Looking at the more recent reworking of WsSession isOpen / isClosed and surrounding logic from BZ66574 I am wondering if it makes sense for OUTPUT_CLOSED to also be checked in the isClosed call? Without a reproducible test case in place it is difficult to determine via static code analysis, but I wanted to mention that we are seeing this after our recent upgrade and did not see it in 9.0.63. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 --- Comment #17 from zhougang --- (In reply to Christopher Schultz from comment #16) > (In reply to Vishwanath Karra from comment #15) > > We are seeing this issue with 8.5.72 - is there some other patch needed for > > the 8.5 codeline ? The fix is confirmed for 9.0.40. > > Are you sure it's the same issue? Please post all your relevant information > (i.e. actual evidence that you have a Websocket session memory leak) to the > user's mailing list to verify this is the same problem. We can see the problem on 8.5.84, the problem is when we configure server.xml connector of the protocol is org. Apache. Coyote. Http11. Http11Nio2Protocol. This is not an issue when changing to the default configuration (Nio). In tests, when Nio mode was set, the problem was fixed after 8.5.60, but when Nio2 mode was set, the problem existed on versions 8.5.60, 8.5.75, 8.5.84, etc. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 --- Comment #16 from Christopher Schultz --- (In reply to Vishwanath Karra from comment #15) > We are seeing this issue with 8.5.72 - is there some other patch needed for > the 8.5 codeline ? The fix is confirmed for 9.0.40. Are you sure it's the same issue? Please post all your relevant information (i.e. actual evidence that you have a Websocket session memory leak) to the user's mailing list to verify this is the same problem. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 --- Comment #15 from Vishwanath Karra --- We are seeing this issue with 8.5.72 - is there some other patch needed for the 8.5 codeline ? The fix is confirmed for 9.0.40. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 --- Comment #14 from Christopher Schultz --- Tomcat 9.0.40 started on 12 November[1]. Feel free to test and vote! [1] https://lists.apache.org/thread.html/rce32f696e12fecfd4b4383a5e8a389f4dac7d357053fd03ac7280868%40%3Cdev.tomcat.apache.org%3E -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 --- Comment #13 from Laszlo Karolyi --- Hi Thomas, According to the last information I got the Tomcat 9.0.40 release was estimated to be ready this week but I haven't seen any update on this. Since our internal release is still in progress this week, I think we may have the chance to integrate the new Tomcat early next week in case it'll be ready by then. Would it be feasible to have the official Tomcat 9.0.40 build in the beginning of next week? -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 --- Comment #12 from Laszlo Karolyi --- Ok, it sounds promising. Thanks for the quick feedback. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 --- Comment #11 from Mark Thomas --- Soon. Need to resolve the remaining open issues then we can tag. Hopefully early next week. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 --- Comment #10 from Laszlo Karolyi --- I couldn't find any information about the publication date of 9.0.40 but it would help us if we knew when it is expected as we would include it in our next release. Please let me know if you happen to have any update on the next Tomcat version's release date. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 --- Comment #9 from Laszlo Karolyi --- I can confirm that the issue I reported is indeed fixed in Tomcat 9.0.40 so thanks for the quick remediation. Do you happen to have an estimation as to when the 9.0.40 version of Tomcat is expected to be released? -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 Mark Thomas changed: What|Removed |Added Resolution|--- |FIXED Status|REOPENED|RESOLVED --- Comment #8 from Mark Thomas --- Fixed in: - 10.0.x for 10.0.0-M10 onwards - 9.0.x for 9.0.40 onwards - 8.5.x for 8.5.60 onwards -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 Mark Thomas changed: What|Removed |Added Status|RESOLVED|REOPENED Resolution|FIXED |--- --- Comment #7 from Mark Thomas --- The issue as reproduced by the provided test case is fixed in 9.0.39. Entries are removed from the waitingProcessor list when an event is processed for the socket (and returned once processing is complete if the connection is still open). If the writes from the server are made from an application thread (rather then in response to a message from the client and on the same thread) then it looks like it is possible that an unclean close will result in problem described here. Re-opening while I investigate a potential unit test for this. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 --- Comment #6 from Laszlo Karolyi --- After re-testing the reported scenario with 9.0.39 Tomcat version (core + websocket) the WsSession objects still seem to get stuck on the heap. Upon investigating the Tomcat source code a bit deeper it's not entirely clear how a particular Processor can be removed from the waitingProcessors set located in Http11NioProtocol class (more accurately in its heir, AbstractProtocol). According to the heap dump snapshot enclosed with the original report, a WsSession object is held by a WsHttpUpgradeHandler object that is used within UpgradeProcessorInternal class. UpgradeProcessorInternal inherits from Processor, the type whose objects seem stored in waitingProcessors. If sending a message block fails in WsRemoteEndpointImplBase during the invocation of writeMessagePart() method (as the referred https://github.com/apache/tomcat/commit/5707895bafa38c1443460356a268932a967b79e9 pull request also contains) then even though the WebSocket session gets closed followed by re-throwing of the caught exception, does it mean that the WsHttpUpgradeHandler object associated to the closed WebSocket session should eventually be purged from the above mentioned waitingProcessors set too? In other words, under which conditions should a Processor be removed from waitingProcessors? -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 Mark Thomas changed: What|Removed |Added Resolution|--- |FIXED Status|NEEDINFO|RESOLVED --- Comment #5 from Mark Thomas --- This bug has already been fixed in the 9.0.38 onwards. https://github.com/apache/tomcat/commit/5707895bafa38c1443460356a268932a967b79e9 -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 --- Comment #4 from Mark Thomas --- I'm able to recreate this with the 3 provided Maven projects. I ran the client on a local VM with VMware Workstation configured to limit the network connection to 10Mbps with 20ms latency inbound and outbound. Next step is to update to the latest Tomcat 9 code and see if the issue persists. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 --- Comment #3 from Mark Thomas --- Actually, scratch that last comment. Tomcat detects that and will throw an Exception. I'm still looking into this but there are changes between 9.0.36 and 9./0.39 that might help so that feedback would still be very helpful. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 --- Comment #2 from Mark Thomas --- The other thing to check is whether you have multiple threads writing to the same WebSocket session concurrently. That is not supported by the Java WebSocket API and could lead to the issue you describe. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 64848] WsSession objects in OUTPUT_CLOSED state are implicitly held by waitingProcessors and GC cannot purge them from the JVM heap
https://bz.apache.org/bugzilla/show_bug.cgi?id=64848 Mark Thomas changed: What|Removed |Added OS||All Status|NEW |NEEDINFO --- Comment #1 from Mark Thomas --- Please re-test with 9.0.39. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
