Re: Possible memory leak with Tomcat?
Hi Martin, Yes looks like the same issue at least one of them - perhaps the second is a knock on effect. When will 7.14.0 be released? On Tue, May 21, 2019 at 8:44 AM Martin Grigorov wrote: > Hi Wayne, > > I think you are facing https://issues.apache.org/jira/browse/WICKET-6639 > It is fixed in 7.14.0 and 8.4.0. > > On Mon, May 20, 2019 at 6:48 PM Wayne W > wrote: > > > Hi Sven, > > > > I'm having trouble replicating locally. We just saw it in production and > > then had to roll back as it was effecting customers. The log is full of > > these: > > > > May 16, 2019 6:39:39 AM org.apache.catalina.session.StandardSession > > setAttribute > > SEVERE: Session binding event listener threw exception > > java.lang.NullPointerException > > at > > > > > org.apache.wicket.page.PageStoreManager$SessionEntry.clear(PageStoreManager.java:351) > > at > > > > > org.apache.wicket.page.PageStoreManager$SessionEntry.valueUnbound(PageStoreManager.java:340) > > at > > > > > org.apache.catalina.session.StandardSession.setAttribute(StandardSession.java:1496) > > at > > > > > org.redisson.tomcat.RedissonSession.superSetAttribute(RedissonSession.java:257) > > at > > > > > org.redisson.tomcat.RedissonSessionManager$2.onMessage(RedissonSessionManager.java:284) > > at > > > > > org.redisson.tomcat.RedissonSessionManager$2.onMessage(RedissonSessionManager.java:251) > > at > > > org.redisson.PubSubMessageListener.onMessage(PubSubMessageListener.java:79) > > at > > > > > org.redisson.client.RedisPubSubConnection.onMessage(RedisPubSubConnection.java:78) > > at > > > > > org.redisson.client.handler.CommandPubSubDecoder.lambda$enqueueMessage$0(CommandPubSubDecoder.java:184) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at > > > > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > > at java.lang.Thread.run(Thread.java:745) > > > > > > Then about 2 hours later a huge number of these started to appear in the > > logs: > > > > 2019-05-16 08:52:47,851 ERROR - Wicket-PageSavingThread > > > > > org.apache.wicket.pageStore.DiskDataStore$SessionEntry.getFileChannel(DiskDataStore.java:434) > > 434 DiskDataStore - > > > > > /ec/var/tomcat/gc1/work2/ROOT/wicket.hub-filestore/5168/1931/547BB9E7E255B16D5B2864F42B4F9C28.gc1/data > > (No such file or directory) > > java.io.FileNotFoundException: > > > > > /ec/var/tomcat/gc1/work2/ROOT/wicket.hub-filestore/5168/1931/547BB9E7E255B16D5B2864F42B4F9C28.gc1/data > > (No such file or directory) > > at java.io.RandomAccessFile.open0(Native Method) > > at java.io.RandomAccessFile.open(RandomAccessFile.java:316) > > at java.io.RandomAccessFile.(RandomAccessFile.java:243) > > at > > > > > org.apache.wicket.pageStore.DiskDataStore$SessionEntry.getFileChannel(DiskDataStore.java:428) > > at > > > > > org.apache.wicket.pageStore.DiskDataStore$SessionEntry.savePage(DiskDataStore.java:346) > > at > > > org.apache.wicket.pageStore.DiskDataStore.storeData(DiskDataStore.java:185) > > at > > > > > org.apache.wicket.pageStore.AsynchronousDataStore$PageSavingRunnable.run(AsynchronousDataStore.java:355) > > at java.lang.Thread.run(Thread.java:745) > > > > > > Whats odd is that it effected some user but not others at all. When we > > rollback to 7.8.0 these go away. I'm not sure how I'm going to get the > the > > bottom of this > > > > > > > > > > On Fri, May 17, 2019 at 11:03 PM Sven Meier wrote: > > > > > Hi, > > > > > > WICKET-6564 was about applications explicitly calling #clear() on the > > > PageStoreManager. > > > This doesn't has anything to do with session expiration. > > > > > > Could you please explain what issues you're facing? A quickstart would > > > be perfect, or at least some explanations how we're able to reproduce > > > the problem. > > > > > > Regards > > > Sven > > > > > > > > > Am 17.05.19 um 13:23 schrieb Wayne W: > > > > Hello Sven, > > > > > > > > the fix you did for WICKET-6564 ( > > > > > > > > > > https://git-wip-us.apache.org/repos/asf?p=wicket.git;a=commitdiff;h=7665dc5;hp=3a1602d308a366e00948dbf91dbd96bc40cae167 > > > ) > > > > doesn't work for us. Its causing lots of issues. We use redisson for > > > > managing our sessions in tomcat, so that we can failover without > > loosing > > > > user session. > > > > > > > > When a session attribute is updated redisson calls setAttribute on > the > > > > session. Looking at the code for > > > > org.apache.catalina.session.StandardSession.setAttribute I see: > > > > > > > > // Replace or add this attribute > > > > > > > > Object unbound = attributes.put(name, value); > > > > > > > > > > > > // Call the valueUnbound() method if necessary > > > > > > > > *if* (notify && (unbound != *null*) && (unbound != value) && > > > > > > > > (unbound *instanceof* HttpSessionBindingListener)) { > > > > > > > >
Re: Possible memory leak with Tomcat?
Hi Wayne, I think you are facing https://issues.apache.org/jira/browse/WICKET-6639 It is fixed in 7.14.0 and 8.4.0. On Mon, May 20, 2019 at 6:48 PM Wayne W wrote: > Hi Sven, > > I'm having trouble replicating locally. We just saw it in production and > then had to roll back as it was effecting customers. The log is full of > these: > > May 16, 2019 6:39:39 AM org.apache.catalina.session.StandardSession > setAttribute > SEVERE: Session binding event listener threw exception > java.lang.NullPointerException > at > > org.apache.wicket.page.PageStoreManager$SessionEntry.clear(PageStoreManager.java:351) > at > > org.apache.wicket.page.PageStoreManager$SessionEntry.valueUnbound(PageStoreManager.java:340) > at > > org.apache.catalina.session.StandardSession.setAttribute(StandardSession.java:1496) > at > > org.redisson.tomcat.RedissonSession.superSetAttribute(RedissonSession.java:257) > at > > org.redisson.tomcat.RedissonSessionManager$2.onMessage(RedissonSessionManager.java:284) > at > > org.redisson.tomcat.RedissonSessionManager$2.onMessage(RedissonSessionManager.java:251) > at > org.redisson.PubSubMessageListener.onMessage(PubSubMessageListener.java:79) > at > > org.redisson.client.RedisPubSubConnection.onMessage(RedisPubSubConnection.java:78) > at > > org.redisson.client.handler.CommandPubSubDecoder.lambda$enqueueMessage$0(CommandPubSubDecoder.java:184) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.lang.Thread.run(Thread.java:745) > > > Then about 2 hours later a huge number of these started to appear in the > logs: > > 2019-05-16 08:52:47,851 ERROR - Wicket-PageSavingThread > > org.apache.wicket.pageStore.DiskDataStore$SessionEntry.getFileChannel(DiskDataStore.java:434) > 434 DiskDataStore - > > /ec/var/tomcat/gc1/work2/ROOT/wicket.hub-filestore/5168/1931/547BB9E7E255B16D5B2864F42B4F9C28.gc1/data > (No such file or directory) > java.io.FileNotFoundException: > > /ec/var/tomcat/gc1/work2/ROOT/wicket.hub-filestore/5168/1931/547BB9E7E255B16D5B2864F42B4F9C28.gc1/data > (No such file or directory) > at java.io.RandomAccessFile.open0(Native Method) > at java.io.RandomAccessFile.open(RandomAccessFile.java:316) > at java.io.RandomAccessFile.(RandomAccessFile.java:243) > at > > org.apache.wicket.pageStore.DiskDataStore$SessionEntry.getFileChannel(DiskDataStore.java:428) > at > > org.apache.wicket.pageStore.DiskDataStore$SessionEntry.savePage(DiskDataStore.java:346) > at > org.apache.wicket.pageStore.DiskDataStore.storeData(DiskDataStore.java:185) > at > > org.apache.wicket.pageStore.AsynchronousDataStore$PageSavingRunnable.run(AsynchronousDataStore.java:355) > at java.lang.Thread.run(Thread.java:745) > > > Whats odd is that it effected some user but not others at all. When we > rollback to 7.8.0 these go away. I'm not sure how I'm going to get the the > bottom of this > > > > > On Fri, May 17, 2019 at 11:03 PM Sven Meier wrote: > > > Hi, > > > > WICKET-6564 was about applications explicitly calling #clear() on the > > PageStoreManager. > > This doesn't has anything to do with session expiration. > > > > Could you please explain what issues you're facing? A quickstart would > > be perfect, or at least some explanations how we're able to reproduce > > the problem. > > > > Regards > > Sven > > > > > > Am 17.05.19 um 13:23 schrieb Wayne W: > > > Hello Sven, > > > > > > the fix you did for WICKET-6564 ( > > > > > > https://git-wip-us.apache.org/repos/asf?p=wicket.git;a=commitdiff;h=7665dc5;hp=3a1602d308a366e00948dbf91dbd96bc40cae167 > > ) > > > doesn't work for us. Its causing lots of issues. We use redisson for > > > managing our sessions in tomcat, so that we can failover without > loosing > > > user session. > > > > > > When a session attribute is updated redisson calls setAttribute on the > > > session. Looking at the code for > > > org.apache.catalina.session.StandardSession.setAttribute I see: > > > > > > // Replace or add this attribute > > > > > > Object unbound = attributes.put(name, value); > > > > > > > > > // Call the valueUnbound() method if necessary > > > > > > *if* (notify && (unbound != *null*) && (unbound != value) && > > > > > > (unbound *instanceof* HttpSessionBindingListener)) { > > > > > > *try* { > > > > > > ((HttpSessionBindingListener) unbound).valueUnbound > > > > > > (*new* HttpSessionBindingEvent(getSession(), > name)); > > > > > > } *catch* (Throwable t) { > > > > > > ExceptionUtils.*handleThrowable*(t); > > > > > > manager.getContext().getLogger().error > > > > > > (*sm*.getString("standardSession.bindingEvent"), > t); > > > > > > } > > > > > > } > > > > > > The
Re: Possible memory leak with Tomcat?
Hi Sven, I'm having trouble replicating locally. We just saw it in production and then had to roll back as it was effecting customers. The log is full of these: May 16, 2019 6:39:39 AM org.apache.catalina.session.StandardSession setAttribute SEVERE: Session binding event listener threw exception java.lang.NullPointerException at org.apache.wicket.page.PageStoreManager$SessionEntry.clear(PageStoreManager.java:351) at org.apache.wicket.page.PageStoreManager$SessionEntry.valueUnbound(PageStoreManager.java:340) at org.apache.catalina.session.StandardSession.setAttribute(StandardSession.java:1496) at org.redisson.tomcat.RedissonSession.superSetAttribute(RedissonSession.java:257) at org.redisson.tomcat.RedissonSessionManager$2.onMessage(RedissonSessionManager.java:284) at org.redisson.tomcat.RedissonSessionManager$2.onMessage(RedissonSessionManager.java:251) at org.redisson.PubSubMessageListener.onMessage(PubSubMessageListener.java:79) at org.redisson.client.RedisPubSubConnection.onMessage(RedisPubSubConnection.java:78) at org.redisson.client.handler.CommandPubSubDecoder.lambda$enqueueMessage$0(CommandPubSubDecoder.java:184) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:745) Then about 2 hours later a huge number of these started to appear in the logs: 2019-05-16 08:52:47,851 ERROR - Wicket-PageSavingThread org.apache.wicket.pageStore.DiskDataStore$SessionEntry.getFileChannel(DiskDataStore.java:434) 434 DiskDataStore - /ec/var/tomcat/gc1/work2/ROOT/wicket.hub-filestore/5168/1931/547BB9E7E255B16D5B2864F42B4F9C28.gc1/data (No such file or directory) java.io.FileNotFoundException: /ec/var/tomcat/gc1/work2/ROOT/wicket.hub-filestore/5168/1931/547BB9E7E255B16D5B2864F42B4F9C28.gc1/data (No such file or directory) at java.io.RandomAccessFile.open0(Native Method) at java.io.RandomAccessFile.open(RandomAccessFile.java:316) at java.io.RandomAccessFile.(RandomAccessFile.java:243) at org.apache.wicket.pageStore.DiskDataStore$SessionEntry.getFileChannel(DiskDataStore.java:428) at org.apache.wicket.pageStore.DiskDataStore$SessionEntry.savePage(DiskDataStore.java:346) at org.apache.wicket.pageStore.DiskDataStore.storeData(DiskDataStore.java:185) at org.apache.wicket.pageStore.AsynchronousDataStore$PageSavingRunnable.run(AsynchronousDataStore.java:355) at java.lang.Thread.run(Thread.java:745) Whats odd is that it effected some user but not others at all. When we rollback to 7.8.0 these go away. I'm not sure how I'm going to get the the bottom of this On Fri, May 17, 2019 at 11:03 PM Sven Meier wrote: > Hi, > > WICKET-6564 was about applications explicitly calling #clear() on the > PageStoreManager. > This doesn't has anything to do with session expiration. > > Could you please explain what issues you're facing? A quickstart would > be perfect, or at least some explanations how we're able to reproduce > the problem. > > Regards > Sven > > > Am 17.05.19 um 13:23 schrieb Wayne W: > > Hello Sven, > > > > the fix you did for WICKET-6564 ( > > > https://git-wip-us.apache.org/repos/asf?p=wicket.git;a=commitdiff;h=7665dc5;hp=3a1602d308a366e00948dbf91dbd96bc40cae167 > ) > > doesn't work for us. Its causing lots of issues. We use redisson for > > managing our sessions in tomcat, so that we can failover without loosing > > user session. > > > > When a session attribute is updated redisson calls setAttribute on the > > session. Looking at the code for > > org.apache.catalina.session.StandardSession.setAttribute I see: > > > > // Replace or add this attribute > > > > Object unbound = attributes.put(name, value); > > > > > > // Call the valueUnbound() method if necessary > > > > *if* (notify && (unbound != *null*) && (unbound != value) && > > > > (unbound *instanceof* HttpSessionBindingListener)) { > > > > *try* { > > > > ((HttpSessionBindingListener) unbound).valueUnbound > > > > (*new* HttpSessionBindingEvent(getSession(), name)); > > > > } *catch* (Throwable t) { > > > > ExceptionUtils.*handleThrowable*(t); > > > > manager.getContext().getLogger().error > > > > (*sm*.getString("standardSession.bindingEvent"), t); > > > > } > > > > } > > > > The valueUnbound is call to notify the object its no longer in the > session. > > However looking at the change you did for > > PageStoreManager.valueUnbound its actually > > removing the session by calling clear() > > > > Doesn't seem correct?? or am I missing something? > > > > > > > > On Fri, May 10, 2019 at 1:43 PM Wayne W > wrote: > > > >> Thanks Sven. > >> > >> On Thu, May 9, 2019 at 8:24 PM Sven Meier wrote: > >> > >>> Hi, > >>> > >>> we've had two
Re: Possible memory leak with Tomcat?
Hi, WICKET-6564 was about applications explicitly calling #clear() on the PageStoreManager. This doesn't has anything to do with session expiration. Could you please explain what issues you're facing? A quickstart would be perfect, or at least some explanations how we're able to reproduce the problem. Regards Sven Am 17.05.19 um 13:23 schrieb Wayne W: Hello Sven, the fix you did for WICKET-6564 ( https://git-wip-us.apache.org/repos/asf?p=wicket.git;a=commitdiff;h=7665dc5;hp=3a1602d308a366e00948dbf91dbd96bc40cae167) doesn't work for us. Its causing lots of issues. We use redisson for managing our sessions in tomcat, so that we can failover without loosing user session. When a session attribute is updated redisson calls setAttribute on the session. Looking at the code for org.apache.catalina.session.StandardSession.setAttribute I see: // Replace or add this attribute Object unbound = attributes.put(name, value); // Call the valueUnbound() method if necessary *if* (notify && (unbound != *null*) && (unbound != value) && (unbound *instanceof* HttpSessionBindingListener)) { *try* { ((HttpSessionBindingListener) unbound).valueUnbound (*new* HttpSessionBindingEvent(getSession(), name)); } *catch* (Throwable t) { ExceptionUtils.*handleThrowable*(t); manager.getContext().getLogger().error (*sm*.getString("standardSession.bindingEvent"), t); } } The valueUnbound is call to notify the object its no longer in the session. However looking at the change you did for PageStoreManager.valueUnbound its actually removing the session by calling clear() Doesn't seem correct?? or am I missing something? On Fri, May 10, 2019 at 1:43 PM Wayne W wrote: Thanks Sven. On Thu, May 9, 2019 at 8:24 PM Sven Meier wrote: Hi, we've had two fixed related to page stores (WICKET-6457, WICKET-6564) so please upgrade to 7.13.0 first. Have fun Sven Am 09.05.19 um 21:10 schrieb Wayne W: Hi, I've take a memory dump of one of our production instances and I'm seeing about 500MB of memory consumed just by the DiskDataStore which doesn't seem correct to me. Here is a screen shot: https://customerservices.glasscubes.com/share/s/qmvnvrdtm36amme6sdrqg4dp2g I've highlighted the item - 524,288 items doesn't seem correct for ~500 http sessions?. It seems to me that removeData(*final* String sessionId) is probably not being called and clearing up the sessionEntryMap. Looking at the call hierarchy I can see how Jetty calls it but I cannot see how tomcat is calling it. Any ideas whats happening? Wicket 7.8.0, Tomcat 8.0.32 Thanks - To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org For additional commands, e-mail: users-h...@wicket.apache.org - To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org For additional commands, e-mail: users-h...@wicket.apache.org
Re: Possible memory leak with Tomcat?
Hello Sven, the fix you did for WICKET-6564 ( https://git-wip-us.apache.org/repos/asf?p=wicket.git;a=commitdiff;h=7665dc5;hp=3a1602d308a366e00948dbf91dbd96bc40cae167) doesn't work for us. Its causing lots of issues. We use redisson for managing our sessions in tomcat, so that we can failover without loosing user session. When a session attribute is updated redisson calls setAttribute on the session. Looking at the code for org.apache.catalina.session.StandardSession.setAttribute I see: // Replace or add this attribute Object unbound = attributes.put(name, value); // Call the valueUnbound() method if necessary *if* (notify && (unbound != *null*) && (unbound != value) && (unbound *instanceof* HttpSessionBindingListener)) { *try* { ((HttpSessionBindingListener) unbound).valueUnbound (*new* HttpSessionBindingEvent(getSession(), name)); } *catch* (Throwable t) { ExceptionUtils.*handleThrowable*(t); manager.getContext().getLogger().error (*sm*.getString("standardSession.bindingEvent"), t); } } The valueUnbound is call to notify the object its no longer in the session. However looking at the change you did for PageStoreManager.valueUnbound its actually removing the session by calling clear() Doesn't seem correct?? or am I missing something? On Fri, May 10, 2019 at 1:43 PM Wayne W wrote: > Thanks Sven. > > On Thu, May 9, 2019 at 8:24 PM Sven Meier wrote: > >> Hi, >> >> we've had two fixed related to page stores (WICKET-6457, WICKET-6564) so >> please upgrade to 7.13.0 first. >> >> Have fun >> Sven >> >> >> Am 09.05.19 um 21:10 schrieb Wayne W: >> > Hi, >> > >> > I've take a memory dump of one of our production instances and I'm >> seeing >> > about 500MB of memory consumed just by the DiskDataStore which doesn't >> seem >> > correct to me. Here is a screen shot: >> > >> https://customerservices.glasscubes.com/share/s/qmvnvrdtm36amme6sdrqg4dp2g >> > >> > I've highlighted the item - 524,288 items doesn't seem correct for ~500 >> > http sessions?. It seems to me that removeData(*final* String >> sessionId) is >> > probably not being called and clearing up the sessionEntryMap. Looking >> at >> > the call hierarchy I can see how Jetty calls it but I cannot see how >> tomcat >> > is calling it. >> > >> > Any ideas whats happening? >> > Wicket 7.8.0, Tomcat 8.0.32 >> > >> > Thanks >> > >> >> - >> To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org >> For additional commands, e-mail: users-h...@wicket.apache.org >> >>
Re: Possible memory leak with Tomcat?
Thanks Sven. On Thu, May 9, 2019 at 8:24 PM Sven Meier wrote: > Hi, > > we've had two fixed related to page stores (WICKET-6457, WICKET-6564) so > please upgrade to 7.13.0 first. > > Have fun > Sven > > > Am 09.05.19 um 21:10 schrieb Wayne W: > > Hi, > > > > I've take a memory dump of one of our production instances and I'm seeing > > about 500MB of memory consumed just by the DiskDataStore which doesn't > seem > > correct to me. Here is a screen shot: > > > https://customerservices.glasscubes.com/share/s/qmvnvrdtm36amme6sdrqg4dp2g > > > > I've highlighted the item - 524,288 items doesn't seem correct for ~500 > > http sessions?. It seems to me that removeData(*final* String sessionId) > is > > probably not being called and clearing up the sessionEntryMap. Looking at > > the call hierarchy I can see how Jetty calls it but I cannot see how > tomcat > > is calling it. > > > > Any ideas whats happening? > > Wicket 7.8.0, Tomcat 8.0.32 > > > > Thanks > > > > - > To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org > For additional commands, e-mail: users-h...@wicket.apache.org > >
Re: Possible memory leak with Tomcat?
Hi, we've had two fixed related to page stores (WICKET-6457, WICKET-6564) so please upgrade to 7.13.0 first. Have fun Sven Am 09.05.19 um 21:10 schrieb Wayne W: Hi, I've take a memory dump of one of our production instances and I'm seeing about 500MB of memory consumed just by the DiskDataStore which doesn't seem correct to me. Here is a screen shot: https://customerservices.glasscubes.com/share/s/qmvnvrdtm36amme6sdrqg4dp2g I've highlighted the item - 524,288 items doesn't seem correct for ~500 http sessions?. It seems to me that removeData(*final* String sessionId) is probably not being called and clearing up the sessionEntryMap. Looking at the call hierarchy I can see how Jetty calls it but I cannot see how tomcat is calling it. Any ideas whats happening? Wicket 7.8.0, Tomcat 8.0.32 Thanks - To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org For additional commands, e-mail: users-h...@wicket.apache.org
Possible memory leak with Tomcat?
Hi, I've take a memory dump of one of our production instances and I'm seeing about 500MB of memory consumed just by the DiskDataStore which doesn't seem correct to me. Here is a screen shot: https://customerservices.glasscubes.com/share/s/qmvnvrdtm36amme6sdrqg4dp2g I've highlighted the item - 524,288 items doesn't seem correct for ~500 http sessions?. It seems to me that removeData(*final* String sessionId) is probably not being called and clearing up the sessionEntryMap. Looking at the call hierarchy I can see how Jetty calls it but I cannot see how tomcat is calling it. Any ideas whats happening? Wicket 7.8.0, Tomcat 8.0.32 Thanks