Re: Equivalent for PerSessionPageStore in Wicket 9
Hi Sven, I was thinking about this as well. SoftReferences worked well in my application. G1GC seems to start to evict them when -XX:InitiatingHeapOccupancyPercent is reached. In my case, when the heap is around 60% full. But as you said, there is no real control over which references are evicted and it is much harder to monitor the current state of the memory map. We are already using Caffeine extensively and your suggested solution would allow me much more control over the cache. We could still use SoftReferences with Caffeine if we wanted to in addition to setting a global limit on cache size and an eviction policy. To avoid calling an overridable method from the constructor you could add a Supplier> argument. I usually choose a supplier-based approach in such cases. Let's go for this solution! Best regards, Thomas On Sun, Apr 12, 2020 at 6:57 PM Sven Meier wrote: > Hi Thomas, > > I've did a little research on using SoftReferences for caches: > > https://stackoverflow.com/questions/264582/is-there-a-softhashmap-in-java > > http://jeremymanson.blogspot.com/2009/07/how-hotspot-decides-to-clear_07.html > > The experts seem to agree that depending on the GC to clean up your > cache is a bad idea: > > - you can't control which elements in the cache are evicted first > - eviction happens only when the system is low on memory > > Best option would be using Guava's CacheBuilder: > >https://github.com/google/guava/wiki/CachesExplained > > IMHO these are very special solutions and we don't actually need to > integrate one of them into Wicket. > Instead we can leave that decision to your application, by adding an > overridable method to InMemoryPageStore: > > /** > * Create a map to hold memory data for all sessions. > * > * @return a {@link ConcurrentHashMap} by default > */ > protected Map newMemoryMap() > { > return new ConcurrentHashMap<>(); > } > > (Yes, it would be called from the constructor which is a bad practice by > itself, but this is the simplest solution.) > > What do you think? > > Sven > > > On 12.04.20 10:34, Thomas Heigl wrote: > > Hi Sven, > > > > That's good to hear! Please let me know when you have an implementation > and > > I'll give it another go. > > > > Best regards, > > > > Thomas > > > > On Sat, Apr 11, 2020 at 11:01 PM Sven Meier wrote: > > > >> Hi Thomas, > >> > >> actually not bad news at all (for Wicket 9 at least). > >> > >> The old page manager implementation had so many special concepts and > >> solutions, it's easy to miss one. > >> > >> A soft reference feature can easily be added/restored. I'm already > >> checking where it fits best. > >> > >> Thanks for your thorough testing. > >> > >> Best regards > >> Sven > >> > >> > >> On 11.04.20 10:58, Thomas Heigl wrote: > >>> Hi all, > >>> > >>> Bad news. My application was caught in a GC loop after running for 8 > >> hours. > >>> The old generation was exhausted. > >>> > >>> I couldn't get a heap dump at that time but restarted the application, > >> took > >>> a heap dump after about an hour, and reverted back to Wicket 8. > >>> > >>> The problem is this: The heap was full of objects referencing > >>> InMemoryPageStore, i.e. the in-memory 2nd-level cache for pages. My > first > >>> thought was that there is something wrong with the implementation of > that > >>> store and pages do not get limited or removed correctly. So I debugged > >>> it locally but everything is working fine. > >>> > >>> Then I noticed that there are a lot of instances of Hibernate entities > on > >>> the heap. So there definitely is an issue with models somewhere in my > >>> application. To be sure that this is not a new issue, I took another > heap > >>> dump from production - now running Wicket 8 again - and compared it. > >>> There are undetached entity models on the heap as well. > >>> > >>> So why does it not OOM with Wicket 8? Well, the PerSessionPageStore > >>> (roughly the equivalent of InMemoryPageStore in Wicket 9) uses > >>> SoftReferences for storing pages. InMemoryPageStore does not and GC > >> cannot > >>> reclaim memory from it. > >>> > >>> So while this surfaced some issues in my application that I just > fixed, I > >>> believe that InMemoryPageStore should use SoftReferences or another > >>> implementation based on SoftReferences should be added to Wicket 9. A > >> cache > >>> should not consume all the memory if it can easily re-fetch > >>> the value from persistent storage. > >>> > >>> I guess the reason for not using SoftReferences in InMemoryPageStore is > >>> that it can theoretically be used as a "persistent" store for pages. If > >> that > >>> behavior is really required, I suggest adding another implementation > >> using > >>> SoftReferences. > >>> > >>> Best regards, > >>> > >>> Thomas > >>> > >>> On Fri, Apr 10, 2020 at 7:19 PM Martin Grigorov > >>> wrote: > >>> > On Fri, Apr 10, 2020 at 4:01 PM Thomas Heigl > >> wrote: > > FYI: I deployed Wicket 9.0.0-M
Re: Equivalent for PerSessionPageStore in Wicket 9
Hi Thomas, I've did a little research on using SoftReferences for caches: https://stackoverflow.com/questions/264582/is-there-a-softhashmap-in-java http://jeremymanson.blogspot.com/2009/07/how-hotspot-decides-to-clear_07.html The experts seem to agree that depending on the GC to clean up your cache is a bad idea: - you can't control which elements in the cache are evicted first - eviction happens only when the system is low on memory Best option would be using Guava's CacheBuilder: https://github.com/google/guava/wiki/CachesExplained IMHO these are very special solutions and we don't actually need to integrate one of them into Wicket. Instead we can leave that decision to your application, by adding an overridable method to InMemoryPageStore: /** * Create a map to hold memory data for all sessions. * * @return a {@link ConcurrentHashMap} by default */ protected Map newMemoryMap() { return new ConcurrentHashMap<>(); } (Yes, it would be called from the constructor which is a bad practice by itself, but this is the simplest solution.) What do you think? Sven On 12.04.20 10:34, Thomas Heigl wrote: Hi Sven, That's good to hear! Please let me know when you have an implementation and I'll give it another go. Best regards, Thomas On Sat, Apr 11, 2020 at 11:01 PM Sven Meier wrote: Hi Thomas, actually not bad news at all (for Wicket 9 at least). The old page manager implementation had so many special concepts and solutions, it's easy to miss one. A soft reference feature can easily be added/restored. I'm already checking where it fits best. Thanks for your thorough testing. Best regards Sven On 11.04.20 10:58, Thomas Heigl wrote: Hi all, Bad news. My application was caught in a GC loop after running for 8 hours. The old generation was exhausted. I couldn't get a heap dump at that time but restarted the application, took a heap dump after about an hour, and reverted back to Wicket 8. The problem is this: The heap was full of objects referencing InMemoryPageStore, i.e. the in-memory 2nd-level cache for pages. My first thought was that there is something wrong with the implementation of that store and pages do not get limited or removed correctly. So I debugged it locally but everything is working fine. Then I noticed that there are a lot of instances of Hibernate entities on the heap. So there definitely is an issue with models somewhere in my application. To be sure that this is not a new issue, I took another heap dump from production - now running Wicket 8 again - and compared it. There are undetached entity models on the heap as well. So why does it not OOM with Wicket 8? Well, the PerSessionPageStore (roughly the equivalent of InMemoryPageStore in Wicket 9) uses SoftReferences for storing pages. InMemoryPageStore does not and GC cannot reclaim memory from it. So while this surfaced some issues in my application that I just fixed, I believe that InMemoryPageStore should use SoftReferences or another implementation based on SoftReferences should be added to Wicket 9. A cache should not consume all the memory if it can easily re-fetch the value from persistent storage. I guess the reason for not using SoftReferences in InMemoryPageStore is that it can theoretically be used as a "persistent" store for pages. If that behavior is really required, I suggest adding another implementation using SoftReferences. Best regards, Thomas On Fri, Apr 10, 2020 at 7:19 PM Martin Grigorov wrote: On Fri, Apr 10, 2020 at 4:01 PM Thomas Heigl wrote: FYI: I deployed Wicket 9.0.0-M5 to production an hour ago. 100k requests served and no issues so far. Awesome! Thank you for testing it! Great work! Thomas On Wed, Apr 8, 2020 at 3:13 PM Sven Meier wrote: Many thanks Maxim! Sven On 08.04.20 14:29, Maxim Solodovnik wrote: Released :) On Wed, 8 Apr 2020 at 15:41, Maxim Solodovnik wrote: OK Will start new release process in couple of hours Please stop me if you will find any blocker :) On Wed, 8 Apr 2020 at 14:36, Thomas Heigl wrote: Hi Maxim, It works for me now! Thomas On Wed, Apr 8, 2020 at 9:17 AM Maxim Solodovnik < solomax...@gmail.com> wrote: Thanks a million! On Wed, 8 Apr 2020 at 14:10, Thomas Heigl wrote: Hi Maxim, I'm testing against the snapshot now. Will get back to you shortly. Thomas On Wed, Apr 8, 2020 at 2:53 AM Maxim Solodovnik < solomax...@gmail.com> wrote: Hello All, M5 seems to be broken (deploy has failed more than 10 times during my build attempts) I have to start another release Could you please tell when can I start? On Wed, 8 Apr 2020 at 07:01, Maxim Solodovnik < solomax...@gmail.com wrote: Hello Thomas, Please test M6-SNAPSHOT (so I don't have to release M5.2 : On Wed, 8 Apr 2020 at 02:39, Thomas Heigl wrote: Hi Maxim, That would be great. I want to do some more extensive testing and then deploy M5 into production. ;) Thomas On Tue, Apr 7, 2020
Re: Equivalent for PerSessionPageStore in Wicket 9
Hi Sven, That's good to hear! Please let me know when you have an implementation and I'll give it another go. Best regards, Thomas On Sat, Apr 11, 2020 at 11:01 PM Sven Meier wrote: > Hi Thomas, > > actually not bad news at all (for Wicket 9 at least). > > The old page manager implementation had so many special concepts and > solutions, it's easy to miss one. > > A soft reference feature can easily be added/restored. I'm already > checking where it fits best. > > Thanks for your thorough testing. > > Best regards > Sven > > > On 11.04.20 10:58, Thomas Heigl wrote: > > Hi all, > > > > Bad news. My application was caught in a GC loop after running for 8 > hours. > > The old generation was exhausted. > > > > I couldn't get a heap dump at that time but restarted the application, > took > > a heap dump after about an hour, and reverted back to Wicket 8. > > > > The problem is this: The heap was full of objects referencing > > InMemoryPageStore, i.e. the in-memory 2nd-level cache for pages. My first > > thought was that there is something wrong with the implementation of that > > store and pages do not get limited or removed correctly. So I debugged > > it locally but everything is working fine. > > > > Then I noticed that there are a lot of instances of Hibernate entities on > > the heap. So there definitely is an issue with models somewhere in my > > application. To be sure that this is not a new issue, I took another heap > > dump from production - now running Wicket 8 again - and compared it. > > There are undetached entity models on the heap as well. > > > > So why does it not OOM with Wicket 8? Well, the PerSessionPageStore > > (roughly the equivalent of InMemoryPageStore in Wicket 9) uses > > SoftReferences for storing pages. InMemoryPageStore does not and GC > cannot > > reclaim memory from it. > > > > So while this surfaced some issues in my application that I just fixed, I > > believe that InMemoryPageStore should use SoftReferences or another > > implementation based on SoftReferences should be added to Wicket 9. A > cache > > should not consume all the memory if it can easily re-fetch > > the value from persistent storage. > > > > I guess the reason for not using SoftReferences in InMemoryPageStore is > > that it can theoretically be used as a "persistent" store for pages. If > that > > behavior is really required, I suggest adding another implementation > using > > SoftReferences. > > > > Best regards, > > > > Thomas > > > > On Fri, Apr 10, 2020 at 7:19 PM Martin Grigorov > > wrote: > > > >> On Fri, Apr 10, 2020 at 4:01 PM Thomas Heigl > wrote: > >> > >>> FYI: I deployed Wicket 9.0.0-M5 to production an hour ago. 100k > requests > >>> served and no issues so far. > >>> > >> Awesome! > >> Thank you for testing it! > >> > >> > >>> Great work! > >>> > >>> Thomas > >>> > >>> On Wed, Apr 8, 2020 at 3:13 PM Sven Meier wrote: > >>> > Many thanks Maxim! > > Sven > > On 08.04.20 14:29, Maxim Solodovnik wrote: > > Released :) > > > > On Wed, 8 Apr 2020 at 15:41, Maxim Solodovnik > wrote: > >> OK > >> > >> Will start new release process in couple of hours > >> Please stop me if you will find any blocker :) > >> > >> On Wed, 8 Apr 2020 at 14:36, Thomas Heigl > >>> wrote: > >>> Hi Maxim, > >>> > >>> It works for me now! > >>> > >>> Thomas > >>> > >>> On Wed, Apr 8, 2020 at 9:17 AM Maxim Solodovnik < > >>> solomax...@gmail.com> > >>> wrote: > >>> > Thanks a million! > > On Wed, 8 Apr 2020 at 14:10, Thomas Heigl > wrote: > > Hi Maxim, > > > > I'm testing against the snapshot now. Will get back to you > >> shortly. > > Thomas > > > > On Wed, Apr 8, 2020 at 2:53 AM Maxim Solodovnik < > solomax...@gmail.com> > > wrote: > > > >> Hello All, > >> > >> M5 seems to be broken (deploy has failed more than 10 times > >> during > my > >> build attempts) > >> I have to start another release > >> Could you please tell when can I start? > >> > >> On Wed, 8 Apr 2020 at 07:01, Maxim Solodovnik < > >>> solomax...@gmail.com > >> wrote: > >>> Hello Thomas, > >>> > >>> Please test M6-SNAPSHOT (so I don't have to release M5.2 : > >>> > >>> On Wed, 8 Apr 2020 at 02:39, Thomas Heigl wrote: > Hi Maxim, > > That would be great. I want to do some more extensive testing > >>> and > then > deploy M5 into production. ;) > > Thomas > > On Tue, Apr 7, 2020 at 7:50 PM Maxim Solodovnik < > solomax...@gmail.com> > wrote: > > > I can pack another release > > later this week ... > > > > On Wed, 8 Apr 2020 at 00:48, Th