Re: Equivalent for PerSessionPageStore in Wicket 9
Hi Sven, That's good to hear! Please let me know when you have an implementation and I'll give it another go. Best regards, Thomas On Sat, Apr 11, 2020 at 11:01 PM Sven Meier wrote: > Hi Thomas, > > actually not bad news at all (for Wicket 9 at least). > > The old page manager implementation had so many special concepts and > solutions, it's easy to miss one. > > A soft reference feature can easily be added/restored. I'm already > checking where it fits best. > > Thanks for your thorough testing. > > Best regards > Sven > > > On 11.04.20 10:58, Thomas Heigl wrote: > > Hi all, > > > > Bad news. My application was caught in a GC loop after running for 8 > hours. > > The old generation was exhausted. > > > > I couldn't get a heap dump at that time but restarted the application, > took > > a heap dump after about an hour, and reverted back to Wicket 8. > > > > The problem is this: The heap was full of objects referencing > > InMemoryPageStore, i.e. the in-memory 2nd-level cache for pages. My first > > thought was that there is something wrong with the implementation of that > > store and pages do not get limited or removed correctly. So I debugged > > it locally but everything is working fine. > > > > Then I noticed that there are a lot of instances of Hibernate entities on > > the heap. So there definitely is an issue with models somewhere in my > > application. To be sure that this is not a new issue, I took another heap > > dump from production - now running Wicket 8 again - and compared it. > > There are undetached entity models on the heap as well. > > > > So why does it not OOM with Wicket 8? Well, the PerSessionPageStore > > (roughly the equivalent of InMemoryPageStore in Wicket 9) uses > > SoftReferences for storing pages. InMemoryPageStore does not and GC > cannot > > reclaim memory from it. > > > > So while this surfaced some issues in my application that I just fixed, I > > believe that InMemoryPageStore should use SoftReferences or another > > implementation based on SoftReferences should be added to Wicket 9. A > cache > > should not consume all the memory if it can easily re-fetch > > the value from persistent storage. > > > > I guess the reason for not using SoftReferences in InMemoryPageStore is > > that it can theoretically be used as a "persistent" store for pages. If > that > > behavior is really required, I suggest adding another implementation > using > > SoftReferences. > > > > Best regards, > > > > Thomas > > > > On Fri, Apr 10, 2020 at 7:19 PM Martin Grigorov > > wrote: > > > >> On Fri, Apr 10, 2020 at 4:01 PM Thomas Heigl > wrote: > >> > >>> FYI: I deployed Wicket 9.0.0-M5 to production an hour ago. 100k > requests > >>> served and no issues so far. > >>> > >> Awesome! > >> Thank you for testing it! > >> > >> > >>> Great work! > >>> > >>> Thomas > >>> > >>> On Wed, Apr 8, 2020 at 3:13 PM Sven Meier wrote: > >>> > Many thanks Maxim! > > Sven > > On 08.04.20 14:29, Maxim Solodovnik wrote: > > Released :) > > > > On Wed, 8 Apr 2020 at 15:41, Maxim Solodovnik > wrote: > >> OK > >> > >> Will start new release process in couple of hours > >> Please stop me if you will find any blocker :) > >> > >> On Wed, 8 Apr 2020 at 14:36, Thomas Heigl > >>> wrote: > >>> Hi Maxim, > >>> > >>> It works for me now! > >>> > >>> Thomas > >>> > >>> On Wed, Apr 8, 2020 at 9:17 AM Maxim Solodovnik < > >>> solomax...@gmail.com> > >>> wrote: > >>> > Thanks a million! > > On Wed, 8 Apr 2020 at 14:10, Thomas Heigl > wrote: > > Hi Maxim, > > > > I'm testing against the snapshot now. Will get back to you > >> shortly. > > Thomas > > > > On Wed, Apr 8, 2020 at 2:53 AM Maxim Solodovnik < > solomax...@gmail.com> > > wrote: > > > >> Hello All, > >> > >> M5 seems to be broken (deploy has failed more than 10 times > >> during > my > >> build attempts) > >> I have to start another release > >> Could you please tell when can I start? > >> > >> On Wed, 8 Apr 2020 at 07:01, Maxim Solodovnik < > >>> solomax...@gmail.com > >> wrote: > >>> Hello Thomas, > >>> > >>> Please test M6-SNAPSHOT (so I don't have to release M5.2 : > >>> > >>> On Wed, 8 Apr 2020 at 02:39, Thomas Heigl wrote: > Hi Maxim, > > That would be great. I want to do some more extensive testing > >>> and > then > deploy M5 into production. ;) > > Thomas > > On Tue, Apr 7, 2020 at 7:50 PM Maxim Solodovnik < > solomax...@gmail.com> > wrote: > > > I can pack another release > > later this week ... > > > > On Wed, 8 Apr 2020 at 00:48,
Re: Equivalent for PerSessionPageStore in Wicket 9
Hi Thomas,
I've did a little research on using SoftReferences for caches:
https://stackoverflow.com/questions/264582/is-there-a-softhashmap-in-java
http://jeremymanson.blogspot.com/2009/07/how-hotspot-decides-to-clear_07.html
The experts seem to agree that depending on the GC to clean up your
cache is a bad idea:
- you can't control which elements in the cache are evicted first
- eviction happens only when the system is low on memory
Best option would be using Guava's CacheBuilder:
https://github.com/google/guava/wiki/CachesExplained
IMHO these are very special solutions and we don't actually need to
integrate one of them into Wicket.
Instead we can leave that decision to your application, by adding an
overridable method to InMemoryPageStore:
/**
* Create a map to hold memory data for all sessions.
*
* @return a {@link ConcurrentHashMap} by default
*/
protected Map newMemoryMap()
{
return new ConcurrentHashMap<>();
}
(Yes, it would be called from the constructor which is a bad practice by
itself, but this is the simplest solution.)
What do you think?
Sven
On 12.04.20 10:34, Thomas Heigl wrote:
Hi Sven,
That's good to hear! Please let me know when you have an implementation and
I'll give it another go.
Best regards,
Thomas
On Sat, Apr 11, 2020 at 11:01 PM Sven Meier wrote:
Hi Thomas,
actually not bad news at all (for Wicket 9 at least).
The old page manager implementation had so many special concepts and
solutions, it's easy to miss one.
A soft reference feature can easily be added/restored. I'm already
checking where it fits best.
Thanks for your thorough testing.
Best regards
Sven
On 11.04.20 10:58, Thomas Heigl wrote:
Hi all,
Bad news. My application was caught in a GC loop after running for 8
hours.
The old generation was exhausted.
I couldn't get a heap dump at that time but restarted the application,
took
a heap dump after about an hour, and reverted back to Wicket 8.
The problem is this: The heap was full of objects referencing
InMemoryPageStore, i.e. the in-memory 2nd-level cache for pages. My first
thought was that there is something wrong with the implementation of that
store and pages do not get limited or removed correctly. So I debugged
it locally but everything is working fine.
Then I noticed that there are a lot of instances of Hibernate entities on
the heap. So there definitely is an issue with models somewhere in my
application. To be sure that this is not a new issue, I took another heap
dump from production - now running Wicket 8 again - and compared it.
There are undetached entity models on the heap as well.
So why does it not OOM with Wicket 8? Well, the PerSessionPageStore
(roughly the equivalent of InMemoryPageStore in Wicket 9) uses
SoftReferences for storing pages. InMemoryPageStore does not and GC
cannot
reclaim memory from it.
So while this surfaced some issues in my application that I just fixed, I
believe that InMemoryPageStore should use SoftReferences or another
implementation based on SoftReferences should be added to Wicket 9. A
cache
should not consume all the memory if it can easily re-fetch
the value from persistent storage.
I guess the reason for not using SoftReferences in InMemoryPageStore is
that it can theoretically be used as a "persistent" store for pages. If
that
behavior is really required, I suggest adding another implementation
using
SoftReferences.
Best regards,
Thomas
On Fri, Apr 10, 2020 at 7:19 PM Martin Grigorov
wrote:
On Fri, Apr 10, 2020 at 4:01 PM Thomas Heigl
wrote:
FYI: I deployed Wicket 9.0.0-M5 to production an hour ago. 100k
requests
served and no issues so far.
Awesome!
Thank you for testing it!
Great work!
Thomas
On Wed, Apr 8, 2020 at 3:13 PM Sven Meier wrote:
Many thanks Maxim!
Sven
On 08.04.20 14:29, Maxim Solodovnik wrote:
Released :)
On Wed, 8 Apr 2020 at 15:41, Maxim Solodovnik
wrote:
OK
Will start new release process in couple of hours
Please stop me if you will find any blocker :)
On Wed, 8 Apr 2020 at 14:36, Thomas Heigl
wrote:
Hi Maxim,
It works for me now!
Thomas
On Wed, Apr 8, 2020 at 9:17 AM Maxim Solodovnik <
solomax...@gmail.com>
wrote:
Thanks a million!
On Wed, 8 Apr 2020 at 14:10, Thomas Heigl
wrote:
Hi Maxim,
I'm testing against the snapshot now. Will get back to you
shortly.
Thomas
On Wed, Apr 8, 2020 at 2:53 AM Maxim Solodovnik <
solomax...@gmail.com>
wrote:
Hello All,
M5 seems to be broken (deploy has failed more than 10 times
during
my
build attempts)
I have to start another release
Could you please tell when can I start?
On Wed, 8 Apr 2020 at 07:01, Maxim Solodovnik <
solomax...@gmail.com
wrote:
Hello Thomas,
Please test M6-SNAPSHOT (so I don't have to release M5.2 :
On Wed, 8 Apr 2020 at 02:39, Thomas Heigl
wrote:
Hi Maxim,
That would be great. I want to do some more extensive testing
and
then
deploy M5 into production. ;)
Thomas
On Tue, Apr 7,
Re: Equivalent for PerSessionPageStore in Wicket 9
Hi Sven,
I was thinking about this as well.
SoftReferences worked well in my application. G1GC seems to start to evict
them when -XX:InitiatingHeapOccupancyPercent is reached. In my case, when
the heap is around 60% full.
But as you said, there is no real control over which references are evicted
and it is much harder to monitor the current state of the memory map.
We are already using Caffeine extensively and your suggested solution would
allow me much more control over the cache.
We could still use SoftReferences with Caffeine if we wanted to in addition
to setting a global limit on cache size and an eviction policy.
To avoid calling an overridable method from the constructor you could add a
Supplier> argument. I usually choose a
supplier-based approach in such cases.
Let's go for this solution!
Best regards,
Thomas
On Sun, Apr 12, 2020 at 6:57 PM Sven Meier wrote:
> Hi Thomas,
>
> I've did a little research on using SoftReferences for caches:
>
> https://stackoverflow.com/questions/264582/is-there-a-softhashmap-in-java
>
> http://jeremymanson.blogspot.com/2009/07/how-hotspot-decides-to-clear_07.html
>
> The experts seem to agree that depending on the GC to clean up your
> cache is a bad idea:
>
> - you can't control which elements in the cache are evicted first
> - eviction happens only when the system is low on memory
>
> Best option would be using Guava's CacheBuilder:
>
>https://github.com/google/guava/wiki/CachesExplained
>
> IMHO these are very special solutions and we don't actually need to
> integrate one of them into Wicket.
> Instead we can leave that decision to your application, by adding an
> overridable method to InMemoryPageStore:
>
> /**
> * Create a map to hold memory data for all sessions.
> *
> * @return a {@link ConcurrentHashMap} by default
> */
> protected Map newMemoryMap()
> {
> return new ConcurrentHashMap<>();
> }
>
> (Yes, it would be called from the constructor which is a bad practice by
> itself, but this is the simplest solution.)
>
> What do you think?
>
> Sven
>
>
> On 12.04.20 10:34, Thomas Heigl wrote:
> > Hi Sven,
> >
> > That's good to hear! Please let me know when you have an implementation
> and
> > I'll give it another go.
> >
> > Best regards,
> >
> > Thomas
> >
> > On Sat, Apr 11, 2020 at 11:01 PM Sven Meier wrote:
> >
> >> Hi Thomas,
> >>
> >> actually not bad news at all (for Wicket 9 at least).
> >>
> >> The old page manager implementation had so many special concepts and
> >> solutions, it's easy to miss one.
> >>
> >> A soft reference feature can easily be added/restored. I'm already
> >> checking where it fits best.
> >>
> >> Thanks for your thorough testing.
> >>
> >> Best regards
> >> Sven
> >>
> >>
> >> On 11.04.20 10:58, Thomas Heigl wrote:
> >>> Hi all,
> >>>
> >>> Bad news. My application was caught in a GC loop after running for 8
> >> hours.
> >>> The old generation was exhausted.
> >>>
> >>> I couldn't get a heap dump at that time but restarted the application,
> >> took
> >>> a heap dump after about an hour, and reverted back to Wicket 8.
> >>>
> >>> The problem is this: The heap was full of objects referencing
> >>> InMemoryPageStore, i.e. the in-memory 2nd-level cache for pages. My
> first
> >>> thought was that there is something wrong with the implementation of
> that
> >>> store and pages do not get limited or removed correctly. So I debugged
> >>> it locally but everything is working fine.
> >>>
> >>> Then I noticed that there are a lot of instances of Hibernate entities
> on
> >>> the heap. So there definitely is an issue with models somewhere in my
> >>> application. To be sure that this is not a new issue, I took another
> heap
> >>> dump from production - now running Wicket 8 again - and compared it.
> >>> There are undetached entity models on the heap as well.
> >>>
> >>> So why does it not OOM with Wicket 8? Well, the PerSessionPageStore
> >>> (roughly the equivalent of InMemoryPageStore in Wicket 9) uses
> >>> SoftReferences for storing pages. InMemoryPageStore does not and GC
> >> cannot
> >>> reclaim memory from it.
> >>>
> >>> So while this surfaced some issues in my application that I just
> fixed, I
> >>> believe that InMemoryPageStore should use SoftReferences or another
> >>> implementation based on SoftReferences should be added to Wicket 9. A
> >> cache
> >>> should not consume all the memory if it can easily re-fetch
> >>> the value from persistent storage.
> >>>
> >>> I guess the reason for not using SoftReferences in InMemoryPageStore is
> >>> that it can theoretically be used as a "persistent" store for pages. If
> >> that
> >>> behavior is really required, I suggest adding another implementation
> >> using
> >>> SoftReferences.
> >>>
> >>> Best regards,
> >>>
> >>> Thomas
> >>>
> >>> On Fri, Apr 10, 2020 at 7:19 PM Martin Grigorov
> >>> wrote:
> >>>
> On Fri, Apr 10, 2020 at 4:01 PM Thomas Heigl
> >> wrote:
> > FYI: I deployed Wicket
