Thanks for your answer, Kirk. If we persist the unsent events in a persistent region then the memory consumed would not be as high but still it would not solve our problem with long lasting split brain as the persistent region would take some memory too to store those events even if they were overflown.
Ideally it should be backed up in a queue that does not use any memory. Best regards, Alberto ________________________________ From: Kirk Lund <kl...@apache.org> Sent: Thursday, July 2, 2020 7:41 PM To: dev@geode.apache.org <dev@geode.apache.org> Subject: Re: Question about gateway sender stopped and memory consumption I would have expected unsent events to be stored in a queue that is backed by a persistent region or something on disk. If that's not currently true, then it seems like a good direction might be to make tmpDroppedEvents use a durable queue of some sort that overflows to disk. On Thu, Jul 2, 2020 at 10:33 AM Alberto Gomez <alberto.go...@est.tech> wrote: > Hi, > > We have observed that when a gateway sender is stopped in a site, all the > events received while it is stopped are stored in the > 'AbstractGatewaySender.tmpDroppedEvents' queue of the primary sender. The > elements of this queue are not removed from this queue until the sender is > started back again. > > This behavior implies that if the gateway sender is stopped for a long > time, there is a risk of heap exhaustion in the members hosting primary > senders. > > Under split brain situations, if lasting long enough, there could be heap > exhaustion problems in servers due to the memory used by the gateway sender > queues, even if overflow to disk is used -given that part of the event is > always stored in memory. > For those situations we had thought about stopping gateway senders when > the memory used by the gateway sender queues reached a certain memory > threshold. But according to the above, stopping the gateway senders would > only make things worse. > > Would it make sense for the gateway sender not to store the received > events in tmpDroppedEvents while it is stopped? > > Any suggestion on how to approach the problem of heap exhaustion due to > the growth of gateway sender queues in long lasting split brain situations? > > Thanks in advance, > > Alberto G. > > >
