Following up on the conflation thing - Anil and I did an experiment. Conflation definitely *does* happen on everything in the queue, not just the last batch. But we didn't see destroys get conflated with updates.
So one thing that might make this use case work is to conflate the destroys with the updates. Then the disk space would be freed up when the expiration events are conflated in the queue. -Dan On Tue, Mar 26, 2019 at 8:19 AM Bruce Schuchardt <bschucha...@pivotal.io> wrote: > I've been thinking along those lines as well Suranjan. Since conflation > and expiry-forwarding don't solve the problem of running out of disk > space the solution needs to involve the dispatch thread. > > For the session-state caching scenario that raised this whole issue I > think what you've described will work. Looking at it with a wider lens > I'm a little concerned about a TTL on the queue because multiple regions > can feed into the same queue and you might not have the same TTL > settings on all of those regions. > > On 3/25/19 4:53 PM, Suranjan Kumar wrote: > > Hi, > > I think the one approach for a user would be to 'filter' the events > while > > dispatching. If I remember correctly, we can attach a filter at dispatch > > time and filter the events based on creationTime of the GatewayEvent. We > > can provide a pre created filter and use it based on some so that user > > doesn't have to write his/her own. > > > > Something like, > > /** > > All the events which spend timeToLive or more time in queue will be > deleted > > from the queue > > and will not be sent to remote site. > > Possible consequence is that two sites can be inconsistent in case > > */ > > public GatewaySenderFactory setEntryTimeToLive(long timeToLive); > > > > As queues will be read in LRU way this would be faster too. Only drawback > > is that there will be only one thread (not sure if we have concurrent > > dispatcher yet) clearing the queue. > > > > As Udo/Dan mentioned above, user needs to be aware of the consequences. > > > > > > On Tue, Mar 26, 2019 at 3:09 AM Bruce Schuchardt <bschucha...@pivotal.io > > > > wrote: > > > >> I've walked through the code to forward expiration actions to async > >> event listeners & don't see how to apply it to removal of queue entries > >> for WAN. The current implementation just queues the expiration > >> actions. If we wanted to remove any queued events associated with the > >> expired region entry we'd have to scan the whole queue, which would be > >> too slow if we're overflowing the queue to disk. > >> > >> I've also walked through the conflation code. It applies only to the > >> current batch being processed by the gateway sender. The data structure > >> used to perform conflation is just a Map that is created in the sender's > >> batch processing method and then thrown away. > >> > >> On 3/20/19 11:15 AM, Dan Smith wrote: > >>>> 2) The developer wants to replicate _state_. This means that implicit > >>>> state changes (expiration or eviction w/ destroy) could allow us to > >>>> optimize the queue size. This is very similar to conflation, just a > >>>> different kind of optimization. > >>>> > >>>> For this second case, does it make sense to allow the user to specify > a > >>>> different TTL than the underlying region? It seems like what the user > >>>> wants is to not replicate stale data and having an extra TTL attribute > >>>> would just be another value to mis-configure. What do you think about > >> just > >>>> providing a boolean flag? > >>>> > >>>> > >>> This kinda jogged my memory. AsyncEventQueues actually *do* have a > >> boolean > >>> flag to allow you to forward expiration events to the queue. I have no > >> idea > >>> how this interacts with conflation though - > >>> > >> > https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventQueueFactory.html#setForwardExpirationDestroy-boolean- >
