If all that the customer is concerned about is that the receiving side
gets the "latest" data, conflation is definitely the best bet. How do we
classify old? The only classification that I have of old (in this
context) is that there is a newer version of a data entry. This
classification is not time based, as a TTL approach would require, but
state.
What is the use of WAN replication if we are proposing to only replicate
*some* of the data. How would the clusters ever know if they are
in-sync? I believe we are just opening a door that we will never be able
to close again and will cause us endless issues.
--Udo
On 3/20/19 10:56, Bruce Schuchardt wrote:
Udo, in the cases I've looked at the user is okay with inconsistency
because they don't really care about the old data. They're most
interested in getting the newest data and keeping the sending site
from going down. I guess the docs for TTL should make it very clear
that it will cause inconsistencies.
Conflation does seem like an appropriate thing to try if the same keys
are being updated - I'll do some investigation and see why it wasn't
appropriate.
On 3/20/19 10:51 AM, Udo Kohlmeyer wrote:
-1, I don't believe this is a feature that we should support. IF a
client is experiencing slow WAN replication and users only care about
the "latest" data, then maybe the user should use "conflation".
With a TTL model, we are messing with our consistency tenet. I'm am
NOT in support of a setting that can cause inconsistency.
Dead-letter queues is another area that WILL cause data/site
inconsistency. I think we really have to take a step back, think
about WHAT tenets are important to GEODE and then act accordingly.
--Udo
On 3/20/19 10:46, Bruce Schuchardt wrote:
IDK Anil, we'll figure that out in the implementation. I was
thinking it would be in the dispatch threads, so if distribution is
need that will happen as it does now. I'm hopeful that this won't
perturb the code too much.
One thing that was brought up to me in person was the Dead Letter
Queue
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80452478>
initiative that seems to have stalled. That seems like a very
similar idea though it's reacting to errors coming from the
receiving side and not a local condition. I like the callback,
stats, gfsh and cluster config integration in that write-up & think
they might be useful here. There is also relevant discussion in
that page about things like PDX registrations. Is that initiative
going to move forward at some point or is it off the boards?
On 3/20/19 10:25 AM, Anilkumar Gingade wrote:
+1. Will the expiration (destroy) be applied on local queues or the
expiration will be replicated (for both serial and parallel)?
-Anil.
On Wed, Mar 20, 2019 at 8:59 AM Bruce Schuchardt
<bschucha...@pivotal.io>
wrote:
We've seen situations where the receiving side of a WAN gateway is
slow
to accept data or is not accepting any data. This can cause
queues to
fill up on the sending side. If disk-overflow is being used this can
even lead to an outage. Some users are concerned more with the
latest
data and don't really care if old data is thrown away in this
situation. They may have set a TTL on their Regions and would
like to
be able to do the same thing with their GatewaySenders.
With that in mind I'd like to add this method to
GatewaySenderFactory:
/** * Sets the timeToLive expiration attribute for queue entries
for the
next * {@code GatewaySender} created. * * @param timeToLive the
timeToLive ExpirationAttributes for entries in this region *
@return a
reference to this GatewaySenderFactory object * @throws
IllegalArgumentException if timeToLive is null * @see
RegionFactory#setEntryTimeToLive */ public GatewaySenderFactory
setEntryTimeToLive(ExpirationAttributes timeToLive);
The exact implementation may not be the same as for Regions since we
probably want to expire the oldest entries first and make sure we
do so
in their order in the queue.
