Re: [infinispan-dev] Local state transfer before going over network

2011-06-08 Thread Manik Surtani
On 24 May 2011, at 11:33, Mircea Markus wrote: Merkle trees would do much better than key-by-key comparison. +1. Is there a JIRA for this? Anyone wants to take this on, I would target this at 6.0 though. -- Manik Surtani ma...@jboss.org twitter.com/maniksurtani Lead, Infinispan

Re: [infinispan-dev] Local state transfer before going over network

2011-05-24 Thread Mircea Markus
I think in both cases (repl and dist) it still may make sense in some cases. E.g., in dist, if a node joins, existing owners could, rather than push data to the joiner, just push a list of {key: version} tuples, which may be significantly smaller than the values. The joiner can then

Re: [infinispan-dev] Local state transfer before going over network

2011-05-20 Thread Galder Zamarreño
I've just created http://community.jboss.org/docs/DOC-16815 The document title is temporary and I'm open to suggestions for different names. Feel free to add more thoughts/comments to it. I'll add a link from the main wiki now. On May 19, 2011, at 12:18 PM, Manik Surtani wrote: On 19 May

Re: [infinispan-dev] Local state transfer before going over network

2011-05-19 Thread Bela Ban
As someone mentioned, the biggest issue would be to make sure the data read from the file system isn't stale. If a node has been down for an extended period of time, then this process might slow things down. We'd have to implement some rsync like algorithm, which checks the local data against

Re: [infinispan-dev] Local state transfer before going over network

2011-05-17 Thread Galder Zamarreño
On May 16, 2011, at 1:18 PM, Sanne Grinovero wrote: 2011/5/16 Galder Zamarreño gal...@redhat.com: Not sure if the idea has come up but while at GeeCON last week I was discussing to one of the attendees about state transfer improvements in replicated environments: The idea is that in a

Re: [infinispan-dev] Local state transfer before going over network

2011-05-17 Thread Sanne Grinovero
2011/5/17 Galder Zamarreño gal...@redhat.com: On May 16, 2011, at 1:18 PM, Sanne Grinovero wrote: 2011/5/16 Galder Zamarreño gal...@redhat.com: Not sure if the idea has come up but while at GeeCON last week I was discussing to one of the attendees about state transfer improvements in

Re: [infinispan-dev] Local state transfer before going over network

2011-05-17 Thread Manik Surtani
Interesting discussions. Another approach may be to version data using lamport clocks or vector clocks. Then at the start of a rehash, a digest of keys and versions can be pushed, and the receiver 'decides' which keys are out of date and needs to be pulled from across the network. On 17 May

Re: [infinispan-dev] Local state transfer before going over network

2011-05-17 Thread Bela Ban
This is exactly what JGroups digests do On 05/17/2011 10:38 AM, Manik Surtani wrote: Interesting discussions. Another approach may be to version data using lamport clocks or vector clocks. Then at the start of a rehash, a digest of keys and versions can be pushed, and the receiver

[infinispan-dev] Local state transfer before going over network

2011-05-16 Thread Galder Zamarreño
Not sure if the idea has come up but while at GeeCON last week I was discussing to one of the attendees about state transfer improvements in replicated environments: The idea is that in a replicated environment, if a cache manager shuts down, it would dump its memory contents to a cache store