Hi Galder, sorry it took so long to reply. On Mon, Oct 24, 2011 at 4:16 PM, Galder Zamarreño <gal...@redhat.com> wrote: > Btw, forgot to attach the log: > > > > > On Oct 24, 2011, at 3:13 PM, Galder Zamarreño wrote: > >> Hi Dan, >> >> Re: http://goo.gl/TGwrP >> >> There's a few of this in the Hot Rod server+client testsuites. It's easy to >> replicate it locally. Seems like cache operations right after a cache has >> started are rather problematic. >> >> In local execution of HotRodReplicationTest, I was able to replicate the >> issue when trying to test topology changes. Please find attached the log >> file, but here're the interesting bits: >> >> 1. A new view installation is being prepared with NodeA and NodeB: >> 2011-10-24 14:36:09,046 4221 TRACE >> [org.infinispan.cacheviews.CacheViewsManagerImpl] >> (OOB-1,Infinispan-Cluster,NodeB-15806:___hotRodTopologyCache) >> ___hotRodTopologyCache: Preparing cache view CacheView{viewId=4, >> members=[NodeA-63227, NodeB-15806]}, committed view is CacheView{viewId=3, >> members=[NodeA-63227, NodeB-15806, NodeC-17654]} >> … >> 2011-10-24 14:36:09,047 4222 DEBUG >> [org.infinispan.statetransfer.StateTransferLockImpl] >> (OOB-1,Infinispan-Cluster,NodeB-15806:___hotRodTopologyCache) Blocking new >> transactions >> 2011-10-24 14:36:09,047 4222 TRACE >> [org.infinispan.statetransfer.StateTransferLockImpl] >> (OOB-1,Infinispan-Cluster,NodeB-15806:___hotRodTopologyCache) Acquiring >> exclusive state transfer shared lock, shared holders: 0 >> 2011-10-24 14:36:09,047 4222 TRACE >> [org.infinispan.statetransfer.StateTransferLockImpl] >> (OOB-1,Infinispan-Cluster,NodeB-15806:___hotRodTopologyCache) Acquired state >> transfer lock in exclusive mode >> >> 2. The cluster coordinator discovers a view change and requests NodeA and >> NodeB to remove NodeC from the topology view: >> 2011-10-24 14:36:09,048 4223 TRACE >> [org.infinispan.interceptors.InvocationContextInterceptor] >> (OOB-3,Infinispan-Cluster,NodeB-15806:___hotRodTopologyCache) Invoked with >> command RemoveCommand{key=NodeC-17654, value=null, flags=null} and >> InvocationContext [NonTxInvocationContext{flags=null}] >> >> 3. NodeB has not yet finished installing the cache view, so that remove >> times out: >> 2011-10-24 14:36:09,049 4224 ERROR >> [org.infinispan.interceptors.InvocationContextInterceptor] >> (OOB-3,Infinispan-Cluster,NodeB-15806:___hotRodTopologyCache) ISPN000136: >> Execution error >> org.infinispan.distribution.RehashInProgressException: Timed out waiting for >> the transaction lock >> >> A way to solve this is to avoid relying on cluster view changes, but instead >> wait for the cache view to be installed, and then do the operations then. Is >> there any way to wait till then? >> >> One way would be to have some CacheView installed callbacks or similar. This >> could be a good option cos I could have a CacheView listener for the hot rod >> topology cache whose callbacks I can check for isPre=false and then do the >> cache ops safely. >>
Initially I was thinking of allowing multiple cache view listeners for each cache and making StateTransferManager one of them but I decided against it because I realized it needs a different interface than our regular listeners. I know that it was only a matter of time until someone needed it... An alternative solution would be to retry all operations, like we do with commits now, when we receive a RehashInProgressException exception from the remote node. That's what I was planning to do first as it helps in other use cases as well. >> Otherwise, code like this the one I used for keeping the Hot Rod topology is >> gonna be racing against your cache view installation code. >> >> You seem to have some pieces in place for this, i.e. CacheViewListener, but >> it seems only designed for internal core/ work. >> >> Any other suggestions? >> >> Cheers, >> -- >> Galder Zamarreño >> Sr. Software Engineer >> Infinispan, JBoss Cache >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
