Re: [infinispan-dev] Why DistTxInterceptor in use with Hot Rod ?

2011-09-16 Thread Galder Zamarreño 

On Sep 15, 2011, at 1:31 PM, Dan Berindei wrote:

 On Thu, Sep 15, 2011 at 12:25 PM, Galder Zamarreño gal...@redhat.com wrote:
 
 On Sep 14, 2011, at 11:40 AM, Dan Berindei wrote:
 
 Going back to your original question Galder, the exception is most
 likely thrown because of this sequence of events:
 
 0. Given a cluster {A, B}, a key k and a node C joining.
 1. Put acquires the transaction lock on node A (blocking rehashing)
 2. Put acquires lock for key k on node A
 3. Rehashing starts on node B, blocking transactions
 4. Put tries to acquire transaction lock on node B
 
 Since it's impossible to finish rehashing while the put operation
 keeps the transaction lock on node A, the best option was to kill the
 put operation by throwing a RehashInProgressException.
 
 I was thinking in the context of transactions when I wrote this code
 (see 
 https://github.com/danberindei/infinispan/commit/6ed94d3b2e184d4a48d4e781db8d404baf5915a3,
 this scenario became just a footnote to the generic case with multiple
 caches), but the scenario also occurs without transactions. Actually I
 renamed it state transfer lock and I moved it to a separate
 interceptor in my ISPN-1194 branch.
 
 Right, but it shouldn't happen when transactions are off, right?
 
 
 It will still happen with transactions off, because state transfer
 will acquire the state transfer lock in exclusive mode and
 WriteCommands will acquire the state transfer lock in shared mode
 instead of PrepareCommands with transactions on.
 
 If we don't acquire the state transfer lock than we run the risk of
 pushing stale data to other nodes, are you saying that without
 transactions enabled this risk is acceptable?

Hmmm, at first glance no, it does not look acceptable. Refactoring though, as 
you did, is in order though to move it somewhere else, cos clearly it's logic 
that's required by both transactional and non-transactional caches.

 
 
 Maybe the locking changes in 5.1 will eliminate this scenario, but
 otherwise we could improve the user experience by retrying the command
 after the rehashing finishes.
 
 I'd prefer that, otherwise users are likely to code similar logic which is 
 not nice.
 
 
 After looking at the scenario again it became clear that step 2. is
 not necessary at all and the locking enhancements will not change
 anything.
 
 We could change DistributionInterceptor to not hold the state transfer
 lock during remote calls, which will allow rehashing to proceed on the
 origin. But then the ownership of the keys might change during the
 call, so we'll need a retry phase anyway.
 
 Dan
 
 ___
 infinispan-dev mailing list
 infinispan-dev@lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache


___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Why DistTxInterceptor in use with Hot Rod ?

2011-09-15 Thread Galder Zamarreño 

On Sep 14, 2011, at 11:40 AM, Dan Berindei wrote:

 Going back to your original question Galder, the exception is most
 likely thrown because of this sequence of events:
 
 0. Given a cluster {A, B}, a key k and a node C joining.
 1. Put acquires the transaction lock on node A (blocking rehashing)
 2. Put acquires lock for key k on node A
 3. Rehashing starts on node B, blocking transactions
 4. Put tries to acquire transaction lock on node B
 
 Since it's impossible to finish rehashing while the put operation
 keeps the transaction lock on node A, the best option was to kill the
 put operation by throwing a RehashInProgressException.
 
 I was thinking in the context of transactions when I wrote this code
 (see 
 https://github.com/danberindei/infinispan/commit/6ed94d3b2e184d4a48d4e781db8d404baf5915a3,
 this scenario became just a footnote to the generic case with multiple
 caches), but the scenario also occurs without transactions. Actually I
 renamed it state transfer lock and I moved it to a separate
 interceptor in my ISPN-1194 branch.

Right, but it shouldn't happen when transactions are off, right?

 
 Maybe the locking changes in 5.1 will eliminate this scenario, but
 otherwise we could improve the user experience by retrying the command
 after the rehashing finishes.

I'd prefer that, otherwise users are likely to code similar logic which is not 
nice.

 
 Dan
 
 
 On Tue, Sep 13, 2011 at 8:11 PM, Galder Zamarreño gal...@redhat.com wrote:
 
 On Sep 13, 2011, at 6:38 PM, Mircea Markus wrote:
 
 
 On 13 Sep 2011, at 17:22, Galder Zamarreño wrote:
 
 Hi,
 
 I'm looking at this failure http://goo.gl/NQw4h and I'm wondering why 
 org.infinispan.distribution.RehashInProgressException: Timed out waiting 
 for the transaction lock is thrown?
 
 This is thrown DistTxInterceptor which is added by the 
 InterceptorChainFactory:
 
 if (configuration.getCacheMode().isDistributed())

 interceptorChain.appendInterceptor(createInterceptor(DistTxInterceptor.class));
 else

 interceptorChain.appendInterceptor(createInterceptor(TxInterceptor.class));
 
 However, this is a Hot Rod test and the cache is not configured with a 
 transaction manager, so is DistTxInterceptor really needed?
 
 In fact, why a TxInterceptor if no transaction manager is configured? 
 (this kinda goes back to Mircea's email earlier today about a transactions 
 enabled/disabled flag)
 All valid concerns, IMO.
 5.1 will clarify this by not adding Tx interceptors for non tx caches.
 
 Is this captured in some other JIRA? I'd like to reference the test and the 
 JIRA from ISPN-1123 (stabilising testsuite jira)
 
 
 
 Cheers,
 --
 Galder Zamarreño
 Sr. Software Engineer
 Infinispan, JBoss Cache
 
 
 ___
 infinispan-dev mailing list
 infinispan-dev@lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/infinispan-dev
 
 
 ___
 infinispan-dev mailing list
 infinispan-dev@lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/infinispan-dev
 
 --
 Galder Zamarreño
 Sr. Software Engineer
 Infinispan, JBoss Cache
 
 
 ___
 infinispan-dev mailing list
 infinispan-dev@lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/infinispan-dev
 
 
 ___
 infinispan-dev mailing list
 infinispan-dev@lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache


___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Why DistTxInterceptor in use with Hot Rod ?

2011-09-15 Thread Dan Berindei 
On Thu, Sep 15, 2011 at 12:25 PM, Galder Zamarreño gal...@redhat.com wrote:

 On Sep 14, 2011, at 11:40 AM, Dan Berindei wrote:

 Going back to your original question Galder, the exception is most
 likely thrown because of this sequence of events:

 0. Given a cluster {A, B}, a key k and a node C joining.
 1. Put acquires the transaction lock on node A (blocking rehashing)
 2. Put acquires lock for key k on node A
 3. Rehashing starts on node B, blocking transactions
 4. Put tries to acquire transaction lock on node B

 Since it's impossible to finish rehashing while the put operation
 keeps the transaction lock on node A, the best option was to kill the
 put operation by throwing a RehashInProgressException.

 I was thinking in the context of transactions when I wrote this code
 (see 
 https://github.com/danberindei/infinispan/commit/6ed94d3b2e184d4a48d4e781db8d404baf5915a3,
 this scenario became just a footnote to the generic case with multiple
 caches), but the scenario also occurs without transactions. Actually I
 renamed it state transfer lock and I moved it to a separate
 interceptor in my ISPN-1194 branch.

 Right, but it shouldn't happen when transactions are off, right?


It will still happen with transactions off, because state transfer
will acquire the state transfer lock in exclusive mode and
WriteCommands will acquire the state transfer lock in shared mode
instead of PrepareCommands with transactions on.

If we don't acquire the state transfer lock than we run the risk of
pushing stale data to other nodes, are you saying that without
transactions enabled this risk is acceptable?


 Maybe the locking changes in 5.1 will eliminate this scenario, but
 otherwise we could improve the user experience by retrying the command
 after the rehashing finishes.

 I'd prefer that, otherwise users are likely to code similar logic which is 
 not nice.


After looking at the scenario again it became clear that step 2. is
not necessary at all and the locking enhancements will not change
anything.

We could change DistributionInterceptor to not hold the state transfer
lock during remote calls, which will allow rehashing to proceed on the
origin. But then the ownership of the keys might change during the
call, so we'll need a retry phase anyway.

Dan

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Why DistTxInterceptor in use with Hot Rod ?

2011-09-14 Thread Dan Berindei 
Going back to your original question Galder, the exception is most
likely thrown because of this sequence of events:

0. Given a cluster {A, B}, a key k and a node C joining.
1. Put acquires the transaction lock on node A (blocking rehashing)
2. Put acquires lock for key k on node A
3. Rehashing starts on node B, blocking transactions
4. Put tries to acquire transaction lock on node B

Since it's impossible to finish rehashing while the put operation
keeps the transaction lock on node A, the best option was to kill the
put operation by throwing a RehashInProgressException.

I was thinking in the context of transactions when I wrote this code
(see 
https://github.com/danberindei/infinispan/commit/6ed94d3b2e184d4a48d4e781db8d404baf5915a3,
this scenario became just a footnote to the generic case with multiple
caches), but the scenario also occurs without transactions. Actually I
renamed it state transfer lock and I moved it to a separate
interceptor in my ISPN-1194 branch.

Maybe the locking changes in 5.1 will eliminate this scenario, but
otherwise we could improve the user experience by retrying the command
after the rehashing finishes.

Dan


On Tue, Sep 13, 2011 at 8:11 PM, Galder Zamarreño gal...@redhat.com wrote:

 On Sep 13, 2011, at 6:38 PM, Mircea Markus wrote:


 On 13 Sep 2011, at 17:22, Galder Zamarreño wrote:

 Hi,

 I'm looking at this failure http://goo.gl/NQw4h and I'm wondering why 
 org.infinispan.distribution.RehashInProgressException: Timed out waiting 
 for the transaction lock is thrown?

 This is thrown DistTxInterceptor which is added by the 
 InterceptorChainFactory:

     if (configuration.getCacheMode().isDistributed())
        
 interceptorChain.appendInterceptor(createInterceptor(DistTxInterceptor.class));
     else
        
 interceptorChain.appendInterceptor(createInterceptor(TxInterceptor.class));

 However, this is a Hot Rod test and the cache is not configured with a 
 transaction manager, so is DistTxInterceptor really needed?

 In fact, why a TxInterceptor if no transaction manager is configured? (this 
 kinda goes back to Mircea's email earlier today about a transactions 
 enabled/disabled flag)
 All valid concerns, IMO.
 5.1 will clarify this by not adding Tx interceptors for non tx caches.

 Is this captured in some other JIRA? I'd like to reference the test and the 
 JIRA from ISPN-1123 (stabilising testsuite jira)



 Cheers,
 --
 Galder Zamarreño
 Sr. Software Engineer
 Infinispan, JBoss Cache


 ___
 infinispan-dev mailing list
 infinispan-dev@lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/infinispan-dev


 ___
 infinispan-dev mailing list
 infinispan-dev@lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/infinispan-dev

 --
 Galder Zamarreño
 Sr. Software Engineer
 Infinispan, JBoss Cache


 ___
 infinispan-dev mailing list
 infinispan-dev@lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/infinispan-dev


___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

[infinispan-dev] Why DistTxInterceptor in use with Hot Rod ?

2011-09-13 Thread Galder Zamarreño 
Hi,

I'm looking at this failure http://goo.gl/NQw4h and I'm wondering why 
org.infinispan.distribution.RehashInProgressException: Timed out waiting for 
the transaction lock is thrown?

This is thrown DistTxInterceptor which is added by the InterceptorChainFactory:

  if (configuration.getCacheMode().isDistributed())
 
interceptorChain.appendInterceptor(createInterceptor(DistTxInterceptor.class));
  else
 
interceptorChain.appendInterceptor(createInterceptor(TxInterceptor.class));

However, this is a Hot Rod test and the cache is not configured with a 
transaction manager, so is DistTxInterceptor really needed?

In fact, why a TxInterceptor if no transaction manager is configured? (this 
kinda goes back to Mircea's email earlier today about a transactions 
enabled/disabled flag)

Cheers,
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache


___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Why DistTxInterceptor in use with Hot Rod ?

2011-09-13 Thread Mircea Markus 

On 13 Sep 2011, at 17:22, Galder Zamarreño wrote:

 Hi,
 
 I'm looking at this failure http://goo.gl/NQw4h and I'm wondering why 
 org.infinispan.distribution.RehashInProgressException: Timed out waiting for 
 the transaction lock is thrown?
 
 This is thrown DistTxInterceptor which is added by the 
 InterceptorChainFactory:
 
  if (configuration.getCacheMode().isDistributed())
 
 interceptorChain.appendInterceptor(createInterceptor(DistTxInterceptor.class));
  else
 
 interceptorChain.appendInterceptor(createInterceptor(TxInterceptor.class));
 
 However, this is a Hot Rod test and the cache is not configured with a 
 transaction manager, so is DistTxInterceptor really needed?
 
 In fact, why a TxInterceptor if no transaction manager is configured? (this 
 kinda goes back to Mircea's email earlier today about a transactions 
 enabled/disabled flag)
All valid concerns, IMO.
5.1 will clarify this by not adding Tx interceptors for non tx caches. 

 
 Cheers,
 --
 Galder Zamarreño
 Sr. Software Engineer
 Infinispan, JBoss Cache
 
 
 ___
 infinispan-dev mailing list
 infinispan-dev@lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/infinispan-dev


___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Why DistTxInterceptor in use with Hot Rod ?

2011-09-13 Thread Galder Zamarreño 

On Sep 13, 2011, at 6:38 PM, Mircea Markus wrote:

 
 On 13 Sep 2011, at 17:22, Galder Zamarreño wrote:
 
 Hi,
 
 I'm looking at this failure http://goo.gl/NQw4h and I'm wondering why 
 org.infinispan.distribution.RehashInProgressException: Timed out waiting 
 for the transaction lock is thrown?
 
 This is thrown DistTxInterceptor which is added by the 
 InterceptorChainFactory:
 
 if (configuration.getCacheMode().isDistributed())

 interceptorChain.appendInterceptor(createInterceptor(DistTxInterceptor.class));
 else

 interceptorChain.appendInterceptor(createInterceptor(TxInterceptor.class));
 
 However, this is a Hot Rod test and the cache is not configured with a 
 transaction manager, so is DistTxInterceptor really needed?
 
 In fact, why a TxInterceptor if no transaction manager is configured? (this 
 kinda goes back to Mircea's email earlier today about a transactions 
 enabled/disabled flag)
 All valid concerns, IMO.
 5.1 will clarify this by not adding Tx interceptors for non tx caches. 

Is this captured in some other JIRA? I'd like to reference the test and the 
JIRA from ISPN-1123 (stabilising testsuite jira)

 
 
 Cheers,
 --
 Galder Zamarreño
 Sr. Software Engineer
 Infinispan, JBoss Cache
 
 
 ___
 infinispan-dev mailing list
 infinispan-dev@lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/infinispan-dev
 
 
 ___
 infinispan-dev mailing list
 infinispan-dev@lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache


___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev