Hi Dharam- I was pinged by the team with questions about Apache Geode cache transactions inside a *Spring* application context, specifically using SDG to configure and enable Apache Geode's cache transaction management capabilities, i.e. by using the Geode's o.a.g.cache.CacheTransactionManager [1] under-the-hood, wrapped conveniently and elegantly in *Spring's Transaction Management* [2] infrastructure, and SDG's extension [3] of that to handle Geode cache transactions in @Transactional application service methods.\
Anyway, I filed a JIRA ticket [4] (DATAGEODE-153 <https://jira.spring.io/browse/DATAGEODE-153> - "*Add test asserting 2 concurrent threads entering a transaction on the same entry leads to the proper outcome*" [4]) and wrote the corresponding test class <https://github.com/spring-projects/spring-data-geode/blob/master/src/test/java/org/springframework/data/gemfire/transaction/CommitConflictExceptionTransactionalIntegrationTests.java> [5] containing a test case asserting the proper behavior in this scenario. It is not unlike the code snippet *Swapnil* shared with you, but is a bit more sophisticated. This class makes use of the *MultithreadedTC Test Framework* [6] for properly coordinate the interactions between 2 or more threads. Though, no longer maintained, it provides a much improved experience over using Barriers, Latches along with other threading facilities and primitives provided in java.util.concurrent, IMO. Have a look at the test and if you have any questions, let me know. This test class demonstrates all is well, as expected. Regards, John [1] http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/CacheTransactionManager.html [2] https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction [3] https://docs.spring.io/spring-data/geode/docs/current/reference/html/#apis:transaction-management [4] https://jira.spring.io/browse/DATAGEODE-153 [5] https://github.com/spring-projects/spring-data-geode/blob/master/src/test/java/org/springframework/data/gemfire/transaction/CommitConflictExceptionTransactionalIntegrationTests.java [6] https://www.cs.umd.edu/projects/PL/multithreadedtc/overview.html On Thu, Oct 25, 2018 at 1:39 PM, Eric Shu <[email protected]> wrote: > After some digging, I find out Exception was thrown by Geode and then > wrapped and re-thrown by Spring data. > > // Let's create 2 callables and try to update same copy of > customer > > > // Update address from A1 to A2 > > Callable<String> t1 = new Callable<String>() { > > > @Override > > public String call() throws Exception { > > try { > > service.updateCustomer("1234", > "A2"); > > } catch (Exception e) { > > System.out.println("update A2 > failed with Exception " + e); > > throw e; > > } > > System.out.println("t1 done"); > > return "T1 Done"; > > } > > }; > > > // Update address from A1 to A3 > > Callable<String> t2 = new Callable<String>() { > > > @Override > > public String call() throws Exception { > > try { > > service.updateCustomer("1234", > "A3"); > > } catch (Exception e) { > > System.out.println("update A3 > failed with Exception " + e); > > throw e; > > } > > System.out.println("t2 done"); > > return "T2 Done"; > > } > > }; > > Here is the logging shown after run the example. > update A3 failed with Exception org.springframework.data. > gemfire.transaction.GemfireTransactionCommitException: Unexpected failure > occurred on commit of local cache transaction; nested exception is > org.apache.geode.cache.CommitConflictException: Concurrent transaction > commit detected The key 1234 in region /Customers was being modified by > another transaction locally. > > Based on this, I think there is no issue in spring data or geode for your > test case. > > -Eric > > > On Mon, Oct 22, 2018 at 4:52 PM Eric Shu <[email protected]> wrote: > >> Hi Dharam, >> >> Only <pool-3-thread-1> tid=0x4b got the lock and apply its changes >> to Geode. >> >> [debug 2018/10/19 09:36:48.608 IST <pool-3-thread-1> tid=0x4b] txApplyPut >> cbEvent=EntryEventImpl[op=UPDATE;region=/Customers;key=1234;oldValue=null;newValue=com.example.geode.model.Customer@3491dea3;callbackArg=null;originRemote=false;originMember=192.168.31.62(SampleServer:9006)<ec><v0>:1024;id=EventIDid=30bytes;threadID=3;sequenceID=1]] >> >> <pool-3-thread-2> tid=0x4c did not get the lock and transaction failed with >> CommitConflictException: >> >> [debug 2018/10/19 09:36:48.609 IST <pool-3-thread-2> tid=0x4c] >> <DLockRequestProcessor 13 waiting for 1 replies from >> [192.168.31.62(SampleServer:9006)<ec><v0>:1024]> got >> process(DLockRequestProcessor.DLockResponseMessage responding >> TRY_LOCK_FAILED; serviceName=DTLS(version 0); objectName=[TXLockBatch: >> txLockId=TXLockId: 192.168.31.62(SampleServer:9006)<ec><v0>:1024-3; >> reqs=[regionPath=/Customers keys=[1234]]; participants=[]]; responseCode=4; >> keyIfFailed=The key 1234 in region /Customers was being modified by >> another transaction locally.; leaseExpireTime=0; processorId=13; lockId=13) >> from 192.168.31.62(SampleServer:9006)<ec><v0>:1024 >> >> We do not see the txApplyPut in the logs for thread-2 which means the >> transaction does not write to cache. >> >> The following code snip shows that transaction is failed with >> CommitConflictException in TXLockServiceImpl.txLock() >> >> logger.debug("[TXLockServiceImpl.txLock] acquire try-locks for {}", >> batch); >> >> >> // TODO: get a readWriteLock to make the following block atomic... >> >> Object[] keyIfFail = new Object[1]; >> >> gotLocks = this.dlock.acquireTryLocks(batch, TIMEOUT_MILLIS, >> LEASE_MILLIS, keyIfFail); >> >> if (gotLocks) { // ...otherwise race can occur between tryLocks and >> readLock >> >> acquireRecoveryReadLock(); >> >> } else if (keyIfFail[0] != null) { >> >> --> throw new CommitConflictException( >> >> String.format("Concurrent transaction commit detected %s", >> >> keyIfFail[0])); >> >> } else { >> >> throw new CommitConflictException( >> >> String.format("Failed to request try locks from grantor: %s", >> >> this.dlock.getLockGrantorId())); >> >> >> I will try to find out why your application does not get the >> CommitConflictException thrown by Geode. >> >> -Eric >> >> >> >> On Sat, Oct 20, 2018 at 8:10 AM Dharam Thacker <[email protected]> >> wrote: >> >>> Hello Swapil & Team, >>> >>> Did you get chance to look into this? Ideally it's a plain case of >>> *optimistic >>> locking*. It works exactly fine with *JPA using @Version* property and >>> *optimistic >>> locking*. [Same example by changing Customer model to be compatible >>> with database table] >>> >>> As I mentioned that *both transactions are working exactly together* >>> reading first cut copy of Customer and still passing. I believe in geode, >>> we already have optimistic locking implemented in transactions. Could you >>> validate debug logs and/or POC project which I had send earlier? >>> >>> It's similar to below customer bank account analogy I have account with >>> bank balance as 0 initially. >>> >>> Tx1: When I started, customer balance was 0, so let me add $500 and end >>> result will be $500 >>> >>> Tx2: When I started, customer balance was 0, so let me add $100 and end >>> result will be $100 . >>> >>> Both transactions start at the same time, they read customer account >>> object from cache, with balance as 0 initially which is nothing but a >>> snapshot copy when transaction started. >>> But at least 1 of the transaction should be proved wrong while >>> committing. That's not case here and it's becoming $100. >>> >>> *Note*: 1 second of delay which I introduced to simulate this scenario >>> in my poc project is just to simulate external service call which makes >>> sense for real life project. >>> >>> Thanks, >>> - Dharam Thacker >>> >>> >>> On Fri, Oct 19, 2018 at 9:45 AM Dharam Thacker < >>> [email protected]> wrote: >>> >>>> Thank you Swapnil! I understand your point. If I start threads as you >>>> mentioned it does work. >>>> >>>> But if you see my attachment, both transactions *actually start >>>> together with thread pool and reads same copy of customer from cache* >>>> as I have added 1 second of delay in update service as soon as they read >>>> the copy of customer from cache. >>>> >>>> I have attached *DEBUG level logs* *in file(Debug_Logs.txt)* as well >>>> more information. I am sure something is not right here. >>>> >>>> After that I was expecting 1 transaction to win as other thread has >>>> definitely older copy of customer data. Info logs shown below for quick >>>> information. >>>> >>>> *INFO Logs for quick glance:* >>>> >>>> pool-3-thread-2 : Entered into update method >>>> pool-3-thread-1 : Entered into update method >>>> pool-3-thread-2 : CurrentCopy >> { >>>> "cid" : "1234", >>>> *"name" : "A1",* >>>> "address" : "India" >>>> } >>>> pool-3-thread-1 : CurrentCopy >> { >>>> "cid" : "1234", >>>> *"name" : "A1",* >>>> "address" : "India" >>>> } >>>> pool-3-thread-2 : Current customer name >> A1 >>>> pool-3-thread-1 : Current customer name >> A1 >>>> pool-3-thread-2 : Final Data >> { >>>> "cid" : "1234", >>>> "name" : "A3", >>>> "address" : "India" >>>> } >>>> pool-3-thread-1 : Final Data >> { >>>> "cid" : "1234", >>>> "name" : "A2", >>>> "address" : "India" >>>> } >>>> pool-3-thread-1 : Exiting from update method >>>> pool-3-thread-2 : Exiting from update method >>>> Main thread is going to wait for 5 seconds... >>>> Main thread is going to shutdown pool... >>>> >>>> >>>> Thanks, >>>> Dharam >>>> >>>> >>>> >>>> On Fri, Oct 19, 2018, 03:25 Swapnil Bawaskar <[email protected]> >>>> wrote: >>>> >>>>> Hi Dharam, >>>>> If there are two requests that read and write the same entry, you can >>>>> trust Geode to throw a CommitConflictException if the changes are going to >>>>> overwrite one another. My above example will throw this exception. >>>>> If that exception is not thrown, then your transactions actually >>>>> happened one after the other, in which case the second transaction should >>>>> be able to read the change made by the first transaction. >>>>> >>>>> >>>>> On Thu, Oct 18, 2018 at 11:30 AM Dharam Thacker < >>>>> [email protected]> wrote: >>>>> >>>>>> Thank you Swapnil! >>>>>> >>>>>> But how can we guarantee this? >>>>>> >>>>>> I have a spring boot geode client as web app. It has several >>>>>> emdpoints to accept add/update/delete actions. >>>>>> >>>>>> If 2 users are trying to change same customer with undefined email as >>>>>> initial state in 2 different rest requests in almost same time, of course >>>>>> by reading original customer with no email and changing emails, how could >>>>>> this be ensured? >>>>>> >>>>>> It may happen that user1's transaction succeded due to many possible >>>>>> reasons, it's confirmed now that user2 has stale data and he will >>>>>> overwtite >>>>>> changes. >>>>>> >>>>>> Could you suggest how we can handle it? >>>>>> >>>>>> Thanks, >>>>>> Dharam >>>>>> >>>>>> On Thu, Oct 18, 2018, 23:46 Swapnil Bawaskar <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> If you just throw two transactions in a thread pool, you may or may >>>>>>> not get an exception since one transaction may have completed before the >>>>>>> other transaction begins. To deterministically make one transaction >>>>>>> fail, >>>>>>> you have to ensure that both threads have read the initial value and >>>>>>> have >>>>>>> made change to the original value before either transaction commits. >>>>>>> >>>>>>> region.put("K", "V"); >>>>>>> >>>>>>> CountDownLatch latch1 = new CountDownLatch(1); >>>>>>> CountDownLatch latch2 = new CountDownLatch(1); >>>>>>> Thread th1 = new Thread(() -> { >>>>>>> mgr.being(); >>>>>>> region.put("K", "V1"); >>>>>>> latch1.countDown(); >>>>>>> latch2.await(); >>>>>>> mgr.commit(); >>>>>>> >>>>>>> }); >>>>>>> Thread th2 = new Thread(() -> { >>>>>>> mgr.begin(); >>>>>>> region.put("K", "V2"); >>>>>>> latch1.await(); >>>>>>> latch2.countDown(); >>>>>>> mgr.commit(); >>>>>>> }); >>>>>>> th1.start(); >>>>>>> th2.start(); >>>>>>> >>>>>>> >>>>>>> On Thu, Oct 18, 2018 at 10:28 AM Dharam Thacker < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hello Team, >>>>>>>> >>>>>>>> I am trying to understand *transaction behavior* with apache >>>>>>>> geode. I have created a simple demo as attached here in with this >>>>>>>> email. >>>>>>>> It's a simple spring data geode application with apache geode 1.6.0 >>>>>>>> >>>>>>>> Spring Startup Listener class >> SimpleApplicationListener .java >>>>>>>> Service class >> CustomerService.java >>>>>>>> >>>>>>>> 2 thread are simultaneously trying to update customer=1234 by >>>>>>>> changing name from [Thread-T1 - (From A1->To A2)] and [Thread-T2 - >>>>>>>> (From >>>>>>>> A1->To A3)] >>>>>>>> I assume that at least 1 thread has stale copy of data and should >>>>>>>> throw commit conflict exception but I think I am missing something and >>>>>>>> it's >>>>>>>> not behaving as expected. >>>>>>>> >>>>>>>> Could some one help me to understand what I am missing here? >>>>>>>> [Attached tar.gz file] >>>>>>>> >>>>>>>> Thanks & Regards, >>>>>>>> - Dharam Thacker >>>>>>>> >>>>>>> -- -John john.blum10101 (skype)
