Hello James, I was thinking about the statement "we haven't had that many large scale deployments and the scalability issues associated with multiple parallel calls haven't come to the fore, or haven't come back to the list."
I remember that we faced issues with load testing on savings accounts BUT this is not related to Idempotency (in fact I think that the concurrency is handled in Fineract with the Optimistic logic, I share some line of sample code already in Fineract), so then *****Idempotency is not about large scale deployment nor scalability issues****. Idempotency should not be confused with issues related to concurrency, scalability nor high volume of transactions per second. I would propose another way to view the "Idempotency Concern" on Apache Fineract: "Strategy for design and building a Fault Tolerant API Rest Layer for non safe and/or non idempotent Http methods in Apache Fineract" (it could be shorter or fancier like "Apache Fineract Idempotency" :P ) As reference Safe and Idempotent comes from the http 1.1 spec https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html HTTP MethodIdempotentSafe OPTIONS yes yes GET yes yes HEAD yes yes PUT yes no POST no no DELETE yes no PATCH no no Let's talk then, Saransh and everyone interested in the topic. Well... just the code about Txn Concurrency Management in Apache Fineract https://github.com/apache/fineract/blob/develop/fineract-provider/src/main/java/org/apache/fineract/portfolio/savings/api/SavingsAccountTransactionsApiResource.java "... } catch (ObjectOptimisticLockingFailureException lockingFailureException) { throw new PlatformDataIntegrityException("error.msg.savings.concurrent.operations", "Concurrent Transactions being made on this savings account: " + lockingFailureException.getMessage(), lockingFailureException); } catch (CannotAcquireLockException cannotAcquireLockException) { throw new PlatformDataIntegrityException("error.msg.savings.concurrent.operations.unable.to.acquire.lock", "Unable to acquir lock for this transaction: " + cannotAcquireLockException.getMessage(), cannotAcquireLockException); } ..." El lun, 25 oct 2021 a las 23:37, VICTOR MANUEL ROMERO RODRIGUEZ (< [email protected]>) escribió: > Saransh > > Thanks, I'll be there. > > Regards > > Victor > > El lun, 25 oct 2021 a las 23:37, Saransh Sharma (<[email protected]>) > escribió: > >> Apache Fineract Dev Discussion >> Thursday, 28 Oct · 9:50 – 10:50 AM >> Google Meet joining info >> Video call link: https://meet.google.com/pdq-ybcs-adf >> Or dial: +1 617-675-4444 PIN: 1480810204623 >> More phone numbers: https://tel.meet/pdq-ybcs-adf?pin=1480810204623 >> >> On Tuesday, October 26, 2021, Saransh Sharma <[email protected]> >> wrote: >> >>> Let’s do that let me know your availability we can stream it also ! >>> >>> Let me know your preferred platform ! >>> >>> >>> >>> On Tuesday, October 26, 2021, VICTOR MANUEL ROMERO RODRIGUEZ < >>> [email protected]> wrote: >>> >>>> Saransh >>>> >>>> I would like to have a discussion, if we can record it, publish it in >>>> youtube for example and share the link here again, in order to allow the >>>> community to be in the loop. >>>> >>>> Does it work? >>>> >>>> Regards >>>> >>>> El lun, 25 oct 2021 a las 22:37, Saransh Sharma (<[email protected]>) >>>> escribió: >>>> >>>>> Timestamping could also be added as part of the key. >>>>> >>>>> I agree with Victor here, we need to keep it simple, Hey @Victor >>>>> Romero <[email protected]> would you like to discuss this >>>>> and flesh it out together ? >>>>> >>>>> On Tue, Oct 26, 2021 at 9:01 AM VICTOR MANUEL ROMERO RODRIGUEZ < >>>>> [email protected]> wrote: >>>>> >>>>>> I will extend the previous email, >>>>>> >>>>>> At asynchronous transactions: >>>>>> >>>>>> The trigger (or POST) for doing asynchronous transactions while >>>>>> sending notifications (email, sms, push) over MQ/JMS queues, the >>>>>> user/customer experience will be affected by receiving twice or more >>>>>> payment reminders, payment notification (they can think that the payment >>>>>> was applied twice), account creations, loan applications, etc and the >>>>>> cost >>>>>> associated for each message sent over the notification channel. >>>>>> >>>>>> At journey entry: >>>>>> >>>>>> The trigger (or POST) for doing any transactions (monetary or non >>>>>> monetary) only 1 entry must be recorded in all the systems so then the >>>>>> logging and monitoring systems (fraud, alerting, etc) can correlate the >>>>>> transaction (using surrogate keys) and later the data scientist or data >>>>>> engineers won't be blaming the CBS engineering team. >>>>>> >>>>>> At transaction caching: >>>>>> >>>>>> We got a use case where the loan origination process was taking some >>>>>> time to execute the process and because of the demand of the financial >>>>>> product people was trying to execute several times to have a loan, so >>>>>> then >>>>>> the business requirement was to keep on cache the same "loan request" if >>>>>> the people ID was the same (in our case the National Population ID) and >>>>>> not >>>>>> executing the query to the external systems (Credit Bureau, Black List >>>>>> had >>>>>> a high cost per each query). So then we saved execution time and money. >>>>>> >>>>>> ****TL;TR**** >>>>>> What I am trying to express is that the level, use cases, and user >>>>>> history (or whatever it can be the concept about is it use) have a wide >>>>>> range of "idempotent" implementations for the potential solutions should >>>>>> be >>>>>> evaluated carefully depending on them. >>>>>> >>>>>> I think that the ID of the "idempotent" key should have enough >>>>>> entropy to avoid any collision in the middleware, cache, db or messaging >>>>>> level and the recording of it must be at UTC time either if it handles >>>>>> synchronous or asynchronous API calls. >>>>>> >>>>>> ******** >>>>>> >>>>>> Best regards >>>>>> >>>>>> Victor >>>>>> >>>>>> >>>>>> El lun, 25 oct 2021 a las 19:39, VICTOR MANUEL ROMERO RODRIGUEZ (< >>>>>> [email protected]>) escribió: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I agree that a key must be used on the server side (Consumer) to >>>>>>> avoid any duplicate record, most of the time I used to look at the >>>>>>> Software >>>>>>> Patterns like the Martin Fowler's articles: >>>>>>> >>>>>>> >>>>>>> https://martinfowler.com/articles/patterns-of-distributed-systems/idempotent-receiver.html >>>>>>> >>>>>>> >>>>>>> The pattern described there is more complex than the >>>>>>> Saransh's proposal (which is a good starting point), there are many >>>>>>> approaches to add this feature (Idempotency) on Apache Fineract, which I >>>>>>> think should be flexible enough to handle business rules. >>>>>>> >>>>>>> Idempotency can be at different levels (even using the POST/PUT), >>>>>>> example: >>>>>>> >>>>>>> At DB level: >>>>>>> >>>>>>> The POST can be executed to create a new customer and the DB table >>>>>>> has as a key or unique the email used in the account creating, then a >>>>>>> duplicated entry at DB level should be thrown and the API Rest should >>>>>>> send >>>>>>> this error. Then this API Rest is Idempotent because at DB level there >>>>>>> is a >>>>>>> restriction. >>>>>>> >>>>>>> At API REST per business rule level: >>>>>>> >>>>>>> The POST for a login should be idempotent because only one session >>>>>>> is created at a time per user/customer and in case of X number of failed >>>>>>> retries the account must be locked for a period of time or unlocked by >>>>>>> a >>>>>>> manual/automatic process. Once the session is created, if a POST for >>>>>>> login >>>>>>> sends a 200 status, then for a period of time the possibility to have >>>>>>> another login must be blocked until the current active session expires >>>>>>> (expiry time is set by the digital delivery channel owner like Internet >>>>>>> or >>>>>>> Mobile banking). >>>>>>> >>>>>>> At API REST per consistency rule level: >>>>>>> >>>>>>> The POST for doing a payment (third party same bank or interbank >>>>>>> transfer) the idempotency key must be used for keeping the track of the >>>>>>> transaction status, holding the amount, settlement or reversal of the >>>>>>> amount, digital receipt, digital invoice generation, accounting record, >>>>>>> etc) in order to avoid duplication of charges in the origin account. >>>>>>> Even >>>>>>> with confirmation/warning pop ups/windows errors occur. So then the >>>>>>> handle >>>>>>> of the idempotency key must be handled and implemented in the Front end >>>>>>> application, the Server (and the internal consumer). >>>>>>> >>>>>>> And by the way the transactions must be done at UTC time, not at TZ >>>>>>> (another change in the Apache Fineract's tables should be done for the >>>>>>> offset time zone recording) . >>>>>>> >>>>>>> Best regards >>>>>>> >>>>>>> Victor >>>>>>> >>>>>>> El sáb, 23 oct 2021 a las 0:05, Saransh Sharma (< >>>>>>> [email protected]>) escribió: >>>>>>> >>>>>>>> This is a curious problem in computer science and especially in >>>>>>>> distributed networks. >>>>>>>> >>>>>>>> API Idempotency is a major issue for software like Fineract, >>>>>>>> unless the API handles drop off and recognize the second call or any n >>>>>>>> call as either duplicate or important in any given case, like how >>>>>>>> stripe >>>>>>>> does this by specifying a key from the client end ensures that even in >>>>>>>> case >>>>>>>> of failure the charge does not happen twice! >>>>>>>> >>>>>>>> This can lead to financial loss in case of real payment taking >>>>>>>> place! >>>>>>>> >>>>>>>> I think, as of right now there is no such mechanism in Fineract 1.x >>>>>>>> that eliminates such risk. Except, on the validation side, but that’s >>>>>>>> not >>>>>>>> enough. >>>>>>>> >>>>>>>> To have this implemented we have done some work on this last year , >>>>>>>> maybe we can present some code also ! >>>>>>>> >>>>>>>> I have used some cryptography libs to have the client-side send a >>>>>>>> unique key that’s generated by the private key infrastructure and the >>>>>>>> client-side has to send these as headers and on the server-side, the >>>>>>>> signatures are validated thus making the idempotency in POST and PUT >>>>>>>> request. >>>>>>>> >>>>>>>> The code on the client side >>>>>>>> >>>>>>>> var keys = { >>>>>>>> alice: >>>>>>>> '38f93bdda21a5c4a7bae4eb75bb7811cbc3eb627176805c1009ff2099263c6ad', >>>>>>>> bob: >>>>>>>> '09880c962437080d72f72c8c63a69efd65d086c9e7851a87b76373eb6ce9aab5'}; >>>>>>>> // GET >>>>>>>> for(k in keys) { >>>>>>>> var url = 'APIURL'; >>>>>>>> var dataToSign = url; >>>>>>>> var options = { >>>>>>>> url: url, >>>>>>>> headers: { >>>>>>>> 'x-identity': auth.getPublicKeyFromPrivateKey(keys[k]), >>>>>>>> 'x-signature': auth.sign(dataToSign, keys[k]) >>>>>>>> } >>>>>>>> }; >>>>>>>> >>>>>>>> >>>>>>>> The above code generates these Keys are now passed on each request >>>>>>>> to the server, and the server using the middleware in case of Fineract >>>>>>>> have to do two things >>>>>>>> >>>>>>>> 1. VerifySignature >>>>>>>> 2. GetSinFromThePublicKey >>>>>>>> >>>>>>>> I started writing some code for this and quickly realized that it's >>>>>>>> actually doable, it requires around 20 hours of work, There are some >>>>>>>> dependency that needs to be added on the server side and some quick >>>>>>>> changes >>>>>>>> on the middleware and its done. >>>>>>>> >>>>>>>> We presented a paper on this last year on Apache Con ! Remember >>>>>>>> something about using zero knowledge of proof and password less auth . >>>>>>>> >>>>>>>> Let me know , I would like to make sure that this gets added in >>>>>>>> this upcoming release. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Friday, October 22, 2021, James Dailey <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Devs - >>>>>>>>> >>>>>>>>> Through a conversation with a computer science person, I've >>>>>>>>> learned that Fineract should have a clearly articulated approach with >>>>>>>>> regard to idempotency. This is particularly true when thinking about >>>>>>>>> how >>>>>>>>> fineract interacts with payment systems. >>>>>>>>> >>>>>>>>> "When building a system to move money, it is paramount that >>>>>>>>> operations that move money are idempotent. Failure to do this might >>>>>>>>> result >>>>>>>>> in accidentally double charging your customer, or paying a vendor >>>>>>>>> multiple >>>>>>>>> times. The risk of this happening is elevated based on the way >>>>>>>>> software is >>>>>>>>> typically built today, since developers take advantage of scalable >>>>>>>>> systems >>>>>>>>> that process multiple items in parallel. " >>>>>>>>> ( >>>>>>>>> https://www.moderntreasury.com/journal/why-idempotency-matters-in-payments >>>>>>>>> ) >>>>>>>>> >>>>>>>>> Or this article on Stripe engineering calling for liberal use of >>>>>>>>> Idempotency: >>>>>>>>> >>>>>>>>> "The easiest way to address inconsistencies in distributed state >>>>>>>>> caused by failures is to implement server endpoints so that they’re >>>>>>>>> idempotent, which means that they can be called any number of times >>>>>>>>> while >>>>>>>>> guaranteeing that side effects only occur once." ( >>>>>>>>> https://stripe.com/blog/idempotency) >>>>>>>>> >>>>>>>>> and this article in Apache Camel >>>>>>>>> >>>>>>>>> https://camel.apache.org/camel-kafka-connector/latest/user-guide/idempotency.html >>>>>>>>> >>>>>>>>> So, my understanding may be wrong and we've got this covered, or >>>>>>>>> maybe we just consider this overkill. Perhaps there is something in >>>>>>>>> the >>>>>>>>> "magic" that's developed in the CQRS plus in-memory database queuing >>>>>>>>> and >>>>>>>>> our clever transaction state management, but... I don't know. >>>>>>>>> >>>>>>>>> Or perhaps we haven't really had to think about this because we >>>>>>>>> haven't had that many large scale deployments and the scalability >>>>>>>>> issues >>>>>>>>> associated with multiple parallel calls haven't come to the fore, or >>>>>>>>> haven't come back to the list. So, that's the intent of this email. >>>>>>>>> >>>>>>>>> Do we have a need now for layering on Idempotency checks? What is >>>>>>>>> the right approach? Has anyone already implemented this, or decided >>>>>>>>> not to >>>>>>>>> for very good reasons? What would be the right approach to this? >>>>>>>>> Implement where? Time limited to 3 hrs? Etc... >>>>>>>>> >>>>>>>>> So, this is a call to discuss. I'm far from an expert on this. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> >>>>> -- >>>>> >>>>> Saransh Sharma >>>>> *Research Partner* >>>>> *Muellner Internet Pvt Ltd * >>>>> >>>>> >>>>> ---------------------------------------------------------------------------------------------- >>>>> >>>>> The idea of separation of me and you is amazing. >>>>> >>>>> ---------------------------------------------------------------------------------------------- >>>>> *Company Website <https://www.muellners.com/> **Company Linkedin >>>>> <https://www.linkedin.com/company/muellners> * >>>>> Company Facebook <https://www.facebook.com/muellners> >>>>> >>>>> This mail is governed by Muellner® Internet Private Limited's IT >>>>> policy. >>>>> The information contained in this e-mail and any accompanying >>>>> documents may contain information that is confidential or otherwise >>>>> protected from disclosure. If you are not the intended recipient of this >>>>> message, or if this message has been addressed to you in error, please >>>>> immediately alert the sender by reply e-mail and then delete this message, >>>>> including any attachments. Any dissemination, distribution or other use of >>>>> the contents of this message by anyone other than the intended recipient >>>>> is >>>>> strictly prohibited. All messages sent to and from this e-mail address may >>>>> be monitored as permitted by applicable law and regulations to ensure >>>>> compliance with our internal policies and to protect our business. E-mails >>>>> are not secure and cannot be guaranteed to be error free as they can be >>>>> intercepted, amended, lost or destroyed, or contain viruses. You are >>>>> deemed >>>>> to have accepted these risks if you communicate with us by e-mail. >>>>> >>>> >>> >>> -- >>> Sent from Gmail Mobile >>> >> >> >> -- >> Sent from Gmail Mobile >> >
