Hey, If you guys have time we can discuss this over a call? On Tue, Oct 26, 2021 at 6:09 AM VICTOR MANUEL ROMERO RODRIGUEZ < [email protected]> wrote:
> Hello, > > I agree that a key must be used on the server side (Consumer) to avoid any > duplicate record, most of the time I used to look at the Software Patterns > like the Martin Fowler's articles: > > > https://martinfowler.com/articles/patterns-of-distributed-systems/idempotent-receiver.html > > > The pattern described there is more complex than the Saransh's proposal > (which is a good starting point), there are many approaches to add this > feature (Idempotency) on Apache Fineract, which I think should be flexible > enough to handle business rules. > > Idempotency can be at different levels (even using the POST/PUT), example: > > At DB level: > > The POST can be executed to create a new customer and the DB table has as > a key or unique the email used in the account creating, then a duplicated > entry at DB level should be thrown and the API Rest should send this error. > Then this API Rest is Idempotent because at DB level there is a restriction. > > At API REST per business rule level: > > The POST for a login should be idempotent because only one session is > created at a time per user/customer and in case of X number of failed > retries the account must be locked for a period of time or unlocked by a > manual/automatic process. Once the session is created, if a POST for login > sends a 200 status, then for a period of time the possibility to have > another login must be blocked until the current active session expires > (expiry time is set by the digital delivery channel owner like Internet or > Mobile banking). > > At API REST per consistency rule level: > > The POST for doing a payment (third party same bank or interbank transfer) > the idempotency key must be used for keeping the track of the transaction > status, holding the amount, settlement or reversal of the amount, digital > receipt, digital invoice generation, accounting record, etc) in order to > avoid duplication of charges in the origin account. Even with > confirmation/warning pop ups/windows errors occur. So then the handle of > the idempotency key must be handled and implemented in the Front end > application, the Server (and the internal consumer). > > And by the way the transactions must be done at UTC time, not at TZ > (another change in the Apache Fineract's tables should be done for the > offset time zone recording) . > > Best regards > > Victor > > El sáb, 23 oct 2021 a las 0:05, Saransh Sharma (<[email protected]>) > escribió: > >> This is a curious problem in computer science and especially in >> distributed networks. >> >> API Idempotency is a major issue for software like Fineract, unless the >> API handles drop off and recognize the second call or any n call as either >> duplicate or important in any given case, like how stripe does this by >> specifying a key from the client end ensures that even in case of failure >> the charge does not happen twice! >> >> This can lead to financial loss in case of real payment taking place! >> >> I think, as of right now there is no such mechanism in Fineract 1.x that >> eliminates such risk. Except, on the validation side, but that’s not >> enough. >> >> To have this implemented we have done some work on this last year , maybe >> we can present some code also ! >> >> I have used some cryptography libs to have the client-side send a unique >> key that’s generated by the private key infrastructure and the client-side >> has to send these as headers and on the server-side, the signatures are >> validated thus making the idempotency in POST and PUT request. >> >> The code on the client side >> >> var keys = { >> alice: '38f93bdda21a5c4a7bae4eb75bb7811cbc3eb627176805c1009ff2099263c6ad', >> bob: '09880c962437080d72f72c8c63a69efd65d086c9e7851a87b76373eb6ce9aab5'}; >> // GET >> for(k in keys) { >> var url = 'APIURL'; >> var dataToSign = url; >> var options = { >> url: url, >> headers: { >> 'x-identity': auth.getPublicKeyFromPrivateKey(keys[k]), >> 'x-signature': auth.sign(dataToSign, keys[k]) >> } >> }; >> >> >> The above code generates these Keys are now passed on each request to the >> server, and the server using the middleware in case of Fineract have to do >> two things >> >> 1. VerifySignature >> 2. GetSinFromThePublicKey >> >> I started writing some code for this and quickly realized that it's >> actually doable, it requires around 20 hours of work, There are some >> dependency that needs to be added on the server side and some quick changes >> on the middleware and its done. >> >> We presented a paper on this last year on Apache Con ! Remember something >> about using zero knowledge of proof and password less auth . >> >> Let me know , I would like to make sure that this gets added in this >> upcoming release. >> >> >> >> >> >> On Friday, October 22, 2021, James Dailey <[email protected]> wrote: >> >>> Devs - >>> >>> Through a conversation with a computer science person, I've learned that >>> Fineract should have a clearly articulated approach with regard to >>> idempotency. This is particularly true when thinking about how fineract >>> interacts with payment systems. >>> >>> "When building a system to move money, it is paramount that operations >>> that move money are idempotent. Failure to do this might result in >>> accidentally double charging your customer, or paying a vendor multiple >>> times. The risk of this happening is elevated based on the way software is >>> typically built today, since developers take advantage of scalable systems >>> that process multiple items in parallel. " >>> ( >>> https://www.moderntreasury.com/journal/why-idempotency-matters-in-payments >>> ) >>> >>> Or this article on Stripe engineering calling for liberal use of >>> Idempotency: >>> >>> "The easiest way to address inconsistencies in distributed state caused >>> by failures is to implement server endpoints so that they’re idempotent, >>> which means that they can be called any number of times while guaranteeing >>> that side effects only occur once." ( >>> https://stripe.com/blog/idempotency) >>> >>> and this article in Apache Camel >>> >>> https://camel.apache.org/camel-kafka-connector/latest/user-guide/idempotency.html >>> >>> So, my understanding may be wrong and we've got this covered, or maybe >>> we just consider this overkill. Perhaps there is something in the "magic" >>> that's developed in the CQRS plus in-memory database queuing and our clever >>> transaction state management, but... I don't know. >>> >>> Or perhaps we haven't really had to think about this because we haven't >>> had that many large scale deployments and the scalability issues associated >>> with multiple parallel calls haven't come to the fore, or haven't come back >>> to the list. So, that's the intent of this email. >>> >>> Do we have a need now for layering on Idempotency checks? What is the >>> right approach? Has anyone already implemented this, or decided not to for >>> very good reasons? What would be the right approach to this? Implement >>> where? Time limited to 3 hrs? Etc... >>> >>> So, this is a call to discuss. I'm far from an expert on this. >>> >>> >>> -- Saransh Sharma *Research Partner* *Muellner Internet Pvt Ltd * ---------------------------------------------------------------------------------------------- The idea of separation of me and you is amazing. ---------------------------------------------------------------------------------------------- *Company Website <https://www.muellners.com/> **Company Linkedin <https://www.linkedin.com/company/muellners> *Company Facebook <https://www.facebook.com/muellners> This mail is governed by Muellner® Internet Private Limited's IT policy. The information contained in this e-mail and any accompanying documents may contain information that is confidential or otherwise protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message, including any attachments. Any dissemination, distribution or other use of the contents of this message by anyone other than the intended recipient is strictly prohibited. All messages sent to and from this e-mail address may be monitored as permitted by applicable law and regulations to ensure compliance with our internal policies and to protect our business. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, lost or destroyed, or contain viruses. You are deemed to have accepted these risks if you communicate with us by e-mail.
