I will extend the previous email, At asynchronous transactions:
The trigger (or POST) for doing asynchronous transactions while sending notifications (email, sms, push) over MQ/JMS queues, the user/customer experience will be affected by receiving twice or more payment reminders, payment notification (they can think that the payment was applied twice), account creations, loan applications, etc and the cost associated for each message sent over the notification channel. At journey entry: The trigger (or POST) for doing any transactions (monetary or non monetary) only 1 entry must be recorded in all the systems so then the logging and monitoring systems (fraud, alerting, etc) can correlate the transaction (using surrogate keys) and later the data scientist or data engineers won't be blaming the CBS engineering team. At transaction caching: We got a use case where the loan origination process was taking some time to execute the process and because of the demand of the financial product people was trying to execute several times to have a loan, so then the business requirement was to keep on cache the same "loan request" if the people ID was the same (in our case the National Population ID) and not executing the query to the external systems (Credit Bureau, Black List had a high cost per each query). So then we saved execution time and money. ****TL;TR**** What I am trying to express is that the level, use cases, and user history (or whatever it can be the concept about is it use) have a wide range of "idempotent" implementations for the potential solutions should be evaluated carefully depending on them. I think that the ID of the "idempotent" key should have enough entropy to avoid any collision in the middleware, cache, db or messaging level and the recording of it must be at UTC time either if it handles synchronous or asynchronous API calls. ******** Best regards Victor El lun, 25 oct 2021 a las 19:39, VICTOR MANUEL ROMERO RODRIGUEZ (< [email protected]>) escribió: > Hello, > > I agree that a key must be used on the server side (Consumer) to avoid any > duplicate record, most of the time I used to look at the Software Patterns > like the Martin Fowler's articles: > > > https://martinfowler.com/articles/patterns-of-distributed-systems/idempotent-receiver.html > > > The pattern described there is more complex than the Saransh's proposal > (which is a good starting point), there are many approaches to add this > feature (Idempotency) on Apache Fineract, which I think should be flexible > enough to handle business rules. > > Idempotency can be at different levels (even using the POST/PUT), example: > > At DB level: > > The POST can be executed to create a new customer and the DB table has as > a key or unique the email used in the account creating, then a duplicated > entry at DB level should be thrown and the API Rest should send this error. > Then this API Rest is Idempotent because at DB level there is a restriction. > > At API REST per business rule level: > > The POST for a login should be idempotent because only one session is > created at a time per user/customer and in case of X number of failed > retries the account must be locked for a period of time or unlocked by a > manual/automatic process. Once the session is created, if a POST for login > sends a 200 status, then for a period of time the possibility to have > another login must be blocked until the current active session expires > (expiry time is set by the digital delivery channel owner like Internet or > Mobile banking). > > At API REST per consistency rule level: > > The POST for doing a payment (third party same bank or interbank transfer) > the idempotency key must be used for keeping the track of the transaction > status, holding the amount, settlement or reversal of the amount, digital > receipt, digital invoice generation, accounting record, etc) in order to > avoid duplication of charges in the origin account. Even with > confirmation/warning pop ups/windows errors occur. So then the handle of > the idempotency key must be handled and implemented in the Front end > application, the Server (and the internal consumer). > > And by the way the transactions must be done at UTC time, not at TZ > (another change in the Apache Fineract's tables should be done for the > offset time zone recording) . > > Best regards > > Victor > > El sáb, 23 oct 2021 a las 0:05, Saransh Sharma (<[email protected]>) > escribió: > >> This is a curious problem in computer science and especially in >> distributed networks. >> >> API Idempotency is a major issue for software like Fineract, unless the >> API handles drop off and recognize the second call or any n call as either >> duplicate or important in any given case, like how stripe does this by >> specifying a key from the client end ensures that even in case of failure >> the charge does not happen twice! >> >> This can lead to financial loss in case of real payment taking place! >> >> I think, as of right now there is no such mechanism in Fineract 1.x that >> eliminates such risk. Except, on the validation side, but that’s not >> enough. >> >> To have this implemented we have done some work on this last year , maybe >> we can present some code also ! >> >> I have used some cryptography libs to have the client-side send a unique >> key that’s generated by the private key infrastructure and the client-side >> has to send these as headers and on the server-side, the signatures are >> validated thus making the idempotency in POST and PUT request. >> >> The code on the client side >> >> var keys = { >> alice: '38f93bdda21a5c4a7bae4eb75bb7811cbc3eb627176805c1009ff2099263c6ad', >> bob: '09880c962437080d72f72c8c63a69efd65d086c9e7851a87b76373eb6ce9aab5'}; >> // GET >> for(k in keys) { >> var url = 'APIURL'; >> var dataToSign = url; >> var options = { >> url: url, >> headers: { >> 'x-identity': auth.getPublicKeyFromPrivateKey(keys[k]), >> 'x-signature': auth.sign(dataToSign, keys[k]) >> } >> }; >> >> >> The above code generates these Keys are now passed on each request to the >> server, and the server using the middleware in case of Fineract have to do >> two things >> >> 1. VerifySignature >> 2. GetSinFromThePublicKey >> >> I started writing some code for this and quickly realized that it's >> actually doable, it requires around 20 hours of work, There are some >> dependency that needs to be added on the server side and some quick changes >> on the middleware and its done. >> >> We presented a paper on this last year on Apache Con ! Remember something >> about using zero knowledge of proof and password less auth . >> >> Let me know , I would like to make sure that this gets added in this >> upcoming release. >> >> >> >> >> >> On Friday, October 22, 2021, James Dailey <[email protected]> wrote: >> >>> Devs - >>> >>> Through a conversation with a computer science person, I've learned that >>> Fineract should have a clearly articulated approach with regard to >>> idempotency. This is particularly true when thinking about how fineract >>> interacts with payment systems. >>> >>> "When building a system to move money, it is paramount that operations >>> that move money are idempotent. Failure to do this might result in >>> accidentally double charging your customer, or paying a vendor multiple >>> times. The risk of this happening is elevated based on the way software is >>> typically built today, since developers take advantage of scalable systems >>> that process multiple items in parallel. " >>> ( >>> https://www.moderntreasury.com/journal/why-idempotency-matters-in-payments >>> ) >>> >>> Or this article on Stripe engineering calling for liberal use of >>> Idempotency: >>> >>> "The easiest way to address inconsistencies in distributed state caused >>> by failures is to implement server endpoints so that they’re idempotent, >>> which means that they can be called any number of times while guaranteeing >>> that side effects only occur once." ( >>> https://stripe.com/blog/idempotency) >>> >>> and this article in Apache Camel >>> >>> https://camel.apache.org/camel-kafka-connector/latest/user-guide/idempotency.html >>> >>> So, my understanding may be wrong and we've got this covered, or maybe >>> we just consider this overkill. Perhaps there is something in the "magic" >>> that's developed in the CQRS plus in-memory database queuing and our clever >>> transaction state management, but... I don't know. >>> >>> Or perhaps we haven't really had to think about this because we haven't >>> had that many large scale deployments and the scalability issues associated >>> with multiple parallel calls haven't come to the fore, or haven't come back >>> to the list. So, that's the intent of this email. >>> >>> Do we have a need now for layering on Idempotency checks? What is the >>> right approach? Has anyone already implemented this, or decided not to for >>> very good reasons? What would be the right approach to this? Implement >>> where? Time limited to 3 hrs? Etc... >>> >>> So, this is a call to discuss. I'm far from an expert on this. >>> >>> >>>
