Apache Fineract Dev Discussion Thursday, 28 Oct · 9:50 – 10:50 AM Google Meet joining info Video call link: https://meet.google.com/pdq-ybcs-adf Or dial: +1 617-675-4444 PIN: 1480810204623 More phone numbers: https://tel.meet/pdq-ybcs-adf?pin=1480810204623
On Tuesday, October 26, 2021, Saransh Sharma <[email protected]> wrote: > Let’s do that let me know your availability we can stream it also ! > > Let me know your preferred platform ! > > > > On Tuesday, October 26, 2021, VICTOR MANUEL ROMERO RODRIGUEZ < > [email protected]> wrote: > >> Saransh >> >> I would like to have a discussion, if we can record it, publish it in >> youtube for example and share the link here again, in order to allow the >> community to be in the loop. >> >> Does it work? >> >> Regards >> >> El lun, 25 oct 2021 a las 22:37, Saransh Sharma (<[email protected]>) >> escribió: >> >>> Timestamping could also be added as part of the key. >>> >>> I agree with Victor here, we need to keep it simple, Hey @Victor Romero >>> <[email protected]> would you like to discuss this and >>> flesh it out together ? >>> >>> On Tue, Oct 26, 2021 at 9:01 AM VICTOR MANUEL ROMERO RODRIGUEZ < >>> [email protected]> wrote: >>> >>>> I will extend the previous email, >>>> >>>> At asynchronous transactions: >>>> >>>> The trigger (or POST) for doing asynchronous transactions while sending >>>> notifications (email, sms, push) over MQ/JMS queues, the user/customer >>>> experience will be affected by receiving twice or more payment >>>> reminders, payment notification (they can think that the payment was >>>> applied twice), account creations, loan applications, etc and the cost >>>> associated for each message sent over the notification channel. >>>> >>>> At journey entry: >>>> >>>> The trigger (or POST) for doing any transactions (monetary or non >>>> monetary) only 1 entry must be recorded in all the systems so then the >>>> logging and monitoring systems (fraud, alerting, etc) can correlate the >>>> transaction (using surrogate keys) and later the data scientist or data >>>> engineers won't be blaming the CBS engineering team. >>>> >>>> At transaction caching: >>>> >>>> We got a use case where the loan origination process was taking some >>>> time to execute the process and because of the demand of the financial >>>> product people was trying to execute several times to have a loan, so then >>>> the business requirement was to keep on cache the same "loan request" if >>>> the people ID was the same (in our case the National Population ID) and not >>>> executing the query to the external systems (Credit Bureau, Black List had >>>> a high cost per each query). So then we saved execution time and money. >>>> >>>> ****TL;TR**** >>>> What I am trying to express is that the level, use cases, and user >>>> history (or whatever it can be the concept about is it use) have a wide >>>> range of "idempotent" implementations for the potential solutions should be >>>> evaluated carefully depending on them. >>>> >>>> I think that the ID of the "idempotent" key should have enough >>>> entropy to avoid any collision in the middleware, cache, db or messaging >>>> level and the recording of it must be at UTC time either if it handles >>>> synchronous or asynchronous API calls. >>>> >>>> ******** >>>> >>>> Best regards >>>> >>>> Victor >>>> >>>> >>>> El lun, 25 oct 2021 a las 19:39, VICTOR MANUEL ROMERO RODRIGUEZ (< >>>> [email protected]>) escribió: >>>> >>>>> Hello, >>>>> >>>>> I agree that a key must be used on the server side (Consumer) to avoid >>>>> any duplicate record, most of the time I used to look at the Software >>>>> Patterns like the Martin Fowler's articles: >>>>> >>>>> https://martinfowler.com/articles/patterns-of-distributed- >>>>> systems/idempotent-receiver.html >>>>> >>>>> The pattern described there is more complex than the >>>>> Saransh's proposal (which is a good starting point), there are many >>>>> approaches to add this feature (Idempotency) on Apache Fineract, which I >>>>> think should be flexible enough to handle business rules. >>>>> >>>>> Idempotency can be at different levels (even using the POST/PUT), >>>>> example: >>>>> >>>>> At DB level: >>>>> >>>>> The POST can be executed to create a new customer and the DB table has >>>>> as a key or unique the email used in the account creating, then a >>>>> duplicated entry at DB level should be thrown and the API Rest should send >>>>> this error. Then this API Rest is Idempotent because at DB level there is >>>>> a >>>>> restriction. >>>>> >>>>> At API REST per business rule level: >>>>> >>>>> The POST for a login should be idempotent because only one session is >>>>> created at a time per user/customer and in case of X number of failed >>>>> retries the account must be locked for a period of time or unlocked by a >>>>> manual/automatic process. Once the session is created, if a POST for login >>>>> sends a 200 status, then for a period of time the possibility to have >>>>> another login must be blocked until the current active session expires >>>>> (expiry time is set by the digital delivery channel owner like Internet or >>>>> Mobile banking). >>>>> >>>>> At API REST per consistency rule level: >>>>> >>>>> The POST for doing a payment (third party same bank or interbank >>>>> transfer) the idempotency key must be used for keeping the track of the >>>>> transaction status, holding the amount, settlement or reversal of the >>>>> amount, digital receipt, digital invoice generation, accounting record, >>>>> etc) in order to avoid duplication of charges in the origin account. Even >>>>> with confirmation/warning pop ups/windows errors occur. So then the >>>>> handle >>>>> of the idempotency key must be handled and implemented in the Front end >>>>> application, the Server (and the internal consumer). >>>>> >>>>> And by the way the transactions must be done at UTC time, not at TZ >>>>> (another change in the Apache Fineract's tables should be done for the >>>>> offset time zone recording) . >>>>> >>>>> Best regards >>>>> >>>>> Victor >>>>> >>>>> El sáb, 23 oct 2021 a las 0:05, Saransh Sharma (<[email protected]>) >>>>> escribió: >>>>> >>>>>> This is a curious problem in computer science and especially in >>>>>> distributed networks. >>>>>> >>>>>> API Idempotency is a major issue for software like Fineract, unless >>>>>> the API handles drop off and recognize the second call or any n call as >>>>>> either duplicate or important in any given case, like how stripe does >>>>>> this >>>>>> by specifying a key from the client end ensures that even in case of >>>>>> failure the charge does not happen twice! >>>>>> >>>>>> This can lead to financial loss in case of real payment taking place! >>>>>> >>>>>> I think, as of right now there is no such mechanism in Fineract 1.x >>>>>> that eliminates such risk. Except, on the validation side, but that’s not >>>>>> enough. >>>>>> >>>>>> To have this implemented we have done some work on this last year , >>>>>> maybe we can present some code also ! >>>>>> >>>>>> I have used some cryptography libs to have the client-side send a >>>>>> unique key that’s generated by the private key infrastructure and the >>>>>> client-side has to send these as headers and on the server-side, the >>>>>> signatures are validated thus making the idempotency in POST and PUT >>>>>> request. >>>>>> >>>>>> The code on the client side >>>>>> >>>>>> var keys = { >>>>>> alice: >>>>>> '38f93bdda21a5c4a7bae4eb75bb7811cbc3eb627176805c1009ff2099263c6ad', >>>>>> bob: >>>>>> '09880c962437080d72f72c8c63a69efd65d086c9e7851a87b76373eb6ce9aab5'}; >>>>>> // GET >>>>>> for(k in keys) { >>>>>> var url = 'APIURL'; >>>>>> var dataToSign = url; >>>>>> var options = { >>>>>> url: url, >>>>>> headers: { >>>>>> 'x-identity': auth.getPublicKeyFromPrivateKey(keys[k]), >>>>>> 'x-signature': auth.sign(dataToSign, keys[k]) >>>>>> } >>>>>> }; >>>>>> >>>>>> >>>>>> The above code generates these Keys are now passed on each request to >>>>>> the server, and the server using the middleware in case of Fineract have >>>>>> to do two things >>>>>> >>>>>> 1. VerifySignature >>>>>> 2. GetSinFromThePublicKey >>>>>> >>>>>> I started writing some code for this and quickly realized that it's >>>>>> actually doable, it requires around 20 hours of work, There are some >>>>>> dependency that needs to be added on the server side and some quick >>>>>> changes >>>>>> on the middleware and its done. >>>>>> >>>>>> We presented a paper on this last year on Apache Con ! Remember >>>>>> something about using zero knowledge of proof and password less auth . >>>>>> >>>>>> Let me know , I would like to make sure that this gets added in this >>>>>> upcoming release. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Friday, October 22, 2021, James Dailey <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Devs - >>>>>>> >>>>>>> Through a conversation with a computer science person, I've learned >>>>>>> that Fineract should have a clearly articulated approach with regard to >>>>>>> idempotency. This is particularly true when thinking about how fineract >>>>>>> interacts with payment systems. >>>>>>> >>>>>>> "When building a system to move money, it is paramount that >>>>>>> operations that move money are idempotent. Failure to do this might >>>>>>> result >>>>>>> in accidentally double charging your customer, or paying a vendor >>>>>>> multiple >>>>>>> times. The risk of this happening is elevated based on the way software >>>>>>> is >>>>>>> typically built today, since developers take advantage of scalable >>>>>>> systems >>>>>>> that process multiple items in parallel. " >>>>>>> (https://www.moderntreasury.com/journal/why-idempotency-matt >>>>>>> ers-in-payments) >>>>>>> >>>>>>> Or this article on Stripe engineering calling for liberal use of >>>>>>> Idempotency: >>>>>>> >>>>>>> "The easiest way to address inconsistencies in distributed state >>>>>>> caused by failures is to implement server endpoints so that they’re >>>>>>> idempotent, which means that they can be called any number of times >>>>>>> while >>>>>>> guaranteeing that side effects only occur once." ( >>>>>>> https://stripe.com/blog/idempotency) >>>>>>> >>>>>>> and this article in Apache Camel >>>>>>> https://camel.apache.org/camel-kafka-connector/latest/user- >>>>>>> guide/idempotency.html >>>>>>> >>>>>>> So, my understanding may be wrong and we've got this covered, or >>>>>>> maybe we just consider this overkill. Perhaps there is something in the >>>>>>> "magic" that's developed in the CQRS plus in-memory database queuing and >>>>>>> our clever transaction state management, but... I don't know. >>>>>>> >>>>>>> Or perhaps we haven't really had to think about this because we >>>>>>> haven't had that many large scale deployments and the scalability issues >>>>>>> associated with multiple parallel calls haven't come to the fore, or >>>>>>> haven't come back to the list. So, that's the intent of this email. >>>>>>> >>>>>>> Do we have a need now for layering on Idempotency checks? What is >>>>>>> the right approach? Has anyone already implemented this, or decided >>>>>>> not to >>>>>>> for very good reasons? What would be the right approach to this? >>>>>>> Implement where? Time limited to 3 hrs? Etc... >>>>>>> >>>>>>> So, this is a call to discuss. I'm far from an expert on this. >>>>>>> >>>>>>> >>>>>>> >>> >>> -- >>> >>> Saransh Sharma >>> *Research Partner* >>> *Muellner Internet Pvt Ltd * >>> >>> ------------------------------------------------------------ >>> ---------------------------------- >>> >>> The idea of separation of me and you is amazing. >>> ------------------------------------------------------------ >>> ---------------------------------- >>> *Company Website <https://www.muellners.com/> **Company Linkedin >>> <https://www.linkedin.com/company/muellners> * >>> Company Facebook <https://www.facebook.com/muellners> >>> >>> This mail is governed by Muellner® Internet Private Limited's IT >>> policy. >>> The information contained in this e-mail and any accompanying documents >>> may contain information that is confidential or otherwise protected from >>> disclosure. If you are not the intended recipient of this message, or if >>> this message has been addressed to you in error, please immediately alert >>> the sender by reply e-mail and then delete this message, including any >>> attachments. Any dissemination, distribution or other use of the contents >>> of this message by anyone other than the intended recipient is strictly >>> prohibited. All messages sent to and from this e-mail address may be >>> monitored as permitted by applicable law and regulations to ensure >>> compliance with our internal policies and to protect our business. E-mails >>> are not secure and cannot be guaranteed to be error free as they can be >>> intercepted, amended, lost or destroyed, or contain viruses. You are deemed >>> to have accepted these risks if you communicate with us by e-mail. >>> >> > > -- > Sent from Gmail Mobile > -- Sent from Gmail Mobile
