Saransh Thanks, I'll be there.
Regards Victor El lun, 25 oct 2021 a las 23:37, Saransh Sharma (<[email protected]>) escribió: > Apache Fineract Dev Discussion > Thursday, 28 Oct · 9:50 – 10:50 AM > Google Meet joining info > Video call link: https://meet.google.com/pdq-ybcs-adf > Or dial: +1 617-675-4444 PIN: 1480810204623 > More phone numbers: https://tel.meet/pdq-ybcs-adf?pin=1480810204623 > > On Tuesday, October 26, 2021, Saransh Sharma <[email protected]> > wrote: > >> Let’s do that let me know your availability we can stream it also ! >> >> Let me know your preferred platform ! >> >> >> >> On Tuesday, October 26, 2021, VICTOR MANUEL ROMERO RODRIGUEZ < >> [email protected]> wrote: >> >>> Saransh >>> >>> I would like to have a discussion, if we can record it, publish it in >>> youtube for example and share the link here again, in order to allow the >>> community to be in the loop. >>> >>> Does it work? >>> >>> Regards >>> >>> El lun, 25 oct 2021 a las 22:37, Saransh Sharma (<[email protected]>) >>> escribió: >>> >>>> Timestamping could also be added as part of the key. >>>> >>>> I agree with Victor here, we need to keep it simple, Hey @Victor Romero >>>> <[email protected]> would you like to discuss this and >>>> flesh it out together ? >>>> >>>> On Tue, Oct 26, 2021 at 9:01 AM VICTOR MANUEL ROMERO RODRIGUEZ < >>>> [email protected]> wrote: >>>> >>>>> I will extend the previous email, >>>>> >>>>> At asynchronous transactions: >>>>> >>>>> The trigger (or POST) for doing asynchronous transactions while >>>>> sending notifications (email, sms, push) over MQ/JMS queues, the >>>>> user/customer experience will be affected by receiving twice or more >>>>> payment reminders, payment notification (they can think that the payment >>>>> was applied twice), account creations, loan applications, etc and the cost >>>>> associated for each message sent over the notification channel. >>>>> >>>>> At journey entry: >>>>> >>>>> The trigger (or POST) for doing any transactions (monetary or non >>>>> monetary) only 1 entry must be recorded in all the systems so then the >>>>> logging and monitoring systems (fraud, alerting, etc) can correlate the >>>>> transaction (using surrogate keys) and later the data scientist or data >>>>> engineers won't be blaming the CBS engineering team. >>>>> >>>>> At transaction caching: >>>>> >>>>> We got a use case where the loan origination process was taking some >>>>> time to execute the process and because of the demand of the financial >>>>> product people was trying to execute several times to have a loan, so then >>>>> the business requirement was to keep on cache the same "loan request" if >>>>> the people ID was the same (in our case the National Population ID) and >>>>> not >>>>> executing the query to the external systems (Credit Bureau, Black List had >>>>> a high cost per each query). So then we saved execution time and money. >>>>> >>>>> ****TL;TR**** >>>>> What I am trying to express is that the level, use cases, and user >>>>> history (or whatever it can be the concept about is it use) have a wide >>>>> range of "idempotent" implementations for the potential solutions should >>>>> be >>>>> evaluated carefully depending on them. >>>>> >>>>> I think that the ID of the "idempotent" key should have enough >>>>> entropy to avoid any collision in the middleware, cache, db or messaging >>>>> level and the recording of it must be at UTC time either if it handles >>>>> synchronous or asynchronous API calls. >>>>> >>>>> ******** >>>>> >>>>> Best regards >>>>> >>>>> Victor >>>>> >>>>> >>>>> El lun, 25 oct 2021 a las 19:39, VICTOR MANUEL ROMERO RODRIGUEZ (< >>>>> [email protected]>) escribió: >>>>> >>>>>> Hello, >>>>>> >>>>>> I agree that a key must be used on the server side (Consumer) to >>>>>> avoid any duplicate record, most of the time I used to look at the >>>>>> Software >>>>>> Patterns like the Martin Fowler's articles: >>>>>> >>>>>> >>>>>> https://martinfowler.com/articles/patterns-of-distributed-systems/idempotent-receiver.html >>>>>> >>>>>> >>>>>> The pattern described there is more complex than the >>>>>> Saransh's proposal (which is a good starting point), there are many >>>>>> approaches to add this feature (Idempotency) on Apache Fineract, which I >>>>>> think should be flexible enough to handle business rules. >>>>>> >>>>>> Idempotency can be at different levels (even using the POST/PUT), >>>>>> example: >>>>>> >>>>>> At DB level: >>>>>> >>>>>> The POST can be executed to create a new customer and the DB table >>>>>> has as a key or unique the email used in the account creating, then a >>>>>> duplicated entry at DB level should be thrown and the API Rest should >>>>>> send >>>>>> this error. Then this API Rest is Idempotent because at DB level there >>>>>> is a >>>>>> restriction. >>>>>> >>>>>> At API REST per business rule level: >>>>>> >>>>>> The POST for a login should be idempotent because only one session is >>>>>> created at a time per user/customer and in case of X number of failed >>>>>> retries the account must be locked for a period of time or unlocked by a >>>>>> manual/automatic process. Once the session is created, if a POST for >>>>>> login >>>>>> sends a 200 status, then for a period of time the possibility to have >>>>>> another login must be blocked until the current active session expires >>>>>> (expiry time is set by the digital delivery channel owner like Internet >>>>>> or >>>>>> Mobile banking). >>>>>> >>>>>> At API REST per consistency rule level: >>>>>> >>>>>> The POST for doing a payment (third party same bank or interbank >>>>>> transfer) the idempotency key must be used for keeping the track of the >>>>>> transaction status, holding the amount, settlement or reversal of the >>>>>> amount, digital receipt, digital invoice generation, accounting record, >>>>>> etc) in order to avoid duplication of charges in the origin account. Even >>>>>> with confirmation/warning pop ups/windows errors occur. So then the >>>>>> handle >>>>>> of the idempotency key must be handled and implemented in the Front end >>>>>> application, the Server (and the internal consumer). >>>>>> >>>>>> And by the way the transactions must be done at UTC time, not at TZ >>>>>> (another change in the Apache Fineract's tables should be done for the >>>>>> offset time zone recording) . >>>>>> >>>>>> Best regards >>>>>> >>>>>> Victor >>>>>> >>>>>> El sáb, 23 oct 2021 a las 0:05, Saransh Sharma (< >>>>>> [email protected]>) escribió: >>>>>> >>>>>>> This is a curious problem in computer science and especially in >>>>>>> distributed networks. >>>>>>> >>>>>>> API Idempotency is a major issue for software like Fineract, unless >>>>>>> the API handles drop off and recognize the second call or any n call as >>>>>>> either duplicate or important in any given case, like how stripe does >>>>>>> this >>>>>>> by specifying a key from the client end ensures that even in case of >>>>>>> failure the charge does not happen twice! >>>>>>> >>>>>>> This can lead to financial loss in case of real payment taking place! >>>>>>> >>>>>>> I think, as of right now there is no such mechanism in Fineract 1.x >>>>>>> that eliminates such risk. Except, on the validation side, but that’s >>>>>>> not >>>>>>> enough. >>>>>>> >>>>>>> To have this implemented we have done some work on this last year , >>>>>>> maybe we can present some code also ! >>>>>>> >>>>>>> I have used some cryptography libs to have the client-side send a >>>>>>> unique key that’s generated by the private key infrastructure and the >>>>>>> client-side has to send these as headers and on the server-side, the >>>>>>> signatures are validated thus making the idempotency in POST and PUT >>>>>>> request. >>>>>>> >>>>>>> The code on the client side >>>>>>> >>>>>>> var keys = { >>>>>>> alice: >>>>>>> '38f93bdda21a5c4a7bae4eb75bb7811cbc3eb627176805c1009ff2099263c6ad', >>>>>>> bob: >>>>>>> '09880c962437080d72f72c8c63a69efd65d086c9e7851a87b76373eb6ce9aab5'}; >>>>>>> // GET >>>>>>> for(k in keys) { >>>>>>> var url = 'APIURL'; >>>>>>> var dataToSign = url; >>>>>>> var options = { >>>>>>> url: url, >>>>>>> headers: { >>>>>>> 'x-identity': auth.getPublicKeyFromPrivateKey(keys[k]), >>>>>>> 'x-signature': auth.sign(dataToSign, keys[k]) >>>>>>> } >>>>>>> }; >>>>>>> >>>>>>> >>>>>>> The above code generates these Keys are now passed on each request >>>>>>> to the server, and the server using the middleware in case of Fineract >>>>>>> have to do two things >>>>>>> >>>>>>> 1. VerifySignature >>>>>>> 2. GetSinFromThePublicKey >>>>>>> >>>>>>> I started writing some code for this and quickly realized that it's >>>>>>> actually doable, it requires around 20 hours of work, There are some >>>>>>> dependency that needs to be added on the server side and some quick >>>>>>> changes >>>>>>> on the middleware and its done. >>>>>>> >>>>>>> We presented a paper on this last year on Apache Con ! Remember >>>>>>> something about using zero knowledge of proof and password less auth . >>>>>>> >>>>>>> Let me know , I would like to make sure that this gets added in this >>>>>>> upcoming release. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Friday, October 22, 2021, James Dailey <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Devs - >>>>>>>> >>>>>>>> Through a conversation with a computer science person, I've learned >>>>>>>> that Fineract should have a clearly articulated approach with regard to >>>>>>>> idempotency. This is particularly true when thinking about how >>>>>>>> fineract >>>>>>>> interacts with payment systems. >>>>>>>> >>>>>>>> "When building a system to move money, it is paramount that >>>>>>>> operations that move money are idempotent. Failure to do this might >>>>>>>> result >>>>>>>> in accidentally double charging your customer, or paying a vendor >>>>>>>> multiple >>>>>>>> times. The risk of this happening is elevated based on the way >>>>>>>> software is >>>>>>>> typically built today, since developers take advantage of scalable >>>>>>>> systems >>>>>>>> that process multiple items in parallel. " >>>>>>>> ( >>>>>>>> https://www.moderntreasury.com/journal/why-idempotency-matters-in-payments >>>>>>>> ) >>>>>>>> >>>>>>>> Or this article on Stripe engineering calling for liberal use of >>>>>>>> Idempotency: >>>>>>>> >>>>>>>> "The easiest way to address inconsistencies in distributed state >>>>>>>> caused by failures is to implement server endpoints so that they’re >>>>>>>> idempotent, which means that they can be called any number of times >>>>>>>> while >>>>>>>> guaranteeing that side effects only occur once." ( >>>>>>>> https://stripe.com/blog/idempotency) >>>>>>>> >>>>>>>> and this article in Apache Camel >>>>>>>> >>>>>>>> https://camel.apache.org/camel-kafka-connector/latest/user-guide/idempotency.html >>>>>>>> >>>>>>>> So, my understanding may be wrong and we've got this covered, or >>>>>>>> maybe we just consider this overkill. Perhaps there is something in >>>>>>>> the >>>>>>>> "magic" that's developed in the CQRS plus in-memory database queuing >>>>>>>> and >>>>>>>> our clever transaction state management, but... I don't know. >>>>>>>> >>>>>>>> Or perhaps we haven't really had to think about this because we >>>>>>>> haven't had that many large scale deployments and the scalability >>>>>>>> issues >>>>>>>> associated with multiple parallel calls haven't come to the fore, or >>>>>>>> haven't come back to the list. So, that's the intent of this email. >>>>>>>> >>>>>>>> Do we have a need now for layering on Idempotency checks? What is >>>>>>>> the right approach? Has anyone already implemented this, or decided >>>>>>>> not to >>>>>>>> for very good reasons? What would be the right approach to this? >>>>>>>> Implement where? Time limited to 3 hrs? Etc... >>>>>>>> >>>>>>>> So, this is a call to discuss. I'm far from an expert on this. >>>>>>>> >>>>>>>> >>>>>>>> >>>> >>>> -- >>>> >>>> Saransh Sharma >>>> *Research Partner* >>>> *Muellner Internet Pvt Ltd * >>>> >>>> >>>> ---------------------------------------------------------------------------------------------- >>>> >>>> The idea of separation of me and you is amazing. >>>> >>>> ---------------------------------------------------------------------------------------------- >>>> *Company Website <https://www.muellners.com/> **Company Linkedin >>>> <https://www.linkedin.com/company/muellners> * >>>> Company Facebook <https://www.facebook.com/muellners> >>>> >>>> This mail is governed by Muellner® Internet Private Limited's IT >>>> policy. >>>> The information contained in this e-mail and any accompanying documents >>>> may contain information that is confidential or otherwise protected from >>>> disclosure. If you are not the intended recipient of this message, or if >>>> this message has been addressed to you in error, please immediately alert >>>> the sender by reply e-mail and then delete this message, including any >>>> attachments. Any dissemination, distribution or other use of the contents >>>> of this message by anyone other than the intended recipient is strictly >>>> prohibited. All messages sent to and from this e-mail address may be >>>> monitored as permitted by applicable law and regulations to ensure >>>> compliance with our internal policies and to protect our business. E-mails >>>> are not secure and cannot be guaranteed to be error free as they can be >>>> intercepted, amended, lost or destroyed, or contain viruses. You are deemed >>>> to have accepted these risks if you communicate with us by e-mail. >>>> >>> >> >> -- >> Sent from Gmail Mobile >> > > > -- > Sent from Gmail Mobile >
