Saransh, Please reschedule. I was unable to attend.
Regards El mié, 27 oct 2021 a las 23:26, Saransh Sharma (<[email protected]>) escribió: > Hey, I am waiting @Victor Romero <[email protected]> are you > joining, let me know or we can reschedule later so that others can join > > Thank > > On Thu, Oct 28, 2021 at 9:51 AM Saransh Sharma <[email protected]> > wrote: > >> Sorry, >> https://streamyard.com/caisuehywi >> >> @Victor Romero <[email protected]> Join here >> >> >> >> On Tuesday, October 26, 2021, VICTOR MANUEL ROMERO RODRIGUEZ < >> [email protected]> wrote: >> >>> Hello James, >>> >>> I was thinking about the statement "we haven't had that many large scale >>> deployments and the scalability issues associated with multiple parallel >>> calls haven't come to the fore, or haven't come back to the list." >>> >>> I remember that we faced issues with load testing on savings accounts >>> BUT this is not related to Idempotency (in fact I think that the >>> concurrency is handled in Fineract with the Optimistic logic, I share some >>> line of sample code already in Fineract), so then *****Idempotency is not >>> about large scale deployment nor scalability issues****. >>> >>> Idempotency should not be confused with issues related to concurrency, >>> scalability nor high volume of transactions per second. >>> >>> I would propose another way to view the "Idempotency Concern" on Apache >>> Fineract: >>> >>> "Strategy for design and building a Fault Tolerant API Rest Layer for >>> non safe and/or non idempotent Http methods in Apache Fineract" (it >>> could be shorter or fancier like "Apache Fineract Idempotency" :P ) >>> >>> As reference Safe and Idempotent comes from the http 1.1 spec >>> https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html >>> HTTP MethodIdempotentSafe >>> OPTIONS yes yes >>> GET yes yes >>> HEAD yes yes >>> PUT yes no >>> POST no no >>> DELETE yes no >>> PATCH no no >>> Let's talk then, Saransh and everyone interested in the topic. >>> >>> >>> Well... just the code about Txn Concurrency Management in Apache Fineract >>> >>> >>> https://github.com/apache/fineract/blob/develop/fineract-provider/src/main/java/org/apache/fineract/portfolio/savings/api/SavingsAccountTransactionsApiResource.java >>> >>> "... >>> } catch (ObjectOptimisticLockingFailureException >>> lockingFailureException) { >>> throw new >>> PlatformDataIntegrityException("error.msg.savings.concurrent.operations", >>> "Concurrent Transactions being made on this savings >>> account: " + lockingFailureException.getMessage(), >>> lockingFailureException); >>> } catch (CannotAcquireLockException cannotAcquireLockException) { >>> throw new >>> PlatformDataIntegrityException("error.msg.savings.concurrent.operations.unable.to.acquire.lock", >>> "Unable to acquir lock for this transaction: " + >>> cannotAcquireLockException.getMessage(), cannotAcquireLockException); >>> } >>> >>> ..." >>> >>> El lun, 25 oct 2021 a las 23:37, VICTOR MANUEL ROMERO RODRIGUEZ (< >>> [email protected]>) escribió: >>> >>>> Saransh >>>> >>>> Thanks, I'll be there. >>>> >>>> Regards >>>> >>>> Victor >>>> >>>> El lun, 25 oct 2021 a las 23:37, Saransh Sharma (<[email protected]>) >>>> escribió: >>>> >>>>> Apache Fineract Dev Discussion >>>>> Thursday, 28 Oct · 9:50 – 10:50 AM >>>>> Google Meet joining info >>>>> Video call link: https://meet.google.com/pdq-ybcs-adf >>>>> Or dial: +1 617-675-4444 PIN: 1480810204623 >>>>> More phone numbers: https://tel.meet/pdq-ybcs-adf?pin=1480810204623 >>>>> >>>>> On Tuesday, October 26, 2021, Saransh Sharma <[email protected]> >>>>> wrote: >>>>> >>>>>> Let’s do that let me know your availability we can stream it also ! >>>>>> >>>>>> Let me know your preferred platform ! >>>>>> >>>>>> >>>>>> >>>>>> On Tuesday, October 26, 2021, VICTOR MANUEL ROMERO RODRIGUEZ < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Saransh >>>>>>> >>>>>>> I would like to have a discussion, if we can record it, publish it >>>>>>> in youtube for example and share the link here again, in order to allow >>>>>>> the >>>>>>> community to be in the loop. >>>>>>> >>>>>>> Does it work? >>>>>>> >>>>>>> Regards >>>>>>> >>>>>>> El lun, 25 oct 2021 a las 22:37, Saransh Sharma (< >>>>>>> [email protected]>) escribió: >>>>>>> >>>>>>>> Timestamping could also be added as part of the key. >>>>>>>> >>>>>>>> I agree with Victor here, we need to keep it simple, Hey @Victor >>>>>>>> Romero <[email protected]> would you like to discuss >>>>>>>> this and flesh it out together ? >>>>>>>> >>>>>>>> On Tue, Oct 26, 2021 at 9:01 AM VICTOR MANUEL ROMERO RODRIGUEZ < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> I will extend the previous email, >>>>>>>>> >>>>>>>>> At asynchronous transactions: >>>>>>>>> >>>>>>>>> The trigger (or POST) for doing asynchronous transactions while >>>>>>>>> sending notifications (email, sms, push) over MQ/JMS queues, the >>>>>>>>> user/customer experience will be affected by receiving twice or more >>>>>>>>> payment reminders, payment notification (they can think that the >>>>>>>>> payment >>>>>>>>> was applied twice), account creations, loan applications, etc and the >>>>>>>>> cost >>>>>>>>> associated for each message sent over the notification channel. >>>>>>>>> >>>>>>>>> At journey entry: >>>>>>>>> >>>>>>>>> The trigger (or POST) for doing any transactions (monetary or non >>>>>>>>> monetary) only 1 entry must be recorded in all the systems so then the >>>>>>>>> logging and monitoring systems (fraud, alerting, etc) can correlate >>>>>>>>> the >>>>>>>>> transaction (using surrogate keys) and later the data scientist or >>>>>>>>> data >>>>>>>>> engineers won't be blaming the CBS engineering team. >>>>>>>>> >>>>>>>>> At transaction caching: >>>>>>>>> >>>>>>>>> We got a use case where the loan origination process was taking >>>>>>>>> some time to execute the process and because of the demand of the >>>>>>>>> financial >>>>>>>>> product people was trying to execute several times to have a loan, so >>>>>>>>> then >>>>>>>>> the business requirement was to keep on cache the same "loan request" >>>>>>>>> if >>>>>>>>> the people ID was the same (in our case the National Population ID) >>>>>>>>> and not >>>>>>>>> executing the query to the external systems (Credit Bureau, Black >>>>>>>>> List had >>>>>>>>> a high cost per each query). So then we saved execution time and >>>>>>>>> money. >>>>>>>>> >>>>>>>>> ****TL;TR**** >>>>>>>>> What I am trying to express is that the level, use cases, and user >>>>>>>>> history (or whatever it can be the concept about is it use) have a >>>>>>>>> wide >>>>>>>>> range of "idempotent" implementations for the potential solutions >>>>>>>>> should be >>>>>>>>> evaluated carefully depending on them. >>>>>>>>> >>>>>>>>> I think that the ID of the "idempotent" key should have enough >>>>>>>>> entropy to avoid any collision in the middleware, cache, db or >>>>>>>>> messaging >>>>>>>>> level and the recording of it must be at UTC time either if it handles >>>>>>>>> synchronous or asynchronous API calls. >>>>>>>>> >>>>>>>>> ******** >>>>>>>>> >>>>>>>>> Best regards >>>>>>>>> >>>>>>>>> Victor >>>>>>>>> >>>>>>>>> >>>>>>>>> El lun, 25 oct 2021 a las 19:39, VICTOR MANUEL ROMERO RODRIGUEZ (< >>>>>>>>> [email protected]>) escribió: >>>>>>>>> >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> I agree that a key must be used on the server side (Consumer) to >>>>>>>>>> avoid any duplicate record, most of the time I used to look at the >>>>>>>>>> Software >>>>>>>>>> Patterns like the Martin Fowler's articles: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://martinfowler.com/articles/patterns-of-distributed-systems/idempotent-receiver.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The pattern described there is more complex than the >>>>>>>>>> Saransh's proposal (which is a good starting point), there are many >>>>>>>>>> approaches to add this feature (Idempotency) on Apache Fineract, >>>>>>>>>> which I >>>>>>>>>> think should be flexible enough to handle business rules. >>>>>>>>>> >>>>>>>>>> Idempotency can be at different levels (even using the POST/PUT), >>>>>>>>>> example: >>>>>>>>>> >>>>>>>>>> At DB level: >>>>>>>>>> >>>>>>>>>> The POST can be executed to create a new customer and the DB >>>>>>>>>> table has as a key or unique the email used in the account creating, >>>>>>>>>> then a >>>>>>>>>> duplicated entry at DB level should be thrown and the API Rest >>>>>>>>>> should send >>>>>>>>>> this error. Then this API Rest is Idempotent because at DB level >>>>>>>>>> there is a >>>>>>>>>> restriction. >>>>>>>>>> >>>>>>>>>> At API REST per business rule level: >>>>>>>>>> >>>>>>>>>> The POST for a login should be idempotent because only one >>>>>>>>>> session is created at a time per user/customer and in case of X >>>>>>>>>> number of >>>>>>>>>> failed retries the account must be locked for a period of time or >>>>>>>>>> unlocked >>>>>>>>>> by a manual/automatic process. Once the session is created, if a >>>>>>>>>> POST for >>>>>>>>>> login sends a 200 status, then for a period of time the possibility >>>>>>>>>> to have >>>>>>>>>> another login must be blocked until the current active session >>>>>>>>>> expires >>>>>>>>>> (expiry time is set by the digital delivery channel owner like >>>>>>>>>> Internet or >>>>>>>>>> Mobile banking). >>>>>>>>>> >>>>>>>>>> At API REST per consistency rule level: >>>>>>>>>> >>>>>>>>>> The POST for doing a payment (third party same bank or interbank >>>>>>>>>> transfer) the idempotency key must be used for keeping the track of >>>>>>>>>> the >>>>>>>>>> transaction status, holding the amount, settlement or reversal of the >>>>>>>>>> amount, digital receipt, digital invoice generation, accounting >>>>>>>>>> record, >>>>>>>>>> etc) in order to avoid duplication of charges in the origin account. >>>>>>>>>> Even >>>>>>>>>> with confirmation/warning pop ups/windows errors occur. So then the >>>>>>>>>> handle >>>>>>>>>> of the idempotency key must be handled and implemented in the Front >>>>>>>>>> end >>>>>>>>>> application, the Server (and the internal consumer). >>>>>>>>>> >>>>>>>>>> And by the way the transactions must be done at UTC time, not at >>>>>>>>>> TZ (another change in the Apache Fineract's tables should be done >>>>>>>>>> for the >>>>>>>>>> offset time zone recording) . >>>>>>>>>> >>>>>>>>>> Best regards >>>>>>>>>> >>>>>>>>>> Victor >>>>>>>>>> >>>>>>>>>> El sáb, 23 oct 2021 a las 0:05, Saransh Sharma (< >>>>>>>>>> [email protected]>) escribió: >>>>>>>>>> >>>>>>>>>>> This is a curious problem in computer science and especially in >>>>>>>>>>> distributed networks. >>>>>>>>>>> >>>>>>>>>>> API Idempotency is a major issue for software like Fineract, >>>>>>>>>>> unless the API handles drop off and recognize the second call or >>>>>>>>>>> any n >>>>>>>>>>> call as either duplicate or important in any given case, like how >>>>>>>>>>> stripe >>>>>>>>>>> does this by specifying a key from the client end ensures that even >>>>>>>>>>> in case >>>>>>>>>>> of failure the charge does not happen twice! >>>>>>>>>>> >>>>>>>>>>> This can lead to financial loss in case of real payment taking >>>>>>>>>>> place! >>>>>>>>>>> >>>>>>>>>>> I think, as of right now there is no such mechanism in Fineract >>>>>>>>>>> 1.x that eliminates such risk. Except, on the validation side, but >>>>>>>>>>> that’s >>>>>>>>>>> not enough. >>>>>>>>>>> >>>>>>>>>>> To have this implemented we have done some work on this last >>>>>>>>>>> year , maybe we can present some code also ! >>>>>>>>>>> >>>>>>>>>>> I have used some cryptography libs to have the client-side send >>>>>>>>>>> a unique key that’s generated by the private key infrastructure and >>>>>>>>>>> the >>>>>>>>>>> client-side has to send these as headers and on the server-side, the >>>>>>>>>>> signatures are validated thus making the idempotency in POST and PUT >>>>>>>>>>> request. >>>>>>>>>>> >>>>>>>>>>> The code on the client side >>>>>>>>>>> >>>>>>>>>>> var keys = { >>>>>>>>>>> alice: >>>>>>>>>>> '38f93bdda21a5c4a7bae4eb75bb7811cbc3eb627176805c1009ff2099263c6ad', >>>>>>>>>>> bob: >>>>>>>>>>> '09880c962437080d72f72c8c63a69efd65d086c9e7851a87b76373eb6ce9aab5'}; >>>>>>>>>>> // GET >>>>>>>>>>> for(k in keys) { >>>>>>>>>>> var url = 'APIURL'; >>>>>>>>>>> var dataToSign = url; >>>>>>>>>>> var options = { >>>>>>>>>>> url: url, >>>>>>>>>>> headers: { >>>>>>>>>>> 'x-identity': auth.getPublicKeyFromPrivateKey(keys[k]), >>>>>>>>>>> 'x-signature': auth.sign(dataToSign, keys[k]) >>>>>>>>>>> } >>>>>>>>>>> }; >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The above code generates these Keys are now passed on each >>>>>>>>>>> request to the server, and the server using the middleware in case >>>>>>>>>>> of >>>>>>>>>>> Fineract have to do two things >>>>>>>>>>> >>>>>>>>>>> 1. VerifySignature >>>>>>>>>>> 2. GetSinFromThePublicKey >>>>>>>>>>> >>>>>>>>>>> I started writing some code for this and quickly realized that >>>>>>>>>>> it's actually doable, it requires around 20 hours of work, There >>>>>>>>>>> are some >>>>>>>>>>> dependency that needs to be added on the server side and some quick >>>>>>>>>>> changes >>>>>>>>>>> on the middleware and its done. >>>>>>>>>>> >>>>>>>>>>> We presented a paper on this last year on Apache Con ! Remember >>>>>>>>>>> something about using zero knowledge of proof and password less >>>>>>>>>>> auth . >>>>>>>>>>> >>>>>>>>>>> Let me know , I would like to make sure that this gets added in >>>>>>>>>>> this upcoming release. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Friday, October 22, 2021, James Dailey < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Devs - >>>>>>>>>>>> >>>>>>>>>>>> Through a conversation with a computer science person, I've >>>>>>>>>>>> learned that Fineract should have a clearly articulated approach >>>>>>>>>>>> with >>>>>>>>>>>> regard to idempotency. This is particularly true when thinking >>>>>>>>>>>> about how >>>>>>>>>>>> fineract interacts with payment systems. >>>>>>>>>>>> >>>>>>>>>>>> "When building a system to move money, it is paramount that >>>>>>>>>>>> operations that move money are idempotent. Failure to do this >>>>>>>>>>>> might result >>>>>>>>>>>> in accidentally double charging your customer, or paying a vendor >>>>>>>>>>>> multiple >>>>>>>>>>>> times. The risk of this happening is elevated based on the way >>>>>>>>>>>> software is >>>>>>>>>>>> typically built today, since developers take advantage of scalable >>>>>>>>>>>> systems >>>>>>>>>>>> that process multiple items in parallel. " >>>>>>>>>>>> ( >>>>>>>>>>>> https://www.moderntreasury.com/journal/why-idempotency-matters-in-payments >>>>>>>>>>>> ) >>>>>>>>>>>> >>>>>>>>>>>> Or this article on Stripe engineering calling for liberal use >>>>>>>>>>>> of Idempotency: >>>>>>>>>>>> >>>>>>>>>>>> "The easiest way to address inconsistencies in distributed >>>>>>>>>>>> state caused by failures is to implement server endpoints so that >>>>>>>>>>>> they’re >>>>>>>>>>>> idempotent, which means that they can be called any number of >>>>>>>>>>>> times while >>>>>>>>>>>> guaranteeing that side effects only occur once." ( >>>>>>>>>>>> https://stripe.com/blog/idempotency) >>>>>>>>>>>> >>>>>>>>>>>> and this article in Apache Camel >>>>>>>>>>>> >>>>>>>>>>>> https://camel.apache.org/camel-kafka-connector/latest/user-guide/idempotency.html >>>>>>>>>>>> >>>>>>>>>>>> So, my understanding may be wrong and we've got this covered, >>>>>>>>>>>> or maybe we just consider this overkill. Perhaps there is >>>>>>>>>>>> something in the >>>>>>>>>>>> "magic" that's developed in the CQRS plus in-memory database >>>>>>>>>>>> queuing and >>>>>>>>>>>> our clever transaction state management, but... I don't know. >>>>>>>>>>>> >>>>>>>>>>>> Or perhaps we haven't really had to think about this because we >>>>>>>>>>>> haven't had that many large scale deployments and the scalability >>>>>>>>>>>> issues >>>>>>>>>>>> associated with multiple parallel calls haven't come to the fore, >>>>>>>>>>>> or >>>>>>>>>>>> haven't come back to the list. So, that's the intent of this email. >>>>>>>>>>>> >>>>>>>>>>>> Do we have a need now for layering on Idempotency checks? What >>>>>>>>>>>> is the right approach? Has anyone already implemented this, or >>>>>>>>>>>> decided not >>>>>>>>>>>> to for very good reasons? What would be the right approach to >>>>>>>>>>>> this? >>>>>>>>>>>> Implement where? Time limited to 3 hrs? Etc... >>>>>>>>>>>> >>>>>>>>>>>> So, this is a call to discuss. I'm far from an expert on this. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Saransh Sharma >>>>>>>> *Research Partner* >>>>>>>> *Muellner Internet Pvt Ltd * >>>>>>>> >>>>>>>> >>>>>>>> ---------------------------------------------------------------------------------------------- >>>>>>>> >>>>>>>> The idea of separation of me and you is amazing. >>>>>>>> >>>>>>>> ---------------------------------------------------------------------------------------------- >>>>>>>> *Company Website <https://www.muellners.com/> **Company Linkedin >>>>>>>> <https://www.linkedin.com/company/muellners> * >>>>>>>> Company Facebook <https://www.facebook.com/muellners> >>>>>>>> >>>>>>>> This mail is governed by Muellner® Internet Private Limited's IT >>>>>>>> policy. >>>>>>>> The information contained in this e-mail and any accompanying >>>>>>>> documents may contain information that is confidential or otherwise >>>>>>>> protected from disclosure. If you are not the intended recipient of >>>>>>>> this >>>>>>>> message, or if this message has been addressed to you in error, please >>>>>>>> immediately alert the sender by reply e-mail and then delete this >>>>>>>> message, >>>>>>>> including any attachments. Any dissemination, distribution or other >>>>>>>> use of >>>>>>>> the contents of this message by anyone other than the intended >>>>>>>> recipient is >>>>>>>> strictly prohibited. All messages sent to and from this e-mail address >>>>>>>> may >>>>>>>> be monitored as permitted by applicable law and regulations to ensure >>>>>>>> compliance with our internal policies and to protect our business. >>>>>>>> E-mails >>>>>>>> are not secure and cannot be guaranteed to be error free as they can be >>>>>>>> intercepted, amended, lost or destroyed, or contain viruses. You are >>>>>>>> deemed >>>>>>>> to have accepted these risks if you communicate with us by e-mail. >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Sent from Gmail Mobile >>>>>> >>>>> >>>>> >>>>> -- >>>>> Sent from Gmail Mobile >>>>> >>>> > > -- > > Saransh Sharma > *Research Partner* > *Muellner Internet Pvt Ltd * > > > ---------------------------------------------------------------------------------------------- > > The idea of separation of me and you is amazing. > > ---------------------------------------------------------------------------------------------- > *Company Website <https://www.muellners.com/> **Company Linkedin > <https://www.linkedin.com/company/muellners> *Company Facebook > <https://www.facebook.com/muellners> > > This mail is governed by Muellner® Internet Private Limited's IT policy. > The information contained in this e-mail and any accompanying documents > may contain information that is confidential or otherwise protected from > disclosure. If you are not the intended recipient of this message, or if > this message has been addressed to you in error, please immediately alert > the sender by reply e-mail and then delete this message, including any > attachments. Any dissemination, distribution or other use of the contents > of this message by anyone other than the intended recipient is strictly > prohibited. All messages sent to and from this e-mail address may be > monitored as permitted by applicable law and regulations to ensure > compliance with our internal policies and to protect our business. E-mails > are not secure and cannot be guaranteed to be error free as they can be > intercepted, amended, lost or destroyed, or contain viruses. You are deemed > to have accepted these risks if you communicate with us by e-mail. >
