Sorry, https://streamyard.com/caisuehywi
@Victor Romero <[email protected]> Join here On Tuesday, October 26, 2021, VICTOR MANUEL ROMERO RODRIGUEZ < [email protected]> wrote: > Hello James, > > I was thinking about the statement "we haven't had that many large scale > deployments and the scalability issues associated with multiple parallel > calls haven't come to the fore, or haven't come back to the list." > > I remember that we faced issues with load testing on savings accounts BUT > this is not related to Idempotency (in fact I think that the concurrency is > handled in Fineract with the Optimistic logic, I share some line of sample > code already in Fineract), so then *****Idempotency is not about large > scale deployment nor scalability issues****. > > Idempotency should not be confused with issues related to concurrency, > scalability nor high volume of transactions per second. > > I would propose another way to view the "Idempotency Concern" on Apache > Fineract: > > "Strategy for design and building a Fault Tolerant API Rest Layer for non > safe and/or non idempotent Http methods in Apache Fineract" (it could be > shorter or fancier like "Apache Fineract Idempotency" :P ) > > As reference Safe and Idempotent comes from the http 1.1 spec > https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html > HTTP MethodIdempotentSafe > OPTIONS yes yes > GET yes yes > HEAD yes yes > PUT yes no > POST no no > DELETE yes no > PATCH no no > Let's talk then, Saransh and everyone interested in the topic. > > > Well... just the code about Txn Concurrency Management in Apache Fineract > > > https://github.com/apache/fineract/blob/develop/fineract-provider/src/main/java/org/apache/fineract/portfolio/savings/api/SavingsAccountTransactionsApiResource.java > > "... > } catch (ObjectOptimisticLockingFailureException lockingFailureException) > { > throw new > PlatformDataIntegrityException("error.msg.savings.concurrent.operations", > "Concurrent Transactions being made on this savings > account: " + lockingFailureException.getMessage(), > lockingFailureException); > } catch (CannotAcquireLockException cannotAcquireLockException) { > throw new > PlatformDataIntegrityException("error.msg.savings.concurrent.operations.unable.to.acquire.lock", > "Unable to acquir lock for this transaction: " + > cannotAcquireLockException.getMessage(), cannotAcquireLockException); > } > > ..." > > El lun, 25 oct 2021 a las 23:37, VICTOR MANUEL ROMERO RODRIGUEZ (< > [email protected]>) escribió: > >> Saransh >> >> Thanks, I'll be there. >> >> Regards >> >> Victor >> >> El lun, 25 oct 2021 a las 23:37, Saransh Sharma (<[email protected]>) >> escribió: >> >>> Apache Fineract Dev Discussion >>> Thursday, 28 Oct · 9:50 – 10:50 AM >>> Google Meet joining info >>> Video call link: https://meet.google.com/pdq-ybcs-adf >>> Or dial: +1 617-675-4444 PIN: 1480810204623 >>> More phone numbers: https://tel.meet/pdq-ybcs-adf?pin=1480810204623 >>> >>> On Tuesday, October 26, 2021, Saransh Sharma <[email protected]> >>> wrote: >>> >>>> Let’s do that let me know your availability we can stream it also ! >>>> >>>> Let me know your preferred platform ! >>>> >>>> >>>> >>>> On Tuesday, October 26, 2021, VICTOR MANUEL ROMERO RODRIGUEZ < >>>> [email protected]> wrote: >>>> >>>>> Saransh >>>>> >>>>> I would like to have a discussion, if we can record it, publish it in >>>>> youtube for example and share the link here again, in order to allow the >>>>> community to be in the loop. >>>>> >>>>> Does it work? >>>>> >>>>> Regards >>>>> >>>>> El lun, 25 oct 2021 a las 22:37, Saransh Sharma (< >>>>> [email protected]>) escribió: >>>>> >>>>>> Timestamping could also be added as part of the key. >>>>>> >>>>>> I agree with Victor here, we need to keep it simple, Hey @Victor >>>>>> Romero <[email protected]> would you like to discuss >>>>>> this and flesh it out together ? >>>>>> >>>>>> On Tue, Oct 26, 2021 at 9:01 AM VICTOR MANUEL ROMERO RODRIGUEZ < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> I will extend the previous email, >>>>>>> >>>>>>> At asynchronous transactions: >>>>>>> >>>>>>> The trigger (or POST) for doing asynchronous transactions while >>>>>>> sending notifications (email, sms, push) over MQ/JMS queues, the >>>>>>> user/customer experience will be affected by receiving twice or more >>>>>>> payment reminders, payment notification (they can think that the payment >>>>>>> was applied twice), account creations, loan applications, etc and the >>>>>>> cost >>>>>>> associated for each message sent over the notification channel. >>>>>>> >>>>>>> At journey entry: >>>>>>> >>>>>>> The trigger (or POST) for doing any transactions (monetary or non >>>>>>> monetary) only 1 entry must be recorded in all the systems so then the >>>>>>> logging and monitoring systems (fraud, alerting, etc) can correlate the >>>>>>> transaction (using surrogate keys) and later the data scientist or data >>>>>>> engineers won't be blaming the CBS engineering team. >>>>>>> >>>>>>> At transaction caching: >>>>>>> >>>>>>> We got a use case where the loan origination process was taking some >>>>>>> time to execute the process and because of the demand of the financial >>>>>>> product people was trying to execute several times to have a loan, so >>>>>>> then >>>>>>> the business requirement was to keep on cache the same "loan request" if >>>>>>> the people ID was the same (in our case the National Population ID) and >>>>>>> not >>>>>>> executing the query to the external systems (Credit Bureau, Black List >>>>>>> had >>>>>>> a high cost per each query). So then we saved execution time and money. >>>>>>> >>>>>>> ****TL;TR**** >>>>>>> What I am trying to express is that the level, use cases, and user >>>>>>> history (or whatever it can be the concept about is it use) have a wide >>>>>>> range of "idempotent" implementations for the potential solutions >>>>>>> should be >>>>>>> evaluated carefully depending on them. >>>>>>> >>>>>>> I think that the ID of the "idempotent" key should have enough >>>>>>> entropy to avoid any collision in the middleware, cache, db or messaging >>>>>>> level and the recording of it must be at UTC time either if it handles >>>>>>> synchronous or asynchronous API calls. >>>>>>> >>>>>>> ******** >>>>>>> >>>>>>> Best regards >>>>>>> >>>>>>> Victor >>>>>>> >>>>>>> >>>>>>> El lun, 25 oct 2021 a las 19:39, VICTOR MANUEL ROMERO RODRIGUEZ (< >>>>>>> [email protected]>) escribió: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I agree that a key must be used on the server side (Consumer) to >>>>>>>> avoid any duplicate record, most of the time I used to look at the >>>>>>>> Software >>>>>>>> Patterns like the Martin Fowler's articles: >>>>>>>> >>>>>>>> >>>>>>>> https://martinfowler.com/articles/patterns-of-distributed-systems/idempotent-receiver.html >>>>>>>> >>>>>>>> >>>>>>>> The pattern described there is more complex than the >>>>>>>> Saransh's proposal (which is a good starting point), there are many >>>>>>>> approaches to add this feature (Idempotency) on Apache Fineract, which >>>>>>>> I >>>>>>>> think should be flexible enough to handle business rules. >>>>>>>> >>>>>>>> Idempotency can be at different levels (even using the POST/PUT), >>>>>>>> example: >>>>>>>> >>>>>>>> At DB level: >>>>>>>> >>>>>>>> The POST can be executed to create a new customer and the DB table >>>>>>>> has as a key or unique the email used in the account creating, then a >>>>>>>> duplicated entry at DB level should be thrown and the API Rest should >>>>>>>> send >>>>>>>> this error. Then this API Rest is Idempotent because at DB level there >>>>>>>> is a >>>>>>>> restriction. >>>>>>>> >>>>>>>> At API REST per business rule level: >>>>>>>> >>>>>>>> The POST for a login should be idempotent because only one session >>>>>>>> is created at a time per user/customer and in case of X number of >>>>>>>> failed >>>>>>>> retries the account must be locked for a period of time or unlocked >>>>>>>> by a >>>>>>>> manual/automatic process. Once the session is created, if a POST for >>>>>>>> login >>>>>>>> sends a 200 status, then for a period of time the possibility to have >>>>>>>> another login must be blocked until the current active session expires >>>>>>>> (expiry time is set by the digital delivery channel owner like >>>>>>>> Internet or >>>>>>>> Mobile banking). >>>>>>>> >>>>>>>> At API REST per consistency rule level: >>>>>>>> >>>>>>>> The POST for doing a payment (third party same bank or interbank >>>>>>>> transfer) the idempotency key must be used for keeping the track of the >>>>>>>> transaction status, holding the amount, settlement or reversal of the >>>>>>>> amount, digital receipt, digital invoice generation, accounting record, >>>>>>>> etc) in order to avoid duplication of charges in the origin account. >>>>>>>> Even >>>>>>>> with confirmation/warning pop ups/windows errors occur. So then the >>>>>>>> handle >>>>>>>> of the idempotency key must be handled and implemented in the Front end >>>>>>>> application, the Server (and the internal consumer). >>>>>>>> >>>>>>>> And by the way the transactions must be done at UTC time, not at TZ >>>>>>>> (another change in the Apache Fineract's tables should be done for the >>>>>>>> offset time zone recording) . >>>>>>>> >>>>>>>> Best regards >>>>>>>> >>>>>>>> Victor >>>>>>>> >>>>>>>> El sáb, 23 oct 2021 a las 0:05, Saransh Sharma (< >>>>>>>> [email protected]>) escribió: >>>>>>>> >>>>>>>>> This is a curious problem in computer science and especially in >>>>>>>>> distributed networks. >>>>>>>>> >>>>>>>>> API Idempotency is a major issue for software like Fineract, >>>>>>>>> unless the API handles drop off and recognize the second call or any >>>>>>>>> n >>>>>>>>> call as either duplicate or important in any given case, like how >>>>>>>>> stripe >>>>>>>>> does this by specifying a key from the client end ensures that even >>>>>>>>> in case >>>>>>>>> of failure the charge does not happen twice! >>>>>>>>> >>>>>>>>> This can lead to financial loss in case of real payment taking >>>>>>>>> place! >>>>>>>>> >>>>>>>>> I think, as of right now there is no such mechanism in Fineract >>>>>>>>> 1.x that eliminates such risk. Except, on the validation side, but >>>>>>>>> that’s >>>>>>>>> not enough. >>>>>>>>> >>>>>>>>> To have this implemented we have done some work on this last year >>>>>>>>> , maybe we can present some code also ! >>>>>>>>> >>>>>>>>> I have used some cryptography libs to have the client-side send a >>>>>>>>> unique key that’s generated by the private key infrastructure and the >>>>>>>>> client-side has to send these as headers and on the server-side, the >>>>>>>>> signatures are validated thus making the idempotency in POST and PUT >>>>>>>>> request. >>>>>>>>> >>>>>>>>> The code on the client side >>>>>>>>> >>>>>>>>> var keys = { >>>>>>>>> alice: >>>>>>>>> '38f93bdda21a5c4a7bae4eb75bb7811cbc3eb627176805c1009ff2099263c6ad', >>>>>>>>> bob: >>>>>>>>> '09880c962437080d72f72c8c63a69efd65d086c9e7851a87b76373eb6ce9aab5'}; >>>>>>>>> // GET >>>>>>>>> for(k in keys) { >>>>>>>>> var url = 'APIURL'; >>>>>>>>> var dataToSign = url; >>>>>>>>> var options = { >>>>>>>>> url: url, >>>>>>>>> headers: { >>>>>>>>> 'x-identity': auth.getPublicKeyFromPrivateKey(keys[k]), >>>>>>>>> 'x-signature': auth.sign(dataToSign, keys[k]) >>>>>>>>> } >>>>>>>>> }; >>>>>>>>> >>>>>>>>> >>>>>>>>> The above code generates these Keys are now passed on each request >>>>>>>>> to the server, and the server using the middleware in case of >>>>>>>>> Fineract >>>>>>>>> have to do two things >>>>>>>>> >>>>>>>>> 1. VerifySignature >>>>>>>>> 2. GetSinFromThePublicKey >>>>>>>>> >>>>>>>>> I started writing some code for this and quickly realized that >>>>>>>>> it's actually doable, it requires around 20 hours of work, There are >>>>>>>>> some >>>>>>>>> dependency that needs to be added on the server side and some quick >>>>>>>>> changes >>>>>>>>> on the middleware and its done. >>>>>>>>> >>>>>>>>> We presented a paper on this last year on Apache Con ! Remember >>>>>>>>> something about using zero knowledge of proof and password less auth . >>>>>>>>> >>>>>>>>> Let me know , I would like to make sure that this gets added in >>>>>>>>> this upcoming release. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Friday, October 22, 2021, James Dailey <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Devs - >>>>>>>>>> >>>>>>>>>> Through a conversation with a computer science person, I've >>>>>>>>>> learned that Fineract should have a clearly articulated approach with >>>>>>>>>> regard to idempotency. This is particularly true when thinking >>>>>>>>>> about how >>>>>>>>>> fineract interacts with payment systems. >>>>>>>>>> >>>>>>>>>> "When building a system to move money, it is paramount that >>>>>>>>>> operations that move money are idempotent. Failure to do this might >>>>>>>>>> result >>>>>>>>>> in accidentally double charging your customer, or paying a vendor >>>>>>>>>> multiple >>>>>>>>>> times. The risk of this happening is elevated based on the way >>>>>>>>>> software is >>>>>>>>>> typically built today, since developers take advantage of scalable >>>>>>>>>> systems >>>>>>>>>> that process multiple items in parallel. " >>>>>>>>>> ( >>>>>>>>>> https://www.moderntreasury.com/journal/why-idempotency-matters-in-payments >>>>>>>>>> ) >>>>>>>>>> >>>>>>>>>> Or this article on Stripe engineering calling for liberal use of >>>>>>>>>> Idempotency: >>>>>>>>>> >>>>>>>>>> "The easiest way to address inconsistencies in distributed state >>>>>>>>>> caused by failures is to implement server endpoints so that they’re >>>>>>>>>> idempotent, which means that they can be called any number of times >>>>>>>>>> while >>>>>>>>>> guaranteeing that side effects only occur once." ( >>>>>>>>>> https://stripe.com/blog/idempotency) >>>>>>>>>> >>>>>>>>>> and this article in Apache Camel >>>>>>>>>> >>>>>>>>>> https://camel.apache.org/camel-kafka-connector/latest/user-guide/idempotency.html >>>>>>>>>> >>>>>>>>>> So, my understanding may be wrong and we've got this covered, or >>>>>>>>>> maybe we just consider this overkill. Perhaps there is something in >>>>>>>>>> the >>>>>>>>>> "magic" that's developed in the CQRS plus in-memory database queuing >>>>>>>>>> and >>>>>>>>>> our clever transaction state management, but... I don't know. >>>>>>>>>> >>>>>>>>>> Or perhaps we haven't really had to think about this because we >>>>>>>>>> haven't had that many large scale deployments and the scalability >>>>>>>>>> issues >>>>>>>>>> associated with multiple parallel calls haven't come to the fore, or >>>>>>>>>> haven't come back to the list. So, that's the intent of this email. >>>>>>>>>> >>>>>>>>>> Do we have a need now for layering on Idempotency checks? What is >>>>>>>>>> the right approach? Has anyone already implemented this, or decided >>>>>>>>>> not to >>>>>>>>>> for very good reasons? What would be the right approach to this? >>>>>>>>>> Implement where? Time limited to 3 hrs? Etc... >>>>>>>>>> >>>>>>>>>> So, this is a call to discuss. I'm far from an expert on this. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Saransh Sharma >>>>>> *Research Partner* >>>>>> *Muellner Internet Pvt Ltd * >>>>>> >>>>>> >>>>>> ---------------------------------------------------------------------------------------------- >>>>>> >>>>>> The idea of separation of me and you is amazing. >>>>>> >>>>>> ---------------------------------------------------------------------------------------------- >>>>>> *Company Website <https://www.muellners.com/> **Company Linkedin >>>>>> <https://www.linkedin.com/company/muellners> * >>>>>> Company Facebook <https://www.facebook.com/muellners> >>>>>> >>>>>> This mail is governed by Muellner® Internet Private Limited's IT >>>>>> policy. >>>>>> The information contained in this e-mail and any accompanying >>>>>> documents may contain information that is confidential or otherwise >>>>>> protected from disclosure. If you are not the intended recipient of this >>>>>> message, or if this message has been addressed to you in error, please >>>>>> immediately alert the sender by reply e-mail and then delete this >>>>>> message, >>>>>> including any attachments. Any dissemination, distribution or other use >>>>>> of >>>>>> the contents of this message by anyone other than the intended recipient >>>>>> is >>>>>> strictly prohibited. All messages sent to and from this e-mail address >>>>>> may >>>>>> be monitored as permitted by applicable law and regulations to ensure >>>>>> compliance with our internal policies and to protect our business. >>>>>> E-mails >>>>>> are not secure and cannot be guaranteed to be error free as they can be >>>>>> intercepted, amended, lost or destroyed, or contain viruses. You are >>>>>> deemed >>>>>> to have accepted these risks if you communicate with us by e-mail. >>>>>> >>>>> >>>> >>>> -- >>>> Sent from Gmail Mobile >>>> >>> >>> >>> -- >>> Sent from Gmail Mobile >>> >>
