I agree that performance tuning should be driven by non functional requirements/goals and determined by performance testing of the application. There is nothing wrong with the current technology stack if developers adopt best practices and write efficient algorithms and functions. Maybe a good place to start is to profile the application and see where the bottlenecks exist(Memory, CPU, I/O). Identify which layer of the applications needs to be scaled and how to scale both horizontally and vertically.
On Thu, Nov 7, 2019 at 5:51 AM Niklas Uhrberg <[email protected]> wrote: > Hi all, I think the scalability question calls for a systematic and > analytical way of working. First things to understand is what are the > different functional parts that have individual scalability requirements, > and then what is the behaviour/properties of the current implementation. > > Does the current implementation meet future requirements? > If no (and it seems this is the case for at least some parts) understand > what the limitations of the current architecture is. If the current > architecture will not be able to support the requirements then it must > change. > > As previous posts have pointed out: since (or if ?) the java layer can be > scaled horizontally it will probably not be a bottleneck. But since there > is a common database just scaling the java layer may not be sufficient. > > In any case setting up a potent test environment is the only way to really > get the correct answers to questions about where the limits are. > > In previous thread there was a discussion about to what extent a NewSQL db > could be used to increase the scalability. The only thing I want to say > here is that there are use cases where by definition distributed > transactions guaranteeing strong consistency and availability will never > scale as well as eventual consistency architecture á la e.g. actor model > with only aggregate level strong consistency guarantees. > > I think it is crucial to first identify the scalability requirements and > understand how the system can be partitioned (including database) to meet > the requirements and take the CAP theorem into account doing this. > > Best regards Niklas > > > > > > Niklas Uhrberg > Konsult > IT Development | Loans & Credit > > > > Resurs Bank AB > > > > Växel: +46 42 38 20 00 > E-post: [email protected] > Webb: www.resursbank.se > > > > ------------------------------ > *From:* Giorgio Zoppi [[email protected]] > *Sent:* Thursday, November 07, 2019 9:51 AM > *To:* [email protected] > *Cc:* Mifos software development > *Subject:* Re: [Mifos-developer] Scalability & Performance Working Group > > Hi Nazeer, > i was thinking about introduce Redis in the current architecture. Do you > think it feasible? > BR. > Giorgio > > El jue., 7 nov. 2019 a las 9:24, Nazeer Hussain Shaik (< > [email protected]>) escribió: > >> Hi all, >> >> Before thinking about moving micro services in different technologies >> please think about getting the required skill set people and the >> complexities around those technologies. >> I believe scaling the micro services horizontally with proper hardware >> configurations and by using some kind of load balance service (NGINX) may >> be good idea. >> In my current company we have more than 80 micro services and most of >> them are written in Java technologies. We deploy our services on AWS >> (Simulating SAAS model). >> We use kubernetes for docker orchestration. Currently we are able to >> scale 2 million requests. We are working on to scale to 4 million requests >> at least. So I don't think Java is bottleneck here. >> Please check whether the DB layer and JPA is performing as expected. I >> believe you need to look into client facing API(s) instead of hitting the >> direct micro services, better to use some kind of Indexers (Elastic Search, >> etc..). Or aggregate the data in client facing services properly. >> >> Please correct me if my thinking is wrong. >> >> Regards, >> Nazeer >> >> On Thu, Nov 7, 2019 at 12:54 PM Apoorva M.K <[email protected]> >> wrote: >> >>> Hi Ed, >>> I'm interested in helping out in this, I have gone through the list of >>> next steps. I will start working on what I can. >>> >>> Regards, >>> Apoorva >>> >>> On Thu 24 Oct, 2019, 1:57 AM Ed Cable, <[email protected]> wrote: >>> >>>> Hi Mifos and Fineract communities, >>>> >>>> In follow up to my previous comments on Joseph's thread I wanted to >>>> initiate a collaborative community-wide effort to help address the ongoing >>>> and growing need for helping Mifos/Fineract scale and sustain high load >>>> environments. >>>> >>>> This really represent an area where we can demonstrate that together as >>>> Open Source Community we are much stronger than each individual or partner >>>> trying to tackle this alone. >>>> >>>> This would be for both Fineract and Fineract CN. >>>> >>>> *Why the need for this Working Group?* >>>> The formation of a working group focused on this subject is needed as >>>> evident from various mail thread or support requests trying to optimize the >>>> system for high load/volume environments and a growing number of prospects >>>> eager to use the platform but needing more visibility into its performance >>>> and ability to meet high TPS requirements. >>>> >>>> Secondly, there are many partner-led implementations and deployments of >>>> Fineract/Mifos supporting millions of clients and it would be valuable to >>>> share that knowledge across the community. >>>> >>>> Lastly, it's been quite some time since we've had public performance >>>> testing done, the efforts led by eSolve in 2017 (see wiki page) and >>>> IBM/Conflux >>>> in 2015 >>>> <https://www.ibm.com/partnerworld/page/stg_ast_sys-mifos-x-on-ibm-powerlinux-servers>both >>>> pre-date the recent change from Hibernate to OpenJPA. Kumaranath with >>>> support from Avik of Fynarfin worked on performance related issues during >>>> 2018 GSOC - >>>> https://docs.google.com/document/d/18_awblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW_-In0 >>>> <https://smex12-5-en-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2fdocs.google.com%2fdocument%2fd%2f18%5fawblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW%5f%2dIn0&umid=ba389591-3c1d-4752-9f0f-b1566097d2f0&auth=5570038ba5c1d86c75907f538183887803b1409d-66acc5307e319e338515ff148c4394046f4e061b> >>>> >>>> >>>> *What/How* >>>> >>>> 1. Create a set of reproducible tools to enable automation of this >>>> on an ongoing basis. >>>> 1. Performance testing related data sets - i.e. DB dumps, >>>> scripts to generate test data >>>> 2. Postman scripts to run load tests >>>> 3. Perhaps we can create a repo on github for this? >>>> 2. Access to Resources >>>> 1. If you have hardware or cloud environments available to >>>> conduct these load/performance tests please share. >>>> 3. Culture & Transparency - Establish this as a priority area of >>>> the community to collectively address. >>>> 4. Documentation (likely on the Fineract Wiki) - I created this >>>> page to start: https://cwiki.apache.org/confluence/x/khD3Bw >>>> 1. Share and document existing results of performance testing to >>>> date. Share the details of environments you have set up to address >>>> high >>>> load needs. >>>> 2. Document the scenarios and TPS requirements that need to be >>>> tested >>>> 3. Document typical improvements to address performance (at a >>>> configuration, database, code, level, etc.) >>>> 5. Identify issues to fix >>>> 1. Log new issues, update existing issues, and tag >>>> appropriately. >>>> 6. Share code and fixes to address performance issues >>>> 1. Make these a priority and plan out in our release >>>> roadmapping. >>>> >>>> *Next Steps* >>>> >>>> - If interested in joining the group, respond to this thread. >>>> - Begin sharing your inputs on the wiki page. >>>> - Start fixing existing issues. >>>> - Help in creating and executing on our performance testing plan. >>>> >>>> Thanks, >>>> >>>> Ed >>>> >>>> Mifos-developer mailing list >>>> [email protected] >>>> Unsubscribe or change settings at: >>>> https://lists.sourceforge.net/lists/listinfo/mifos-developer >>> >>> > > -- > Life is a chess game - Anonymous. > >
