+1 On Fri, Nov 8, 2019 at 9:02 AM Niklas Uhrberg <[email protected]> wrote:
> Nazeer, I think you sum important things up very well. > > When using the term "java layer" that can be scaled horizontally I simply > meant the java microservices in Fineract CN or monolith java application in > Fineract 1 excluding the database. Just to convey what I was actually > thinking of, sorry if it was a bit vague. > > (The quote is "As previous posts have pointed out: since (or if ?) the > java layer can be scaled horizontally it will probably not be a bottleneck. > But since there is a common database just scaling the java layer may not be > sufficient." ) > > It boild down to that it is pointless to scale the application logic alone > if the db becomes a bottleneck. > > I also think that having each microservice "own" its database in the sense > that it is the only user of it makes a lot of sense. This is indeed that > default when doing microservices. I suspect also that splitting the > database is far more difficult than just splitting a monolith into > multiple services. > > As I pointed out in the previous thread about NewSQL, there can be > eventually consistent read models that contain data from multiple services > serving the use cases where you effectively need data from multiple > services in the same database. (For performance reasons.) > > Best regards Niklas > > > > > Niklas Uhrberg > Konsult > IT Development | Loans & Credit > > > > Resurs Bank AB > > > > Växel: +46 42 38 20 00 > E-post: [email protected] > Webb: www.resursbank.se > > > > ------------------------------ > *From:* Nazeer Hussain Shaik [[email protected]] > *Sent:* Friday, November 08, 2019 7:03 AM > *To:* [email protected] > *Subject:* Re: [Mifos-developer] Scalability & Performance Working Group > > Hi Niklas, > > I agree with you we have to approach systematic way. Small correction. > Java doesn't scale horizontally. Micro services should be scaled. Having a > single instance of DB (single cluster with minimal read replicas) for all > micro services will be a big bottleneck on production environments. Should > be clustered for each micro services based on the load. Usually different > micro services might have different DB types. For example one micro service > might have Graph DB, another might have Mongo, etc.. As you know no > architecture is perfect, we need to a take the approach which will give > optimal benefits with some small drawbacks. > > Note: I am not saying for every deployment ,we need to have DB clusters > for each micro service and load balancers. That depends upon the how much > through put that deployment should take. But for client facing API(s) it > should always better to have data aggregated and ready to serve the request > immediately than constructing the data on the fly by reading different > schema. If you need to have good performance it should always better to use > eventual consistency in clustered environments. But some use cases might > require strong consistency. So the configurations changes based on micro > service to micro service. > > As all suggested, better to profile current implementation to find out > where is the bottleneck. > > Regards, > Nazeer > > On Fri, Nov 8, 2019 at 5:58 AM Tonio O <[email protected]> wrote: > >> I agree that performance tuning should be driven by non functional >> requirements/goals and determined by performance testing of the >> application. There is nothing wrong with the current technology stack if >> developers adopt best practices and write efficient algorithms and >> functions. Maybe a good place to start is to profile the application and >> see where the bottlenecks exist(Memory, CPU, I/O). Identify which layer of >> the applications needs to be scaled and how to scale both horizontally and >> vertically. >> >> >> On Thu, Nov 7, 2019 at 5:51 AM Niklas Uhrberg <[email protected]> >> wrote: >> >>> Hi all, I think the scalability question calls for a systematic and >>> analytical way of working. First things to understand is what are the >>> different functional parts that have individual scalability requirements, >>> and then what is the behaviour/properties of the current implementation. >>> >>> Does the current implementation meet future requirements? >>> If no (and it seems this is the case for at least some parts) understand >>> what the limitations of the current architecture is. If the current >>> architecture will not be able to support the requirements then it must >>> change. >>> >>> As previous posts have pointed out: since (or if ?) the java layer can >>> be scaled horizontally it will probably not be a bottleneck. But since >>> there is a common database just scaling the java layer may not be >>> sufficient. >>> >>> In any case setting up a potent test environment is the only way to >>> really get the correct answers to questions about where the limits are. >>> >>> In previous thread there was a discussion about to what extent a NewSQL >>> db could be used to increase the scalability. The only thing I want to say >>> here is that there are use cases where by definition distributed >>> transactions guaranteeing strong consistency and availability will never >>> scale as well as eventual consistency architecture á la e.g. actor model >>> with only aggregate level strong consistency guarantees. >>> >>> I think it is crucial to first identify the scalability requirements and >>> understand how the system can be partitioned (including database) to meet >>> the requirements and take the CAP theorem into account doing this. >>> >>> Best regards Niklas >>> >>> >>> >>> >>> >>> Niklas Uhrberg >>> Konsult >>> IT Development | Loans & Credit >>> >>> >>> >>> Resurs Bank AB >>> >>> >>> >>> Växel: +46 42 38 20 00 >>> E-post: [email protected] >>> Webb: www.resursbank.se >>> >>> >>> >>> ------------------------------ >>> *From:* Giorgio Zoppi [[email protected]] >>> *Sent:* Thursday, November 07, 2019 9:51 AM >>> *To:* [email protected] >>> *Cc:* Mifos software development >>> *Subject:* Re: [Mifos-developer] Scalability & Performance Working Group >>> >>> Hi Nazeer, >>> i was thinking about introduce Redis in the current architecture. Do you >>> think it feasible? >>> BR. >>> Giorgio >>> >>> El jue., 7 nov. 2019 a las 9:24, Nazeer Hussain Shaik (< >>> [email protected]>) escribió: >>> >>>> Hi all, >>>> >>>> Before thinking about moving micro services in different technologies >>>> please think about getting the required skill set people and the >>>> complexities around those technologies. >>>> I believe scaling the micro services horizontally with proper hardware >>>> configurations and by using some kind of load balance service (NGINX) may >>>> be good idea. >>>> In my current company we have more than 80 micro services and most of >>>> them are written in Java technologies. We deploy our services on AWS >>>> (Simulating SAAS model). >>>> We use kubernetes for docker orchestration. Currently we are able to >>>> scale 2 million requests. We are working on to scale to 4 million requests >>>> at least. So I don't think Java is bottleneck here. >>>> Please check whether the DB layer and JPA is performing as expected. I >>>> believe you need to look into client facing API(s) instead of hitting the >>>> direct micro services, better to use some kind of Indexers (Elastic Search, >>>> etc..). Or aggregate the data in client facing services properly. >>>> >>>> Please correct me if my thinking is wrong. >>>> >>>> Regards, >>>> Nazeer >>>> >>>> On Thu, Nov 7, 2019 at 12:54 PM Apoorva M.K <[email protected]> >>>> wrote: >>>> >>>>> Hi Ed, >>>>> I'm interested in helping out in this, I have gone through the list of >>>>> next steps. I will start working on what I can. >>>>> >>>>> Regards, >>>>> Apoorva >>>>> >>>>> On Thu 24 Oct, 2019, 1:57 AM Ed Cable, <[email protected]> wrote: >>>>> >>>>>> Hi Mifos and Fineract communities, >>>>>> >>>>>> In follow up to my previous comments on Joseph's thread I wanted to >>>>>> initiate a collaborative community-wide effort to help address the >>>>>> ongoing >>>>>> and growing need for helping Mifos/Fineract scale and sustain high load >>>>>> environments. >>>>>> >>>>>> This really represent an area where we can demonstrate that together >>>>>> as Open Source Community we are much stronger than each individual or >>>>>> partner trying to tackle this alone. >>>>>> >>>>>> This would be for both Fineract and Fineract CN. >>>>>> >>>>>> *Why the need for this Working Group?* >>>>>> The formation of a working group focused on this subject is needed as >>>>>> evident from various mail thread or support requests trying to optimize >>>>>> the >>>>>> system for high load/volume environments and a growing number of >>>>>> prospects >>>>>> eager to use the platform but needing more visibility into its >>>>>> performance >>>>>> and ability to meet high TPS requirements. >>>>>> >>>>>> Secondly, there are many partner-led implementations and deployments >>>>>> of Fineract/Mifos supporting millions of clients and it would be valuable >>>>>> to share that knowledge across the community. >>>>>> >>>>>> Lastly, it's been quite some time since we've had public performance >>>>>> testing done, the efforts led by eSolve in 2017 (see wiki page) and >>>>>> IBM/Conflux >>>>>> in 2015 >>>>>> <https://www.ibm.com/partnerworld/page/stg_ast_sys-mifos-x-on-ibm-powerlinux-servers>both >>>>>> pre-date the recent change from Hibernate to OpenJPA. Kumaranath with >>>>>> support from Avik of Fynarfin worked on performance related issues during >>>>>> 2018 GSOC - >>>>>> https://docs.google.com/document/d/18_awblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW_-In0 >>>>>> <https://smex12-5-en-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2fdocs.google.com%2fdocument%2fd%2f18%5fawblHsI3uZmc7f80Q5HljAGwc2XhIpYOlOtW%5f%2dIn0&umid=ba389591-3c1d-4752-9f0f-b1566097d2f0&auth=5570038ba5c1d86c75907f538183887803b1409d-66acc5307e319e338515ff148c4394046f4e061b> >>>>>> >>>>>> >>>>>> *What/How* >>>>>> >>>>>> 1. Create a set of reproducible tools to enable automation of >>>>>> this on an ongoing basis. >>>>>> 1. Performance testing related data sets - i.e. DB dumps, >>>>>> scripts to generate test data >>>>>> 2. Postman scripts to run load tests >>>>>> 3. Perhaps we can create a repo on github for this? >>>>>> 2. Access to Resources >>>>>> 1. If you have hardware or cloud environments available to >>>>>> conduct these load/performance tests please share. >>>>>> 3. Culture & Transparency - Establish this as a priority area of >>>>>> the community to collectively address. >>>>>> 4. Documentation (likely on the Fineract Wiki) - I created this >>>>>> page to start: https://cwiki.apache.org/confluence/x/khD3Bw >>>>>> 1. Share and document existing results of performance testing >>>>>> to date. Share the details of environments you have set up to >>>>>> address high >>>>>> load needs. >>>>>> 2. Document the scenarios and TPS requirements that need to be >>>>>> tested >>>>>> 3. Document typical improvements to address performance (at a >>>>>> configuration, database, code, level, etc.) >>>>>> 5. Identify issues to fix >>>>>> 1. Log new issues, update existing issues, and tag >>>>>> appropriately. >>>>>> 6. Share code and fixes to address performance issues >>>>>> 1. Make these a priority and plan out in our release >>>>>> roadmapping. >>>>>> >>>>>> *Next Steps* >>>>>> >>>>>> - If interested in joining the group, respond to this thread. >>>>>> - Begin sharing your inputs on the wiki page. >>>>>> - Start fixing existing issues. >>>>>> - Help in creating and executing on our performance testing plan. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Ed >>>>>> >>>>>> Mifos-developer mailing list >>>>>> [email protected] >>>>>> Unsubscribe or change settings at: >>>>>> https://lists.sourceforge.net/lists/listinfo/mifos-developer >>>>> >>>>> >>> >>> -- >>> Life is a chess game - Anonymous. >>> >>>
