Hi Aleks and Courage, Will it make sense to have a Docker Compose setup run on the Apache VM to enable the Front end apps for GSoC? Rather than wait for the Swarm thing which may need more resources? Given there are 2 VMs, we could use one for the Datastores and another for the Services? How complicated is this to do?
What do you think? On Thu, Mar 7, 2019 at 1:37 AM Isaac Kamga <[email protected]> wrote: > Hello fineracters, > > +Aleksandar Vidakovic <[email protected]> , +courage angeh > <[email protected]> > > Is there any progress on this front ? > > Most front-end projects (from last year's GSoC students and other > contributors) need this public demo-server in order to do rigorous testing. > > At Your Service, > Isaac Kamga. > > On Mon, Feb 25, 2019 at 12:33 AM Aleksandar Vidakovic < > [email protected]> wrote: > > > Great, Courage! I'll have a look at this. > > > > On Fri, Feb 22, 2019 at 4:49 PM Courage Angeh <[email protected]> > > wrote: > > > > > Hi Alek, > > > > > > I have migrated the Docker files to their respective Fineract > > repositories. > > > Here are draft PRs for three microservices. Please review it and I will > > > appreciate your feedback on any updates: > > > https://github.com/apache/fineract-cn-identity/pull/8 > > > https://github.com/apache/fineract-cn-office/pull/8 > > > https://github.com/apache/fineract-cn-customer/pull/9 > > > > > > You could review already built docker images on > > > https://cloud.docker.com/u/anh3h/repository/list > > > It's just a rough sample though. > > > > > > In the main time, I am updating the Docker-compose file so it spins up > a > > > swarm cluster. > > > > > > Thanks, > > > Courage. > > > > > > On Fri, Feb 1, 2019 at 8:00 PM Courage Angeh <[email protected]> > > > wrote: > > > > > >> Hi Alek, > > >> > > >> I think we can start with that. > > >> We can connect Docker Hub-Jenkins-GitHub. > > >> If Fineract can't use Jenkins at the moment, then > > >> we can connect GitHub directly to Docker Hub. > > >> > > >> Thanks, > > >> Courage > > >> > > >> On Thu, Jan 31, 2019 at 9:44 PM Aleksandar Vidakovic < > > >> [email protected]> wrote: > > >> > > >>> Hi Courage, > > >>> > > >>> ... would be great if you could help out... especially with your > > >>> knowledge > > >>> about Docker. > > >>> > > >>> Preparing the Git repositories should be fairly easy... another nice > > >>> thing > > >>> to have: some kind of CI server to build and push images to Docker > hub; > > >>> not > > >>> sure if Fineract is currently using Jenkins at Apache... in any case > > not > > >>> a > > >>> big thing... setting up things with Travis or similar is not a big > > deal. > > >>> > > >>> Anything else you can think of? > > >>> > > >>> On Thu, Jan 31, 2019 at 6:27 PM Courage Angeh < > [email protected]> > > >>> wrote: > > >>> > > >>> > Hi Aleksandar, > > >>> > > > >>> > I can work with you on migrating the Fineract services from Docker > > >>> compose > > >>> > to Docker Swarm. > > >>> > Then pushing the Fineract images to Docker Hub so it's easily > > >>> accessible. > > >>> > > > >>> > I think that will require Ed to create a Docker Hub account for > > >>> > Mifos/Fineract. > > >>> > > > >>> > Thanks, > > >>> > Courage. > > >>> > > > >>> > On Sat, Jan 19, 2019 at 1:49 PM Aleksandar Vidakovic < > > >>> > [email protected]> wrote: > > >>> > > > >>> > > Hi Juan, > > >>> > > > > >>> > > ... thanks for chiming in... and thanks for the pointer to the > Git > > >>> > > repository with the Dockerfiles; that makes life already a lot > > >>> easier. > > >>> > > > > >>> > > You are right, once you have the docker-compose.yml files lined > up > > >>> it's > > >>> > not > > >>> > > a big stretch to setup a Swarm cluster. After running such a > > cluster > > >>> for > > >>> > a > > >>> > > client of mine I just want to add: > > >>> > > > > >>> > > - a Swarm cluster with less than 3 nodes will not run very > stable; > > >>> I'm > > >>> > > saying this, because at the moment we have 2 servers at our > > disposal > > >>> (I > > >>> > > think) > > >>> > > - colleagues told me that Swarm cluster worked less reliable for > > them > > >>> > than > > >>> > > other solutions; I don't think for a demo system that is too much > > of > > >>> a > > >>> > > concern, but again I had one running in a production environment > > and > > >>> had > > >>> > no > > >>> > > major problems > > >>> > > > > >>> > > If you want then let's connect ([email protected]) and figure out > > >>> how we > > >>> > > can > > >>> > > proceed with this... before the next GSoC season begins ;-) > > >>> > > > > >>> > > Cheers > > >>> > > > > >>> > > > > >>> > > On Sat, Jan 19, 2019 at 4:03 PM Juhan Aasaru <[email protected]> > > >>> wrote: > > >>> > > > > >>> > > > Hi Aleks > > >>> > > > > > >>> > > > Thanks for your work on pushing the demo server to live. > > >>> > > > I have played around with the containers also and I add my > > >>> feedback and > > >>> > > > ideas. > > >>> > > > > > >>> > > > > - it would come in handy to have default Fineract CN Docker > > >>> images > > >>> > > > published on Docker Hub > > >>> > > > > > >>> > > > I think this is a way to go. If we want to promote adoption of > > >>> > > Fineract-CN > > >>> > > > then public images > > >>> > > > lower the burden to anyone to download and get going with the > > >>> project. > > >>> > > > Does the CI server already exist that could potentially build > the > > >>> > images? > > >>> > > > > > >>> > > > > - I suggest to add a Dockerfile in every microservice Git > > >>> > repository > > >>> > > > (e. > > >>> > > > > g. fineract-cn-customer, fineract-cn-teller, > > >>> fineract-cn-payroll) > > >>> > > and > > >>> > > > let a > > >>> > > > > CI server build and publish Docker images of these > > >>> > > > > > >>> > > > Yes. Most of the Dockerfiles already exist here > > >>> > > > https://github.com/openMF/fineract-cn-containers > > >>> > > > But they logically belong to the application's own code base > so I > > >>> see > > >>> > no > > >>> > > > harm in adding all > > >>> > > > Dockerfiles to the app's own repository. > > >>> > > > > > >>> > > > - I am assuming that we **don't** want to go all the way to > > >>> setup a > > >>> > > > Kubernetes (or even a Docker Swarm) cluster; the goal is to > just > > >>> have a > > >>> > > set > > >>> > > > > > >>> > > > If we plan to operate with docker-compose already (and run in > two > > >>> > > servers) > > >>> > > > then I, in my opinion, it wouldn't be much overhead to create a > > >>> Swarm > > >>> > > > cluster. > > >>> > > > If I look at the instructions ( > > >>> > > https://docs.docker.com/get-started/part4/) > > >>> > > > it doesn't seem like a lot of work. > > >>> > > > Also if something happens then Swarm can detect and relaunch > > >>> > containers. > > >>> > > > But I'm no system administrator myself so I might be mistaken > in > > >>> terms > > >>> > of > > >>> > > > how much work it requires. > > >>> > > > > > >>> > > > > - to avoid code changes or Docker image rebuilds we should > > >>> introduce > > >>> > > > > environment variables in the application.yml files of these > > >>> > > > microservice > > >>> > > > > projects; e. g.: > > >>> > > > > cassandra: > > >>> > > > > clusterName: staging_cluster > > >>> > > > /---/ > > >>> > > > > ... should look something like this: > > >>> > > > > cassandra: > > >>> > > > > clusterName: > > >>> > > > ${FINERACT_CUSTOMER_CASSANDRA_CLUSTER_NAME:staging_cluster} > > >>> > > > /---/ > > >>> > > > > > >>> > > > I think there is no need to change application.yml files. > > >>> > > > In docker-compose.yml you can overwrite any application.yml > > >>> property in > > >>> > > > "environment" section like this: > > >>> > > > > > >>> > > > environment: > > >>> > > > - > > >>> > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > "cassandra.clusterName=${FINERACT_CUSTOMER_CASSANDRA_CLUSTER_NAME:staging_cluster}" > > >>> > > > > > >>> > > > Juhan > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > Kontakt Aleksandar Vidakovic (<[email protected]>) > > >>> kirjutas > > >>> > > > kuupäeval N, 17. jaanuar 2019 kell 03:46: > > >>> > > > > > >>> > > > > Hi everyone, > > >>> > > > > > > >>> > > > > ... hope you all enjoyed the holidays and had a good start > into > > >>> the > > >>> > new > > >>> > > > > year :-) > > >>> > > > > > > >>> > > > > I have to appologize for my radio silence concerning the demo > > >>> server, > > >>> > > > but I > > >>> > > > > got a bit steam rolled by work in the last 6 months. > > >>> > > > > > > >>> > > > > Anyway, I just wanted to get this effort going again and > would > > >>> like > > >>> > to > > >>> > > > > discuss it with anyone interested. > > >>> > > > > > > >>> > > > > The current status: > > >>> > > > > > > >>> > > > > - we have 2 (quite big) servers provided by the Apache > > >>> Foundation > > >>> > to > > >>> > > > run > > >>> > > > > the demo setup > > >>> > > > > - initially I tried to get it running on one, but was not > > >>> enough > > >>> > > (even > > >>> > > > > with 32GB of RAM and some swap configuration tricks) > > >>> > > > > - I've used the demo server module with some minor > > >>> modifications > > >>> > to > > >>> > > > turn > > >>> > > > > off non-essential components (thanks Myrle) > > >>> > > > > > > >>> > > > > Trying all of this took quite some time... even on the beefy > > >>> machine > > >>> > > from > > >>> > > > > Apache it took (as far as I remember) 30-40min until the demo > > >>> server > > >>> > > > > startup would ultimately fail. > > >>> > > > > > > >>> > > > > Instead of going down that route again I'd like to propose a > > >>> > different > > >>> > > > > strategy: > > >>> > > > > > > >>> > > > > - I am assuming that we **don't** want to go all the way > to > > >>> setup > > >>> > a > > >>> > > > > Kubernetes (or even a Docker Swarm) cluster; the goal is > to > > >>> just > > >>> > > have > > >>> > > > a > > >>> > > > > set > > >>> > > > > of docker-compose.yml files to start the system > > >>> > > > > - it would come in handy to have default Fineract CN > Docker > > >>> images > > >>> > > > > published on Docker Hub > > >>> > > > > - I suggest to add a Dockerfile in every microservice Git > > >>> > repository > > >>> > > > (e. > > >>> > > > > g. fineract-cn-customer, fineract-cn-teller, > > >>> fineract-cn-payroll) > > >>> > > and > > >>> > > > > let a > > >>> > > > > CI server build and publish Docker images of these > > >>> > > > > - to avoid code changes or Docker image rebuilds we should > > >>> > introduce > > >>> > > > > environment variables in the application.yml files of > these > > >>> > > > microservice > > >>> > > > > projects; e. g.: > > >>> > > > > > > >>> > > > > [code] > > >>> > > > > ... > > >>> > > > > cassandra: > > >>> > > > > clusterName: staging_cluster > > >>> > > > > contactPoints: 127.0.0.1:9042,127.0.0.2:9042, > 127.0.0.3:9042 > > >>> > > > > keyspace: seshat > > >>> > > > > cl: > > >>> > > > > read: LOCAL_QUORUM > > >>> > > > > write: LOCAL_QUORUM > > >>> > > > > delete: LOCAL_QUORUM > > >>> > > > > ... > > >>> > > > > [/code] > > >>> > > > > > > >>> > > > > ... should look something like this: > > >>> > > > > > > >>> > > > > [code] > > >>> > > > > ... > > >>> > > > > cassandra: > > >>> > > > > clusterName: > > >>> > > > ${FINERACT_CUSTOMER_CASSANDRA_CLUSTER_NAME:staging_cluster} > > >>> > > > > contactPoints: > > ${FINERACT_CUSTOMER_CASSANDRA_CONTACT_ENDPOINTS: > > >>> > > > > 127.0.0.1:9042,127.0.0.2:9042,127.0.0.3:9042} > > >>> > > > > keyspace: ${FINERACT_CUSTOMER_CASSANDRA_KEYSPACE:seshat} > > >>> > > > > cl: > > >>> > > > > read: LOCAL_QUORUM > > >>> > > > > write: LOCAL_QUORUM > > >>> > > > > delete: LOCAL_QUORUM > > >>> > > > > ... > > >>> > > > > [/config] > > >>> > > > > > > >>> > > > > - with the above changes we could then define > > >>> docker-compose.yml > > >>> > > files > > >>> > > > > like this (pseudo file for customer microservice): > > >>> > > > > > > >>> > > > > [code] > > >>> > > > > version: '3.6' > > >>> > > > > > > >>> > > > > services: > > >>> > > > > customer: > > >>> > > > > image: > > >>> > nexus.pelotoninnovations.com/rspndr/server-in-memory:latest > > >>> > > > > depends_on: > > >>> > > > > - mongo > > >>> > > > > env_file: > > >>> > > > > - ./customer.env > > >>> > > > > ports: > > >>> > > > > - "10000:10000" > > >>> > > > > command: sh -c "java -Xmx1024m -Duser.timezone=UTC > > >>> > > > > -Dlogging.config=./logback.xml -jar > > >>> -Djava.net.preferIPv4Stack=true > > >>> > > > > fineract-cn-customer.jar" > > >>> > > > > [/code] > > >>> > > > > > > >>> > > > > ... and the customer.env file would contain something like > > this: > > >>> > > > > > > >>> > > > > [code] > > >>> > > > > FINERACT_CUSTOMER_CASSANDRA_CLUSTER_NAME=prod_cluster > > >>> > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > FINERACT_CUSTOMER_CASSANDRA_CONTACT_ENDPOINTS=server1:9042,server2:9042,server3:9042 > > >>> > > > > FINERACT_CUSTOMER_CASSANDRA_KEYSPACE=seshat > > >>> > > > > [/code] > > >>> > > > > > > >>> > > > > - we would provide templates for those env files (e. g. > > >>> > > > > "customer.env.template"); custom configurations (e. g. > > >>> > > "customer.env") > > >>> > > > > should not be checked into Git > > >>> > > > > - if no environment variables are provided then the > defaults > > >>> in > > >>> > the > > >>> > > > > application.yml config files kick in with reasonable > values > > >>> for a > > >>> > > > local > > >>> > > > > dev > > >>> > > > > machine deployment (given the required resources unlikely > > for > > >>> most > > >>> > > > devs) > > >>> > > > > > > >>> > > > > > > >>> > > > > Advantages: > > >>> > > > > > > >>> > > > > - ready to consume Fineract CN Docker images > > >>> > > > > - no lengthy builds > > >>> > > > > - no re-build (Gradle, Docker) for configuration changes > > >>> > > > > - no requirement to do cluster (Swarm, Kubernetes) setup > > >>> > > > > - the Docker images could still be used as the basic > > building > > >>> > blocks > > >>> > > > of > > >>> > > > > more complex architectures (Kubernetes) > > >>> > > > > - every service can be started/stopped separately which > > makes > > >>> > life a > > >>> > > > lot > > >>> > > > > easier when we have to figure out the right configuration > > for > > >>> the > > >>> > > demo > > >>> > > > > setup (I guess it would make it also easier for others > that > > >>> would > > >>> > > like > > >>> > > > > to > > >>> > > > > setup their own environments) > > >>> > > > > > > >>> > > > > I am using most (if not all) of the required bits and pieces > > for > > >>> this > > >>> > > > setup > > >>> > > > > on a daily basis and I think it should be not too complicated > > to > > >>> get > > >>> > > this > > >>> > > > > working. And it would not interfere (too much) with the > > existing > > >>> Git > > >>> > > > > repositories. > > >>> > > > > > > >>> > > > > Please let me know what you think... > > >>> > > > > > > >>> > > > > Cheers, > > >>> > > > > > > >>> > > > > Aleks > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >> > > >
