Hello Avik, On Wed, Oct 10, 2018 at 5:24 PM Avik Ganguly <[email protected]> wrote:
> Hi guys, > > I am assuming that the suggested form of development for Fineract CN is to > write unit tests to test the code which you just wrote locally and then > push the code to a remote cluster or a massive VM and wait 40+ minutes to > run the API / integration tests there. A lot of the confusion regarding the > stacktraces in identity, rhythm, timeouts and auto-shutdown might or might > not have anything to do with the resource crunch in my local. (i7, 8GB, > SSD) > > Maybe a FAQ section in "How to build Fineract CN" to help devs with local > setup would go a long way to bring in contributions to the core modules. > With the right specifications on your local (i7, 16GB, SSD), you could test your changes instead of having to use a remote VM. So far, what we've done is to ask developers like Graham and Karthik with new knowledge to edit their experiences into the page. A FAQ section in https://cwiki.apache.org/confluence/display/FINERACT/How+To+Build+Apache+Fineract+CN is very welcome and helpful. Maybe you could start by synthesizing the above paragraph into the first Q and A. > > 1. Change demo server orchestration logic from running Identity, > Organization, Customer, Accounting, Portfolio, Deposit, Teller, Reporting, > Cheque, Payroll, Group and Notification services to running only Identity, > Organization, Customer and working set of your choice (Portfolio and > Accounting OR Deposit and Accounting OR Group and Portfolio) > > 2. Leave out Rhythm from orchestration logic for now. A JIRA ticket > tracking ability of Rhythm to run with low resource footprint will probably > help. > > > @Myrle, can you elaborate on FINCN-25? Is the motivation of this ticket to > reduce resource footprint only or also to reduce the startup time from 40 > minutes to a reasonable couple of minutes? Do you have any debugging tips > with respect to the slow startup caused by services is due to registering > with eureka or provisioning or sequential startup? > > Maybe we should also follow up in Cassandra mailing list regarding > configuring Cassandra to run within 8GB of RAM and include the same config > as part of the FAQ. > > > Regards, > Avik. > ᐧ > > On Wed, Sep 19, 2018 at 2:23 AM Courage Angeh <[email protected]> > wrote: > > > Thanks for the update. Ed. > > > > If you need an extra hand please let me know Aleks. > > > > On Tue, Sep 18, 2018, 4:27 PM Ed Cable <[email protected]> wrote: > > > > > Hi all, > > > > > > Just to bring this thread back up to top of everyone's inbox. Aleks > > > estimates he has about a day left to get the demo server up and running > > is > > > hoping to tackle it by tomorrow. Here are the two approaches he > suggested > > > now that we have the second VM allocated: > > > > > > - continue with the JUnit based startup sequence of the demo module; > the > > > one thing that I would need to change is to split up the startup > sequence > > > and recompile > > > - create a set of Docker Compose files to do the startup on the 2 VM > > > instances (maybe I could borrow some stuff from Courage) > > > > > > I would prefer the latter, because you have more fine grained control > and > > > it's easier move service instances around... and you don't have to > > > recompile anything. > > > > > > Ed > > > > > > On Tue, Jul 17, 2018 at 4:09 AM Myrle Krantz <[email protected]> wrote: > > > > > > > Hey Aleks, > > > > > > > > I don’t see anything in that log that would explain the behavior > you’re > > > > observing. Markus did most of the work on deposits. Markus? Do you > have > > > any > > > > ideas? > > > > > > > > Best Regards, > > > > Myrle > > > > > > > > > > > > > > > > On Fri 13. Jul 2018 at 07:06 Aleksandar Vidakovic < > > > > [email protected]> > > > > wrote: > > > > > > > > > @Myrle: with your proposed changes things look a bit better. I've > > > > disabled > > > > > Rhythm as you suggested and get past all microservice startups > with a > > > lot > > > > > less exceptions (one is coming repeatedly from Netty native > > transport, > > > > > because epoll is not available). > > > > > > > > > > The one exception (actually it happened on 2 occasions) that is > still > > > > > there: > > > > > > > > > > [snip] > > > > > > > > > > 13:33:45.080 [main] INFO > > > > o.a.f.c.d.s.DepositAccountManagementApplication - > > > > > Started DepositAccountManagementApplication in 55.187 seconds (JVM > > > > running > > > > > for 62.772) > > > > > 13:34:45.730 [AsyncResolver-bootstrap-executor-0] INFO > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints > via > > > > > configuration > > > > > Resolving artifact > > > > > org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT > > > > > Resolving metadata > > > > > > > > > > > > > > > > > > > > org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml > > > > > from /root/.m2/repository (enhanced) > > > > > Resolved metadata > > > > > > > > > > > > > > > > > > > > org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml > > > > > from /root/.m2/repository (enhanced) > > > > > Resolved artifact > > > > > org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT > > > from > > > > > /root/.m2/repository (enhanced) > > > > > 13:34:56.800 [DiscoveryClient-0] ERROR > > > c.n.discovery.TimedSupervisorTask > > > > - > > > > > task supervisor timed out > > > > > java.util.concurrent.TimeoutException: null > > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205) > > > > > at > > > > > > > > > > > > > > > com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64) > > > > > at > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > > at > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > > > > > at > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > > > > > at > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > > > at > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > > at java.lang.Thread.run(Thread.java:748) > > > > > 13:34:56.801 [DiscoveryClient-1] ERROR > > > c.n.discovery.TimedSupervisorTask > > > > - > > > > > task supervisor timed out > > > > > java.util.concurrent.TimeoutException: null > > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205) > > > > > at > > > > > > > > > > > > > > > com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64) > > > > > at > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > > at > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > > > > > at > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > > > > > at > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > > > at > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > > at java.lang.Thread.run(Thread.java:748) > > > > > 13:34:56.916 [DiscoveryClient-2] ERROR > > > c.n.discovery.TimedSupervisorTask > > > > - > > > > > task supervisor timed out > > > > > java.util.concurrent.TimeoutException: null > > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205) > > > > > at > > > > > > > > > > > > > > > com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64) > > > > > at > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > > at > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > > > > > at > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > > > > > at > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > > > at > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > > at java.lang.Thread.run(Thread.java:748) > > > > > 13:35:08.586 [main] INFO > o.s.c.a.AnnotationConfigApplicationContext > > - > > > > > Refreshing > > > > > > > > > > > > > > > > > > > > org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4 > > > > > : > > > > > startup date [Fri Jul 13 13:35:08 UTC 2018]; root of context > > hierarchy > > > > > 13:35:10.182 [main] INFO > > > o.s.b.f.a.AutowiredAnnotationBeanPostProcessor > > > > - > > > > > JSR-330 'javax.inject.Inject' annotation found and supported for > > > > autowiring > > > > > 13:35:10.371 [main] INFO > > > > > o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker > - > > > Bean > > > > > 'configurationPropertiesRebinderAutoConfiguration' of type [class > > > > > > > > > > > > > > > > > > > > org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$ffaf96f9] > > > > > is not eligible for getting processed by all BeanPostProcessors > (for > > > > > example: not eligible for auto-proxying) > > > > > 13:35:10.435 [background-preinit] INFO > > > > o.h.validator.internal.util.Version > > > > > - HV000001: Hibernate Validator 5.3.0.Final > > > > > > > > > > [/snip] > > > > > > > > > > It looks like the whole startup procedure goes through and I can > see > > > some > > > > > kind of "summary" that lists all REST endpoints. But it stays in > that > > > > state > > > > > for only a minute or so and then a cascade of shutdowns are > triggered > > > > > again. The only thing that stays alive is Eureka which complains > > about > > > > the > > > > > inaccessible services. > > > > > > > > > > Any idea what else we could do other than just ordering an > additional > > > > > machine? > > > > > > > > > > Cheers, > > > > > > > > > > Aleks > > > > > > > > > > On Mon, Jun 18, 2018 at 5:42 PM Myrle Krantz <[email protected]> > > wrote: > > > > > > > > > > > Hey Aleks, > > > > > > > > > > > > It looks like that exception is coming from Rhythm. I suspect > it's > > > not > > > > > the > > > > > > cause of your difficulties, since you say your other services are > > > going > > > > > > down too. You can run most of the Fineract server without > Rhythm. > > > You > > > > > > could edit the demo-server script to leave it out for the > purposes > > of > > > > > > testing. The consequence will be a lack of interest calculations > > for > > > > > > disbursed loans, but for finding out what the real problem is, > it'd > > > be > > > > > good > > > > > > to get this one out of the way. > > > > > > > > > > > > Best Regards, > > > > > > Myrle > > > > > > > > > > > > (FYI: I've removed the non-list subscribers from the to. Most of > > > them > > > > > are > > > > > > already subscribed, and those who aren't, don't want to be.) > > > > > > > > > > > > On Wed, Jun 13, 2018 at 1:40 AM Aleksandar Vidakovic < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > I managed to further improve the startup procedure (aka less > > > > exceptions > > > > > > > than before)... there are still some timeouts, but not as many > as > > > > > before. > > > > > > > > > > > > > > The demo server still dies after this exception: > > > > > > > > > > > > > > 23:27:41.896 [AsyncResolver-bootstrap-executor-0] INFO > > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka > endpoints > > > via > > > > > > > configuration > > > > > > > 23:27:44.564 [AsyncResolver-bootstrap-executor-0] INFO > > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka > endpoints > > > via > > > > > > > configuration > > > > > > > 23:28:43.669 [AsyncResolver-bootstrap-executor-0] INFO > > > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka > endpoints > > > via > > > > > > > configuration > > > > > > > 23:28:45.990 [pool-9-thread-1] ERROR > > > > > > o.s.s.s.TaskUtils$LoggingErrorHandler > > > > > > > - Unexpected error occurred in scheduled task. > > > > > > > > org.springframework.transaction.CannotCreateTransactionException: > > > > Could > > > > > > not > > > > > > > open JPA EntityManager for transaction; nested exception is > > > > > > > javax.persistence.PersistenceException: > > > > > > > java.sql.SQLNonTransientConnectionException: Could not read > > > > resultset: > > > > > > > unexpected end of stream, read 0 bytes from 4 > > > > > > > Query is : set autocommit=0 > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.fineract.cn.rhythm.service.internal.service.Drummer$$EnhancerBySpringCGLIB$$70dfc8e3.checkForBeatsNeeded(<generated>) > > > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) > > > > > > > at > > > > > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > > > > > at > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > Caused by: javax.persistence.PersistenceException: > > > > > > > java.sql.SQLNonTransientConnectionException: Could not read > > > > resultset: > > > > > > > unexpected end of stream, read 0 bytes from 4 > > > > > > > Query is : set autocommit=0 > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700) > > > > > > > at > > > > > > > > > > > > > > > > org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380) > > > > > > > ... 20 common frames omitted > > > > > > > Caused by: java.sql.SQLNonTransientConnectionException: Could > not > > > > read > > > > > > > resultset: unexpected end of stream, read 0 bytes from 4 > > > > > > > Query is : set autocommit=0 > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512) > > > > > > > at > > > > > > > > > > > > > > > > org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45) > > > > > > > ... 22 common frames omitted > > > > > > > Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: > > Could > > > > not > > > > > > > read resultset: unexpected end of stream, read 0 bytes from 4 > > > > > > > Query is : set autocommit=0 > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261) > > > > > > > ... 31 common frames omitted > > > > > > > Caused by: java.io.EOFException: unexpected end of stream, > read 0 > > > > bytes > > > > > > > from 4 > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:168) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900) > > > > > > > ... 33 common frames omitted > > > > > > > 23:29:05.500 [qtp6422064-20] INFO > > > > o.a.f.c.l.c.ServiceExceptionFilter - > > > > > > > Responding with a service error ServiceError{code=409, > > > > > > message='Application > > > > > > > identity-v1 already exists!'} > > > > > > > 23:29:21.282 [Thread-15] INFO > > > > > > > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext - > Closing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26 > > > > > > > : > > > > > > > startup date [Tue Jun 12 23:02:02 UTC 2018]; parent: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4 > > > > > > > 23:29:23.717 [Thread-15] INFO > o.s.c.s.DefaultLifecycleProcessor > > - > > > > > > Stopping > > > > > > > beans in phase 2147483647 > > > > > > > 23:29:24.267 [Thread-15] INFO > o.s.c.s.DefaultLifecycleProcessor > > - > > > > > > Stopping > > > > > > > beans in phase 0 > > > > > > > 23:29:24.557 [Thread-15] INFO > > o.s.b.a.e.jmx.EndpointMBeanExporter > > > - > > > > > > > Unregistering JMX-exposed beans on shutdown > > > > > > > 23:29:24.579 [Thread-15] INFO > > o.s.b.a.e.jmx.EndpointMBeanExporter > > > - > > > > > > > Unregistering JMX-exposed beans > > > > > > > > > > > > > > The only thing I've left: start the databases (aka Cassandra) > on > > a > > > > > > separate > > > > > > > machine (e. g. on Digital Ocean) and see how this is working > > then. > > > > > > > > > > > > > > ... FYI. > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > Aleks > > > > > > > > > > > > > > On Tue, Jun 12, 2018 at 11:16 PM Aleksandar Vidakovic < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > @Rajan: it's not running, because the demo server dies... See > > > logs > > > > > from > > > > > > > > previous message... > > > > > > > > > > > > > > > > We just have to figure that one out and we should be good to > > go. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 12, 2018, 9:11 PM Rajan Maurya < > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > >> Thanks, Alex, > > > > > > > >> > > > > > > > >> I can see web app is running but I can't log in with the > > > > credentials > > > > > > > that > > > > > > > >> you shared earlier > > > > > > > >> > > > > > > > >> Tenant : playground > > > > > > > >> Username : operator > > > > > > > >> Password : init1@l > > > > > > > >> > > > > > > > >> Thanks > > > > > > > >> > > > > > > > >> On Wed, Jun 13, 2018 at 12:32 AM Aleksandar Vidakovic < > > > > > > > >> [email protected]> wrote: > > > > > > > >> > > > > > > > >> > Hi all, > > > > > > > >> > > > > > > > > >> > ... sweeeet! I got a considerable step further. > > > > > > > >> > > > > > > > > >> > @Markus: thanks for the conversation... gave me an idea > how > > to > > > > fix > > > > > > > (most > > > > > > > >> > of) it! > > > > > > > >> > > > > > > > > >> > The problem was - as suspected - the memory (or lack of). > > The > > > VM > > > > > > > >> instance > > > > > > > >> > at Apache has net 30G available. 16G are needed for > > Cassandra > > > > and > > > > > in > > > > > > > my > > > > > > > >> > previous tests I tried to run the demo server with up to > > 16G. > > > > The > > > > > > > >> problem > > > > > > > >> > is that the demo server needs considerably more memory > than > > > 16G. > > > > > > > >> According > > > > > > > >> > to my observation it's around 23-24G. Even if you limit > the > > > demo > > > > > > > server > > > > > > > >> > memory allocation with "-Xmx" to 12G (for example) it > would > > > just > > > > > > > >> continue > > > > > > > >> > to consume more memory. > > > > > > > >> > > > > > > > > >> > My solution: add a swap file. Not ideal in terms of > > > performance, > > > > > but > > > > > > > it > > > > > > > >> > gets the whole thing going... at least mostly (please read > > > on). > > > > > > > >> > > > > > > > > >> > Current status: > > > > > > > >> > > > > > > > > >> > - Cassandra (and all other Docker containers) are still > > > > > running; > > > > > > > >> > Cassandra died in my previous attempts at the latest > when > > > the > > > > > > > Teller > > > > > > > >> > application started, but with the swap file it's still > > > > running > > > > > > > >> > - all Spring Boot apps are starting now > > > > > > > >> > - I get once in a while some spurious timeouts (I think > > > when > > > > > > either > > > > > > > >> the > > > > > > > >> > Eureka server is contacted or the config server) > > > > > > > >> > - there are also sometimes exceptions concerning MySQL > > > > > > connections, > > > > > > > >> but > > > > > > > >> > doesn't seem to matter overall > > > > > > > >> > - I could issue an authentication request with Postman > to > > > > > > > >> > http://fineract-vm.apache.org:2021/identity/v1; just > to > > > test > > > > > if > > > > > > > >> > anything > > > > > > > >> > is responding > > > > > > > >> > - PROBLEM: but the demo server (the Spring Boot apps) > > > > > eventually > > > > > > > dies > > > > > > > >> > after a couple of minutes; I don't think that the > memory > > is > > > > the > > > > > > > >> problem > > > > > > > >> > at > > > > > > > >> > this point; I think it's more of a timeout problem > > > concerning > > > > > the > > > > > > > >> > communication with Eureka and/or the config server > (maybe > > > > > because > > > > > > > the > > > > > > > >> > system is a bit slow) > > > > > > > >> > > > > > > > > >> > ... and here the memory footprint ("free -h") when > > everything > > > is > > > > > > > >> running, > > > > > > > >> > just FYI: > > > > > > > >> > > > > > > > > >> > total used free shared > > > > > buff/cache > > > > > > > >> > available > > > > > > > >> > Mem: 31G 31G 226M 8.4M > > > > > 169M > > > > > > > >> > 47M > > > > > > > >> > Swap: 15G 6.8G 9.2G > > > > > > > >> > > > > > > > > >> > ... after the Payroll application is started (approx.) I > see > > > > these > > > > > > > >> > exceptions appearing in the logs (a whole series of them): > > > > > > > >> > > > > > > > > >> > 18:19:36.465 [AsyncResolver-bootstrap-executor-0] INFO > > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka > > > endpoints > > > > > via > > > > > > > >> > configuration > > > > > > > >> > Resolving artifact > > > > > > > >> > > > > > org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT > > > > > > > >> > Resolving metadata > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml > > > > > > > >> > from /root/.m2/repository (enhanced) > > > > > > > >> > Resolved metadata > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml > > > > > > > >> > from /root/.m2/repository (enhanced) > > > > > > > >> > Resolved artifact > > > > > > > >> > > > > > org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT > > > > > > > from > > > > > > > >> > /root/.m2/repository (enhanced) > > > > > > > >> > 18:19:44.689 [DiscoveryClient-0] ERROR > > > > > > > >> c.n.discovery.TimedSupervisorTask - > > > > > > > >> > task supervisor timed out > > > > > > > >> > java.util.concurrent.TimeoutException: null > > > > > > > >> > at > java.util.concurrent.FutureTask.get(FutureTask.java:205) > > > > > > > >> > at > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64) > > > > > > > >> > at > > > > > > > >> > > > > > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > > > > > >> > at > java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > > > > >> > at > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > > > > > > > >> > at > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > > > > > > > >> > at > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > > > > > >> > at > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > > > > >> > at java.lang.Thread.run(Thread.java:748) > > > > > > > >> > > > > > > > > >> > ... but it still seems to have no problem starting the > next > > > app > > > > > > > (Group) > > > > > > > >> and > > > > > > > >> > things look good for a while... > > > > > > > >> > > > > > > > > >> > ... until suddenly it seems that app instances are > beginning > > > to > > > > > > > >> shutdown... > > > > > > > >> > and I'm not sure why this happens: > > > > > > > >> > > > > > > > > >> > 18:27:19.603 [AsyncResolver-bootstrap-executor-0] INFO > > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka > > > endpoints > > > > > via > > > > > > > >> > configuration > > > > > > > >> > 18:27:39.974 [qtp1631119258-22] INFO > > > > > > > >> c.datastax.driver.core.ClockFactory - > > > > > > > >> > Using native clock to generate timestamps. > > > > > > > >> > 18:27:39.975 [qtp1631119258-22] WARN > > > > > > > >> c.datastax.driver.core.CodecRegistry > > > > > > > >> > - Ignoring codec LocalDateTimeCodec [timestamp <-> > > > > > > > >> java.time.LocalDateTime] > > > > > > > >> > because it collides with previously registered codec > > > > > > > LocalDateTimeCodec > > > > > > > >> > [timestamp <-> java.time.LocalDateTime] > > > > > > > >> > 18:27:41.309 [qtp1631119258-22] INFO > > > > > > > c.d.d.c.p.DCAwareRoundRobinPolicy > > > > > > > >> - > > > > > > > >> > Using data-center name 'datacenter1' for > > > DCAwareRoundRobinPolicy > > > > > (if > > > > > > > >> this > > > > > > > >> > is incorrect, please provide the correct datacenter name > > with > > > > > > > >> > DCAwareRoundRobinPolicy constructor) > > > > > > > >> > 18:27:41.319 [qtp1631119258-22] INFO > > > > > > > com.datastax.driver.core.Cluster - > > > > > > > >> > New Cassandra host /127.0.0.1:9042 added > > > > > > > >> > 18:27:41.661 [AsyncResolver-bootstrap-executor-0] INFO > > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka > > > endpoints > > > > > via > > > > > > > >> > configuration > > > > > > > >> > 18:27:51.110 [Thread-15] INFO > > > > > > > >> > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext - > > > > Closing > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26 > > > > > > > >> > : > > > > > > > >> > startup date [Tue Jun 12 17:55:35 UTC 2018]; parent: > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4 > > > > > > > >> > 18:27:51.341 [AsyncResolver-bootstrap-executor-0] INFO > > > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka > > > endpoints > > > > > via > > > > > > > >> > configuration > > > > > > > >> > 18:27:53.373 [Thread-15] INFO > > > > o.s.c.s.DefaultLifecycleProcessor - > > > > > > > >> Stopping > > > > > > > >> > beans in phase 2147483647 > > > > > > > >> > 18:27:54.331 [Thread-15] INFO > > > > o.s.c.s.DefaultLifecycleProcessor - > > > > > > > >> Stopping > > > > > > > >> > beans in phase 0 > > > > > > > >> > 18:27:54.736 [Thread-15] INFO > > > > > o.s.b.a.e.jmx.EndpointMBeanExporter - > > > > > > > >> > Unregistering JMX-exposed beans on shutdown > > > > > > > >> > 18:27:54.740 [Thread-15] INFO > > > > > o.s.b.a.e.jmx.EndpointMBeanExporter - > > > > > > > >> > Unregistering JMX-exposed beans > > > > > > > >> > 18:27:54.798 [Thread-15] INFO > > > > o.s.j.e.a.AnnotationMBeanExporter - > > > > > > > >> > Unregistering JMX-exposed beans on shutdown > > > > > > > >> > 18:27:54.800 [Thread-15] INFO > > > > o.s.j.e.a.AnnotationMBeanExporter - > > > > > > > >> > Unregistering JMX-exposed beans > > > > > > > >> > 18:27:54.937 [Thread-15] INFO > > > > > > > >> > o.s.o.j.LocalContainerEntityManagerFactoryBean - Closing > JPA > > > > > > > >> > EntityManagerFactory for persistence unit 'metaPU' > > > > > > > >> > 18:27:58.376 [async-processor-1] INFO > > > > > > > >> o.f.c.internal.util.VersionPrinter - > > > > > > > >> > Flyway 3.2.1 by Boxfuse > > > > > > > >> > 18:27:58.677 [async-processor-1] INFO > > > > > > > >> o.f.c.i.dbsupport.DbSupportFactory - > > > > > > > >> > Database: jdbc:mysql://localhost:3306/seshat (MySQL 10.3) > > > > > > > >> > 18:28:00.620 [async-processor-1] INFO > > > > > > > >> o.f.core.internal.command.DbValidate > > > > > > > >> > - Validated 2 migrations (execution time 00:01.169s) > > > > > > > >> > 18:28:00.901 [Thread-15] INFO > > > > o.e.jetty.server.AbstractConnector > > > > > - > > > > > > > >> Stopped > > > > > > > >> > ServerConnector@e162a35{HTTP/1.1,[http/1.1]}{0.0.0.0:2020 > } > > > > > > > >> > 18:28:01.268 [Thread-15] INFO > > > > > o.e.j.server.handler.ContextHandler - > > > > > > > >> > Stopped o.s.b.c.e.j.JettyEmbeddedWebAppContext@10850d17 > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > {/provisioner/v1,file:///tmp/jetty-docbase.5486166363042759626.2020/,UNAVAILABLE} > > > > > > > >> > > > > > > > > >> > ... the list is actually a lot longer... in the rest of > the > > > log > > > > > you > > > > > > > can > > > > > > > >> see > > > > > > > >> > that one by one all the apps are shutting down. > > > > > > > >> > > > > > > > > >> > The last (error) entries I see are these: > > > > > > > >> > > > > > > > > >> > E18:29:31.278 [DefaultMessageListenerContainer-3] ERROR > > > > > > > >> > o.s.j.l.DefaultMessageListenerContainer - Could not > refresh > > > JMS > > > > > > > >> Connection > > > > > > > >> > for destination 'cheques-v1' - retrying using > > > > > > > >> FixedBackOff{interval=5000, > > > > > > > >> > currentAttempts=0, maxAttempts=unlimited}. Cause: Error > > while > > > > > > > >> attempting to > > > > > > > >> > retrieve a connection from the pool; nested exception is > > > > > > > >> > javax.jms.JMSException: Could not connect to broker URL: > > > > > > > >> > tcp://localhost:61616. Reason: java.net.ConnectException: > > > > > Connection > > > > > > > >> > refused (Connection refused) > > > > > > > >> > > > > > > > > >> > ... I guess an embedded ActiveMQ was also shutdown in the > > > > process > > > > > of > > > > > > > the > > > > > > > >> > previous errors. > > > > > > > >> > > > > > > > > >> > So far from here... any input on the above highly > > appreciated. > > > > > > > Overall I > > > > > > > >> > think we are close. > > > > > > > >> > > > > > > > > >> > Cheers, > > > > > > > >> > > > > > > > > >> > Aleks > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > On Tue, Jun 12, 2018 at 6:16 PM Markus Geiss < > > [email protected] > > > > > > > > > > wrote: > > > > > > > >> > > > > > > > > >> > > One thing we maybe can do, given we are part of the > Apache > > > > > Family, > > > > > > > is > > > > > > > >> to > > > > > > > >> > > ask the Cassandra community for some suggestions. > > > > > > > >> > > > > > > > > > >> > > Cheers > > > > > > > >> > > > > > > > > > >> > > Markus > > > > > > > >> > > > > > > > > > >> > > On Tue, Jun 12, 2018 at 6:04 PM Markus Geiss < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > >> > > > > > > > > > >> > > > Hey all, > > > > > > > >> > > > > > > > > > > >> > > > we are not running Cassandra in a container, we have > > > > dedicated > > > > > > VMs > > > > > > > >> to > > > > > > > >> > run > > > > > > > >> > > > our cluster. > > > > > > > >> > > > > > > > > > > >> > > > Given this we are not the best persons to help here, > > > sorry. > > > > (; > > > > > > > >> > > > > > > > > > > >> > > > Cheers > > > > > > > >> > > > > > > > > > > >> > > > Markus > > > > > > > >> > > > > > > > > > > >> > > > On Tue, Jun 12, 2018 at 10:35 AM Ed Cable < > > > > [email protected]> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > > > > >> > > >> Adding Patric directly to this thread too so he can > > give > > > > his > > > > > > > input > > > > > > > >> > > >> regarding to the challenges with Cassandra. > > > > > > > >> > > >> > > > > > > > >> > > >> Ed > > > > > > > >> > > >> On Tue, Jun 12, 2018 at 12:53 AM Aleksandar > Vidakovic < > > > > > > > >> > > >> [email protected]> wrote: > > > > > > > >> > > >> > > > > > > > >> > > >>> Hi Victor, > > > > > > > >> > > >>> > > > > > > > >> > > >>> .... this is the relevant part of the > > docker-compose.yml > > > > > file: > > > > > > > >> > > >>> > > > > > > > >> > > >>> cassandra0: > > > > > > > >> > > >>> image: cassandra:3.11.1 > > > > > > > >> > > >>> container_name: cassandra0 > > > > > > > >> > > >>> ports: > > > > > > > >> > > >>> - 9042:9042 > > > > > > > >> > > >>> - 9160:9160 > > > > > > > >> > > >>> - 7199:7199 > > > > > > > >> > > >>> - 8778:8778 > > > > > > > >> > > >>> volumes: > > > > > > > >> > > >>> - ./cassandra:/etc/cassandra > > > > > > > >> > > >>> environment: > > > > > > > >> > > >>> - CASSANDRA_START_RPC=true > > > > > > > >> > > >>> - CASSANDRA_SEEDS=cassandra0 > > > > > > > >> > > >>> - CASSANDRA_CLUSTER_NAME=fineract_cluster > > > > > > > >> > > >>> ulimits: > > > > > > > >> > > >>> memlock: -1 > > > > > > > >> > > >>> nproc: 32768 > > > > > > > >> > > >>> nofile: 100000 > > > > > > > >> > > >>> > > > > > > > >> > > >>> ... and just to be complete... here's the Docker > > service > > > > > > > >> > configuration > > > > > > > >> > > >>> ("/lib/systemd/system/docker.service"): > > > > > > > >> > > >>> > > > > > > > >> > > >>> [Unit] > > > > > > > >> > > >>> Description=Docker Application Container Engine > > > > > > > >> > > >>> Documentation=https://docs.docker.com > > > > > > > >> > > >>> After=network-online.target docker.socket > > > > firewalld.service > > > > > > > >> > > >>> Wants=network-online.target > > > > > > > >> > > >>> Requires=docker.socket > > > > > > > >> > > >>> > > > > > > > >> > > >>> [Service] > > > > > > > >> > > >>> Type=notify > > > > > > > >> > > >>> # the default is not to use systemd for cgroups > > because > > > > the > > > > > > > >> delegate > > > > > > > >> > > >>> issues > > > > > > > >> > > >>> still > > > > > > > >> > > >>> # exists and systemd currently does not support the > > > cgroup > > > > > > > feature > > > > > > > >> > set > > > > > > > >> > > >>> required > > > > > > > >> > > >>> # for containers run by docker > > > > > > > >> > > >>> ExecStart=/usr/bin/dockerd -H fd:// > > > > > > > >> > > >>> ExecReload=/bin/kill -s HUP $MAINPID > > > > > > > >> > > >>> LimitNOFILE=1048576 > > > > > > > >> > > >>> # Having non-zero Limit*s causes performance > problems > > > due > > > > to > > > > > > > >> > accounting > > > > > > > >> > > >>> overhead > > > > > > > >> > > >>> # in the kernel. We recommend using cgroups to do > > > > > > > container-local > > > > > > > >> > > >>> accounting. > > > > > > > >> > > >>> LimitNPROC=infinity > > > > > > > >> > > >>> LimitCORE=infinity > > > > > > > >> > > >>> LimitMEMLOCK=infinity > > > > > > > >> > > >>> # Uncomment TasksMax if your systemd version > supports > > > it. > > > > > > > >> > > >>> # Only systemd 226 and above support this version. > > > > > > > >> > > >>> TasksMax=infinity > > > > > > > >> > > >>> TimeoutStartSec=0 > > > > > > > >> > > >>> # set delegate yes so that systemd does not reset > the > > > > > cgroups > > > > > > of > > > > > > > >> > docker > > > > > > > >> > > >>> containers > > > > > > > >> > > >>> Delegate=yes > > > > > > > >> > > >>> # kill only the docker process, not all processes in > > the > > > > > > cgroup > > > > > > > >> > > >>> KillMode=process > > > > > > > >> > > >>> # restart the docker process if it exits prematurely > > > > > > > >> > > >>> Restart=on-failure > > > > > > > >> > > >>> StartLimitBurst=3 > > > > > > > >> > > >>> StartLimitInterval=60s > > > > > > > >> > > >>> > > > > > > > >> > > >>> [Install] > > > > > > > >> > > >>> WantedBy=multi-user.target > > > > > > > >> > > >>> > > > > > > > >> > > >>> ... the one tweak I did there is to set LimitMEMLOCK > > to > > > > > > > >> infinity... > > > > > > > >> > do > > > > > > > >> > > >>> you > > > > > > > >> > > >>> think the problems are file handle related? Should I > > set > > > > > > > >> LimitNOFILE > > > > > > > >> > > also > > > > > > > >> > > >>> to infinity? > > > > > > > >> > > >>> > > > > > > > >> > > >>> Appreciate the help. > > > > > > > >> > > >>> > > > > > > > >> > > >>> Cheers, > > > > > > > >> > > >>> > > > > > > > >> > > >>> Aleks > > > > > > > >> > > >>> > > > > > > > >> > > >>> On Tue, Jun 12, 2018 at 9:09 AM Victor Romero < > > > > > > > >> > > >>> [email protected]> > > > > > > > >> > > >>> wrote: > > > > > > > >> > > >>> > > > > > > > >> > > >>> > Hi Aleks, > > > > > > > >> > > >>> > > > > > > > > >> > > >>> > Can you share the ulimits flags that the > cassandra's > > > > > > container > > > > > > > >> is > > > > > > > >> > > using > > > > > > > >> > > >>> > while it is running? > > > > > > > >> > > >>> > > > > > > > > >> > > >>> > The values are being set in the composer file? Or > in > > > the > > > > > > > >> upstart or > > > > > > > >> > > >>> > systemd docker's deamon config files? > > > > > > > >> > > >>> > > > > > > > > >> > > >>> > > > > > > > > >> > > >>> > > > > > > > > >> > > >>> > Enviado desde TypeApp > > > > > > > >> > > >>> > > > > > > > > >> > > >>> > En jun. 11, 2018 10:02 AM, en 10:02 AM, Aleksandar > > > > > > Vidakovic < > > > > > > > >> > > >>> > [email protected]> escribió: > > > > > > > >> > > >>> > >... and I should have attached the Cassandra log > > > > dump... > > > > > so > > > > > > > >> here > > > > > > > >> > it > > > > > > > >> > > >>> > >is... > > > > > > > >> > > >>> > > > > > > > > > >> > > >>> > >On Mon, Jun 11, 2018 at 4:49 PM Aleksandar > > Vidakovic > > > < > > > > > > > >> > > >>> > >[email protected]> wrote: > > > > > > > >> > > >>> > > > > > > > > > >> > > >>> > >> Hi all, > > > > > > > >> > > >>> > >> > > > > > > > >> > > >>> > >> ... as you might have noticed I did a couple of > > > > > restarts > > > > > > > >> > today... > > > > > > > >> > > >>> the > > > > > > > >> > > >>> > >> problem I am facing now: I can't get Cassandra > to > > > run > > > > > in > > > > > > a > > > > > > > >> > stable > > > > > > > >> > > >>> > >way. > > > > > > > >> > > >>> > >> > > > > > > > >> > > >>> > >> Things look quite OK for a while when running > the > > > > demo > > > > > > > >> server, > > > > > > > >> > but > > > > > > > >> > > >>> > >then > > > > > > > >> > > >>> > >> suddenly Cassandra dies (sometimes it starts > > > > > misbehaving > > > > > > > with > > > > > > > >> > the > > > > > > > >> > > >>> > >deposit > > > > > > > >> > > >>> > >> microservice startup, sometimes with portfolio > or > > > > > > teller). > > > > > > > I > > > > > > > >> > tried > > > > > > > >> > > >>> to > > > > > > > >> > > >>> > >> increase the memory (4G, 8G and 16G) and set > some > > > > > Docker > > > > > > > >> limits > > > > > > > >> > to > > > > > > > >> > > >>> > >> "infinity" (especially LimitMEMLOCK). > > > > > > > >> > > >>> > >> > > > > > > > >> > > >>> > >> I've attached Cassandra's log dump... maybe > > someone > > > > can > > > > > > > help > > > > > > > >> out > > > > > > > >> > > >>> > >here? Is > > > > > > > >> > > >>> > >> it even possible to run Fineract CN on 32GB of > > > > memory? > > > > > > > >> > > >>> > >> > > > > > > > >> > > >>> > >> Other than that the setup would be ready to > go... > > > we > > > > > just > > > > > > > >> need > > > > > > > >> > to > > > > > > > >> > > >>> get > > > > > > > >> > > >>> > >the > > > > > > > >> > > >>> > >> database running more reliably. > > > > > > > >> > > >>> > >> > > > > > > > >> > > >>> > >> Cheers, > > > > > > > >> > > >>> > >> > > > > > > > >> > > >>> > >> Aleks > > > > > > > >> > > >>> > >> > > > > > > > >> > > >>> > >> On Mon, Jun 11, 2018 at 2:38 PM Aleksandar > > > Vidakovic > > > > < > > > > > > > >> > > >>> > >> [email protected]> wrote: > > > > > > > >> > > >>> > >> > > > > > > > >> > > >>> > >>> @Rajan: I had to restart it again (needed to > add > > > > some > > > > > > > >> > additional > > > > > > > >> > > >>> > >reverse > > > > > > > >> > > >>> > >>> proxy configuration for the web UI to work)... > > and > > > > > there > > > > > > > >> were > > > > > > > >> > > more > > > > > > > >> > > >>> > >>> exceptions that I hope will be fixed now. > > > > > > > >> > > >>> > >>> > > > > > > > >> > > >>> > >>> Just FYI > > > > > > > >> > > >>> > >>> > > > > > > > >> > > >>> > >>> On Mon, Jun 11, 2018 at 1:47 PM Aleksandar > > > > Vidakovic < > > > > > > > >> > > >>> > >>> [email protected]> wrote: > > > > > > > >> > > >>> > >>> > > > > > > > >> > > >>> > >>>> No prob.... Let me know how it goes... Can > get > > > back > > > > > to > > > > > > > >> testing > > > > > > > >> > > >>> only > > > > > > > >> > > >>> > >>>> later tonight. > > > > > > > >> > > >>> > >>>> > > > > > > > >> > > >>> > >>>> Cheers > > > > > > > >> > > >>> > >>>> > > > > > > > >> > > >>> > >>>> > > > > > > > >> > > >>> > >>>> On Mon, Jun 11, 2018, 1:45 PM Rajan Maurya > > > > > > > >> > > >>> > ><[email protected]> > > > > > > > >> > > >>> > >>>> wrote: > > > > > > > >> > > >>> > >>>> > > > > > > > >> > > >>> > >>>>> Sorry missed the 30 min, I will test after > 30 > > > min. > > > > > > > >> > > >>> > >>>>> > > > > > > > >> > > >>> > >>>>> Big thanks for this 🙂 > > > > > > > >> > > >>> > >>>>> > > > > > > > >> > > >>> > >>>>> > > > > > > > >> > > >>> > >>>>> > > > > > > > >> > > >>> > >>>>> > > > > > > > >> > > >>> > >>>>> On Mon, Jun 11, 2018 at 5:12 PM Aleksandar > > > > > Vidakovic < > > > > > > > >> > > >>> > >>>>> [email protected]> wrote: > > > > > > > >> > > >>> > >>>>> > > > > > > > >> > > >>> > >>>>> > @Rajan: can't see the image you posted. > > > > > > > >> > > >>> > >>>>> > > > > > > > > >> > > >>> > >>>>> > ... and as I said: the services are still > > > > > > starting... > > > > > > > >> and > > > > > > > >> > the > > > > > > > >> > > >>> > >best > > > > > > > >> > > >>> > >>>>> estimate > > > > > > > >> > > >>> > >>>>> > I have right now (as already mentioned): > > 30min > > > > > > > >> > > >>> > >>>>> > > > > > > > > >> > > >>> > >>>>> > On Mon, Jun 11, 2018 at 1:40 PM Rajan > > Maurya < > > > > > > > >> > > >>> > >>>>> [email protected]> > > > > > > > >> > > >>> > >>>>> > wrote: > > > > > > > >> > > >>> > >>>>> > > > > > > > > >> > > >>> > >>>>> > > [image: image.png] > > > > > > > >> > > >>> > >>>>> > > > > > > > > > >> > > >>> > >>>>> > > > > > > > > > >> > > >>> > >>>>> > > > > > > > > > >> > > >>> > >>>>> > > > > > > > > > >> > > >>> > >>>>> > > > > > > > > > >> > > >>> > >>>>> > > On Mon, Jun 11, 2018 at 5:08 PM Rajan > > > Maurya < > > > > > > > >> > > >>> > >>>>> [email protected]> > > > > > > > >> > > >>> > >>>>> > > wrote: > > > > > > > >> > > >>> > >>>>> > > > > > > > > > >> > > >>> > >>>>> > >> [image: image.png] > > > > > > > >> > > >>> > >>>>> > >> I am getting this. > > > > > > > >> > > >>> > >>>>> > >> > > > > > > > >> > > >>> > >>>>> > >> > > > > > > > >> > > >>> > >>>>> > >> > > > > > > > >> > > >>> > >>>>> > >> > > > > > > > >> > > >>> > >>>>> > >> > > > > > > > >> > > >>> > >>>>> > >> On Mon, Jun 11, 2018 at 5:05 PM > > Aleksandar > > > > > > > Vidakovic > > > > > > > >> < > > > > > > > >> > > >>> > >>>>> > >> [email protected]> wrote: > > > > > > > >> > > >>> > >>>>> > >> > > > > > > > >> > > >>> > >>>>> > >>> Hi all, > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> so... the demo server is (almost) > > ready... > > > > it > > > > > > took > > > > > > > >> me a > > > > > > > >> > > >>> > >moment > > > > > > > >> > > >>> > >>>>> and a > > > > > > > >> > > >>> > >>>>> > >>> couple > > > > > > > >> > > >>> > >>>>> > >>> of restarts to figure out some boot > > > > > failures... > > > > > > > the > > > > > > > >> > > >>> services > > > > > > > >> > > >>> > >are > > > > > > > >> > > >>> > >>>>> quite > > > > > > > >> > > >>> > >>>>> > >>> resource hungry and default settings > > won't > > > > do > > > > > > it. > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> The services are still starting and > this > > > > will > > > > > > > take a > > > > > > > >> > > while > > > > > > > >> > > >>> > >(my > > > > > > > >> > > >>> > >>>>> best > > > > > > > >> > > >>> > >>>>> > guess > > > > > > > >> > > >>> > >>>>> > >>> right now is around 30min or so). > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> You can access the web UI at: > > > > > > > >> > > >>> > >http://fineract-vm.apache.org/login > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> Credentials: > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> Tenant : playground > > > > > > > >> > > >>> > >>>>> > >>> Username : operator > > > > > > > >> > > >>> > >>>>> > >>> Password : init1@l > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> Note (to self): the safest way to > > compile > > > > this > > > > > > app > > > > > > > >> is > > > > > > > >> > > with > > > > > > > >> > > >>> > >NodeJS > > > > > > > >> > > >>> > >>>>> > 8.11.1 > > > > > > > >> > > >>> > >>>>> > >>> (I tried with 10.3.0 before, won't > > work). > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> ... and the webservices at: > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> Identity Service: > > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2021/identity/v1 > > > > > > > >> > > >>> > >>>>> > >>> Office Service: > > > > > > > >> > > >>> http://fineract-vm.apache.org:2023/office/v1 > > > > > > > >> > > >>> > >>>>> > >>> Customer Service: > > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2024/customer/v1 > > > > > > > >> > > >>> > >>>>> > >>> Accounting Service: > > > > > > > >> > > >>> > >>>>> > > > http://fineract-vm.apache.org:2025/accounting/v1 > > > > > > > >> > > >>> > >>>>> > >>> Portfolio Service: > > > > > > > >> > > >>> > >>>>> > > http://fineract-vm.apache.org:2026/portfolio/v1 > > > > > > > >> > > >>> > >>>>> > >>> Deposit Service: > > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2027/deposit/v1 > > > > > > > >> > > >>> > >>>>> > >>> Teller Service: > > > > > > > >> > > >>> http://fineract-vm.apache.org:2028/teller/v1 > > > > > > > >> > > >>> > >>>>> > >>> Reporting Service: > > > > > > > >> > > >>> > >>>>> > > http://fineract-vm.apache.org:2029/reporting/v1 > > > > > > > >> > > >>> > >>>>> > >>> Cheque Service: > > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2030/cheques/v1 > > > > > > > >> > > >>> > >>>>> > >>> Payroll Service: > > > > > > > >> > > >>> > >http://fineract-vm.apache.org:2031/payroll/v1 > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> Note: restarting the services takes > > quite > > > a > > > > > > > >> while... if > > > > > > > >> > > you > > > > > > > >> > > >>> > >>>>> encounter > > > > > > > >> > > >>> > >>>>> > >>> connection problems retry a couple of > > > > minutes > > > > > > > later > > > > > > > >> to > > > > > > > >> > > >>> > >ensure I > > > > > > > >> > > >>> > >>>>> am not > > > > > > > >> > > >>> > >>>>> > >>> currently working on something. > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> For the moment I'll restart the > backend > > > > > services > > > > > > > >> once > > > > > > > >> > per > > > > > > > >> > > >>> > >day or > > > > > > > >> > > >>> > >>>>> so to > > > > > > > >> > > >>> > >>>>> > >>> reset the data; I guess we have to > > figure > > > > out > > > > > > how > > > > > > > we > > > > > > > >> > want > > > > > > > >> > > >>> to > > > > > > > >> > > >>> > >>>>> handle > > > > > > > >> > > >>> > >>>>> > this > > > > > > > >> > > >>> > >>>>> > >>> (also concerning passwords etc.). > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> If you have any suggestions where to > put > > > > this > > > > > > demo > > > > > > > >> > server > > > > > > > >> > > >>> > >>>>> configuration > > > > > > > >> > > >>> > >>>>> > >>> then let me know here. > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> Let me know if you encounter any > > problems > > > (I > > > > > > have > > > > > > > >> not > > > > > > > >> > > >>> > >extensively > > > > > > > >> > > >>> > >>>>> > tested > > > > > > > >> > > >>> > >>>>> > >>> it > > > > > > > >> > > >>> > >>>>> > >>> yet). > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> Cheers, > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> Aleks > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> On Mon, Jun 11, 2018 at 12:00 PM > > > Aleksandar > > > > > > > >> Vidakovic < > > > > > > > >> > > >>> > >>>>> > >>> [email protected]> wrote: > > > > > > > >> > > >>> > >>>>> > >>> > > > > > > > >> > > >>> > >>>>> > >>> > Progress! All the modules seem to be > > > > > starting > > > > > > > fine > > > > > > > >> > > now... > > > > > > > >> > > >>> > >just > > > > > > > >> > > >>> > >>>>> have > > > > > > > >> > >
