I’ve tried to reproduce this today - tried triggering all kinds of operations that I could think of, and monitored the state of the connections. Unfortunately I was not able to reproduce any “frozen” connections. Would be great if someone could give some pointers on how to reproduce this - is it specific operations that result in this, or combinations of operations in parallel, or…?
Thanks Petri > On 9 Dec 2021, at 06:50, Petri Tuomola <[email protected]> wrote: > > Hi > > Interesting... I'd like to troubleshoot this further to see how we can best > fix this. Perhaps the default transaction handling behavior has changed on > one of the recent upgrades (eg Spring Boot, Jersey). > > Would you have any specific example methods where you see this happening? I > haven't come across this so far, so would like to know where to look. > > Thanks > > Regards > Petri > > > On Wed, Dec 8, 2021, 19:16 <[email protected] > <mailto:[email protected]>> wrote: > Bharath and Francis, > > > > The issue with users who are unable to login to their instances seems to be > because of some complex methods in REST controllers are executing multiple > changes in DB through JPA repositories. For some reasons calls to these > interfaces resulted in a growing number of "freezed" active connections, > exhausting the pool. Either annotating these methods as @Transactional …or > enveloping all the logic in a single call to transactional service method > seems to solve the problem. You can apply it to the relevant methods as there > are a growing number of issues surrounding failed connections. > > > > Regards, > > Sifiso > > > > From: hitesh prajapati <[email protected] > <mailto:[email protected]>> > Sent: Sunday, 28 November 2021 4:24 PM > To: [email protected] <mailto:[email protected]> > Subject: Re: Fineract Startup issue - latest develop branch > > > > Hi Petri, > > > > Thanks for your suggestion. > > > > I don't have any data in my database. So I deleted my db and re-created them > again. After that i run bootRun and the application is running fine. > > > > But I forgot to check the c_configurations table for an entry with name = > custom-account-number-length, before deleting it and recreating it. > > > > Thanks, > > Hitesh > > > > > > > > > > On Sun, Nov 28, 2021 at 4:26 PM hitesh prajapati > <[email protected] <mailto:[email protected]>> > wrote: > > Hi Petri, > > > > Thanks for your reply. I will check and let you know. > > > > On Sun, 28 Nov 2021, 16:21 Petri Tuomola, <[email protected] > <mailto:[email protected]>> wrote: > > Hi - can you check if your table c_configurations already has an entry with > name = custom-account-number-length? > > > > For some reason it seems Flyway is trying to insert a config entry that is > already there. Perhaps a previous startup failed? > > > > If the record is there, then remove that row and retry… > > > > Regards > > Petri > > > > On 28 Nov 2021, at 18:26, hitesh prajapati <[email protected] > <mailto:[email protected]>> wrote: > > > > Hi All, > > > > I have taken the latest develop branch from community and run bootRun. The > last commit > > > > https://github.com/apache/fineract/commit/8dc5789607dfa51a9186e1971998550df8e85374 > > <https://github.com/apache/fineract/commit/8dc5789607dfa51a9186e1971998550df8e85374> > > > But the fineract application is failing while starting. Lookslike some issue > with > sql/migrations/core_db/V378__configuration_for_variable_length_acoountNumber.sql > > > > Below the error, I am getting > > > > 2021-11-28 15:48:13.797 INFO 19340 --- [ main] > org.apache.fineract.ServerApplication : Starting ServerApplication using > Java 11.0.12 on DESKTOP-92RI6NG with PID 19340 > (D:\Work\fineract\source\fineract\fineract-provider\build\classes\java\main > started by hites in D:\Work\fineract\source\fineract\fineract-provider) > 2021-11-28 15:48:13.797 INFO 19340 --- [ main] > org.apache.fineract.ServerApplication : No active profile set, falling > back to default profiles: basicauth > 2021-11-28 15:48:21.279 INFO 19340 --- [ main] > o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): > 8080 (http) 8443 (https) > 2021-11-28 15:48:22.950 WARN 19340 --- [ main] > o.f.core.internal.command.DbMigrate : outOfOrder mode is active. > Migration of schema `fineract_tenants` may not be reproducible. > 2021-11-28 15:48:23.963 WARN 19340 --- [ main] > o.f.core.internal.command.DbMigrate : outOfOrder mode is active. > Migration of schema `fineract_default` may not be reproducible. > 2021-11-28 15:48:24.007 WARN 19340 --- [ main] > o.d.jdbc.internal.mysql.MySQLProtocol : Could not execute query > org.drizzle.jdbc.internal.common.query.DrizzleQuery@4e4af370 > <mailto:org.drizzle.jdbc.internal.common.query.DrizzleQuery@4e4af370>: > Duplicate entry 'custom-account-number-length' for key 'name_UNIQUE' > 2021-11-28 15:48:24.039 ERROR 19340 --- [ main] > o.f.core.internal.command.DbMigrate : Migration of schema > `fineract_default` to version "378 - configuration for variable length > acoountNumber" failed! Please restore backups and roll back database and code! > 2021-11-28 15:48:24.097 ERROR 19340 --- [ main] > o.s.b.web.embedded.tomcat.TomcatStarter : Error starting Tomcat context. > Exception: org.springframework.beans.factory.UnsatisfiedDependencyException. > Message: Error creating bean with name 'securityConfig': Unsatisfied > dependency expressed through field 'userDetailsService'; nested exception is > org.springframework.beans.factory.UnsatisfiedDependencyException: Error > creating bean with name 'userDetailsService': Unsatisfied dependency > expressed through field 'platformUserRepository'; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating bean > with name 'appUserRepository' defined in > org.apache.fineract.useradministration.domain.AppUserRepository defined in > @EnableJpaRepositories declared on PersistenceConfig: Cannot create inner > bean '(inner bean)#6f17dd06' of type > [org.springframework.orm.jpa.SharedEntityManagerCreator] while setting bean > property 'entityManager'; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating bean > with name '(inner bean)#6f17dd06': Cannot resolve reference to bean > 'entityManagerFactory' while setting constructor argument; nested exception > is org.springframework.beans.factory.BeanCreationException: Error creating > bean with name 'tenantDatabaseUpgradeService': Invocation of init method > failed; nested exception is org.flywaydb.core.api.FlywayException: Migration > V378__configuration_for_variable_length_acoountNumber.sql failed > -------------------------------------------------------------------------- > SQL State : 23000 > Error Code : 1062 > Message : Duplicate entry 'custom-account-number-length' for key > 'name_UNIQUE' > Location : > sql/migrations/core_db/V378__configuration_for_variable_length_acoountNumber.sql > > (D:\Work\fineract\source\fineract\fineract-provider\build\classes\java\main\sql\migrations\core_db\V378__configuration_for_variable_length_acoountNumber.sql) > Line : 20 > Statement : -- > -- Licensed to the Apache Software Foundation (ASF) under one > -- or more contributor license agreements. See the NOTICE file > -- distributed with this work for additional information > -- regarding copyright ownership. The ASF licenses this file > -- to you under the Apache License, Version 2.0 (the > -- "License"); you may not use this file except in compliance > -- with the License. You may obtain a copy of the License at > -- > -- http://www.apache.org/licenses/LICENSE-2.0 > <http://www.apache.org/licenses/LICENSE-2.0> > -- > -- Unless required by applicable law or agreed to in writing, > -- software distributed under the License is distributed on an > -- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY > -- KIND, either express or implied. See the License for the > -- specific language governing permissions and limitations > -- under the License. > -- > > INSERT INTO c_configuration ( name, description) VALUES ( > 'custom-account-number-length', 'if enabled, the value if this configuration > will set accounnumber length') > ; for Tenant DB URL: jdbc:mysql:thin://localhost:3306/fineract_default, > username: root > 2021-11-28 15:48:24.134 WARN 19340 --- [ main] > o.a.c.loader.WebappClassLoaderBase : The web application > [fineract-provider] appears to have started a thread named [HikariPool-1 > housekeeper] but has failed to stop it. This is very likely to create a > memory leak. Stack trace of thread: > [email protected]/jdk.internal.misc.Unsafe.park(Native > <mailto:[email protected]/jdk.internal.misc.Unsafe.park(Native> Method) > > [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) > > <mailto:[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)> > > [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) > > <mailto:[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)> > > [email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) > > <mailto:[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)> > > [email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) > > <mailto:[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)> > > [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) > > <mailto:[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)> > > [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) > > <mailto:[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)> > > [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > <mailto:[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)> > [email protected]/java.lang.Thread.run(Thread.java:829) > <mailto:[email protected]/java.lang.Thread.run(Thread.java:829)> > 2021-11-28 15:48:24.134 WARN 19340 --- [ main] > o.a.c.loader.WebappClassLoaderBase : The web application > [fineract-provider] appears to have started a thread named [HikariPool-1 > connection adder] but has failed to stop it. This is very likely to create a > memory leak. Stack trace of thread: > [email protected]/jdk.internal.misc.Unsafe.park(Native > <mailto:[email protected]/jdk.internal.misc.Unsafe.park(Native> Method) > > [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) > > <mailto:[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)> > > [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) > > <mailto:[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)> > > [email protected]/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:458) > > <mailto:[email protected]/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:458)> > > [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053) > > <mailto:[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)> > > [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) > > <mailto:[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)> > > [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > <mailto:[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)> > [email protected]/java.lang.Thread.run(Thread.java:829) > <mailto:[email protected]/java.lang.Thread.run(Thread.java:829)> > 2021-11-28 15:48:24.142 WARN 19340 --- [ main] > ConfigServletWebServerApplicationContext : Exception encountered during > context initialization - cancelling refresh attempt: > org.springframework.context.ApplicationContextException: Unable to start web > server; nested exception is > org.springframework.boot.web.server.WebServerException: Unable to start > embedded Tomcat > 2021-11-28 15:48:24.191 ERROR 19340 --- [ main] > o.s.boot.SpringApplication : Application run failed > > org.springframework.context.ApplicationContextException: Unable to start web > server; nested exception is > org.springframework.boot.web.server.WebServerException: Unable to start > embedded Tomcat > at > org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:163) > at > org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:577) > at > org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) > at > org.springframework.boot.SpringApplication.refresh(SpringApplication.java:730) > at > org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:412) > at org.springframework.boot.SpringApplication.run(SpringApplication.java:302) > at > org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:164) > at org.apache.fineract.ServerApplication.main(ServerApplication.java:58) > Caused by: org.springframework.boot.web.server.WebServerException: Unable to > start embedded Tomcat > at > org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:142) > at > org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:104) > at > org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:473) > at > org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:206) > at > org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:182) > at > org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:160) > ... 7 common frames omitted > Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: > Error creating bean with name 'securityConfig': Unsatisfied dependency > expressed through field 'userDetailsService'; nested exception is > org.springframework.beans.factory.UnsatisfiedDependencyException: Error > creating bean with name 'userDetailsService': Unsatisfied dependency > expressed through field 'platformUserRepository'; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating bean > with name 'appUserRepository' defined in > org.apache.fineract.useradministration.domain.AppUserRepository defined in > @EnableJpaRepositories declared on PersistenceConfig: Cannot create inner > bean '(inner bean)#6f17dd06' of type > [org.springframework.orm.jpa.SharedEntityManagerCreator] while setting bean > property 'entityManager'; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating bean > with name '(inner bean)#6f17dd06': Cannot resolve reference to bean > 'entityManagerFactory' while setting constructor argument; nested exception > is org.springframework.beans.factory.BeanCreationException: Error creating > bean with name 'tenantDatabaseUpgradeService': Invocation of init method > failed; nested exception is org.flywaydb.core.api.FlywayException: Migration > V378__configuration_for_variable_length_acoountNumber.sql failed > -------------------------------------------------------------------------- > SQL State : 23000 > Error Code : 1062 > Message : Duplicate entry 'custom-account-number-length' for key > 'name_UNIQUE' > Location : > sql/migrations/core_db/V378__configuration_for_variable_length_acoountNumber.sql > > (D:\Work\fineract\source\fineract\fineract-provider\build\classes\java\main\sql\migrations\core_db\V378__configuration_for_variable_length_acoountNumber.sql) > Line : 20 > Statement : -- > > >
