mrog opened a new issue, #6724:
URL: https://github.com/apache/cloudstack/issues/6724
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete
the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
Database connection
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main
branch.
-->
~~~
4.16, 4.17 (and likely some older versions, too.)
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network,
advanced networking, etc. N/A otherwise
-->
CloudStack 4.16 and MySQL 8.0.28, running on different servers
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
CentOS 7
##### SUMMARY
<!-- Explain the problem/feature briefly -->
Something caused the database connection to be unusable. The root cause is
unknown. The bug is that CloudStack detected the bad database connection and
did nothing to correct it. This exception appeared many times in the log until
we restarted the cloudstack-management service.
```
2022-09-07 13:31:30,626 ERROR [c.c.u.d.ConnectionConcierge]
(ConnectionConcierge-1:ctx-3dd4c5d7) (logid:cda577ad) Unable to keep the db
connection for LockController1
java.sql.SQLNonTransientConnectionException: No operations allowed after
connection closed.
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:73)
at
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:73)
at
com.mysql.cj.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:1659)
at
com.mysql.cj.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:1575)
at
org.apache.commons.dbcp2.DelegatingConnection.prepareStatement(DelegatingConnection.java:301)
at
org.apache.commons.dbcp2.DelegatingConnection.prepareStatement(DelegatingConnection.java:301)
at
com.cloud.utils.db.ConnectionConcierge$ConnectionConciergeManager.testValidity(ConnectionConcierge.java:147)
at
com.cloud.utils.db.ConnectionConcierge$ConnectionConciergeManager$1.runInContext(ConnectionConcierge.java:203)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.mysql.cj.exceptions.ConnectionIsClosedException: No
operations allowed after connection closed.
at
jdk.internal.reflect.GeneratedConstructorAccessor197.newInstance(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
... 22 more
```
I examined the ConnectionConcierge class in CloudStack. I found that it was
checking the health of the database connection, and it has code to reset the
database connection, but the connection reset code doesn't get called from
anywhere. I'm guessing this used to work, but it was broken during refactoring
some time in the past.
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal
test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
~~~
1. Start MySQL
2. Start the CloudStack management service and let it run for a short time.
3. Stop MySQL, wait a couple of minutes, then start it again.
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
The CloudStack management service should reconnect to the database.
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
The CloudStack management service keeps trying to use the broken
connections. The management-server.log file shows lots of exceptions like the
one in the bug description above.
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
