Sorry, I thought we were already on the same page. Clebert, there was a workaround for the MySQL bug which was a change to the connection string to not enable multi-host/replication. I have filed the bug with MySQL Connector/J with a minimal proof of concept (based on the affected artemis-jdbc code) and it has been verified by them. We’re waiting for a fix but we are fine running without multi-host / using a vip for failover.
That bug is https://bugs.mysql.com/bug.php?id=108500 We are not at all concerned with the Artemis reboot. If anything it caught our attention to diagnose the issue. I think I confused things by responding to different people in the thread with different emails, sorry. Hopefully this summarizes everything. Stephen E. Baker From: Clebert Suconic <clebert.suco...@gmail.com> Date: Saturday, September 17, 2022 at 10:20 PM To: users@activemq.apache.org <users@activemq.apache.org> Subject: Re: Critical error sending large messages to mysql Nice catch Justin. Or don’t use MySQL. Just use the journal? On Sat, Sep 17, 2022 at 9:43 PM Justin Bertram <jbert...@apache.org> wrote: > To be clear, the issue in this case has nothing to do with the critical > analyzer. > > The problem is with a bug in the MySQL JDBC driver which is generating an > error interpreted by the broker as a "critical IO error" (as expected) > which causes the broker to shut-down (again, as expected). To resolve the > error the MySQL bug needs to be fixed. > > > Justin > > On Sat, Sep 17, 2022 at 8:39 PM Clebert Suconic <clebert.suco...@gmail.com > > > wrote: > > > The critical analyzer was meant to prevent dead locks in the broker. Say > > the broker failed it would kill itself. > > > > We had a few false positives a few versions ago. > > > > > > Either disable it (which is fine, by setting it to log) > > > > > > The critical analyzer will have no effect out of IO issues. Critical IO > > issues will still stop the broker if they happen. The best action is to > > disable it. > > > > > > > > On Fri, Sep 16, 2022 at 5:24 PM Stephen Baker < > > stephen.ba...@rmssoftwareinc.com> wrote: > > > > > What would the critical analyzer do if it couldn’t write to the > > > large-message or page store now? I didn’t think the restart was that > out > > of > > > line, if a little hopeless in this case. > > > > > > From: Clebert Suconic <clebert.suco...@gmail.com> > > > Date: Friday, September 16, 2022 at 2:38 PM > > > To: users@activemq.apache.org <users@activemq.apache.org> > > > Subject: Re: Critical error sending large messages to mysql > > > I fixed some critical analyzer issues. You should upgrade. > > > > > > Or just turn off the critical analyzer. > > > On Thu, Sep 15, 2022 at 5:01 PM Stephen Baker < > > > stephen.ba...@rmssoftwareinc.com> wrote: > > > > > > > The message size actually doesn’t matter at all. The problem is > simply > > > > calling updateBlob when the connection string contains replication. > It > > > > appears that they have fixed very similar looking issues recently so > > I’m > > > > hopeful it won’t take long on their side. > > > > > > > > From: Vilius Šumskas <vilius.sums...@rivile.lt> > > > > Date: Thursday, September 15, 2022 at 3:14 PM > > > > To: users@activemq.apache.org <users@activemq.apache.org> > > > > Subject: RE: Critical error sending large messages to mysql > > > > Hi, > > > > > > > > how large is the message? Have you checked max_allowed_packet > > > > configuration parameter, specifically > > > > > > > > > > https://dev.mysql.com/doc/mysql-replication-excerpt/8.0/en/replication-features-max-allowed-packet.html > > > > ? > > > > > > > > -- > > > > Vilius > > > > > > > > -----Original Message----- > > > > From: Stephen Baker <stephen.ba...@rmssoftwareinc.com> > > > > Sent: Thursday, September 15, 2022 8:31 PM > > > > To: users@activemq.apache.org > > > > Subject: Re: Critical error sending large messages to mysql > > > > > > > > To follow up, I have tracked this down to a bug in connector/j. I am > > > > working on a simple proof of concept and a ticket for them now. > > > > > > > > For any artemis mysql users that are curious, the problem only > happens > > > > when using a jdbc:mysql:replication:// connection string. It seems > that > > > > updateBlob is broken in that case. > > > > > > > > From: Stephen Baker <stephen.ba...@rmssoftwareinc.com> > > > > Date: Thursday, September 15, 2022 at 10:57 AM > > > > To: users@activemq.apache.org <users@activemq.apache.org> > > > > Subject: Critical error sending large messages to mysql I don’t have > > > exact > > > > reproduction steps yet (otherwise I would have filed an issue), but > on > > > > Artemis 2.22 using a mysql backed journal our QA can reliably send > > > messages > > > > that crash the server: > > > > > > > > ``` > > > > 2022-09-15 10:42:24,843 WARN [org.apache.activemq.artemis.journal] > > > > AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot > be > > > > cast to class com.mysql.cj.jdbc.ClientPreparedStatement > > > > (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement > > are > > > > in unnamed module of loader java.net.URLClassLoader @3a5ed7a6) > > > > 2022-09-15 10:42:24,845 ERROR > [org.apache.activemq.artemis.core.server] > > > > AMQ222010: Critical IO Error, shutting down the server. > > > > > > > > > > file=org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile@3452c05f > > > , > > > > message=Error writing to JDBC file.: java.lang.ClassCastException: > > class > > > > com.sun.proxy.$Proxy21 cannot be cast to class > > > > com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and > > > > com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of > > loader > > > > java.net.URLClassLoader @3a5ed7a6) > > > > at > > > > > > > > > > com.mysql.cj.jdbc.result.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1143) > > > > [mysql-connector-java-8.0.28.jar:8.0.28] > > > > at > > > > > > > > > > com.mysql.cj.jdbc.result.UpdatableResultSet.updateBlob(UpdatableResultSet.java:1283) > > > > [mysql-connector-java-8.0.28.jar:8.0.28] > > > > at > > > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > > > Method) [java.base:] > > > > at > > > > > > > > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > > [java.base:] > > > > at > > > > > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > [java.base:] > > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > > > [java.base:] > > > > at > > > > > > > > > > com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107) > > > > [mysql-connector-java-8.0.28.jar:8.0.28] > > > > at com.sun.proxy.$Proxy22.updateBlob(Unknown Source) > > > > at > > > > > > > > > > org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) > > > > [commons-dbcp2-2.7.0.jar:2.7.0] > > > > at > > > > > > > > > > org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) > > > > [commons-dbcp2-2.7.0.jar:2.7.0] > > > > at > > > > > > > > > > org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) > > > > [commons-dbcp2-2.7.0.jar:2.7.0] > > > > at > > > > > > > > > > org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFileFactoryDriver.writeToFile(JDBCSequentialFileFactoryDriver.java:284) > > > > [artemis-jdbc-store-2.22.0.jar:2.22.0] > > > > at > > > > > > > > > > org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:167) > > > > [artemis-jdbc-store-2.22.0.jar:2.22.0] > > > > at > > > > > > > > > > org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:203) > > > > [artemis-jdbc-store-2.22.0.jar:2.22.0] > > > > at > > > > > > > > > > org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.lambda$scheduleWrite$1(JDBCSequentialFile.java:214) > > > > [artemis-jdbc-store-2.22.0.jar:2.22.0] > > > > at > > > > > > > > > > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) > > > > [artemis-commons-2.22.0.jar:] > > > > at > > > > > > > > > > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) > > > > [artemis-commons-2.22.0.jar:] > > > > at > > > > > > > > > > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) > > > > [artemis-commons-2.22.0.jar:] > > > > at > > > > > > > > > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > > > [java.base:] > > > > at > > > > > > > > > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > > > [java.base:] > > > > at > > > > > > > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > [artemis-commons-2.22.0.jar:] > > > > > > > > 2022-09-15 10:42:24,848 WARN [org.apache.activemq.artemis.journal] > > > > AMQ142021: Error on IO callback, Error writing to JDBC file. > > > > 2022-09-15 10:42:24,854 WARN > [org.apache.activemq.artemis.core.client] > > > > AMQ212004: Failed to connect to server. > > > > 2022-09-15 10:42:24,909 WARN [org.apache.activemq.artemis.journal] > > > > AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot > be > > > > cast to class com.mysql.cj.jdbc.ClientPreparedStatement > > > > (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement > > are > > > > in unnamed module of loader java.net.URLClassLoader @3a5ed7a6) > > > > 2022-09-15 10:42:24,926 WARN [org.apache.activemq.artemis.journal] > > > > AMQ142021: Error on IO callback, Error writing to JDBC file. > > > > ``` > > > > As you can see, whatever the original problem was it is hidden by the > > > > ClassCastException in Artemis. > > > > > > > > When I learn more I will file an issue (unless the problem has been > > > > resolved in a later version) and hopefully a fix. > > > > > > > > Stephen E. Baker > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. > > Do > > > > not click links or open attachments unless you recognize the sender > and > > > > know the content is safe. > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. > > Do > > > > not click links or open attachments unless you recognize the sender > and > > > > know the content is safe. > > > > > > > -- > > > Clebert Suconic > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. > Do > > > not click links or open attachments unless you recognize the sender and > > > know the content is safe. > > > > > -- > > Clebert Suconic > > > -- Clebert Suconic [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
