Is there anything I can do to help get the PR moving? Is it still in draft due to the Travis failures?
From: Clebert Suconic <clebert.suco...@gmail.com> Date: Friday, October 14, 2022 at 9:41 AM To: users@activemq.apache.org <users@activemq.apache.org> Subject: Re: Mirror compatibility across versions It's being done as part of this PR: https://github.com/apache/activemq-artemis/pull/4246 On Fri, Oct 14, 2022 at 4:40 AM Robbie Gemmell <robbie.gemm...@gmail.com> wrote: > > That helper command doesnt exist yet as Clebert said since the idea > only came from discussing something else the other day, but the > pre-existing logging related changes coming for 2.27.0 are covered in > the docs along with diffs of the script changes from 2.26.0, you can > see the current source version of those docs at > https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/versions.md#2270 > . They will be updated to reference the helper once it exits. > > On Thu, 13 Oct 2022 at 18:52, Stephen Baker > <stephen.ba...@rmssoftwareinc.com> wrote: > > > > Because bin/artemis includes references to the jboss logmanager causing > > artemis to fail on startup > > > > Diffing my two instances I see: > > # Set Defaults Properties > > ARTEMIS_LOGGING_CONF="$ARTEMIS_INSTANCE_ETC_URI/logging.properties" > > ARTEMIS_LOG_MANAGER=org.jboss.logmanager.LogManager > > > > > > # finding the Log Manager > > LOG_MANAGER=`ls $ARTEMIS_HOME/lib/jboss-logmanager*jar 2>/dev/null` > > > > if [ -z "$LOG_MANAGER" ] ; then > > # this is the one found when the server was created > > LOG_MANAGER="$ARTEMIS_HOME/lib/jboss-logmanager-2.1.10.Final.jar" > > fi > > > > WILDFLY_COMMON=`ls $ARTEMIS_HOME/lib/wildfly-common*jar 2>/dev/null` > > if [ -z "$WILDFLY_COMMON" ] ; then > > # this is the one found when the server was created > > WILDFLY_COMMON="$ARTEMIS_HOME/lib/wildfly-common-1.5.2.Final.jar" > > fi > > > > -Xbootclasspath/a:"$LOG_MANAGER:$WILDFLY_COMMON" \ > > > > and > > > > -Djava.util.logging.manager="$ARTEMIS_LOG_MANAGER" \ > > -Dlogging.configuration="$ARTEMIS_LOGGING_CONF" \ > > > > all have to go. > > > > I also see a change in the schema for bootstrap.xml (binding child of web), > > and the browse permission added to management.xml > > > > Along with the new log4j.properties you mentioned > > > > Some of those changes might be earlier if they’re backwards compatible, > > I’ve carried forward this configuration for awhile, but in particular the > > shell script changes appear to be manditory. > > > > From: Clebert Suconic <clebert.suco...@gmail.com> > > Date: Thursday, October 13, 2022 at 12:49 PM > > To: users@activemq.apache.org <users@activemq.apache.org> > > Subject: Re: Mirror compatibility across versions > > I think the record would pile up unacked at the source mirror. > > > > > > and @Stephen baker: sorry about my mistake on this fix... > > > > > > Why would the upgrade be difficult on 2.27? it's just adding a > > log4j2.properties.. everything else should be the same. > > > > > > You should probably bring a patched version yourself until we can make > > a release? I'm thinking we should make a release next week. > > > > On Thu, Oct 13, 2022 at 10:27 AM Stephen Baker > > <stephen.ba...@rmssoftwareinc.com> wrote: > > > > > > Your patch does resolve the error. Artemis 2.27 looks like it will be a > > > difficult upgrade, I ended up making a new instance and merging config > > > over. > > > > > > I have pasted a trace in > > > https://issues.apache.org/jira/browse/ARTEMIS-4045 > > > > > > What is the impact of this issue? I’m trying to decide whether to advise > > > our IT team to continue with the planned upgrade or hold off until 2.27. > > > We will definitely encounter this condition in production. From a surface > > > reading, possibly a resource leak? > > > > > > > > > > > > > > > From: Clebert Suconic <clebert.suco...@gmail.com> > > > Date: Wednesday, October 12, 2022 at 9:54 PM > > > To: users@activemq.apache.org <users@activemq.apache.org> > > > Subject: Re: Mirror compatibility across versions > > > Notice that main is now using SLF4j / log4j... (in case you manually > > > upgrade to a snapshot) > > > > > > > > > We are still working the details for an upgrade. > > > > > > > > > if you need to patch your 2.25.0 it's a straight change to make there > > > > > > On Wed, Oct 12, 2022 at 9:52 PM Clebert Suconic > > > <clebert.suco...@gmail.com> wrote: > > > > > > > > I don't know how I would test it yet. It's fairly late in the night > > > > for me.. I will think about it tomorrow. > > > > > > > > > > > > but here is a tentative fix: > > > > > > > > https://github.com/apache/activemq-artemis/pull/4256 > > > > > > > > On Wed, Oct 12, 2022 at 9:30 PM Stephen Baker > > > > <stephen.ba...@rmssoftwareinc.com> wrote: > > > > > > > > > > That’s the full output with regular logging levels. I can reproduce > > > > > at will so I have enabled trace level logging and pasted the result > > > > > in https://issues.apache.org/jira/browse/ARTEMIS-4045 > > > > > > > > > > Let’s take further discussion there? > > > > > > > > > > From: Clebert Suconic <clebert.suco...@gmail.com> > > > > > Date: Wednesday, October 12, 2022 at 9:10 PM > > > > > To: users@activemq.apache.org <users@activemq.apache.org> > > > > > Subject: Re: Mirror compatibility across versions > > > > > is this the actual trace? or you cut some to post here? > > > > > > > > > > > > > > > Just puzzled by skipDelivery calling performAck.. > > > > > > > > > > artemis-test-artemis-1-m-1 | at > > > > > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377) > > > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0] > > > > > artemis-test-artemis-1-m-1 | at > > > > > org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203) > > > > > [artemis-server-2.25.0.jar:2.25.0] > > > > > > > > > > > > > > > can you post the full stack if this is not it? > > > > > > > > > > > > > > > it definitely needs fixing... I'm investigating it. > > > > > > > > > > On Wed, Oct 12, 2022 at 6:05 PM Stephen Baker > > > > > <stephen.ba...@rmssoftwareinc.com> wrote: > > > > > > > > > > > > Having updated both sides to 2.25 I’m seeing this error in the > > > > > > logs, is it a concern that warrants further investigation? > > > > > > > > > > > > artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR > > > > > > [org.apache.activemq.artemis.core.server] AMQ224041: Failed to > > > > > > deliver: java.lang.IllegalStateException: this method requires to > > > > > > be called within the handler, use the executor > > > > > > artemis-test-artemis-1-m-1 | at > > > > > > org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210) > > > > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0] > > > > > > artemis-test-artemis-1-m-1 | at > > > > > > org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197) > > > > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0] > > > > > > artemis-test-artemis-1-m-1 | at > > > > > > org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185) > > > > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0] > > > > > > artemis-test-artemis-1-m-1 | at > > > > > > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125) > > > > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0] > > > > > > artemis-test-artemis-1-m-1 | at > > > > > > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388) > > > > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0] > > > > > > artemis-test-artemis-1-m-1 | at > > > > > > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377) > > > > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0] > > > > > > artemis-test-artemis-1-m-1 | at > > > > > > org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203) > > > > > > [artemis-server-2.25.0.jar:2.25.0] > > > > > > artemis-test-artemis-1-m-1 | at > > > > > > org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932) > > > > > > [artemis-server-2.25.0.jar:2.25.0] > > > > > > artemis-test-artemis-1-m-1 | at > > > > > > org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991) > > > > > > [artemis-server-2.25.0.jar:2.25.0] > > > > > > artemis-test-artemis-1-m-1 | at > > > > > > org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250) > > > > > > [artemis-server-2.25.0.jar:2.25.0] > > > > > > artemis-test-artemis-1-m-1 | at > > > > > > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) > > > > > > [artemis-commons-2.25.0.jar:] > > > > > > artemis-test-artemis-1-m-1 | at > > > > > > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) > > > > > > [artemis-commons-2.25.0.jar:] > > > > > > artemis-test-artemis-1-m-1 | at > > > > > > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) > > > > > > [artemis-commons-2.25.0.jar:] > > > > > > artemis-test-artemis-1-m-1 | at > > > > > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > > > > > [java.base:] > > > > > > artemis-test-artemis-1-m-1 | at > > > > > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > > > > > [java.base:] > > > > > > artemis-test-artemis-1-m-1 | at > > > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > > > [artemis-commons-2.25.0.jar:] > > > > > > > > > > > > > > > > > > From: Stephen Baker <stephen.ba...@rmssoftwareinc.com> > > > > > > Date: Wednesday, October 12, 2022 at 4:43 PM > > > > > > To: users@activemq.apache.org <users@activemq.apache.org> > > > > > > Subject: Re: Mirror compatibility across versions > > > > > > I set up some docker images in this configuration as a preliminary > > > > > > step. One oddity: > > > > > > > > > > > > Configure 2.25 side not to run the reaper > > > > > > Send message to 2.25 side > > > > > > Observe that after expiry the message shows up in the expiry queue > > > > > > on the 2.20 side, but not on the 2.25 side, the message is removed > > > > > > from the original queue on both sides. > > > > > > > > > > > > If the message is originally sent to the 2.20 side it shows up in > > > > > > both queues as expected. > > > > > > > > > > > > There’s probably a reason for it, but I didn’t expect this change. > > > > > > I thought that we would continue to see the old bugs until both > > > > > > sides were updated. > > > > > > > > > > > > From: Clebert Suconic <clebert.suco...@gmail.com> > > > > > > Date: Tuesday, October 11, 2022 at 3:24 PM > > > > > > To: users@activemq.apache.org <users@activemq.apache.org> > > > > > > Subject: Re: Mirror compatibility across versions > > > > > > Yeah.. something like that... not necessarily in there though. but a > > > > > > similar test. > > > > > > > > > > > > On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker > > > > > > <stephen.ba...@rmssoftwareinc.com> wrote: > > > > > > > > > > > > > > Ok, I agree based on a cursory reading of that patch. The extra > > > > > > > ackReason defaults to normal in one direction and isn’t read in > > > > > > > the other direction. Killed, replaced, and expired being > > > > > > > interpreted as normal just means that the 2.20 bugs will persist > > > > > > > until both sides are updated. > > > > > > > > > > > > > > I’ll test it out with different version docker containers. I > > > > > > > suppose as far as writing tests you mean something like the > > > > > > > MultiVersionReplicaTest. > > > > > > > > > > > > > > Stephen E. Baker > > > > > > > > > > > > > > > > > > > > > From: Clebert Suconic <clebert.suco...@gmail.com> > > > > > > > Date: Tuesday, October 11, 2022 at 12:59 PM > > > > > > > To: users@activemq.apache.org <users@activemq.apache.org> > > > > > > > Subject: Re: Mirror compatibility across versions > > > > > > > In theory it should work. > > > > > > > > > > > > > > > > > > > > > Only change that might break compatibility is > > > > > > > https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48 > > > > > > > which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to > > > > > > > determine target operations and fixing Delivering statistics on > > > > > > > Mirror > > > > > > > > > > > > > > > > > > > > > I tried to not break compatibility, but I just realized we should > > > > > > > add > > > > > > > a test to validate compatibility between mirrors. > > > > > > > > > > > > > > > > > > > > > > > > > > > > so, I will say it should be compatible, but I would test it before > > > > > > > doing it in the real system. > > > > > > > > > > > > > > > > > > > > > > > > > > > > if you are willing to contribute to a compatibility test :) > > > > > > > > > > > > > > On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker > > > > > > > <stephen.ba...@rmssoftwareinc.com> wrote: > > > > > > > > > > > > > > > > We are planning our production upgrade from 2.20 to 2.25. These > > > > > > > > upgrades involve a loss of service in the window between > > > > > > > > stopping the live and when the backup instance becomes ready to > > > > > > > > process messages. > > > > > > > > > > > > > > > > I was wondering if the mirror protocol is expected to be > > > > > > > > compatible between those versions. If so we could upgrade our > > > > > > > > cold site, and then wait for a planned failover to avoid any > > > > > > > > additional down time. I know that quite a bit of work was done > > > > > > > > by Clebert in 2.24 so I was hoping he could weigh in. > > > > > > > > > > > > > > > > Stephen E Baker > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Clebert Suconic > > > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile > > > > > > > Safety. Do not click links or open attachments unless you > > > > > > > recognize the sender and know the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Clebert Suconic > > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile > > > > > > Safety. Do not click links or open attachments unless you recognize > > > > > > the sender and know the content is safe. > > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile > > > > > > Safety. Do not click links or open attachments unless you recognize > > > > > > the sender and know the content is safe. > > > > > > > > > > > > > > > > > > > > -- > > > > > Clebert Suconic > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. > > > > > Do not click links or open attachments unless you recognize the > > > > > sender and know the content is safe. > > > > > > > > > > > > > > > > -- > > > > Clebert Suconic > > > > > > > > > > > > -- > > > Clebert Suconic > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do > > > not click links or open attachments unless you recognize the sender and > > > know the content is safe. > > > > > > > > -- > > Clebert Suconic > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do > > not click links or open attachments unless you recognize the sender and > > know the content is safe. -- Clebert Suconic [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
